Upgrade your bastion with a drawbridge

It’s a known best practice to use a bastion host to access your private resources, whether it’s in the cloud or your data centers.

The goal is that only the bastion host is reachable, either directly from the internet or particular IPs. It’s the only entry point to your infrastructure, so it’s easier to secure this single server.

AWS published an excellent Quick Start Reference Deployment Guide: multiple bastion hosts are deployed across Availability Zones with static IPs.

However, do you really need multiple bastions to run at all time?
Maybe you don’t even SSH to your server that often? Because “if you have to SSH into your servers, then your automation has failed”.

So why not remove the entrance to our bastion when we are not using it:

All we have to do is set up the server to terminate when no one is connected and add some convenient way to launch a new one when we need it.

Raise the bridge

With AWS it’s quite easy, here’s a sample cloud formation template.

First, we create an auto scaling group with a single instance and a launch configuration:

LaunchConfig:
  Type: AWS::AutoScaling::LaunchConfiguration
  Properties:
    ImageId: ami-XXXXXXXX
    InstanceType: t2.nano
    IamInstanceProfile: !GetAtt InstanceProfile.Arn
    UserData:
      "Fn::Base64":
        !Sub |
          #!/bin/bash
          yum update -y
          echo "* * * * * root /usr/bin/aws cloudwatch --region ${AWS::Region} put-metric-data --metric-name Users --namespace Bastion --value \$(ps aux | grep sshd | grep -v root | cut -f1 -d' ' | sort | uniq | wc -l)" > /etc/cron.d/cloudwatch
          aws ec2 associate-address --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id) --allocation-id ${EIP.AllocationId}
    AssociatePublicIpAddress: true
    SecurityGroups:
    - sg-XXXXXXXX

Through the instance’s UserData, you can run commands on your instance at launch.

We are going to configure a crontab that sends the number of connected users every minute to the cloudwatch metric Bastion/Users.
Also, we have created an elastic IP that we will associate every time at boot, so we always have the same public IP.

Now we just need a cloudwatch alarm with an auto scaling policy:

ScaleDown:
  Type: AWS::AutoScaling::ScalingPolicy
  Properties:
    AdjustmentType: "ExactCapacity"
    AutoScalingGroupName: !Ref ASG
    ScalingAdjustment: 0

BastionNoUsers:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: BastionUsers
    AlarmDescription: Terminate instance if no users
    Namespace: Bastion
    MetricName: Users
    Statistic: Maximum
    Period: 60
    EvaluationPeriods: 15
    ComparisonOperator: LessThanOrEqualToThreshold
    Threshold: 0
    AlarmActions:
    - !Ref ScaleDown

When our cloudwatch metric Bastion/Users is <= 0 for 15 consecutive minutes, we will initiate the ScaleDown policy which sets the DesiredCapacity of our auto scaling group to 0. Which will terminate the instance.

Lower the bridge

To start a new bastion, the quickest way is aws cli :

aws autoscaling set-desired-capacity --auto-scaling-group-name 'bastion-ASG-XXXXXXX' --desired-capacity 1

But you can find a lot of other ways, I, for instance, like to start it via Slack. I have a dedicated channel for the bastion, so everyone is notified when someone starts it.

Here’s my script for hubot:

robot.respond /^bastion$/i, (msg) ->
  AWS = require('aws-sdk')

  AS = new AWS.AutoScaling({region: 'eu-west-1'})

  AS.describeAutoScalingGroups {}, (err, data) ->
    if err
      console.log err
      msg.send err.message
      return
    for asg in data.AutoScalingGroups
      if /^bastion-ASG-.*/.test(asg.AutoScalingGroupName)
         group_name = asg.AutoScalingGroupName
         if asg.DesiredCapacity == 1
           msg.send "`Bastion` already started."
           return
         break
    AS.setDesiredCapacity {AutoScalingGroupName: group_name,
                           DesiredCapacity: 1}, (err, data) ->
      if err
        console.log err
        msg.send err.message
        return
      msg.send "Starting `Bastion`..."

And you, how do you handle your bastion?