Will's avatar

⬅️ See more posts

SSH Jumping and Bastion Hosts

10 February 2021 (4 minute read)

🔮 This post is also available via Gemini.

100daystooffload technology security

💯 100 Days to Offload

This article is one of a series of posts I have written for the 100 Days to Offload challenge. Disclaimer: The challenge focuses on writing frequency rather than quality, and so posts may not always be fully planned out!

View other posts in this series.

AI generated pixel art of astronauts and cats jumping over computers.

For many small or personal services running on a VPS in the cloud, administration is often done by connecting directly to the server via SSH. Such servers should be hardened with firewalls, employ an SSHd config that denies root and password-based login, run fail2ban, and other services and practices.

Linode has some great getting-started guides on the essentials of securing your server.

Protecting sensitive servers

In more complex production scenarios heightened security can be achieved by isolating application (webapp, API, database, etc.) servers from external internet traffic. This is usually done by placing these “sensitive/protected” servers in a private subnet, without direct internet-facing network interfaces. This means that the server is not reachable from the outside world.

In this type of scenario, outbound traffic from the sensitive server can be routed through a NAT gateway and inbound traffic can be funnelled through a load-balancer or reverse proxy server. In both these cases the NAT gateway and load-balancer would exist in public subnets (with internet-facing network interfaces) and can reach the sensitive server through private network interfaces in order to forward requests (e.g. web traffic).

Diagram of public and private subnets, with a NAT gateway and load balancer

Now the question is around how one does manage the services running on the protected server, since it is no longer available to connect to. Traditionally this is done by introducing bastion hosts into your network.

Bastion hosts

Bastion hosts - like the NAT gateway and load balancers - sit in the public subnet and so they are available to the outside world. They often accept SSH connections, from which one can “jump” through to the protected servers through the bastion’s private networking interface.

Adding a bastion host to the cloud infrastructure

Bastion hosts should be hardened as much as possible (with firewalls and other network rules), and should run a limited set of services - in many cases simply SSHd.

This server then enables administrators to connect through to the protected servers in order to carry out maintenance, upgrades, or other tasks.

Connecting through a bastion host

SSH port-forwarding is a widely-used concept, in which a secure tunnel to a service running on the protected server is opened via a port on the local machine (using ssh’s -L option).

Another option is to use proxy jumping (with the -J option):

ssh -J bastion.company.com protected.company-internal.com

In this example the user connects via the bastion through to the protected server at protected.company-internal.com. Since you should be using key-based authentication to connect, you may also need to specify the private key path (with the -i option), and also tell SSH to forward your agent to the bastion (using -A) so it can continue the connection.

This can make things hard to remember each time. You could write the command in a script, however it’s probably easier to use SSH’s own local configuration. To do so, you can add the following to your local ~/.ssh/config file:

Host *.company-internal.com
  ProxyJump bastion.company.com
  User username
  IdentityFile /home/username/.ssh/identity

With that in place you can now simply run the following when you want to connect to the protected server:

ssh protected.company-internal.com

Note: depending on your system you may need to add the key to your local agent first, but you just need to do this once per login (ssh-add ~/.ssh/identity).

A note on DNS

Generally I would probably avoid assigning public domain names to a bastion host, as this may invite unwanted attention and traffic (even if the host is secured). Instead you can just include the IP address directly in the ProxyJump line of .ssh/config. I used domain names in the examples above to make the process clearer.

Also, in the above example I refer to the company-internal.com domain for use within the private network. This domain should only be resolvable within members of the private network - either by using an internal DNS server or by simply modifying /etc/hosts on the bastion. Alternatively you can just use the private IP address for the protected server on the Host line of .ssh/config.

Additional notes

In setups like this you may also want to consider the following:

Private keys

Don’t provision these on your bastion host. Instead use agent forwarding (as described above). You’ll need to add your public keys to both the bastion and protected servers.

Restrict source network

For extra security you can restrict SSH connections to your bastion only from trusted networks (e.g. your office network or a VPN).

Similarly, restrict protected servers such that they only accept SSH traffic from the bastion, and not from other servers on the network.

Make use of managed services if/when possible

For extra security you can use managed services when they are available. For example, if you use AWS then you can make use of a combination of VPCs, subnets, NAT gateways, elastic load balancing, security groups, Route 53, and other services to secure your hosts and control your network. You can of course set this up on your own servers without relying on managed services.

Either way, I hope this post has helped shed light on some simple ways to improve network security for your applications and services.

✉️ You can reply to this post via email.

📲 Subscribe to updates

If you would like to read more posts like this, then you can subscribe via RSS.