For many small or personal services running on a VPS in the cloud, administration is often done by connecting directly to the server via SSH. Such servers should be hardened with firewalls, employ an SSHd config that denies root and password-based login, run fail2ban, and other services and practices.
Linode has some great getting-started guides on the essentials of securing your server.
Protecting sensitive servers
In more complex production scenarios heightened security can be achieved by isolating application (webapp, API, database, etc.) servers from external internet traffic. This is usually done by placing these “sensitive/protected” servers in a private subnet, without direct internet-facing network interfaces. This means that the server is not reachable from the outside world.
In this type of scenario, outbound traffic from the sensitive server can be routed through a NAT gateway and inbound traffic can be funnelled through a load-balancer or reverse proxy server. In both these cases the NAT gateway and load-balancer would exist in public subnets (with internet-facing network interfaces) and can reach the sensitive server through private network interfaces in order to forward requests (e.g. web traffic).
Now the question is around how one does manage the services running on the protected server, since it is no longer available to connect to. Traditionally this is done by introducing bastion hosts into your network.
Bastion hosts
Bastion hosts - like the NAT gateway and load balancers - sit in the public subnet and so they are available to the outside world. They often accept SSH connections, from which one can “jump” through to the protected servers through the bastion’s private networking interface.
Bastion hosts should be hardened as much as possible (with firewalls and other network rules), and should run a limited set of services - in many cases simply SSHd.
This server then enables administrators to connect through to the protected servers in order to carry out maintenance, upgrades, or other tasks.
Connecting through a bastion host
SSH port-forwarding is a widely-used concept, in which a secure tunnel to a service running on the protected server is opened via a port on the local machine (using ssh’s -L option).
Another option is to use proxy jumping (with the -J option):
ssh -J bastion.company.com protected.company-internal.com
In this example the user connects via the bastion through to the protected server at protected.company-internal.com. Since you should be using key-based authentication to connect, you may also need to specify the private key path (with the -i option), and also tell SSH to forward your agent to the bastion (using -A) so it can continue the connection.
This can make things hard to remember each time. You could write the command in a script, however it’s probably easier to use SSH’s own local configuration. To do so, you can add the following to your local ~/.ssh/config file:
Host *.company-internal.com
ProxyJump bastion.company.com
User username
IdentityFile /home/username/.ssh/identity
With that in place you can now simply run the following when you want to connect to the protected server:
ssh protected.company-internal.com
Note: depending on your system you may need to add the key to your local agent first, but you just need to do this once per login (ssh-add ~/.ssh/identity).
A note on DNS
Generally I would probably avoid assigning public domain names to a bastion host, as this may invite unwanted attention and traffic (even if the host is secured). Instead you can just include the IP address directly in the ProxyJump line of .ssh/config. I used domain names in the examples above to make the process clearer.
Also, in the above example I refer to the company-internal.com domain for use within the private network. This domain should only be resolvable within members of the private network - either by using an internal DNS server or by simply modifying /etc/hosts on the bastion. Alternatively you can just use the private IP address for the protected server on the Host line of .ssh/config.
Additional notes
In setups like this you may also want to consider the following:
Private keys
Don’t provision these on your bastion host. Instead use agent forwarding (as described above). You’ll need to add your public keys to both the bastion and protected servers.
Restrict source network
For extra security you can restrict SSH connections to your bastion only from trusted networks (e.g. your office network or a VPN).
Similarly, restrict protected servers such that they only accept SSH traffic from the bastion, and not from other servers on the network.
Make use of managed services if/when possible
For extra security you can use managed services when they are available. For example, if you use AWS then you can make use of a combination of VPCs, subnets, NAT gateways, elastic load balancing, security groups, Route 53, and other services to secure your hosts and control your network. You can of course set this up on your own servers without relying on managed services.
Either way, I hope this post has helped shed light on some simple ways to improve network security for your applications and services.