ssh-port-forwarding

(how to use plain openssh to do remote port forwarding)

There are many good reasons to do secure port forwarding through ssh. For example if you own two servers in different datacenters and you want to connect them to a single service which is less restricted when accessed locally (e.g. port 25 for SMTP) or you want to forward a service from a system behind a firewall (e.g. a web service on your home server).
Traditionally you would use autossh to manage permanent ssh connections. However through many hours of testing this has prooven unreliable in many ways. When connecting multiple times to the same server autossh by default uses the same ports for monitoring, which leads to the termination of at least one connection. There also were inexplicable cases when sshd remained running on the server, while the client was actually disconnected and could not restore the connection due to the broken process on the server. Even worse, autossh does not check if ssh has built up all forwardings successfully, leading to incomplete connections with partial port forwardings (e.g. if a port on the server is still in use by an other process). Luckily openssh made autossh redundant because it already offers built-in monitoring. No additional monitoring ports are necessary anymore. However, there are quite a few options that you should know about in order to improve security and reliability of such a setup. This is the motivation behind this tutorial.

Disclaimer

When you follow this guide, you can make otherwise protected services accessible to the public Internet (when using ssh -R). This might be an attack vector into your protected network.
When you don't set restrictions properly, an attacker might gain access to your server, either via direct shell access or through forwarding the port of your unprotected service (e.g. a database on localhost).

Server configuration

First become root through sudo su - or su -.

/etc/ssh/sshd_config

Add or change the following lines:

GatewayPorts clientspecified
ClientAliveInterval 5
ClientAliveCountMax 3

This allows any user to forward his local ports to an unprivileged public port on the server (ssh -R). You have to restrict this later on through the permitlisten variable in the authorized_keys file. The other two variables specify how often the server should send keep alive messages and how many missed messages from the client it will tolerate. The same has to be set on the client side, but there it can be done as a command line parameter.

special user account

on debian you can run the following:

adduser ssh-port-forwarding --system
su ssh-port-forwarding -s '/bin/bash' -c 'mkdir ~/.ssh/; chmod 700 ~/.ssh/; touch ~/.ssh/authorized_keys'

/etc/passwd should look similar to this:
ssh-port-forwarding:x:1001:65534::/home/ssh-port-forwarding:/usr/sbin/nologin
and ls -lsha ~ssh-port-forwarding/.ssh/ should look like this:

4,0K drwx------ 2 ssh-port-forwarding nogroup 4,0K 18. Feb 22:18 .
4,0K drwxr-xr-x 3 ssh-port-forwarding nogroup 4,0K 18. Feb 22:18 ..
   0 -rw-r--r-- 1 ssh-port-forwarding nogroup    0 18. Feb 22:18 authorized_keys

Configuring access rights

This is all done in /home/ssh-port-forwarding/.ssh/authorized_keys.

First use the ssh-keygen command to create a private and public key pair on the client side. Don't type any password! Then use cat ~/.ssh/id_rsa.pub to display the content of your newly created public key. After that add a new line in the authorized_keys file on the server. Use the following line as an example. Your clients public key starts at AAAA... and this all needs to be in a single line per key.
restrict,command="",port-forwarding,permitlisten="localhost:9999",permitopen="localhost:80" ssh-rsa AAAA...

• restrict: this disables all available and future forwarding options (we will whitelist what we need)
• command="": don't allow client to send a command, set an empty forced command instead
• port-forwarding: allow port forwarding
• permitlisten="localhost:9999": permit client to create a listening socket (via ssh -R) on port 9999 on the server, which forwards requests to a service on the client
• permitopen="localhost:80": permit client to access (via ssh -L) port 80 port on server, which will then be offered as a local port on the client

The permitopen and permitlisten options can be used multiple times in a row. The syntax is as follows:

• permitlisten="[host:]port
  • port is the port on the server that you want to open locally or towards the Internet
  • host specifies on which interface the server should listen for incoming connections. You should either specify localhost or *
    • localhost binds the port to the loopback device and can only be used by processes on the same server
    • * allows access from everywhere (e.g. from the Internet, if your firewall allows that) if GatewayPorts clientspecified is set in /etc/ssh/sshd_config
• permitopen="host:port"
  • host is the hostname or IP address of the server that the client should be allowed to connect to via your server
  • port is the port number on the host that will be forwarded to the client

Client side configuration

First connect to your server manually, in order to accept the server certificate!
You should run the client side ssh command in a loop because it is tuned to terminate as soon as errors are detected. Don't worry, this is well tested. If you are old school you simply put this into /etc/rc.local:

#!/bin/bash

(while true; do
        ssh ...
        sleep 30
done) &
disown

exit 0

Don't forget to mark the script as executable: chmod +x /etc/rc.local

The client side ssh command looks like:
ssh ssh-port-forwarding@myserver.example.com -TNnqakx -o "TCPKeepAlive yes" -o "ServerAliveInterval 5" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure yes" -R [bind_address:]port:host:hostport -L [bind_address:]port:host:hostport

• -T disable pseudo terminal allocation
• -N don't execute any command on the server
• -n redirect stdin to /dev/null (necessary as we run this command in background)
• -q disable most output
• -a don't forward the authentication agent connection
• -k disable forwarding of GSSAPI credentials
• -x dsable X11 forwarding
• -o "TCPKeepAlive yes" enable SSH's built in self monitring
• -o "ServerAliveInterval 5" send test messages every five seconds
• -o "ServerAliveCountMax 3" mark connection as failed after 3 lost test messages
• -o "ExitOnForwardFailure yes" quit ssh process if self check or any of the forwardings fail (this is a crucial feature missing in autossh)
• -4 (not shown above) is optional to foce ssh to use IPv4 only (in case of problems with IPv6)
• -g (not used here) does the same as [bind_address:] set to *: (with -L the client offers the port publicly; with -R the server offers the port publicly)
• -R [bind_address:]port:host:hostport (see above -> permitlisten; can be repeated multiple times)
  • [bind_address:] address on the server, that the port should be bound to; defaults to localhost:, can be set to *: (optional)
  • port port on the server that should opened for incoming connections
  • host hostname or address that the client should forward the connection to (e.g. localhost)
  • hostport existing port on the host that should be forwarded
• -L [bind_address:]port:host:hostport (see above -> permitopen; can be repeated multiple times)
  • [bind_address:] address on the client, that the port should be bound to; defaults to localhost:, can be set to *: (optional)
  • port port on the client that should be opened
  • host host that the client wants to access through the server (e.g. localhost on the server itself)
  • hostport existing port that should be forwarded to the client

Please beware that the hostname part in the -L and -R options must be spelled exactly the same as in the permitlisten and permitopen variables on the server ("Localhost", "localhost" and "127.0.0.1" are treated different).

Example

In this example we forward a http based service running on the client (port 80) to the server (port 2280). The server can than deliver the service via its own webserver. Port 2280 on the server is only available locally.
Additionally the client accesses the remote mail server (port 25) and provides local access for applications running on the client (through port 2225). That way an application on the client can send mail via the remote server. It is not necessary to to open the remote mail relay to the internet. Requests to port 2225 on the client are treated as if they were done locally on the server.
The last forwarding coud be used to publicly offer a service runninging on the client (port 9999) via the server (port 8080). Port 8080 on the server can be accessed by everyone on the internet and will be forwarded to port 9999 on the client.

Server

/etc/ssh/sshd_config

(changed variables)

GatewayPorts clientspecified
ClientAliveInterval 5
ClientAliveCountMax 3

/home/ssh-port-forwarding/.ssh/authorized_keys

(client public key shortented to AAAA...)

restrict,command="",port-forwarding,permitlisten="localhost:2280",permitopen="localhost:25" ssh-rsa AAAA... root@client

Client

Before running the script, test the command manually. This is also important if you were not connected to this server before from your client. On your first connection you have to accept the server key!

/etc/rc.local

#!/bin/bash

(while true; do
        ssh ssh-port-forwarding@myserver.example.com -TNnqakx -o "TCPKeepAlive yes" -o "ServerAliveInterval 5" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure yes" -R '2280:localhost:80' -L '2225:localhost:25' -R '*:8080:localhost:9999'
        sleep 30
done) &
disown

exit 0