[TUTORIAL] Use Mobile shell (Mosh) with Proxmox

[orwadira]

Dear all. I have done a quick search on these forums and found an old thread, but decided to start a new one with the TUTORIAL prefix, as this is an area where I feel there is not much up-to-date information online.

It is about "Mosh", or the so-called mobile shell, and how to use it to establish sturdy terminal sessions when connecting to your Proxmox containers and VMs, something that I am sure many of you can greatly appreciate through train (actual) tunnel passings. (I live in Germany, so this happens more frequently than I would want, haha)

Someone can describe the script provided at the end of this tutorial as a "jump script" for Mosh when used with Proxmox. (if you do not know what a jump or a bastion server is, fear not! there is a wonderful YouTube video that I highly recommend watching)

For fun and play, I have called the script proxmosh, but please do not let that stop you from renaming it to whatever makes sense to you, including proxmox (as a command name).

What is Mosh?

Mosh, is a technology built independently from SSH (but while relying on it for connection establishment) to provide a terminal session that is stable and reliable even while roaming between different networks, having an IP address that is constantly changing, and also under varying and fluctuating network conditions (sounds like the internet in 2025, eh!).

⮑ Read about it here: https://mosh.org/

How does Mosh do what it does?

The way mosh does that is by using SSH momentarily to bootstrap the connection. For this to work, the host you are connecting to needs to be accessible using SSH. It also have to have more than just an SSH server installed, namely, it needs to have the mosh server binary installed on it, as well (which can be installed easily using apt).

When you connect via Mosh (using mosh user@host) mosh connects to the host using SSH first (through a command invocation that you can fully customise, see below). This is done merely to invoke the mosh server binary on the host you are connecting to, which then establishes the Mosh session on a dedicated UDP port.

Now, that we have a Mosh connection established using UDP, the SSH connection is dropped and we can rely solely on the mosh binaries communicating together via UDP.

To tie this explanation to the mosh CLI and its most important options, I like to show the following snippet:

mosh --help
Usage: mosh [options] [user@]host [command...]
        --ssh=COMMAND        ssh command to run when setting up session
                                (example: "ssh -p 2222")
                                (default: "ssh")
        --server=COMMAND     mosh server command to run on remote machine
                                (default: "mosh-server")

        --port=PORT[:PORT2]  UDP port or range of ports for Mosh sesssions
                                (No effect on SSH port)

The --ssh option help you decide how SSH is invoked, which can use a jump server, but for the Mosh concept to work, you still need a direct access to the remote host using UDP. The --server option, on the other hand, allows you to experiment with the mosh server options, and how it is invoked using SSH.

I believe that these 3 options and the explanation above, summarises what most people need to know about Mosh to be able to start using it.

How is it different from just SSH?

The core concept of Mosh is different from SSH. Mosh is based on the idea of syncing the screen state (as a grid of w x h characters) between the client and the server, rather than forwarding output. This is also what makes it a different beast from SSH. It cannot replace SSH. Rather, it is exploring a smaller problem, and implementing it in a manner that is more suitable in some circumstances.

The practical implication of syncing screen state versus sending output, is that out of the box, Mosh sessions have no way of allowing you to scroll. You can only see what fits within the screen, and that is about it.

⮑ Read about it here: https://mosh.org/mosh-paper.pdf

This tutorial

In this tutorial I am going to teach you the more challenging bits of setting up Mosh in a useful way, so that it could be used along with Proxmox to access your containers and VMs while on the train.

Namely, I won't walk you through installing mosh on your local machine and/or your PVE node (aka the remote). Rather, I will only address some of the challenging bits that will run into after having done just that.

Challenge 1. Setting up the locale


A lot happens when connecting via SSH, which is often not well understood by users. Namely, some of the environmental variables are shared with the remote which are accepted and used throughout your terminal session. The most notable example of these environmental variables is LC_CTYPE.

This variable allows your local terminal and the SSH host to speak the same language, or in more accurate terms, use the same encoding for characters. By default on Mac OS X, this variable can be left unset which can create a potential misconfiguration when connecting via SSH.

Unlike SSH, mosh is quite picky about this stuff and won't let you connect unless you ensure that these variables are set to a locale that the server recognises.

To remedy this in most cases, you need to have this variable (as well as others) set correctly, which can be done using this minimal and sane SSH configuration:

Host *
  SetEnv LC_CTYPE=en_US.UTF-8

(add this to the beginning of ~/.ssh/config and create it if it does not exist)

In Mac OS X, the above would be enough to allow you to connect using Mosh, assuming that you have the option named Set environment locale variables on startup checked in your Terminal settings (check screenshot).

Challenge 2. Can we access the remote using UDP?

Unlike SSH, the only way to use Mosh is to have a UDP port open and available for connections on the remote, which needs to be accessible via a public IP address. This means that using jump servers and other SSH tricks, while useful for bootstrapping the connection, is not enough for allowing you to connect using mosh.

Namely, you would still need to figure out a way to find the public IP address of the remote, and to have the router configured to forward UDP communication to it, over a well-known port range. This is called port-forwarding and many home routers support it in their admin web UI:




Note that you:

1. You only need a single TCP port for bootstrapping the Mosh session using SSH. This can be one of the UDP ports you are using for Mosh but uses TCP instead.
2. You need one UDP port for each simultaneous Mosh session. These are provided as a range of ports which Mosh selects from (see the --port CLI option above)

Challenge 3. The use of the root PVE node as a jump node, similar to SSH

Now that we have a way to connect to the root PVE node, the question really is, how can we bridge to the target container or VM we are interested in establishing a terminal session with?

For this, I have developed a bash script which can be found here.

# Usage: connect to the PVE node
proxmosh
# Usage: connect to a VM called 'hostwithdocker'
proxmosh hostwithdocker

Using the script is simple, you just need to configure the following variables inside the script:

# The public IP address or the Dynamic DNS hostname of the
# proxmox host, which can be resolved to a public IP address
PROXMOX_HOST=host.dyndns.org
# A single TCP port that is used for initiating a connection
PROXMOX_TCP_PORT=60001
# A range of UDP ports that can be used for each Mosh session
PROXMOX_UDP_PORTS=60001:60008

Additionally, if you would like to have more control over which users are used to login to your containers/VMs, as well as having a more uniform way of exiting the sessions, then I highly recommend configuring SSH from the root PVE node, which will be prioritised over using pct console and qm terminal by the script.

Video demonstration

If you want to see the script in action you can check this video I recorded.

PS: this tutorial has been improved and simplified. In a previous version, I went down the rabbit whole of configuring and opening many TCP ports, which was completely unnecessary.