[SOLVED] SSH server restart needed after package upgrades

[keeka]

On my PMG (Proxmox LXC) installation, I am often (possibly always) unable to login via ssh after applying updates apt dist-upgrade.
The message I see on the remote client is:

ssh root@smtp
kex_exchange_identification: read: Connection reset by peer

After restarting ssh via the web UI, all is well. I initially put this down to updates to encryption libraries but it seems to happen after any package upgrade.

I don't recall encountering this problem with other linux installations, including PVE.

EDIT: journal on PMG host shows: fatal: Missing privilege separation directory: /run/sshd

 

[keeka]

I just ran a package upgrade (libpve-http-server-perl) and /run/sshd got deleted.
No other files or dirs under /run or existing ssh sessions are affected. However I need to restart sshd in order for /run/sshd to get (re)created and for new ssh logins to succeed.
This is not something I have encountered before and not sure where to look for the link between apt upgrades and sshd interference.

 

[Stoiko Ivanov]

On my PMG (Proxmox LXC) installation, I am often (possibly always) unable to login via ssh after applying updates apt dist-upgrade.
The message I see on the remote client is:

Cannot reproduce the issue here

* do you have any custom configuration in /etc/apt by any chance?
* unconditionally removing `/run` is nothing that should be done (and libpve-http-server-perl maintainer scripts do not do it)

I hope this helps!

 

[keeka]

* do you have any custom configuration in /etc/apt by any chance?

Thanks that was it! I had (what I thought was a commented-out) custom file in /etc/apt/apt.conf.d.
I'd commented the contents using '#'. I changed these to '//'. Issue resolved, I think.

Some things I have noticed, that have surprised me, whilst looking in to this (this is with a working ssh server in the container):

• /run/sshd is not created until there has been at least one ssh login following boot.
• there appears to be no sshd listening/master process.
• systemctl status ssh shows service inactive, even with an active ssh session running.

This isn't how I remember sshd working in the past. PMG is running in a CT using the Proxmox provided PMG template.

 

[Stoiko Ivanov]

Some things I have noticed, that have surprised me, whilst looking in to this (this is with a working ssh server in the container):

• /run/sshd is not created until there has been at least one ssh login following boot.
• there appears to be no sshd listening/master process.
• systemctl status ssh shows service inactive, even with an active ssh session running.

This is due to socket-activation for ssh being active in the container (I'll ask around if this was intentional) - the details are quite well explained in:
/usr/share/doc/openssh-server/README.Debian.gz (search for systemd)
(this also describes how to disable it if you like - just run systemctl disable ssh.socket) (haven't tried that one myself though - so it's only based on the docs)

I hope this explains it!

 

[keeka]

@Stoiko Ivanov
Many thanks for the help and background information. It's much appreciated.

 

[Stoiko Ivanov]

Thanks that was it! I had (what I thought was a commented-out) custom file in /etc/apt/apt.conf.d.

could I ask which file this was? and what it's purpose was? (wiping /run does not sound like a good idea)

 

[keeka]

I did not lose the entire contents of /run but, post package upgrade, /run/sshd was either deleted or failed to initialise.
The apt customisation I'd attempted, & since abandoned, was to diff files I'd created in /etc/pmg/templates/ with their counterparts in /var/lib/pmg/templates. Whilst it ran successfully standalone, I could not integrate with apt/dpkg. I then believe I incorrectly commented out the line. I have since removed it entirely. This is what I had:
/etc/apt/apt.conf.d/99custom:

DPkg::Post-Invoke { "/usr/local/bin/pmg_template_diff.sh"; };

/usr/local/bin/pmg_template_diff.sh:

#!/bin/env bash
TEMPLATE_DIR=/etc/pmg/templates
TEMPLATE_SYS_DIR=/var/lib/pmg/templates
for p in `find $TEMPLATE_DIR/ -type f -regex '.*\.\(in\|tt\)$'`; do
    echo $p
    diff $p $TEMPLATE_SYS_DIR/$(basename $p)
done

I have just tried quickly restoring a copy of the container from backup, but cannot replicate the problem.
I am fairly certain though the ssh issue arose after running package upgrades.

 

[keeka]

This is due to socket-activation for ssh being active in the container (I'll ask around if this was intentional) - the details are quite well explained in:
/usr/share/doc/openssh-server/README.Debian.gz (search for systemd)
(this also describes how to disable it if you like - just run systemctl disable ssh.socket) (haven't tried that one myself though - so it's only based on the docs)

I was aware of xinetd/inetd but had not realised systemd had this super server functionality. It really does do everything, as claimed ;-)

 

[Stoiko Ivanov]

The apt customisation I'd attempted, & since abandoned, was to diff files I'd created in /etc/pmg/templates/ with their counterparts in /var/lib/pmg/templates.

this is already done through ucf (since pmg 6.1 or 6.2)