LXC SSH confusing habits

[TorbenThorsten]

Hey Guys,
while updating and restarting some of our LXC Containers we got some confusing alarms by our monitoring. We check for some configurations in the sshd_conf to be present (just some basic security stuff and nothing special) and got an alarm that these are missing. So we checked our server, by logging in via ssh an have a look on the present config. The config was ok, but the sshd service wasnt running!? After starting the service the alarm was all right and everything worked fine) So how could we be able to log in? The auth log just tells that we could log in via sshd using public key... You Guys have any explanation for this habit? And maybe how to fix it..

 

[cheiss]

Hi,

as you were still able to login, sshd definitely works.
What distro are you running inside the LXC's?

On a hunch - sshd can also be socket-activated by systemd, i.e. the sshd daemon does not always run by default in the background, but rather gets started by systemd if a connection is made.
Please post the output of systemctl status ssh.service, systemctl status ssh.socket and journalctl -b to confirm this (.. assuming you are running a systemd-based distro inside the LXCs).

 

[TorbenThorsten]

Thanks for the fast reply!
We're running the LXC with Ubuntu 22.04
It seems you're right.

systemctl status ssh.service * ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: inactive (dead) Docs: man:sshd(8) man:sshd_config(5)


systemctl status ssh.socket * ssh.socket - OpenBSD Secure Shell server socket Loaded: loaded (/lib/systemd/system/ssh.socket; enabled; vendor preset: enabled) Active: active (listening) since Mon 2023-08-14 13:10:28 CEST; 47min ago Triggers: * ssh@0-10.x.x.x:22-10.x.x.x:53802.service Listen: [::]:22 (Stream) Accepted: 1; Connected: 1; CGroup: /system.slice/ssh.socket


But the ssh.service is enabled, so i think it should start after rebooting..

 

[cheiss]

Ubuntu seems indeed to use socket-activated sshd since 22.xx.
See https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189

At the end of the blog post there are also instructions how to disable it. But if you don't have any good reason for it, there is not much reason to disable it.

 

[sdettmer]

Ubuntu seems indeed to use socket-activated sshd since 22.xx.
See https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189

At the end of the blog post there are also instructions how to disable it. But if you don't have any good reason for it, there is not much reason to disable it.

Why, does it has disadvantages? In OPs scenario of monitoring I think socket activation helps nothing, as sshd will be launched anyway.
(and I think "systemd instablity making system unrealiable" is a very good reason ).

Interestingly, the feature is said to be implemented for containers (to save resources as they usually never actually launch sshd), but apparently does not work correctly. I repeatedly had issues until I switched back to "service" mode. Thanks to Proxmox cool "Console" feature the biggest problem was entering 20 digits random root console passwords correctly

Anyway, short way to switch back:

systemctl disable --now ssh.socket
systemctl enable --now ssh.service

(but unfortunately thanks to systemd, we have to expect that soon that won't work anymore, and we will meet the problem again )