[SOLVED] ssh madness

[mikem12]

Hi All,

How does ssh into an LXC container work?

I have a Centos 7.9 LXC container that I can ssh into, but I can't figure out what sshd is listening. systemctl start sshd fails with the complaint that it can't bind to port 22, and status shows it's disabled and dead. A third-party sshd also fails for the same reason.

netstat shows that the init process is binding port 22, but ps -ef|grep ssh shows no sshd process... until I try to connect...

When I connect, suddenly there is an sshd process, owned by init. Normally, I would see an sshd listener via ps -ef that looks like: /usr/sbin/sshd
Curiously, when I connect to this phantom sshd, I get prompted by putty to accept a new hostkey. So... something different is listening.

Now, if I kill -9 these init-based sshd's and then systemctl enable sshd, I'll get a normal listener.

What's frustrating is ps -ef of these phantom sshd's only shows "sshd: username" -- it won't show me the full path, so I can't figure out exactly what it is.

This is a brand-new LXC, and the behavior is repeatable across all of the LXC's that I create, so I have to assume it's something proxmox is doing... I just can't figure out what it is.

It's bonkers! Does proxmox start it's own sshd proxy listener or something for a container when it boots up? It looks like it, but I can't tie a process to it. And, if it does, how do I stop it?!?!? I need to be able to start a custom sshd listener within an LXC and that application can't run because something has port 22 bound.

Help Please!!!

 

[mikem12]

By "init-based sshd", this is what I mean:

# netstat -tulpn|grep :22
tcp6 0 0 :::22 :::* LISTEN 1/init

# ps -ef|grep sshd
#

Note how init has port 22 bound, but there is no sshd listener.

Where as a normal sshd listener looks like this:
# netstat -tulpn|grep :22
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1066/sshd
tcp6 0 0 :::22 :::* LISTEN 1066/sshd

# ps -ef|grep sshd
root 1066 1 0 12:08 ? 00:00:00 /usr/sbin/sshd -D
root 1081 1066 0 12:11 ? 00:00:00 sshd: michael [priv]
michael 1083 1081 0 12:11 ? 00:00:00 sshd: michael@pts/4

 

[mikem12]

I removed /usr/sbin/sshd and rebooted. This time, I am unable to login via ssh -- ie. no magical listeners appear. But, netstat still shows init has port 22 bound:
# netstat -tulpn|grep :22
tcp6 0 0 :::22 :::* LISTEN 1/init

What's even weird'er is that it's only listening on tcp6, and tcp6 is supposed to be disabled as I have these in /etc/sysctl.conf:
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1

I'm stumped and really could use some help on this one.

 

[mikem12]

Now I've changed the listen port in sshd_config as well as the ssh entry in /etc/services, and still upon boot, the LXC's init is listening on port 22.

netstat only shows it listening on tcp6 but I can connect to it via tcp4.

This makes no sense. Anyone able to help? Thank you

 

[mikem12]

Ugh... found it. Apparently some genius thought it would be a good idea to use systemd sshd.socket instead of the standard sshd.service mechanism when they built the "LXC default image for Centos 7 (20190926)"

The solution is:
systemctl disable sshd.socket
systemctl stop sshd.socket
systemctl enable sshd
systemctl start sshd

Still bonkers that it would answer on tcp4 but netstat only showed it listening on tcp6.