[Issue] Cannot SSH into container or pct enter CTID

[soundcage]

Hi there,

Looking for some assistance: currently running 5.4-7 and am now unable to SSH into my containers. This has seemed to occur randomly- the last time I had access was a few days ago when I edited a systemd unit service/timer in one of the containers.

The error when I try to use pct enter CTID is:
lxc-attach: 102: attach.c: lxc_attach_run_shell: 1597 No such file or directory - Failed to execute shell

This error appears for all containers, and all containers run as privileged, and share a bindmount for shared files

While i have access to the node itself (and am able to SSH into it as root), as well as login to the web UI; I'm unable to SSH directly into the container nor in the web UI console. In the web UI it asks for a login username, but upon entering a username, it hangs and eventually fails.

Thanks in advance for any help to diagnose this issue. To be specific, there we no issues or this problem prior to this instance, and the containers were running fine since July 1st

 

[soundcage]

Edit: Not sure if this is the issue (I blocked this IP in iptables, but still unable to SSH in; Chinese IP with thousands of SSH login attempts)

Jul 11 18:08:21 e3-2186 sshd[23336]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:08:21 e3-2186 sshd[23336]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:08:24 e3-2186 sshd[23538]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:08:26 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
Jul 11 18:08:29 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
Jul 11 18:08:31 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
Jul 11 18:08:34 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
Jul 11 18:08:37 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
Jul 11 18:08:39 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
Jul 11 18:08:39 e3-2186 sshd[23538]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 38234 ssh2 [preauth]
Jul 11 18:08:39 e3-2186 sshd[23538]: Disconnecting: Too many authentication failures [preauth]
Jul 11 18:08:39 e3-2186 sshd[23538]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:08:39 e3-2186 sshd[23538]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:08:42 e3-2186 sshd[23735]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:08:44 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
Jul 11 18:08:47 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
Jul 11 18:08:50 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
Jul 11 18:08:53 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
Jul 11 18:08:54 e3-2186 sshd[23835]: Received disconnect from 153.36.236.35 port 46520:11:  [preauth]
Jul 11 18:08:54 e3-2186 sshd[23835]: Disconnected from 153.36.236.35 port 46520 [preauth]
Jul 11 18:08:55 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
Jul 11 18:08:58 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
Jul 11 18:08:58 e3-2186 sshd[23735]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 42121 ssh2 [preauth]
Jul 11 18:08:58 e3-2186 sshd[23735]: Disconnecting: Too many authentication failures [preauth]
Jul 11 18:08:58 e3-2186 sshd[23735]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:08:58 e3-2186 sshd[23735]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:09:00 e3-2186 systemd[1]: Starting Proxmox VE replication runner...
Jul 11 18:09:01 e3-2186 systemd[1]: Started Proxmox VE replication runner.
Jul 11 18:09:01 e3-2186 sshd[23914]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:09:01 e3-2186 audit[23951]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"
Jul 11 18:09:01 e3-2186 kernel: audit: type=1400 audit(1562882941.813:564): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"
Jul 11 18:09:03 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
Jul 11 18:09:05 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
Jul 11 18:09:08 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
Jul 11 18:09:11 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
Jul 11 18:09:13 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
Jul 11 18:09:16 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
Jul 11 18:09:16 e3-2186 sshd[23914]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 46017 ssh2 [preauth]
Jul 11 18:09:16 e3-2186 sshd[23914]: Disconnecting: Too many authentication failures [preauth]
Jul 11 18:09:16 e3-2186 sshd[23914]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:09:16 e3-2186 sshd[23914]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:09:19 e3-2186 sshd[24123]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:09:21 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
Jul 11 18:09:24 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
Jul 11 18:09:27 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
Jul 11 18:09:30 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
Jul 11 18:09:32 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
Jul 11 18:09:35 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
Jul 11 18:09:35 e3-2186 sshd[24123]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 49627 ssh2 [preauth]
Jul 11 18:09:35 e3-2186 sshd[24123]: Disconnecting: Too many authentication failures [preauth]
Jul 11 18:09:35 e3-2186 sshd[24123]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:09:35 e3-2186 sshd[24123]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:09:39 e3-2186 sshd[24321]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:09:40 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
Jul 11 18:09:43 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
Jul 11 18:09:46 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
Jul 11 18:09:49 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
Jul 11 18:09:52 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
Jul 11 18:09:55 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
Jul 11 18:09:55 e3-2186 sshd[24321]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 54413 ssh2 [preauth]
Jul 11 18:09:55 e3-2186 sshd[24321]: Disconnecting: Too many authentication failures [preauth]
Jul 11 18:09:55 e3-2186 sshd[24321]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:09:55 e3-2186 sshd[24321]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:09:58 e3-2186 sshd[24508]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:10:00 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
Jul 11 18:10:00 e3-2186 systemd[1]: Starting Proxmox VE replication runner...
Jul 11 18:10:01 e3-2186 systemd[1]: Started Proxmox VE replication runner.
Jul 11 18:10:03 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
Jul 11 18:10:06 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
Jul 11 18:10:07 e3-2186 sshd[24575]: Received disconnect from 153.36.240.126 port 22188:11:  [preauth]
Jul 11 18:10:07 e3-2186 sshd[24575]: Disconnected from 153.36.240.126 port 22188 [preauth]
Jul 11 18:10:08 e3-2186 sshd[24634]: Received disconnect from 159.65.150.212 port 40142:11: Bye Bye [preauth]
Jul 11 18:10:08 e3-2186 sshd[24634]: Disconnected from 159.65.150.212 port 40142 [preauth]
Jul 11 18:10:09 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
Jul 11 18:10:12 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
Jul 11 18:10:15 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
Jul 11 18:10:15 e3-2186 sshd[24508]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 58293 ssh2 [preauth]
Jul 11 18:10:15 e3-2186 sshd[24508]: Disconnecting: Too many authentication failures [preauth]
Jul 11 18:10:15 e3-2186 sshd[24508]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
Jul 11 18:10:15 e3-2186 sshd[24508]: PAM service(sshd) ignoring max retries; 6 > 3
Jul 11 18:10:18 e3-2186 sshd[24679]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root

Something of interest that may be related to the container access problems?

Jul 11 18:09:01 e3-2186 audit[23951]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"
Jul 11 18:09:01 e3-2186 kernel: audit: type=1400 audit(1562882941.813:564): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"

Edit: Looks like the ioclean issue has to do with PHP and LXC containers https://discuss.linuxcontainers.org/t/apparmor-denied-operation-mount/2424

 

[soundcage]

Rebooted the node, problem still exists, can't start one of the containers at all now.

 

[oguz]

hi,

can you try:

lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log

and paste the log here?

 

[soundcage]

Hi oguz,

Thanks for the assistance, can't seem to get the log for the container that won't start (for good measure, I ran it on an already started container that I can't enter into and the returned message is as expected):

root@e3-2186:~# lxc-start -n 103 -F -l DEBUG -o /tmp/lxc-103.log
/sbin/init: error while loading shared libraries: libip4tc.so.0: cannot open shared object file: No such file or directory

root@e3-2186:~# lxc-start -n 102 -F -l DEBUG -o /tmp/lxc-102.log 
lxc-start: 102: tools/lxc_start.c: main: 280 Container is already running

 

[oguz]

hi,

/sbin/init: error while loading shared libraries: libip4tc.so.0: cannot open shared object file: No such file or directory

looks like you messed up your system somehow.

`libip4tc0` seems to be the package for the missing/broken library... maybe try to install/reinstall it? (`apt install libip4tc0 --reinstall`)

see if that changes something and if you can obtain a debug log then.

 

[soundcage]

Sadly I arriving to the same conclusion as you are, something is messed up on the system, missing files or bad links.

Same error after reinstalling that package:

root@e3-2186:~# apt install libip4tc0 --reinstall
Reading package lists... Done
Building dependency tree      
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 67.8 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian stretch/main amd64 libip4tc0 amd64 1.6.0+snapshot20161117-6 [67.8 kB]
Fetched 67.8 kB in 0s (271 kB/s)    
(Reading database ... 52612 files and directories currently installed.)
Preparing to unpack .../libip4tc0_1.6.0+snapshot20161117-6_amd64.deb ...
Unpacking libip4tc0:amd64 (1.6.0+snapshot20161117-6) over (1.6.0+snapshot20161117-6) ...
Setting up libip4tc0:amd64 (1.6.0+snapshot20161117-6) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
root@e3-2186:~# lxc-start -n 103 -F -l DEBUG -o /tmp/lxc-103.log
/sbin/init: error while loading shared libraries: libip4tc.so.0: cannot open shared object file: No such file or directory

Thinking I have to do a complete reinstall of the node, I tried to mount container file system using pct mount 103, which was successful. It turns out that the /usr folder for that container is completely missing, which may explain some of these problems?

The currently live containers do have /usr folders that look normal, but still remain unaccessible via pct enter, SSH, or web console.

What's the next recommend course of action? Seems like I should reinstall the whole PVE ISO and start fresh, but would you like any other logs that would help figure out the cause? As far as I'm aware, there have been no manual actions taken that would have caused borking the system.

 

[oguz]

Seems like I should reinstall the whole PVE ISO and start fresh,

looks like it. you can still try to take backups of your guests and restore them back later.

what i'd do if i were you, would be to take the server offline if possible, and perform some inspections to determine whether there was an intrusion (since you saw a ton of failed logins). maybe you were hacked, or maybe you or your admin did a mistake... i'd start with logs in /var/log/ (for example syslog and access/auth related logs). run some scripts like rkhunter/chkrootkit maybe.

, I tried to mount container file system using pct mount 103, which was successful. It turns out that the /usr folder for that container is completely missing, which may explain some of these problems?

probably, but that's likely a consequence rather than a cause. you can make a tar archive from the mounted directory to restore the files later.