[Issue] Cannot SSH into container or pct enter CTID

Discussion in 'Proxmox VE: Installation and configuration' started by soundcage, Jul 12, 2019.

  1. soundcage

    soundcage New Member

    Joined:
    Jul 11, 2019
    Messages:
    5
    Likes Received:
    0
    Hi there,

    Looking for some assistance: currently running 5.4-7 and am now unable to SSH into my containers. This has seemed to occur randomly- the last time I had access was a few days ago when I edited a systemd unit service/timer in one of the containers.

    The error when I try to use pct enter CTID is:
    lxc-attach: 102: attach.c: lxc_attach_run_shell: 1597 No such file or directory - Failed to execute shell

    This error appears for all containers, and all containers run as privileged, and share a bindmount for shared files

    While i have access to the node itself (and am able to SSH into it as root), as well as login to the web UI; I'm unable to SSH directly into the container nor in the web UI console. In the web UI it asks for a login username, but upon entering a username, it hangs and eventually fails.

    Thanks in advance for any help to diagnose this issue. To be specific, there we no issues or this problem prior to this instance, and the containers were running fine since July 1st
     
    #1 soundcage, Jul 12, 2019
    Last edited: Jul 12, 2019
  2. soundcage

    soundcage New Member

    Joined:
    Jul 11, 2019
    Messages:
    5
    Likes Received:
    0
    Edit: Not sure if this is the issue (I blocked this IP in iptables, but still unable to SSH in; Chinese IP with thousands of SSH login attempts)
    Code:
    Jul 11 18:08:21 e3-2186 sshd[23336]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:08:21 e3-2186 sshd[23336]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:08:24 e3-2186 sshd[23538]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:08:26 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
    Jul 11 18:08:29 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
    Jul 11 18:08:31 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
    Jul 11 18:08:34 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
    Jul 11 18:08:37 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
    Jul 11 18:08:39 e3-2186 sshd[23538]: Failed password for root from 218.92.0.147 port 38234 ssh2
    Jul 11 18:08:39 e3-2186 sshd[23538]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 38234 ssh2 [preauth]
    Jul 11 18:08:39 e3-2186 sshd[23538]: Disconnecting: Too many authentication failures [preauth]
    Jul 11 18:08:39 e3-2186 sshd[23538]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:08:39 e3-2186 sshd[23538]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:08:42 e3-2186 sshd[23735]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:08:44 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
    Jul 11 18:08:47 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
    Jul 11 18:08:50 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
    Jul 11 18:08:53 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
    Jul 11 18:08:54 e3-2186 sshd[23835]: Received disconnect from 153.36.236.35 port 46520:11:  [preauth]
    Jul 11 18:08:54 e3-2186 sshd[23835]: Disconnected from 153.36.236.35 port 46520 [preauth]
    Jul 11 18:08:55 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
    Jul 11 18:08:58 e3-2186 sshd[23735]: Failed password for root from 218.92.0.147 port 42121 ssh2
    Jul 11 18:08:58 e3-2186 sshd[23735]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 42121 ssh2 [preauth]
    Jul 11 18:08:58 e3-2186 sshd[23735]: Disconnecting: Too many authentication failures [preauth]
    Jul 11 18:08:58 e3-2186 sshd[23735]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:08:58 e3-2186 sshd[23735]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:09:00 e3-2186 systemd[1]: Starting Proxmox VE replication runner...
    Jul 11 18:09:01 e3-2186 systemd[1]: Started Proxmox VE replication runner.
    Jul 11 18:09:01 e3-2186 sshd[23914]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:09:01 e3-2186 audit[23951]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"
    Jul 11 18:09:01 e3-2186 kernel: audit: type=1400 audit(1562882941.813:564): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"
    Jul 11 18:09:03 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
    Jul 11 18:09:05 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
    Jul 11 18:09:08 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
    Jul 11 18:09:11 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
    Jul 11 18:09:13 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
    Jul 11 18:09:16 e3-2186 sshd[23914]: Failed password for root from 218.92.0.147 port 46017 ssh2
    Jul 11 18:09:16 e3-2186 sshd[23914]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 46017 ssh2 [preauth]
    Jul 11 18:09:16 e3-2186 sshd[23914]: Disconnecting: Too many authentication failures [preauth]
    Jul 11 18:09:16 e3-2186 sshd[23914]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:09:16 e3-2186 sshd[23914]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:09:19 e3-2186 sshd[24123]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:09:21 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
    Jul 11 18:09:24 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
    Jul 11 18:09:27 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
    Jul 11 18:09:30 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
    Jul 11 18:09:32 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
    Jul 11 18:09:35 e3-2186 sshd[24123]: Failed password for root from 218.92.0.147 port 49627 ssh2
    Jul 11 18:09:35 e3-2186 sshd[24123]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 49627 ssh2 [preauth]
    Jul 11 18:09:35 e3-2186 sshd[24123]: Disconnecting: Too many authentication failures [preauth]
    Jul 11 18:09:35 e3-2186 sshd[24123]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:09:35 e3-2186 sshd[24123]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:09:39 e3-2186 sshd[24321]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:09:40 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
    Jul 11 18:09:43 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
    Jul 11 18:09:46 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
    Jul 11 18:09:49 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
    Jul 11 18:09:52 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
    Jul 11 18:09:55 e3-2186 sshd[24321]: Failed password for root from 218.92.0.147 port 54413 ssh2
    Jul 11 18:09:55 e3-2186 sshd[24321]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 54413 ssh2 [preauth]
    Jul 11 18:09:55 e3-2186 sshd[24321]: Disconnecting: Too many authentication failures [preauth]
    Jul 11 18:09:55 e3-2186 sshd[24321]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:09:55 e3-2186 sshd[24321]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:09:58 e3-2186 sshd[24508]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:10:00 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
    Jul 11 18:10:00 e3-2186 systemd[1]: Starting Proxmox VE replication runner...
    Jul 11 18:10:01 e3-2186 systemd[1]: Started Proxmox VE replication runner.
    Jul 11 18:10:03 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
    Jul 11 18:10:06 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
    Jul 11 18:10:07 e3-2186 sshd[24575]: Received disconnect from 153.36.240.126 port 22188:11:  [preauth]
    Jul 11 18:10:07 e3-2186 sshd[24575]: Disconnected from 153.36.240.126 port 22188 [preauth]
    Jul 11 18:10:08 e3-2186 sshd[24634]: Received disconnect from 159.65.150.212 port 40142:11: Bye Bye [preauth]
    Jul 11 18:10:08 e3-2186 sshd[24634]: Disconnected from 159.65.150.212 port 40142 [preauth]
    Jul 11 18:10:09 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
    Jul 11 18:10:12 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
    Jul 11 18:10:15 e3-2186 sshd[24508]: Failed password for root from 218.92.0.147 port 58293 ssh2
    Jul 11 18:10:15 e3-2186 sshd[24508]: error: maximum authentication attempts exceeded for root from 218.92.0.147 port 58293 ssh2 [preauth]
    Jul 11 18:10:15 e3-2186 sshd[24508]: Disconnecting: Too many authentication failures [preauth]
    Jul 11 18:10:15 e3-2186 sshd[24508]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    Jul 11 18:10:15 e3-2186 sshd[24508]: PAM service(sshd) ignoring max retries; 6 > 3
    Jul 11 18:10:18 e3-2186 sshd[24679]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.147  user=root
    
    Something of interest that may be related to the container access problems?

    Jul 11 18:09:01 e3-2186 audit[23951]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"
    Jul 11 18:09:01 e3-2186 kernel: audit: type=1400 audit(1562882941.813:564): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-103_</var/lib/lxc>" name="/" pid=23951 comm="(ionclean)" flags="rw, rslave"

    Edit: Looks like the ioclean issue has to do with PHP and LXC containers https://discuss.linuxcontainers.org/t/apparmor-denied-operation-mount/2424
     
    #2 soundcage, Jul 12, 2019
    Last edited: Jul 12, 2019
  3. soundcage

    soundcage New Member

    Joined:
    Jul 11, 2019
    Messages:
    5
    Likes Received:
    0
    Rebooted the node, problem still exists, can't start one of the containers at all now.
     
  4. oguz

    oguz Proxmox Staff Member
    Staff Member

    Joined:
    Nov 19, 2018
    Messages:
    603
    Likes Received:
    63
    hi,

    can you try:

    Code:
    lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log
    
    and paste the log here?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. soundcage

    soundcage New Member

    Joined:
    Jul 11, 2019
    Messages:
    5
    Likes Received:
    0
    Hi oguz,

    Thanks for the assistance, can't seem to get the log for the container that won't start (for good measure, I ran it on an already started container that I can't enter into and the returned message is as expected):

    Code:
    root@e3-2186:~# lxc-start -n 103 -F -l DEBUG -o /tmp/lxc-103.log
    /sbin/init: error while loading shared libraries: libip4tc.so.0: cannot open shared object file: No such file or directory
    
    root@e3-2186:~# lxc-start -n 102 -F -l DEBUG -o /tmp/lxc-102.log 
    lxc-start: 102: tools/lxc_start.c: main: 280 Container is already running
    
     
  6. oguz

    oguz Proxmox Staff Member
    Staff Member

    Joined:
    Nov 19, 2018
    Messages:
    603
    Likes Received:
    63
    hi,

    looks like you messed up your system somehow.

    `libip4tc0` seems to be the package for the missing/broken library... maybe try to install/reinstall it? (`apt install libip4tc0 --reinstall`)

    see if that changes something and if you can obtain a debug log then.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. soundcage

    soundcage New Member

    Joined:
    Jul 11, 2019
    Messages:
    5
    Likes Received:
    0
    Sadly I arriving to the same conclusion as you are, something is messed up on the system, missing files or bad links.

    Same error after reinstalling that package:
    Code:
    root@e3-2186:~# apt install libip4tc0 --reinstall
    Reading package lists... Done
    Building dependency tree      
    Reading state information... Done
    0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
    Need to get 67.8 kB of archives.
    After this operation, 0 B of additional disk space will be used.
    Get:1 http://ftp.us.debian.org/debian stretch/main amd64 libip4tc0 amd64 1.6.0+snapshot20161117-6 [67.8 kB]
    Fetched 67.8 kB in 0s (271 kB/s)    
    (Reading database ... 52612 files and directories currently installed.)
    Preparing to unpack .../libip4tc0_1.6.0+snapshot20161117-6_amd64.deb ...
    Unpacking libip4tc0:amd64 (1.6.0+snapshot20161117-6) over (1.6.0+snapshot20161117-6) ...
    Setting up libip4tc0:amd64 (1.6.0+snapshot20161117-6) ...
    Processing triggers for libc-bin (2.24-11+deb9u4) ...
    root@e3-2186:~# lxc-start -n 103 -F -l DEBUG -o /tmp/lxc-103.log
    /sbin/init: error while loading shared libraries: libip4tc.so.0: cannot open shared object file: No such file or directory
    
    Thinking I have to do a complete reinstall of the node, I tried to mount container file system using pct mount 103, which was successful. It turns out that the /usr folder for that container is completely missing, which may explain some of these problems?

    The currently live containers do have /usr folders that look normal, but still remain unaccessible via pct enter, SSH, or web console.

    What's the next recommend course of action? Seems like I should reinstall the whole PVE ISO and start fresh, but would you like any other logs that would help figure out the cause? As far as I'm aware, there have been no manual actions taken that would have caused borking the system.
     
  8. oguz

    oguz Proxmox Staff Member
    Staff Member

    Joined:
    Nov 19, 2018
    Messages:
    603
    Likes Received:
    63
    looks like it. you can still try to take backups of your guests and restore them back later.

    what i'd do if i were you, would be to take the server offline if possible, and perform some inspections to determine whether there was an intrusion (since you saw a ton of failed logins). maybe you were hacked, or maybe you or your admin did a mistake... i'd start with logs in /var/log/ (for example syslog and access/auth related logs). run some scripts like rkhunter/chkrootkit maybe.

    probably, but that's likely a consequence rather than a cause. you can make a tar archive from the mounted directory to restore the files later.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice