/proc and /sys missing for pct enter container but exists for ssh session in

[mathx]

Something funky with pct enter -- this just stared happening, wasnt occuring before. Something's changed (no no packages have been updated on the container that i know of... but obviously something changed while I wasnt looking...)

root@arch:/etc/pve/nodes/arch/lxc# pct enter 909
website:/# ps auxwwf
Error, do this: mount -t proc proc /proc
website:/# mount -t proc proc /proc
mount: proc is write-protected, mounting read-only
mount: cannot mount proc read-only
website:/# mount
mount: failed to read mtab: No such file or directory

website:/# ls -la /sys
total 9
drwxr-xr-x 2 root root 2 Mar 26 2012 .
drwxr-xr-x 23 root root 23 Jun 14 17:50 ..

website:/# ip a | grep inet.*eth0
inet 192.168.1.169/25 brd 199.27.180.255 scope global eth0
website:/# exit
exit
root@arch:/etc/pve/nodes/arch/lxc# ssh -192.168.1.169
Linux website 4.15.18-11-pve #1 SMP PVE 4.15.18-34 (Mon, 25 Feb 2019 14:51:06 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have new mail.
Last login: Thu Aug 29 09:36:27 2019 from arch
website:~# ls -la /proc | wc -l
182
website:~# cat /proc/mounts | grep proc
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0

However something else is also still wrong, mysql wont start (didnt look into it deeply yet).

I have found many other issues with pct enter (it's not a real terminal for eg cant run screen in it) -- should it be avoided in general? This is a new one to me though, a different view of the filesystem to the point where /proc is missing somehow.

Restarting the container fixed it (and mysql).

version: pve-manager/5.3-12/5fbbbaf6 (running kernel: 4.15.18-11-pve)

 

[fiona]

Hi,
'pct enter' does a 'lxc-attach' in the background and does not create a virtual console, which you do get with 'ssh'.
But other than that it shouldn't make much difference, so it's unlikely that 'pct enter' caused the problem.

 

[mathx]

Ok upgraded to PVE6.3 and this is still happening.

/proc went missing in this container when I pve enter.

pve-manager/6.3-2/22f57405 (running kernel: 5.4.73-1-pve)

problem is I fear that restarting daemons or anything else from this shell blindly may also have them inherit this broken security context, and have /proc missing which you can imagine is a major issue.

I think this is related to nesting=1 but this is an older Centos 5.5 container that wouldnt use that.

note that this has been happening from Centos 5.5 to Centos 8 and debian lenny to buster. There's no restriction/version requirement for this /proc issue to pop up (but most of my experience has been on 5.x, only recently getting containers sufficiently moved around to begin upgrading to 6.x)

 

[fiona]

I tried around a bit more and can reproduce this here, but only if I restart the lxcfs.service while the container is running. On upgrades, the service should only be reloaded and this shouldn't happen.

Some questions:

1. Is the issue more likely to appear after upgrades?
2. When the issue appears, does systemctl status lxcfs.service show that the service is up and running and is there anything noticable in the log, i.e. journalctl -u lxcfs.service?
3. What is the output of pveversion -v, which lxc-attach and lxc-attach --version?

In any case, containers should be restarted if the lxcfs.service was restarted (for whatever reason).

 

[mathx]

I am not upgrading or restarting lxcfs.service to cause this. It seems to happen by itself. (Unless OOM pressure has killed it and it has restarted itself or something similar?) Will investigate more.

 

[fiona]

I am not upgrading or restarting lxcfs.service to cause this.

I'm just asking to make sure.

It seems to happen by itself. (Unless OOM pressure has killed it and it has restarted itself or something similar?) Will investigate more.

You might not be the only one running into this (if the culprit is also an lxcfs crash in your case).

Please also try and install the debug symbols apt install lxcfs-dbgsym and systemd-coredump? When the crash happens next time, there should be a core dump in /var/lib/systemd/coredump. It would be great if you could provide that. And please provide the exact version that is installed at that time.