Cannot open /proc/stat: Transport endpoint is not connected

[windinternet]

Yes, it must be in the Service section. Doesn't matter on which line inside the section.

After changing the file, to get it applied you need to do:

systemctl daemon-reload
systemctl restart lxcfs

 

[windinternet]

For info about the cpu-limit, cpu-unit settings look at this thread.

https://forum.proxmox.com/threads/q-lxc-configuration-wizard-what-is-cpu-limit-cpu-unit.22661

 

[windinternet]

Also I have been putting a Proxmox with systemd jessie and centos containers with nginx and php-fpm under pressure, but haven't yet found a reproduce of lxcfs problems.

I am wondering about something though, if a system has this problem what is the output of this on the host:

df /run

 

[SPQRInc]

Hello,

this is the output:

df /run

Dateisystem 1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf

tmpfs 26380888 317044 26063844 2% /run

 

[windinternet]

@SPQRInc,

Ok, so that does not seem out of order or anything. But you are still getting the lockups of htop and php-fpm reloads?

 

[SPQRInc]

I just mentioned that the error happened just now:

Jan 5 19:06:08 wirtssystem kernel: [54271.699087] audit: type=1400 audit(1452017168.545:146): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/" pid=13936 comm="mount" flags="rw, nosuid, nodev, noexec, remount"

Jan 5 19:10:24 wirtssystem pvedaemon[17287]: <root@pam> successful auth for user 'root@pam'

Jan 5 19:12:01 wirtssystem lxcfs[17340]: *** Error in `/usr/bin/lxcfs': realloc(): invalid next size: 0x00007f8700002bf0 ***

Jan 5 19:12:01 wirtssystem systemd[1]: lxcfs.service: main process exited, code=killed, status=6/ABRT

Jan 5 19:12:01 wirtssystem systemd[1]: Unit lxcfs.service entered failed state.

Jan 5 19:12:01 wirtssystem lxcfs[10726]: hierarchies: 1: name=systemd

There was nothing with FPM in this moment.

 

[windinternet]

Hello @SPQRInc

Thanks for reporting it. I think I have a fix on the problem now. The apparmor DENIED message makes me suspect that there is a readonly recursive mount of /sys just before that by systemd, which is being allowed, so that wouldn't show up in the log. Problem is with the way it is mounted to the container by a Proxmox script, that this probably locks up the entire lxcfs system for all containers.

When lxcfs is locked up like that, it probably starts looping around, runs out of stack and starts thrashing the heap, and anything can happen, like being killed by the heap canary that reports the "invalid next size" on realloc. I have inspected the code surrounding that realloc and there seems to be nothing wrong with it that would lead to heap corruption on its own.

It should be comparatively simple to fix by Proxmox if this is the case. If you have access to the container you could look at its mounts, and see if any of them are readonly, especially the cgroup mounts.

 

[SPQRInc]

Hello,

some minutes ago the error appeared again.

This is my mount:

mount
/var/lib/vz/images/102/vm-102-disk-1.raw on / type ext4 (rw,relatime,data=ordered)

none on /dev type tmpfs (rw,relatime,size=100k,mode=755)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)

proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)

proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)

sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)

sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)

fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,size=12k,mode=755)

tmpfs on /sys/fs/cgroup/cgmanager type tmpfs (rw,mode=755)

lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/blkio type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/cpu,cpuacct type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/cpuset type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/devices type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/freezer type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/hugetlb type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/memory type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/systemd type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/net_cls,net_prio type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

lxcfs on /sys/fs/cgroup/perf_event type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)

devpts on /dev/tty1 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)

devpts on /dev/tty2 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)

tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)

mqueue on /dev/mqueue type mqueue (rw,relatime)

tmpfs on /tmp type tmpfs (rw)

hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,relatime)

none on /run/shm type tmpfs (rw,relative)

Syslog shows the following

Jan 6 11:17:01 schwerin CRON[8561]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Jan 6 11:25:36 schwerin systemd-timesyncd[535]: interval/delta/delay/jitter/drift 2048s/-0.000s/0.004s/0.001s/+3ppm

Jan 6 11:27:11 schwerin pvedaemon[24416]: <root@pam> successful auth for user 'root@pam'

Jan 6 11:29:20 schwerin pvedaemon[19316]: worker exit

Jan 6 11:29:20 schwerin pvedaemon[1189]: worker 19316 finished

Jan 6 11:29:20 schwerin pvedaemon[1189]: starting 1 worker(s)

Jan 6 11:29:20 schwerin pvedaemon[1189]: worker 18489 started

Jan 6 11:31:25 schwerin pvedaemon[24416]: worker exit

Jan 6 11:31:25 schwerin pvedaemon[1189]: worker 24416 finished

Jan 6 11:31:25 schwerin pvedaemon[1189]: starting 1 worker(s)

Jan 6 11:31:25 schwerin pvedaemon[1189]: worker 20681 started

Jan 6 11:32:15 schwerin pveproxy[8250]: worker exit

Jan 6 11:32:15 schwerin pveproxy[1197]: worker 8250 finished

Jan 6 11:32:15 schwerin pveproxy[1197]: starting 1 worker(s)

Jan 6 11:32:15 schwerin pveproxy[1197]: worker 21315 started

Jan 6 11:35:32 schwerin rrdcached[975]: flushing old values

Jan 6 11:35:32 schwerin rrdcached[975]: rotating journals

Jan 6 11:35:32 schwerin rrdcached[975]: started new journal /var/lib/rrdcached/journal/rrd.journal.1452076532.484315

Jan 6 11:35:32 schwerin rrdcached[975]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1452069332.484396

Jan 6 11:39:35 schwerin pvedaemon[31944]: worker exit

Jan 6 11:39:35 schwerin pvedaemon[1189]: worker 31944 finished

Jan 6 11:39:35 schwerin pvedaemon[1189]: starting 1 worker(s)

Jan 6 11:39:35 schwerin pvedaemon[1189]: worker 27755 started

Jan 6 11:40:09 schwerin pveproxy[4382]: worker exit

Jan 6 11:40:09 schwerin pveproxy[1197]: worker 4382 finished

Jan 6 11:40:09 schwerin pveproxy[1197]: starting 1 worker(s)

Jan 6 11:40:09 schwerin pveproxy[1197]: worker 28474 started

Jan 6 11:42:11 schwerin pvedaemon[27755]: <root@pam> successful auth for user 'root@pam'

Jan 6 11:46:02 schwerin pveproxy[21315]: worker exit

Jan 6 11:46:02 schwerin pveproxy[1197]: worker 21315 finished

Jan 6 11:46:02 schwerin pveproxy[1197]: starting 1 worker(s)

Jan 6 11:46:02 schwerin pveproxy[1197]: worker 1167 started

Jan 6 11:48:51 schwerin lxcfs[919]: *** Error in `/usr/bin/lxcfs': realloc(): invalid next size: 0x00007feefc001e00 ***

Jan 6 11:48:51 schwerin systemd[1]: lxcfs.service: main process exited, code=killed, status=6/ABRT

Jan 6 11:48:51 schwerin systemd[1]: Unit lxcfs.service entered failed state.

Jan 6 11:48:51 schwerin lxcfs[3635]: hierarchies: 1: name=systemd

Jan 6 11:48:51 schwerin lxcfs[3635]: 2: cpuset

Jan 6 11:48:51 schwerin lxcfs[3635]: 3: cpu,cpuacct

Jan 6 11:48:51 schwerin lxcfs[3635]: 4: blkio

Jan 6 11:48:51 schwerin lxcfs[3635]: 5: memory

Jan 6 11:48:51 schwerin lxcfs[3635]: 6: devices

Jan 6 11:48:51 schwerin lxcfs[3635]: 7: freezer

Jan 6 11:48:51 schwerin lxcfs[3635]: 8: net_cls,net_prio

Jan 6 11:48:51 schwerin lxcfs[3635]: 9: perf_event

Jan 6 11:48:51 schwerin lxcfs[3635]: 10: hugetlb

Jan 6 11:48:51 schwerin kernel: [11607.540174] cgroup: new mount options do not match the existing superblock, will be ignored

Jan 6 11:50:01 schwerin pveproxy[5893]: worker exit

Jan 6 11:50:01 schwerin pveproxy[1197]: worker 5893 finished

Jan 6 11:50:01 schwerin pveproxy[1197]: starting 1 worker(s)

Jan 6 11:50:01 schwerin pveproxy[1197]: worker 4606 started

Jan 6 11:51:45 schwerin systemd[6105]: Starting Paths.

Jan 6 11:51:45 schwerin systemd[6105]: Reached target Paths.

Jan 6 11:51:45 schwerin systemd[6105]: Starting Timers.

Jan 6 11:51:45 schwerin systemd[6105]: Reached target Timers.

Jan 6 11:51:45 schwerin systemd[6105]: Starting Sockets.

Jan 6 11:51:45 schwerin systemd[6105]: Reached target Sockets.

Jan 6 11:51:45 schwerin systemd[6105]: Starting Basic System.

Jan 6 11:51:45 schwerin systemd[6105]: Reached target Basic System.

Jan 6 11:51:45 schwerin systemd[6105]: Starting Default.

Jan 6 11:51:45 schwerin systemd[6105]: Reached target Default.

Jan 6 11:51:45 schwerin systemd[6105]: Startup finished in 7ms.

 

[windinternet]

Hello @SPQRInc

The mounts look good, or at least not any weirder than usual in the containe. I tested for remount problems, as i suggested earlier, but that doesn't seem to be possible. Are the containers hitting their memory limits at the point of failure?

 

[SPQRInc]

Thank you for your answer

No, the containers are using max 40% memory and they have 120GB vSwap. Even CPU-usage is low.

 

[windinternet]

Have to rule something out though. Can you show the configuration files for container 102? Both /etc/pve/lxc/102.conf and /var/lib/lxc/102/config.

In the past Proxmox did kernel memory limiting, and that doesn't show up in the memory usage statistic.

 

[SPQRInc]

Hello winternet, thanks again for your reply,

here they are

arch: amd64
cpulimit: 12
cpuunits: 2048
hostname: example
memory: 24576
net0: bridge=vmbr0,gw=123.123.5.1,hwaddr=36:37:32:34:33:62,ip=123.123.123.1/24,name=eth0,type=veth
net1: bridge=vmbr0,gw=123.123.5.1,hwaddr=32:32:36:38:62:37,ip=123.123.123.2/24,name=eth0.1,type=veth
net2: bridge=vmbr0,gw=123.123.5.1,hwaddr=36:31:62:38:35:38,ip=123.123.123.3/24,name=eth0.2,type=veth
onboot: 1
ostype: debian
rootfs: local:102/vm-102-disk-1.raw,size=400G
swap: 122880

lxc.arch = amd64
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.monitor.unshare = 1
lxc.tty = 2
lxc.environment = TERM=linux
lxc.utsname = example
lxc.cgroup.memory.limit_in_bytes = 25769803776
lxc.cgroup.memory.memsw.limit_in_bytes = 154618822656
lxc.cgroup.cpu.cfs_period_us = 100000
lxc.cgroup.cpu.cfs_quota_us = 1200000
lxc.cgroup.cpu.shares = 2048
lxc.rootfs = /var/lib/lxc/102/rootfs
lxc.network.type = veth
lxc.network.veth.pair = veth102i0
lxc.network.hwaddr = 36:37:32:34:33:62
lxc.network.name = eth0
lxc.network.type = veth
lxc.network.veth.pair = veth102i2
lxc.network.hwaddr = 36:31:62:38:35:38
lxc.network.name = eth0.2
lxc.network.type = veth
lxc.network.veth.pair = veth102i1
lxc.network.hwaddr = 32:32:36:38:62:37
lxc.network.name = eth0.1

 

[hregis]

Hi

I think I have the same problem:

https://forum.proxmox.com/threads/proxmox-4-1-error-proc-must-be-mounted.25509/

 

[SPQRInc]

There is one point I mentioned: NTP was installed and activated by a software which was installed on all LXC-containers. This is - of course - not necessary on LXC, but maybe this could be a point?

I turned off NTP on clients yesterday evening and the error did not occur again for almost 12 hours now.

cat /var/log/syslog.1 | grep realloc | wc -l
3 #this is the count of errors before I changed NTP on all containers

cat /var/log/syslog | grep realloc | wc -l
0 # todays error-count

 

[windinternet]

@hregis

Do you have systemd installed in your debian containers (Debian 8)? Have you looked at memory consumption of your containers?

If I am correct you overcommited your RAM about three times (with all containers running). I also notice that systemd on the host is using a whopping 4GB virtual memory. That looks weird to me.

@SPQRInc

What does your memory overcommit look like?


I just noticed that the memory consumption statistic in Proxmox ONLY shows the amount of userspace memory. It doesn't include usage of kernel structures and disk caches by the container. The virtual swap will only be used when THAT narrowly accounted memory runs out, I think, by which time the container may use double that or more in host RAM.

 

[SPQRInc]

I do not know exactly what you mean with "what does your memory overcommit look like?" - I assume you mean

cat /proc/sys/vm/overcommit_memory

0

?

 

[windinternet]

@SPQRInc

About the NTP. That is very interesting. The LXC containers explicitly drop the capability to set the system time at start up. This might lead to some undefined behaviour when NTP thinks it is running as root user.

The overcommit, I meant the amount of RAM you set for the containers that you run together. If you take the memory statistic from Proxmox, you must be sure to allow for enough spare room for disk caching, the host software, and some kernel structures.

 

[SPQRInc]

Hi @windinternet

well, disabling NTP did unfortunately not solve the problem:

Jan 7 14:14:47 wirtssystem systemd-timesyncd[548]: interval/delta/delay/jitter/drift 2048s/+0.006s/0.011s/0.006s/+4ppm

Jan 7 14:16:38 wirtssystem rrdcached[1006]: flushing old values

Jan 7 14:16:38 wirtssystem rrdcached[1006]: rotating journals

Jan 7 14:16:38 wirtssystem rrdcached[1006]: started new journal /var/lib/rrdcached/journal/rrd.journal.1452172598.799698

Jan 7 14:16:38 wirtssystem rrdcached[1006]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1452165398.799708

Jan 7 14:17:01 wirtssystem CRON[6450]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Jan 7 14:48:55 wirtssystem systemd-timesyncd[548]: interval/delta/delay/jitter/drift 2048s/-0.008s/0.022s/0.006s/+4ppm

Jan 7 14:50:34 wirtssystem lxcfs[29895]: *** Error in `/usr/bin/lxcfs': realloc(): invalid next size: 0x00007f0c8c002d50 ***

Jan 7 14:50:34 wirtssystem systemd[1]: lxcfs.service: main process exited, code=killed, status=6/ABRT

Jan 7 14:50:34 wirtssystem systemd[1]: Unit lxcfs.service entered failed state.

All containers had a maximum ram usage of 70%.

Altogether I assigned the following memory-values to my containers:

102: 24G
103: 40G
104: 16G
105: 24G
200: 1G

= 105GB


Additionally there is a KVM-machine using 8G (fix).

The server got 128G ECC RAM altogether

free -m

total used free shared buffers cached

Mem: 128812 127724 1088 969 5594 89526

-/+ buffers/cache: 32604 96208

Swap: 7812 0 7812

 

[windinternet]

@SPQRInc

Your RAM usage does not look bad. You can easily assign 30G more while keeping enough file cache, if these numbers are typical.

Are you seeing a lot of memory usage by the systemd process on the host also, like @hregis?

Could you do:

cat /proc/1/status
lsof -p 1

 

[windinternet]

Good news. A fix has been made in the lxcfs project for a faulty realloc call. This will be incorporated in lxcfs package for Proxmox and will probably fix the dying of lxcfs by the realloc error.