Cannot open /proc/stat: Transport endpoint is not connected
- Thread starter SPQRInc
- Start date
The Transport error is very likely originated by lxcfs, which is a tool that can allow containers with systemd to still interact with the cgroup system and display correct uptime and memory for the container.
https://linuxcontainers.org/lxcfs/introduction/
https://s3hh.wordpress.com/2015/02/23/introducing-lxcfs/
Is there anything special displayed on the host with:
Also I see some qmgr errors with apparmor DENIED. There is only a few days now a newer kernel on the no subscriptions packages which may fix that error.
https://linuxcontainers.org/lxcfs/introduction/
https://s3hh.wordpress.com/2015/02/23/introducing-lxcfs/
Is there anything special displayed on the host with:
Code:
journalctl -u lxcfsAlso I see some qmgr errors with apparmor DENIED. There is only a few days now a newer kernel on the no subscriptions packages which may fix that error.
Hi windinternet,
thanks a lot for your reply. I installed the new kernel - now I'm waiting for reboot and double-check if the problem persists.
This is the output of journalctl -u lxcfs
Edit: Character-limitation - full log is pasted here: http://d.pr/18cna
thanks a lot for your reply. I installed the new kernel - now I'm waiting for reboot and double-check if the problem persists.
This is the output of journalctl -u lxcfs
Edit: Character-limitation - full log is pasted here: http://d.pr/18cna
Can you tie the recursive rm_dir errors to moments of shutdown of that container?
It's probably a race condition on shutdown, where lxc-stop tries to remove there dirs while the container is still hanging on to them. It sometimes can happen on start too, but only if the container is improperly configured. It might damage later starting of the container. If all containers are stopped you should be able to refresh lxcfs by restarting the lxcfs service.
It's probably a race condition on shutdown, where lxc-stop tries to remove there dirs while the container is still hanging on to them. It sometimes can happen on start too, but only if the container is improperly configured. It might damage later starting of the container. If all containers are stopped you should be able to refresh lxcfs by restarting the lxcfs service.
Its possible that the error appears on these events. I never saw it "live" - I always mentioned it later when I tried to restart services like dovecot or php5-fpm.
So you would suggest to stop all containers, restart lxcfs (service lxcfs restart) and start containers again?
What makes me crazy is that I already restarted the whole system, started the containers and they all are working fine for hours - sometimes for days.
So you would suggest to stop all containers, restart lxcfs (service lxcfs restart) and start containers again?
What makes me crazy is that I already restarted the whole system, started the containers and they all are working fine for hours - sometimes for days.
It may be that one container shutdown ruins things for other running containers. Or it maybe that these error messages are actually totally unrelated. In any case, not being able to restart services and problems with top indicate that lxcfs is not responding anymore, and consequently systemd cannot function because it works with cgroups which are emulated by lxcfs inside the container, and the same with different files for top and uptime.
If that is the case, trying a service lxcfs restart wouldn't hurt.
If that is the case, trying a service lxcfs restart wouldn't hurt.
I looked up some logs, and actually the recursive_rmdir error seems 'normal'.
The send_creds and other errors not however.
The send_creds and other errors not however.
Looking back at your software list, I notice that you have or had lxcfs 0.13-pve1. The current version is lxcfs 0.13-pve2. There have been some fixes in it. Also the newest kernel patch may help with the postfix errors that were visible in the logs.
It may be that you can't reproduce anymore because you did the updates.
It may be that you can't reproduce anymore because you did the updates.
Okay. Is it an error with one container while others continue running? Or do all your systemd lxc containers get stuck, not able to restart services or display top? Did you get the send_creds errors again in the syslog? Any other errors around the same time in the syslog?
In the stuck container:
What is the output of systemctl --version? What is the content of /etc/*-release or /etc/issue?
In the stuck container:
What is the output of systemctl --version? What is the content of /etc/*-release or /etc/issue?
Hello and thanks a lot for your reply 
Well, all containers are running, but all are having the same problem at the same time with top/htop, reloading/starting services using systemd.
The output of systemctl --version on one of the containers (having this error now)
Output of /etc/*-release:
Output of /etc/issue
The syslog shows these lines if I grep for send_creds:
So I did not restart the containers for a while now - maybe that's the reason why there are no more entries in syslog.
I already thought about faulty mounting-options. This is the fstab-file in a container:
Well, all containers are running, but all are having the same problem at the same time with top/htop, reloading/starting services using systemd.
The output of systemctl --version on one of the containers (having this error now)
Output of /etc/*-release:
Output of /etc/issue
The syslog shows these lines if I grep for send_creds:
So I did not restart the containers for a while now - maybe that's the reason why there are no more entries in syslog.
I already thought about faulty mounting-options. This is the fstab-file in a container:
No, I think it is lxcfs not being able to keep up with quickly spawning deamons and a stressed process scheduler that is hurting you. It uses a process in the container to read process ids, and normal processes get scheduled out also.
Maybe it helps to give the busiest container more CPU priority.
Maybe it helps to give the busiest container more CPU priority.
It's a relative setting. If you increase all, it has zero effect. You must interpret this as the weight you give a container with respect to other containers. If they all have the same weight, then they all get the same amount of cpu time.
Maybe it would also make a difference to add a:
In the Service section of the /etc/systemd/system/multi-user.target.wants/lxcfs.service file and restart the lxcfs service.
Code:
Nice=-20In the Service section of the /etc/systemd/system/multi-user.target.wants/lxcfs.service file and restart the lxcfs service.