Cannot open /proc/stat: Transport endpoint is not connected

M

mstgeo

Member
May 15, 2022
40
1
8
Hi Guys,

I started to have a problem on a Proxmox v6 with LXC.

After some time after run of a core server We're getting in VM while top / htop the message:

Cannot open /proc/stat: Transport endpoint is not connected

(top: failed /proc/stat open: Transport endpoint is not connected)

A stop / start for VM helps for a short time and a problem comes back...

The VM is debian 10 while the Proxmox is based on:

Linux server 5.4.174-2-pve #1 SMP PVE 5.4.174-2 (Thu, 10 Mar 2022 15:58:44 +0100) x86_64 GNU/Linux

Anyone has any ideas what could it be and how to get rid of this problem.

Thanks in advance ! ;)

regards,
G.
 
Hi,
maybe try to take a look into the logs of lxcfs:

Code: 
journalctl -u lxcfs
 
HI!

Thx for an answer... I increased ulimit, however I am still getting the below:

Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 996: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 25 20:00:08 linux-server lxcfs[70016]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:08 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:10 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 25 20:00:10 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 25 20:00:10 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 25 20:00:10 linux-server lxcfs[70016]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 25 20:00:14 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 25 20:00:14 linux-server lxcfs[70016]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 25 20:00:14 linux-server lxcfs[70016]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
 
I see a similar error with LXC on proxmox 7, though it fixed after rebooting the container and has not happened since.
 
Still same... Even with increased files limit values...

root@linux-server:~# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3090469
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 500536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3090469
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@linux-server:~#
root@linux-server:~#
root@linux-server:~# more /proc/sys/fs/file-nr
162624 0 2735680




Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 996: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 10:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:08 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:10 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 27 10:00:10 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 27 10:00:10 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 27 10:00:10 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 27 10:00:10 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
Jun 27 10:00:13 linux-server lxcfs[2421]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 27 10:00:14 linux-server lxcfs[2421]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
Jun 27 10:00:14 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated
Jun 27 10:00:14 linux-server lxcfs[2421]: proc_fuse.c: 959: proc_stat_read: Write to cache was truncated




Any idea how to get rid of these ? Thx!

cya!
 
Maybe try to find out what process has so many file handles opened:

Code: 
lsof | awk '{print $1}' | sort | uniq -c | sort -r -n | head

# or to get the PID of the process in the second column you can substitute $1 with $2
lsof | awk '{print $2}' | sort | uniq -c | sort -r -n | head
 
also did you restart the system after chaning the the open file limit?
 
Did you run the command as well to see which process is using so many file handles?
 
4347224 mysqld
270200 java
197123 apache2
25744 kvm
14326 master
13961 node
12609 rsyslogd
8120 systemd
7920 polkitd
7392 unattende

We have a big machine where we have ~100 LXC VMs. mysql eats a lot. However the system limits are still OK and errors / warning should not occur.

Hmm...

Any idea ?

## ulimit

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3090469
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 900536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3090469
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


## sysctl.conf
fs.file-max=9793398


more /proc/sys/fs/file-nr gives:
161280 0 9793398

and still
journalctl -u lxcfs

gives:

Jun 27 18:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 18:00:08 linux-server lxcfs[2421]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: Too many open files
Jun 27 18:00:08 linux-server lxcfs[2421]: utils.c: 338: read_file_fuse: Write to cache was truncated
Jun 27 18:00:08 linux-server lxcfs[2421]: utils.c: 198: wait_for_sock: Too many open files - Failed to create epoll socket
Jun 27 18:00:08 linux-server lxcfs[2421]: utils.c: 291: send_creds: Too many open files - Failed getting reply from server over socketpair: 2
Jun 27 18:00:14 linux-server lxcfs[2421]: proc_fuse.c: 1029: proc_stat_read: Write to cache was truncated
 
Last edited:
Lets try to check the limits of the lxcfs process:

Code: 
$ ps aux | grep lxcfs
root        2125  0.0  0.0 150912  2024 ?        Ssl  08:16   0:00 /usr/bin/lxcfs /var/lib/lxcfs
root     1761479  0.0  0.0   6180   728 pts/0    S+   09:04   0:00 grep lxcfs


$ cat /proc/2125/limits
Limit                     Soft Limit           Hard Limit           Units    
Max cpu time              unlimited            unlimited            seconds  
Max file size             unlimited            unlimited            bytes    
Max data size             unlimited            unlimited            bytes    
Max stack size            8388608              unlimited            bytes    
Max core file size        0                    unlimited            bytes    
Max resident set          unlimited            unlimited            bytes    
Max processes             256220               256220               processes
Max open files            1024                 524288               files    
Max locked memory         65536                65536                bytes    
Max address space         unlimited            unlimited            bytes    
Max file locks            unlimited            unlimited            locks    
Max pending signals       256220               256220               signals  
Max msgqueue size         819200               819200               bytes    
Max nice priority         0                    0                   
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us
 
Catch! ;)

root@linux-server:~# ps aux | grep lxcfs
root 2421 22.7 0.0 50383984 144040 ? Ssl Jun26 493:57 /usr/bin/lxcfs /var/lib/lxcfs
root 34671 0.0 0.0 6136 2332 pts/109 S+ 08:24 0:00 grep lxcfs


root@linux-server:~# cat /proc/2421/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 3090469 3090469 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 3090469 3090469 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
root@linux-server:~#
 
And now we take a look at how many files lxcfs has opened:

Code: 
lsof | awk '{print $1}' | sort | uniq -c | sort -r -n | grep lxcfs
 
So now lets increase the file limit of the lxcfs process, for exmaple to 10240

Code: 
prlimit --nofile=10240: --pid <pid of lxcfs process>
 
DONE!

root@linux-server:~# cat /proc/2421/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 3090469 3090469 processes
Max open files 10240 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 3090469 3090469 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
 
now lets see if you still get the error from lxcfs :). If you get it check how many files are opened with:

Code: 
lsof | awk '{print $1}' | sort | uniq -c | sort -r -n | grep lxcfs

and yes making it permanent is the next step ..., but lets see if it works
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back