lxcfs: utils.c: 331: read_file_fuse: Write to cache was truncated

[olegfusion]

Hello! May anybody please explain what is this error? Getting it in log every 5 seconds.
proxmox 6.1-8
lxcfs: utils.c: 331: read_file_fuse: Write to cache was truncated
Thanks

 

[oguz]

hi,

could you please provide some more context?

-> output of pveversion -v
-> config of CT pct config CTID

also we pushed a new version of lxcfs on our repositories very recently, so its possible that updating will make the problem go away

 

[olegfusion]

root@pve0:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-2
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1


I have many lxc, one of them has mount point. Other don't have and actually are same:
root@pve0:~# pct config 106
arch: amd64
cores: 2
hostname: borg
memory: 1024
mp0: /mnt/pve/backup14tb/BORG,mp=/backup
net0: name=eth0,bridge=vmbr1,gw=172.16.10.1,hwaddr=BE:63:63:9E5:92,ip=172.16.10.85/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: datastore1:subvol-106-disk-0,mountoptions=noatime,size=10G
startup: order=3,up=10
swap: 0
unprivileged: 1

 

[olegfusion]

any help? after reboot nothing is changed. proxmox is latest version

 

[DaryL]

Hi ! I have the same problem since the pve5->6 upgrade, I got this message in syslog 3 times per second :

Apr 16 21:05:25 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:25 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:25 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:26 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:26 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:26 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:27 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:27 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 16 21:05:27 srv lxcfs[738]: utils.c: 331: read_file_fuse: Write to cache was truncated

I tried to reboot (because my OVH server dropped its link negociation to 10Mb/s instead of 1000Mb/s, don't know why), but this message comes again after few hours of uptime.

Here is my pveversion :

proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-9
pve-kernel-5.3: 6.1-6
pve-kernel-4.15: 5.4-16
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-21-pve: 4.15.18-48
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-19
libpve-guest-common-perl: 3.0-7
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.0-2
lxcfs: 4.0.2-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-5
pve-container: 3.1-2
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-5
pve-qemu-kvm: 4.2.0-1
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1

And one of my containers config :

arch: amd64
cores: 2
hostname: yo
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=xxx.xxx.xxx.xxx,gw6=2001:xxx:x:xxff:ff:ff:ff:ff,hwaddr=xx:xx:xx:xx:xx:xx,ip=xxx.xxx.xxx.xxx/32,ip6=2001:xxx:x:xxx::xx/64,type=veth
net1: name=eth1,bridge=vmbr2,hwaddr=EE:80:80:99:DB:8D,ip=192.168.0.157/24,ip6=fd79:1d3e:7310:e22f::157/64,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-157-disk-0,size=100G
swap: 512

 

[SekundenKleber]

The same Problem on my Proxmox-Server ... every one second

Apr 26 09:44:07 pve lxcfs[960]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 26 09:44:08 pve lxcfs[960]: utils.c: 331: read_file_fuse: Write to cache was truncated
Apr 26 09:44:09 pve lxcfs[960]: utils.c: 331: read_file_fuse: Write to cache was truncated

proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[SekundenKleber]

It seems there is a problem with the privileged containers. It is not enough to stop the Conatiner, only after a restart of the server the problem is no longer present.

 

[DaryL]

All my containers are privileged containers (Unprivileged container option set to No).
Do you mean I have to change all my containers to Unprivileged to remove this error ?

 

[Brandon]

I'm also seeing this issue on priveleged containers after upgrading recently.

proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-2-pve: 5.3.13-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 14.2.9-pve1
ceph-fuse: 14.2.9-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

---UPDATE---
It appears that if i just shutdown a handful of priveleged containers and keep it down to less then 5 the errors go away. This might have something to do with cgroup limits?

 

[SekundenKleber]

All my containers are privileged containers (Unprivileged container option set to No).
Do you mean I have to change all my containers to Unprivileged to remove this error ?

I would at least test it on a container first.

 

[greg]

I have the same messages since I upgraded from 5 to 6. I don't know if it's related, but now snapshoting some of the CT freezes, making the CT unresponsive all night (until I just kill -9 the snapshot process).

 

[Gardouille]

I just had this message on a container with a feature (nesting) enable.

Unable to make it disapear, even with :

• Disable the feature
• Stop the container
• Restore the container

The only way was to move it to another host (i kept nesting feature disable). I guess restart the host should also do the job.

Proxmox 6 up to date and privileged LXC container.

 

[grin]

It is not clear to me how lxcfs error related to container config but anyway, I have only unprivileged CTs, and have this spamming error for quite a while now.
Tracking it back there was a reboot (due to various network problems and, eventually, getting fenced) and after reboot and start and running for a short while (30 minutes) one of the busiest CT started logging fancy things:

Sep  7 22:12:27 xx lxcfs[1479]: proc_fuse.c: 881: proc_stat_read: cpu1 from /lxc/901/ns has unexpected cpu time: 30425 in /proc/stat, 30437 in cpuacct.usage_all; unable to determine idle time
Sep  7 22:12:27 xx lxcfs[1479]: proc_fuse.c: 881: proc_stat_read: cpu1 from /lxc/901/ns has unexpected cpu time: 30426 in /proc/stat, 30440 in cpuacct.usage_all; unable to determine idle time
Sep  7 22:12:27 xx lxcfs[1479]: proc_fuse.c: 881: proc_stat_read: cpu1 from /lxc/901/ns has unexpected cpu time: 30428 in /proc/stat, 30443 in cpuacct.usage_all; unable to determine idle time
Sep  7 22:12:28 xx lxcfs[1479]: proc_fuse.c: 881: proc_stat_read: cpu1 from /lxc/901/ns has unexpected cpu time: 30435 in /proc/stat, 30459 in cpuacct.usage_all; unable to determine idle time
Sep  7 22:12:28 xx lxcfs[1479]: proc_fuse.c: 881: proc_stat_read: cpu1 from /lxc/901/ns has unexpected cpu time: 30435 in /proc/stat, 30459 in cpuacct.usage_all; unable to determine idle time

en masse. After about 2400 lines it started complaining about

> lxcfs[1479]: utils.c: 331: read_file_fuse: Write to cache was truncated

first just a few in minutes, then at full throttle. For days. Then the CT was fiddled with (probably snapshotted), and the cpuacct error stopped, while the lxcfs prevailed.

It was eventually "fixed" by restarting lxcfs. For now.


https://bugzilla.proxmox.com/show_bug.cgi?id=3031

 

[lps90]

Any solution?
Same happening with me.
Proxmox is up-to-date.

 

[MK.Persia]

Any updates on this issue? I have the same problem. Stopping and even removing privileged containers didn't improve the situation! host server keeps restarting randomly (between 3h to 48h).

#pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.13-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

#cat /proc/stat
cpu  538499 16756 305104 123201275 261737 0 122574 0 0 0
cpu0 45560 1 14143 7700096 15948 0 87674 0 0 0
cpu1 27841 44 14790 7724252 13528 0 16607 0 0 0
cpu2 48493 2 13963 7704101 15747 0 6638 0 0 0
cpu3 51255 2976 17206 7691258 16891 0 3664 0 0 0
cpu4 45572 4333 22096 7669759 24165 0 2741 0 0 0
cpu5 31604 5 16201 7700183 26982 0 1271 0 0 0
cpu6 8699 9 11762 7756785 13425 0 488 0 0 0
cpu7 61211 4 59751 7560729 16178 0 683 0 0 0
cpu8 24918 30 13665 7724892 14844 0 148 0 0 0
cpu9 44089 2 31280 7648281 20544 0 1253 0 0 0
cpu10 20229 5182 16836 7716102 11968 0 289 0 0 0
cpu11 15822 4126 15537 7725276 10718 0 346 0 0 0
cpu12 30358 22 12768 7720747 14166 0 223 0 0 0
cpu13 30010 6 12940 7721704 14552 0 87 0 0 0
cpu14 26398 4 14979 7721657 16000 0 210 0 0 0
cpu15 26432 4 17182 7715447 16074 0 247 0 0 0
intr 296261112 7 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 84 84 203725 203745 197565 211535 194288 208706 205008 205288 161902 160855 179508 178183 262662 267072 117472 119933 198376 198816 117375 117415 186611 192963 193442 195475 188508 185767 193828 193170 197878 203599 181512 180898 0 824751 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 601141090
btime 1618220212
processes 742298
procs_running 1
procs_blocked 0
softirq 229463190 2 80216610 42139 13389791 1324497 0 9842 84630686 81 49849542

 

[oguz]

Any updates on this issue? I have the same problem. Stopping and even removing privileged containers didn't improve the situation! host server keeps restarting randomly (between 3h to 48h).

i'm unable to reproduce these errors here.

can you try running this in the affected container:

for i in cpuinfo diskstats meminfo stat uptime swaps; do echo "=== $i ==="; cat /proc/$i; done

and posting the output?

you can also try cat these files individually while watching the journal on the host with journalctl -f and noting any errors

but i would guess the reboots are likely happening for another reason, looking at the journal should help figure out what's going on

 

[MK.Persia]

i'm unable to reproduce these errors here.

can you try running this in the affected container:

for i in cpuinfo diskstats meminfo stat uptime swaps; do echo "=== $i ==="; cat /proc/$i; done

and posting the output?

you can also try cat these files individually while watching the journal on the host with journalctl -f and noting any errors

but i would guess the reboots are likely happening for another reason, looking at the journal should help figure out what's going on

Thanks for your quick reply. I couldn't provide the information you asked for because of hardware diagnosis was running by the data center which reported no problems. The problem I've mentioned is not affecting any containers, but the host itself. I've attached the result of the code you asked for and some logs, the cause of the random reboots is totally undeterminable.
Appreciate any help in this regard.

 

[olegfusion]

I get this error after reboot in first 12 hours.
lxcfs[4090]: utils.c: 338: read_file_fuse: Write to cache was truncated

proxmox is latest now
proxmox-ve: 6.4-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.4-4 (running version: 6.4-4/337d6701)
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-1
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-3
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
pve-zsync: 2.0-4
qemu-server: 6.4-1
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Error is still here for 1 year. I want to up this thread.

 

[pocok]

Can confirm the same issue. All my containers are privileged (30+ containers). Virtual Environment 6.4-5

 

[Joe How]

Hi guys, I do have exact same problem
Sep 23 01:58:10 node1 lxcfs[1343]: utils.c: 338: read_file_fuse: Write to cache was truncated

I'm using only unprivileged LXCs and it start happening since PVE 7.0. Restart LXCFS helps, till another reboot or LXC migration.

roxmox-ve: not correctly installed (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-5
pve-kernel-5.3: 6.1-6
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 16.2.5-pve1
ceph-fuse: 16.2.5-pve1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-11
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-3
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1