Syslog is flooded with: lxcfs[2775]: Internal error: truncated write to cache

D

dirk.nilius

Member
Nov 5, 2015
47
0
6
Berlin, Germany
After a while running some containers the syslog is flooded with:

Nov 12 12:21:37 ckc-b-p0004 lxcfs[2775]: Internal error: truncated write to cache

every second.

Does anybody know what this is?
 
Can you watch the log while `cat`-ing the following files and see if any of them in particular triggers that error?
/proc/meminfo
/proc/cpuinfo
/proc/uptime
/proc/stat
/proc/diskstats

You'd probably have to try them in multiple containers.
The content in these files should have a fairly static size, and for some reason lxcfs sticks to a cached size with some margin of error rather than just allowing them to grow.
 
Hi Wolfgang,

did I understand correctly that this is not a critical issue?

I tried your advice but can't trigger the error at the moment. But I had to restart the container to get rid of the messages beacuse they occur every second. So It's hard to see if it's triggered by my action.
 
(apologies for bringing up an old thread, however)

Fresh install of PMX4 (4.4.6-1-pve) last night.. copied over my openVZ backups, restored them, all appears to be well, everything is running.

However, syslog is spammed, multiple times a second, with that same message.

Code: 
Aug  9 00:57:14 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:14 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:14 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:14 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:15 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:15 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:15 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  9 00:57:15 pmx8 lxcfs[5411]: Internal error: truncated write to cache

I stopped almost all the containers (except for two) and the message continued.

Looking back in the log, here's where it started:

Code: 
Aug  8 23:40:47 pmx8 pveproxy[13153]: worker exit
Aug  8 23:40:47 pmx8 pveproxy[5853]: worker 13153 finished
Aug  8 23:40:47 pmx8 pveproxy[5853]: starting 1 worker(s)
Aug  8 23:40:47 pmx8 pveproxy[5853]: worker 27111 started
Aug  8 23:42:49 pmx8 systemd-timesyncd[4347]: interval/delta/delay/jitter/drift 1024s/-0.037s/0.027s/0.025s/-47ppm
Aug  8 23:44:39 pmx8 pvedaemon[29270]: <root@pam> starting task UPID:pmx8:00006E75:000A2851:57A97BD7:vzrestore:204:root@pam:
Aug  8 23:47:37 pmx8 pvedaemon[29270]: worker exit
Aug  8 23:47:37 pmx8 pvedaemon[5785]: worker 29270 finished
Aug  8 23:47:37 pmx8 pvedaemon[5785]: starting 1 worker(s)
Aug  8 23:47:37 pmx8 pvedaemon[5785]: worker 29252 started
Aug  8 23:47:55 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:00 pmx8 pvedaemon[10093]: <root@pam> successful auth for user 'root@pam'
Aug  8 23:48:18 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:19 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:19 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:19 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:20 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:20 pmx8 lxcfs[5411]: Internal error: truncated write to cache
Aug  8 23:48:20 pmx8 lxcfs[5411]: Internal error: truncated write to cache

Any ideas?
 
apt-get update'd yesterday, should be recent enough?

Code: 
root@pmx8:~# pveversion -v
proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve)
pve-manager: 4.2-2 (running version: 4.2-2/725d76f0)
pve-kernel-4.4.6-1-pve: 4.4.6-48
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-39
qemu-server: 4.0-72
pve-firmware: 1.1-8
libpve-common-perl: 4.0-59
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-50
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-14
pve-container: 1.0-62
pve-firewall: 2.0-25
pve-ha-manager: 1.0-28
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve9~jessie
openvswitch-switch: 2.3.0+git20140819-3+deb8u1
 
read again the wiki to the upgrade howto, seems you missed some steps, you run old packages.
 
This was a fresh install from ISO, then update && upgrade && reboot.

Which ones need updated?
 
instead of posting again immediately, please follow the hint and read the wiki.
 
I have the same problem, any suggestions ?

# pveversion -v
proxmox-ve: 4.2-56 (running kernel: 4.4.13-1-pve)
pve-manager: 4.2-17 (running version: 4.2-17/e1400248)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-43
qemu-server: 4.0-85
pve-firmware: 1.1-8
libpve-common-perl: 4.0-71
libpve-access-control: 4.0-18
libpve-storage-perl: 4.0-56
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6-1
pve-container: 1.0-71
pve-firewall: 2.0-29
pve-ha-manager: 1.0-32
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80
openvswitch-switch: 2.5.0-1
 
Tom:

Updated as requested, didn't change behavior in the least.

Code: 
root@pmx8:~# pveversion -v
proxmox-ve: 4.2-60 (running kernel: 4.4.15-1-pve)
pve-manager: 4.2-17 (running version: 4.2-17/e1400248)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.15-1-pve: 4.4.15-60
lvm2: 2.02.116-pve2
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-43
qemu-server: 4.0-85
pve-firmware: 1.1-8
libpve-common-perl: 4.0-71
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-56
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6-1
pve-container: 1.0-72
pve-firewall: 2.0-29
pve-ha-manager: 1.0-33
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.3-4
lxcfs: 2.0.2-pve1
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80
openvswitch-switch: 2.5.0-1

Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Aug 9 09:29:09 pmx8 lxcfs[5165]: Internal error: truncated write to cache
Click to expand...
 
I'm experiencing this same issue with the message being written to journalctl once per second. I'm using a mix of LXC containers and VMs

Were either of you able to resolve this issue?

Code: 
root@VS01:/var/log# pveversion -v
proxmox-ve: 4.4-79 (running kernel: 4.4.35-2-pve)
pve-manager: 4.4-12 (running version: 4.4-12/e71b7a74)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.35-2-pve: 4.4.35-79
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-107
pve-firmware: 1.1-10
libpve-common-perl: 4.0-90
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-73
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-3
pve-qemu-kvm: 2.7.1-1
pve-container: 1.0-93
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-1
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve14~bpo80
 
Adding a "me too" to this.
Mine are all surrounded by systemd-timesyncd[<PID>]: interval/delta/delay/jitter/drift 32s/-0.209s/0.169s/0.217s/-500ppm

Trying to track down a random slowdown issue we're having... Not sure if this is cause or effect.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom