lxcfs: utils.c: 331: read_file_fuse: Write to cache was truncated

[olegfusion]

i'm unable to reproduce these errors here.

can you try running this in the affected container:

for i in cpuinfo diskstats meminfo stat uptime swaps; do echo "=== $i ==="; cat /proc/$i; done

and posting the output?

you can also try cat these files individually while watching the journal on the host with journalctl -f and noting any errors

but i would guess the reboots are likely happening for another reason, looking at the journal should help figure out what's going on

I want to up this thread. Regular log error
Oct 24 15:00:30 pve0 lxcfs[4135927]: utils.c: 338: read_file_fuse: Write to cache was truncated

still exists on current
Virtual Environment 7.0-13
latest kernel.

More information. My last reboot was several days ago. LXCFS service was restarted as always after rebot because of error.

Today was one of rare days when I moved all my lxc and kvm containers to ext4 temporary partition, removed zfs pool and create new one. After migration and starting some of first containers, I found that lxcfs error exists in log again, even not after reboot.

Attaching my cpuinfo diskstats and etc

 

[olegfusion]

hi again, we (probably) found the issue which gives such error. Try to completely disable all pve-zsync tasks, the log error is leaving.

 

[hk@]

Hi,
we do see this strange log also quite often and it _seems_ related to the lxc cpu-time issue, we get this _very_ often

lxcfs[1161]: proc_fuse.c: 1018: proc_stat_read: cpu8 from /lxc/67106/ns has unexpected cpu time: 11596294 in /proc/stat, 13173683 in cpuacct.usage_all; unable to determine idle time

(this we see only for one specific container)

followed by

lxcfs[1161]: utils.c: 338: read_file_fuse: Write to cache was truncated

anyhow, we are not at the latest pve6.4 kernel, but I got the feeling this will not fix it.

pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.162-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-13
pve-kernel-helper: 6.4-13
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.166-1-pve: 5.4.166-1
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1

 

[grin]

anyhow, we are not at the latest pve6.4 kernel, but I got the feeling this will not fix it.

Have you rebooted the node after the last update?
lxcfs daemon seems to be notorously sensitive to restarts, and restaring it may require all containers to be restarted too.

 

[tmikaeld]

I'm seeing the issue on Virtual Environment 7.1-12, running ZFS with Encryption enabled for the CT-Volumes.

Does it affect the services?

Aug 22 15:30:14 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:15 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:16 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:17 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:18 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:19 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:20 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:21 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:22 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:23 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:24 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:25 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:26 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:27 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:28 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:29 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated
Aug 22 15:30:30 devbox lxcfs[1182]: proc_fuse.c: 980: proc_stat_read: Write to cache was truncated

 

[rty]

4 years after this thread was created, I have the same issue. Running Proxmox 8.1.4. Shutting down all VMs/CTs but one privileged CT (NFS server) did not change anything. No ZFS here.

Having three
Feb 22 21:39:14 pve-1 lxcfs[748]: ../src/utils.c: 316: read_file_fuse: Write to cache was truncated
per second in the journal. Blows up the journal and my mind.

Spoiler: for i in cpuinfo diskstats meminfo stat uptime swaps; do echo "=== $i ==="; cat /proc/$i; done

=== cpuinfo ===
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
stepping        : 9
microcode       : 0x21
cpu MHz         : 2335.540
cache size      : 3072 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow fl
expriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_unknown
bogomips        : 4589.53
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
stepping        : 9
microcode       : 0x21
cpu MHz         : 2784.372
cache size      : 3072 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow fl
expriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_unknown
bogomips        : 4589.53
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
stepping        : 9
microcode       : 0x21
cpu MHz         : 3090.218
cache size      : 3072 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow fl
expriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_unknown
bogomips        : 4589.53
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
stepping        : 9
microcode       : 0x21
cpu MHz         : 3242.102
cache size      : 3072 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow fl
expriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds mmio_unknown
bogomips        : 4589.53
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

=== diskstats ===
7       0 loop0 187304 0 13640730 58491 206449 0 3574528 79511 0 618152 176151 0 0 0 0 9411 38147
7       1 loop1 2643652 0 460955008 668964 3551532 0 259771152 5906080 0 10374712 10920650 0 0 0 0 858642 4345605
7       2 loop2 506559 0 70655638 128731 1858028 0 163548872 2244820 0 3291628 3813236 0 0 0 0 312775 1439683
7       3 loop3 91835 0 15630068 18168 5896 0 55000 1294 0 114992 19746 0 0 0 0 120 282
7       4 loop4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7       5 loop5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7       6 loop6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7       7 loop7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
259       0 nvme0n1 3795935 16742 755878039 513905 8075856 1316278 504772964 15031259 0 15563132 16572953 35181 0 478482024 30269 1220592 997518
259       1 nvme0n1p1 3795821 16742 755869175 513839 8075856 1316278 504772964 15031259 0 15563100 15575367 35181 0 478482024 30269 0 0
8       0 sda 3869300 2348108 316906169 4558248 6473917 12013616 426241520 22008727 0 11073492 28777961 20257 81 147554688 7966 1642821 2203019
8       1 sda1 68 0 2272 13 0 0 0 0 0 52 13 0 0 0 0 0 0
8       2 sda2 142 0 13728 67 0 0 0 0 0 152 67 0 0 0 0 0 0
8       3 sda3 3868987 2348108 316883257 4558102 6473916 12013616 426241520 22008727 0 11073440 26574796 20257 81 147554688 7966 0 0
8      32 sdc 5252720 778124 455121026 1608548 637228 1028232 82475232 959210 0 5008564 2690622 174053 0 1185281552 68732 54277 54131
8      33 sdc1 5252623 778124 455113258 1608490 637228 1028232 82475232 959210 0 5008520 2636433 174053 0 1185281552 68732 0 0
8      48 sdd 20224709 1645092 4331703526 72696337 228922 323293 56205240 3558934 0 41747356 77602850 0 0 0 0 54466 1347578
8      49 sdd1 20224466 1645092 4331681410 72693998 228922 323293 56205240 3558934 0 41746940 76252933 0 0 0 0 0 0
8      50 sdd2 131 0 14108 1354 0 0 0 0 0 1476 1354 0 0 0 0 0 0
8      80 sdf 1166925 71827 34648180 35230098 4809662 1738185 4743369080 2906720002 0 95921780 2964400031 0 0 0 0 28580 22449930
8      81 sdf1 1166790 71827 34639724 35229450 4809662 1738185 4743369080 2906720002 0 95921372 2941949452 0 0 0 0 0 0
8      64 sde 4889 248 150317 78976 2 0 16 663 0 43244 79639 0 0 0 0 0 0
8      65 sde1 4753 248 140301 78595 2 0 16 663 0 42864 79258 0 0 0 0 0 0
252       0 dm-0 3064137 0 24521888 972612 5036665 0 40771616 26104572 0 1009040 27077184 0 0 0 0 0 0
252       1 dm-1 611887 0 61286082 1715800 10720717 0 344909560 33934704 0 5115932 35657900 18650 0 145636992 7396 0 0
252       2 dm-2 34128 0 273048 55760 77784 0 531136 49000 0 191352 104760 0 0 0 0 0 0
252       3 dm-3 2144014 0 137909547 9263508 2244295 0 40025372 3596348 0 4828340 12860632 1688 0 1917696 776 0 0
252       4 dm-4 2144014 0 137909547 9266444 2238599 0 40025372 3575824 0 4814236 12843052 1688 0 1917696 784 0 0
252       5 dm-5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
252       6 dm-6 54503 0 1691986 10236 5404 0 43104 19168 0 16424 29404 0 0 0 0 0 0
252       7 dm-7 122747 0 4578710 64888 208983 0 1663224 316692 0 660456 381580 0 0 0 0 0 0
252       8 dm-8 250775 0 8616742 126912 811122 0 9962144 1430436 0 1908452 1557348 0 0 0 0 0 0
252       9 dm-9 202473 0 7030318 139400 150394 0 1221840 178020 0 755016 317420 0 0 0 0 0 0
252      11 dm-11 260009 0 8644294 118948 808137 0 9782976 1632384 0 1929524 1751332 0 0 0 0 0 0
252      12 dm-12 139363 0 5174562 30876 10187 0 81312 24068 0 46668 54944 0 0 0 0 0 0
252      13 dm-13 968560 0 111459191 10551516 232628 0 15772510 1260268 0 319676 11812596 7397 0 5996080 812 0 0
252      14 dm-14 262434 0 33566041 1556932 0 0 0 0 0 24232 1556932 0 0 0 0 0 0
8      16 sdb 7881 0 7088496 49760 1482984 0 759281664 1831038 1 1917144 1880977 0 0 0 0 12 178
=== meminfo ===
MemTotal:       16313752 kB
MemFree:        10798204 kB
MemAvailable:   14829784 kB
Buffers:          179376 kB
Cached:          3887664 kB
SwapCached:       122908 kB
Active:          1317468 kB
Inactive:        3178328 kB
Active(anon):     190828 kB
Inactive(anon):   303452 kB
Active(file):    1126640 kB
Inactive(file):  2874876 kB
Unevictable:      163520 kB
Mlocked:          163520 kB
SwapTotal:      39414204 kB
SwapFree:       38673084 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:              8836 kB
Writeback:             0 kB
AnonPages:        534000 kB
Mapped:           241440 kB
Shmem:             48972 kB
KReclaimable:     364092 kB
Slab:             569668 kB
SReclaimable:     364092 kB
SUnreclaim:       205576 kB
KernelStack:        6016 kB
PageTables:        11376 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    47571080 kB
Committed_AS:    2740924 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       73600 kB
VmallocChunk:          0 kB
Percpu:            95232 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      313228 kB
DirectMap2M:    16429056 kB
=== stat ===
cpu  12291973 47990 6225828 202610980 16556551 0 511470 0 3635414 0
cpu0 2814495 1051 1557209 50933872 4296864 0 152673 0 940639 0
cpu1 3322640 695 1549097 50336404 3995334 0 145088 0 872353 0
cpu2 3122022 1171 1562922 50864850 4148214 0 50770 0 945010 0
cpu3 3032815 45072 1556598 50475853 4116138 0 162938 0 877411 0
intr 1444415730 10 4 0 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 74 31 0 0 1 0 0 0 43282480 723637 4188061 0 0 0 0 2995271 2787928 2924351 2891697 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 44545228 35731813 36535638 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 2987454731
btime 1708031664
processes 4256400
procs_running 1
procs_blocked 1
softirq 1199259786 49 94132356 1454141 435361949 51702763 0 6637137 407780572 4314 202186505
=== uptime ===
602715.64 2026109.80
=== swaps ===
Filename                                Type            Size            Used            Priority
/dev/dm-0                               partition       33554428        741120          -2
/dev/sdd2                               partition       5859776         0               -3

Spoiler: /etc/pve/storage.cfg

dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

dir: nvme0n1
        path /mnt/nvme0n1p1
        content images,rootdir
        nodes pve-1
        prune-backups keep-all=1
        shared 0

dir: backup-passport
        path /mnt/backup-passport/pve-backup
        content backup,images
        prune-backups keep-all=1
        shared 0

dir: backup-pve1
        path /mnt/backup-pve1/backup-pve
        content images,backup
        nodes pve-1
        prune-backups keep-all=1
        shared 0

dir: backup-elements
        path /mnt/backup-elements/backup-pve
        content backup,images
        prune-backups keep-all=1
        shared 0

dir: backup-pve2
        path /mnt/backup-pve2/backup-pve
        content backup,images
        nodes pve-2
        prune-backups keep-all=1
        shared 0

lvmthin: local-lvm2
        thinpool local-lvm2
        vgname local-lvm2
        content rootdir,images
        nodes pve-2

dir: tmp
        path /mnt/tmp
        content backup,iso,vztmpl,snippets,images,rootdir
        nodes pve-1
        prune-backups keep-all=1
        shared 0

Any other advice?

 

[grin]

Feb 22 21:39:14 pve-1 lxcfs[748]: ../src/utils.c: 316: read_file_fuse: Write to cache was truncated
Any other advice?

Yes. You have to reboot the machine. It seems to mess up so many things that nobody ever took the efforts to try to untangle it. Basically major updates of lxcfs are the main reboot magnets in proxmox. That, and kernel updates.

 

[rty]

Thanks for your feedback! It's a pity.

I had just come to the conclusion to accept this and started writing a cronjob script to notify me when it's time for a reboot, assuming that it won't happen that often.

#!/bin/bash

buf=$(mktemp /tmp/`basename "$0"`.XXXXXX)

journalctl --since=-1\ hour --grep=Write\ to\ cache\ was\ truncated --lines 5  > $buf

if [[ $? == 0 ]]; then
    cat $buf | mail -s "Problem Report" -r "Proxmox VE <root@`hostname`>" proxmox@mail.fritz.box
fi

just to find out that after approx 70 minutes uptime, it happened again. No update of anything. No start of anything. 3 VMs and 7 CTs running, only one VM busy with a backup (NFS server data backup using BackInTime). Just happened.

This is really inconvenient and barely desirable!

 

[grin]

just to find out that after approx 70 minutes uptime, it happened again. No update of anything. No start of anything. 3 VMs and 7 CTs running, only one VM busy with a backup (NFS server data backup using BackInTime). Just happened.

This is really inconvenient and barely desirable!

Okay, let's differentiate.

One case is when you upgrade / restart lxcfs and this cause the problem you mentioned. This is what I was talking about: in some cases you cannot safely stop and start lxcfs (I think it's related to various changes in fuse and cgroups but never checked) and it needs a reboot. That's, while not convenient, is not a big problem, as it usually happens only on major upgrades.

What you mention is definitely not normal. It should not "just" happen. If you have rebooted after (major) upgrade, and you haven't fiddled with the system (eg. restarting daemons) then the problem is caused by something else. I am not sure exactly what's messed up in lxcfs when it emits this error, but for that probably either the developers could offer an answer or someone needed to check the source, but that won't be me.

(I don't think NFS can cause this, but no guarantee.)

 

[rty]

I really don't get it. I tried to isolate the problem trigger to replace the vague "just happens" by something more sophisticated and plausibile but failed.

I thought it had something to do with my daily backups: a dedicated archive VM starts at night, collects all data, including all files from NFS export offered by dedicated NFS VM (which afaik needs to be privileged with options `mount=nfs,cifs,nesting=1`) and writes the data to a HDD provided by `qm set my_vmid -scsi1 /dev/backup-daily,backup=0` since this appeared to be somewhat plausible and I had the impression that the lxcfs error series starts while the backup is running (I was setting up my backup algorithm and started the backup very often).

It does not. After detecting the error 14 days ago and rebooting the PVE server, it is now up for 14 days without error. With daily backups.