Backup randomly fails on Synology NAS

M

mirmalag

Member
Proxmox Subscriber
Dec 26, 2021
9
0
21
63
I have troubles backing up an LXC container on PROXMOX VE 9.
I do it as a backup job that saves shapshot images on a NAS.
Strange thing is that it happens sometimes, not always.

Traced error is:
2026-01-10 01:05:02 INFO: zstd: error 70 : Write error : cannot write block : Bad address

Backup NAS is a Synology RS422+ (OS DSM 7.3.2)
Images are saved on a mounted folder shared by the NAS as an SMB volume
Diagnostics on the NAS did not reveal any problem.

More detailed infos are attached.

Can you please help to understand?
 

Attachments

Hi,
not entirely sure if noserverino also helps in your cause, but I'd give it a try:
P

[SOLVED] Thread 'SMB/CIFS mount point stale file handle'

Hi all,

i have a SMB/CIFS MP bound into proxmox and used in the vms/lxc.
In the past week the mount seems to went bad as it return stale file when used on the lxc and even in proxmox itself when i try to cp a file for example.

I unmounted by umount --lazy and removed the dir from /mnt/pve/<share> and set it up again. Error still exists. So i removed the mp from all vm and LXC and removed again from proxmox and set it up again (using cli pvesm add cifs... and webinterface).

if i check with command df i see it succesfully mounted on the path.

Any ideas will be appreciated.
Thanks.
 
fl0moto said:
Hi,
not entirely sure if noserverino also helps in your cause, but I'd give it a try:
P

[SOLVED] Thread 'SMB/CIFS mount point stale file handle'

Hi all,

i have a SMB/CIFS MP bound into proxmox and used in the vms/lxc.
In the past week the mount seems to went bad as it return stale file when used on the lxc and even in proxmox itself when i try to cp a file for example.

I unmounted by umount --lazy and removed the dir from /mnt/pve/<share> and set it up again. Error still exists. So i removed the mp from all vm and LXC and removed again from proxmox and set it up again (using cli pvesm add cifs... and webinterface).

if i check with command df i see it succesfully mounted on the path.

Any ideas will be appreciated.
Thanks.
Click to expand...
Hi fl0moto. I will investigate it but think is not the same problem.
mount's man page says noserverino option has to do with how to treat hardlinks, which are not present among the files I backup.
Thank you anyway.
 
Hi Mirco,
the backup jobs are executed by the pvescheduler (usually run as systemd service).

So for debugging you can try to stop the systemd service and start the scheduler with a strace (and maybe in debug mode). This will generate a lot of output, depending on the others scheduling jobs.

BR, Lucas
 
Hello,

Have you found a fix for your problem ?

I have the exact same problem.

I have a node with 21 VM running on it.
The backup is done daily with the integrated backup system vzdump.
Most of the time it fails randomly on 1 or 2 VMs, always different VM, different time, different place.

105: 2026-03-22 21:13:36 INFO: 52% (130.1 GiB of 250.0 GiB) in 3m 55s, read: 0 B/s, write: 0 B/s
105: 2026-03-22 21:13:36 ERROR: vma_queue_write: write error - Broken pipe
105: 2026-03-22 21:13:36 INFO: aborting backup job
105: 2026-03-22 21:13:36 INFO: resuming VM again
105: 2026-03-22 21:13:37 ERROR: Backup of VM 105 failed - vma_queue_write: write error - Broken pipe

I sometimes also gets the "INFO: zstd: error 70 : Write error : cannot write block : Bad address" message.

I'm running PVE 9.1.6 and the storage is a mounted SMB share hosted on a Synology.

Anyone facing that issue and that might have a solution ?

Thanks
Mike
 
Last edited:
Hi,

same problem here on 9.2.2 and a cluster of two RS3621xs+ with DSM 7.3.2 Update 3.

I tried to strace the problem, but in compliance to Murphys Law, the backup went smooth.

Anyone of you has news on this topic?

Best
Arne
 
Hi - also having the same problem with my 9.2.2 cluster and a QNAP NAS (Mounted SMB share). This problem started with 9.1 and continues with 9.2.
Last night I had 3 backups scheduled - First one failed at 99%, second one was successful (much larger disk) and third also failed at 99%.
Other backups have failed early on in the backup so I don't think it's the length of time or amount of data that's the problem.
It also randomly happens with other nodes and VMs.
When we run the backup manually next day there is no problem. In fact have never had a failure with a manual backup (yet).

144: 2026-06-07 00:55:13 INFO: 99% (99.3 GiB of 100.0 GiB) in 18m 6s, read: 22.2 MiB/s, write: 2.2 MiB/s
144: 2026-06-07 00:55:13 ERROR: vma_queue_write: write error - Broken pipe
144: 2026-06-07 00:55:13 INFO: aborting backup job
144: 2026-06-07 00:55:13 INFO: resuming VM again
144: 2026-06-07 00:55:21 ERROR: Backup of VM 144 failed - vma_queue_write: write error - Broken pipe
 
Last edited:
Good morning. I managed to capture a 2,7 Gb strace of the show. The relevant part:

Code: 
3024886 07:10:42 write(1, "lotsofdata1234"..., 131072 <unfinished ...>
3024887 07:10:42 read(0,  <unfinished ...>
3024886 07:10:42 <... write resumed>)   = -1 EFAULT (Bad address)
3024886 07:10:42 write(2, "zstd: ", 6 <unfinished ...>
3024887 07:10:42 <... read resumed>"lotsofdata1234"..., 126976) = 65536
3024887 07:10:42 read(0,  <unfinished ...>
3024886 07:10:42 <... write resumed>)   = 6
3024857 07:10:42 <... pselect6 resumed>) = 1 (in [5], left {tv_sec=0, tv_nsec=480942946})
3024886 07:10:42 write(2, "error 70 : ", 11) = 11
3024857 07:10:42 read(5,  <unfinished ...>
3024886 07:10:42 write(2, "Write error : cannot write block : Bad address", 46 <unfinished ...>

Can anybody help me out how to proceed from here? What I learned with the help of claude is that piping data back to the parent process failed because the pointer is bad?
 
awiesel said:
Good morning. I managed to capture a 2,7 Gb strace of the show. The relevant part:

Code: 
3024886 07:10:42 write(1, "lotsofdata1234"..., 131072 <unfinished ...>
3024887 07:10:42 read(0,  <unfinished ...>
3024886 07:10:42 <... write resumed>)   = -1 EFAULT (Bad address)
3024886 07:10:42 write(2, "zstd: ", 6 <unfinished ...>
3024887 07:10:42 <... read resumed>"lotsofdata1234"..., 126976) = 65536
3024887 07:10:42 read(0,  <unfinished ...>
3024886 07:10:42 <... write resumed>)   = 6
3024857 07:10:42 <... pselect6 resumed>) = 1 (in [5], left {tv_sec=0, tv_nsec=480942946})
3024886 07:10:42 write(2, "error 70 : ", 11) = 11
3024857 07:10:42 read(5,  <unfinished ...>
3024886 07:10:42 write(2, "Write error : cannot write block : Bad address", 46 <unfinished ...>

Can anybody help me out how to proceed from here? What I learned with the help of claude is that piping data back to the parent process failed because the pointer is bad?
Click to expand...
Check Mount Options: Look at /proc/mounts. If using cache=none or forcedirectio, try switching to default caching (cache=standard) to see if the issue stops.

Kernel Logs (dmesg): CIFS kernel drivers usually log page errors. Run "dmesg -T | grep -i cifs" to see if there are corresponding CIFS VFS: Send error in Write... or get_user_pages warnings at the exact timestamp (07:10:42).
 
awiesel said:
Good morning. I managed to capture a 2,7 Gb strace of the show. The relevant part:

Code: 
3024886 07:10:42 write(1, "lotsofdata1234"..., 131072 <unfinished ...>
3024887 07:10:42 read(0,  <unfinished ...>
3024886 07:10:42 <... write resumed>)   = -1 EFAULT (Bad address)
3024886 07:10:42 write(2, "zstd: ", 6 <unfinished ...>
3024887 07:10:42 <... read resumed>"lotsofdata1234"..., 126976) = 65536
3024887 07:10:42 read(0,  <unfinished ...>
3024886 07:10:42 <... write resumed>)   = 6
3024857 07:10:42 <... pselect6 resumed>) = 1 (in [5], left {tv_sec=0, tv_nsec=480942946})
3024886 07:10:42 write(2, "error 70 : ", 11) = 11
3024857 07:10:42 read(5,  <unfinished ...>
3024886 07:10:42 write(2, "Write error : cannot write block : Bad address", 46 <unfinished ...>

Can anybody help me out how to proceed from here? What I learned with the help of claude is that piping data back to the parent process failed because the pointer is bad?
Click to expand...
Not wanting to be a "Just use XYZ" guy but maybe in case everything else fails you might try to switch to NFS (which the Synology typically offers too).
 
Last edited:
  • Like
Reactions: AlexHK
Good morning,

the error reoccured on sunday. Dmesg shows this:

Code: 
[Sun Sep  6 10:20:13 2026] kworker/u193:13: page allocation failure: order:9, mode:0x40c40(GFP_NOFS|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0-1
[Sun Sep  6 10:20:13 2026] CPU: 42 UID: 0 PID: 3177054 Comm: kworker/u193:13 Tainted: P S         O        7.0.14-14-pve #1 PREEMPT(lazy)
[Sun Sep  6 10:20:13 2026] Tainted: [P]=PROPRIETARY_MODULE, [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE
[Sun Sep  6 10:20:13 2026] Hardware name: Dell Inc. PowerEdge R660/0R5JJC, BIOS 2.8.2 10/30/2025
[Sun Sep  6 10:20:13 2026] Workqueue: writeback wb_workfn (flush-cifs-2)
[Sun Sep  6 10:20:13 2026] Call Trace:
[Sun Sep  6 10:20:13 2026]  <TASK>
[Sun Sep  6 10:20:13 2026]  dump_stack_lvl+0x5f/0x90
[Sun Sep  6 10:20:13 2026]  dump_stack+0x10/0x18
[Sun Sep  6 10:20:13 2026]  warn_alloc+0x171/0x190
[Sun Sep  6 10:20:13 2026]  ? psi_task_change+0xfb/0x110
[Sun Sep  6 10:20:13 2026]  __alloc_frozen_pages_noprof+0xd69/0x1700
[Sun Sep  6 10:20:13 2026]  ? policy_nodemask+0x111/0x190
[Sun Sep  6 10:20:13 2026]  alloc_pages_mpol+0x80/0x180
[Sun Sep  6 10:20:13 2026]  folio_alloc_noprof+0x5b/0xc0
[Sun Sep  6 10:20:13 2026]  netfs_alloc_folioq_buffer+0x157/0x230 [netfs]
[Sun Sep  6 10:20:13 2026]  smb3_init_transform_rq+0xf3/0x3f0 [cifs]
[Sun Sep  6 10:20:13 2026]  smb_send_rqst+0xe3/0x260 [cifs]
[Sun Sep  6 10:20:13 2026]  cifs_call_async+0x175/0x340 [cifs]
[Sun Sep  6 10:20:13 2026]  ? __pfx_smb2_writev_callback+0x10/0x10 [cifs]
[Sun Sep  6 10:20:13 2026]  smb2_async_writev+0x3bc/0x770 [cifs]
[Sun Sep  6 10:20:13 2026]  cifs_issue_write+0x83/0x250 [cifs]
[Sun Sep  6 10:20:13 2026]  ? cifs_issue_write+0x83/0x250 [cifs]
[Sun Sep  6 10:20:13 2026]  netfs_do_issue_write+0x3d/0xe0 [netfs]
[Sun Sep  6 10:20:13 2026]  netfs_advance_write+0x108/0x160 [netfs]
[Sun Sep  6 10:20:13 2026]  ? rolling_buffer_append+0x4d/0xf0 [netfs]
[Sun Sep  6 10:20:13 2026]  netfs_write_folio+0x293/0x890 [netfs]
[Sun Sep  6 10:20:13 2026]  netfs_writepages+0x11c/0x480 [netfs]
[Sun Sep  6 10:20:13 2026]  do_writepages+0xc4/0x180
[Sun Sep  6 10:20:13 2026]  __writeback_single_inode+0x45/0x340
[Sun Sep  6 10:20:13 2026]  writeback_sb_inodes+0x26b/0x5f0
[Sun Sep  6 10:20:13 2026]  __writeback_inodes_wb+0x54/0x100
[Sun Sep  6 10:20:13 2026]  ? queue_io+0x86/0x170
[Sun Sep  6 10:20:13 2026]  wb_writeback+0x1a1/0x360
[Sun Sep  6 10:20:13 2026]  wb_workfn+0x2e9/0x450
[Sun Sep  6 10:20:13 2026]  process_one_work+0x1a9/0x3c0
[Sun Sep  6 10:20:13 2026]  worker_thread+0x1b8/0x360
[Sun Sep  6 10:20:13 2026]  ? __pfx_worker_thread+0x10/0x10
[Sun Sep  6 10:20:13 2026]  kthread+0xf7/0x130
[Sun Sep  6 10:20:13 2026]  ? __pfx_kthread+0x10/0x10
[Sun Sep  6 10:20:13 2026]  ret_from_fork+0x2da/0x3a0
[Sun Sep  6 10:20:13 2026]  ? __pfx_kthread+0x10/0x10
[Sun Sep  6 10:20:13 2026]  ret_from_fork_asm+0x1a/0x30
[Sun Sep  6 10:20:13 2026]  </TASK>
[Sun Sep  6 10:20:13 2026] Mem-Info:
[Sun Sep  6 10:20:13 2026] active_anon:8562003 inactive_anon:169188 isolated_anon:0
                            active_file:27062 inactive_file:19588350 isolated_file:0
                            unevictable:38521 dirty:3475241 writeback:2048
                            slab_reclaimable:288706 slab_unreclaimable:272734
                            mapped:55642 shmem:22010 pagetables:20911
                            sec_pagetables:8131 bounce:0
                            kernel_misc_reclaimable:0
                            free:411522 free_pcp:4176 free_cma:11748
[Sun Sep  6 10:20:13 2026] Node 0 active_anon:13702612kB inactive_anon:190612kB active_file:75864kB inactive_file:41849984kB unevictable:13204kB isolated(anon):0kB isolated(file):0kB mapped:113384kB dirty:7355100kB writeback:8192kB shmem:39384kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:13271040kB kernel_stack:10864kB pagetables:38524kB sec_pagetables:24400kB all_unreclaimable? no Balloon:0kB
[Sun Sep  6 10:20:13 2026] Node 1 active_anon:20546368kB inactive_anon:486140kB active_file:32356kB inactive_file:36484192kB unevictable:140880kB isolated(anon):0kB isolated(file):0kB mapped:109212kB dirty:6547744kB writeback:0kB shmem:48656kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:20275200kB kernel_stack:8576kB pagetables:45120kB sec_pagetables:8124kB all_unreclaimable? no Balloon:0kB
[Sun Sep  6 10:20:13 2026] Node 0 DMA free:11264kB boost:0kB min:8kB low:20kB high:32kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB zspages:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sun Sep  6 10:20:13 2026] lowmem_reserve[]: 0 1480 63895 63895 63895
[Sun Sep  6 10:20:13 2026] Node 0 DMA32 free:251308kB boost:0kB min:824kB low:1944kB high:3064kB reserved_highatomic:0KB free_highatomic:0KB active_anon:2048kB inactive_anon:0kB active_file:0kB inactive_file:1255824kB unevictable:0kB writepending:192kB zspages:0kB present:1583100kB managed:1515672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:45920kB
[Sun Sep  6 10:20:13 2026] lowmem_reserve[]: 0 0 62415 62415 62415
[Sun Sep  6 10:20:13 2026] Node 0 Normal free:861296kB boost:0kB min:45344kB low:106860kB high:168376kB reserved_highatomic:0KB free_highatomic:0KB active_anon:13701048kB inactive_anon:190612kB active_file:75864kB inactive_file:40595376kB unevictable:13204kB writepending:7364340kB zspages:0kB present:65011712kB managed:63912980kB mlocked:10132kB bounce:0kB free_pcp:12128kB local_pcp:0kB free_cma:968kB
[Sun Sep  6 10:20:13 2026] lowmem_reserve[]: 0 0 0 0 0
[Sun Sep  6 10:20:13 2026] Node 1 Normal free:589908kB boost:6144kB min:50072kB low:109664kB high:169256kB reserved_highatomic:0KB free_highatomic:0KB active_anon:20549272kB inactive_anon:486140kB active_file:32356kB inactive_file:36483808kB unevictable:140880kB writepending:6547792kB zspages:0kB present:67108864kB managed:66046972kB mlocked:140880kB bounce:0kB free_pcp:16480kB local_pcp:0kB free_cma:8600kB
[Sun Sep  6 10:20:13 2026] lowmem_reserve[]: 0 0 0 0 0
[Sun Sep  6 10:20:13 2026] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
[Sun Sep  6 10:20:13 2026] Node 0 DMA32: 79*4kB (UME) 78*8kB (UMEC) 78*16kB (MEC) 99*32kB (UMEC) 93*64kB (EC) 87*128kB (UMEC) 78*256kB (UE) 64*512kB (EC) 48*1024kB (UEC) 28*2048kB (EC) 17*4096kB (UEC) = 251308kB
[Sun Sep  6 10:20:13 2026] Node 0 Normal: 51138*4kB (UMEC) 32655*8kB (UME) 13237*16kB (UME) 1026*32kB (UME) 216*64kB (UME) 141*128kB (UME) 459*256kB (ME) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 859792kB
[Sun Sep  6 10:20:13 2026] Node 1 Normal: 42359*4kB (UME) 18487*8kB (UMEC) 11075*16kB (UMEC) 1719*32kB (UMEC) 214*64kB (UMEC) 36*128kB (UMEC) 15*256kB (UMEC) 1*512kB (C) 3*1024kB (MC) 3*2048kB (MC) 2*4096kB (M) = 589604kB
[Sun Sep  6 10:20:13 2026] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Sep  6 10:20:13 2026] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Sep  6 10:20:13 2026] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Sep  6 10:20:13 2026] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Sep  6 10:20:13 2026] 19636225 total pagecache pages
[Sun Sep  6 10:20:13 2026] 0 pages in swap cache
[Sun Sep  6 10:20:13 2026] Free swap  = 0kB
[Sun Sep  6 10:20:13 2026] Total swap = 0kB
[Sun Sep  6 10:20:13 2026] 33429918 pages RAM
[Sun Sep  6 10:20:13 2026] 0 pages HighMem/MovableOnly
[Sun Sep  6 10:20:13 2026] 557172 pages reserved
[Sun Sep  6 10:20:13 2026] 0 pages cma reserved
[Sun Sep  6 10:20:13 2026] 0 pages hwpoisoned
[Sun Sep  6 10:20:13 2026] Memory cgroup min protection 0kB -- low protection 0kB
[Sun Sep  6 10:34:22 2026] kworker/u193:0: page allocation failure: order:9, mode:0x40c40(GFP_NOFS|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0-1
[Sun Sep  6 10:34:22 2026] CPU: 4 UID: 0 PID: 3192410 Comm: kworker/u193:0 Tainted: P S         O        7.0.14-14-pve #1 PREEMPT(lazy)
[Sun Sep  6 10:34:22 2026] Tainted: [P]=PROPRIETARY_MODULE, [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE
[Sun Sep  6 10:34:22 2026] Hardware name: Dell Inc. PowerEdge R660/0R5JJC, BIOS 2.8.2 10/30/2025
[Sun Sep  6 10:34:22 2026] Workqueue: writeback wb_workfn (flush-cifs-2)
[Sun Sep  6 10:34:22 2026] Call Trace:
[Sun Sep  6 10:34:22 2026]  <TASK>
[Sun Sep  6 10:34:22 2026]  dump_stack_lvl+0x5f/0x90
[Sun Sep  6 10:34:22 2026]  dump_stack+0x10/0x18
[Sun Sep  6 10:34:22 2026]  warn_alloc+0x171/0x190
[Sun Sep  6 10:34:22 2026]  ? psi_task_change+0xfb/0x110
[Sun Sep  6 10:34:22 2026]  __alloc_frozen_pages_noprof+0xd69/0x1700
[Sun Sep  6 10:34:22 2026]  ? policy_nodemask+0x111/0x190
[Sun Sep  6 10:34:22 2026]  alloc_pages_mpol+0x80/0x180
[Sun Sep  6 10:34:22 2026]  folio_alloc_noprof+0x5b/0xc0
[Sun Sep  6 10:34:22 2026]  netfs_alloc_folioq_buffer+0x157/0x230 [netfs]
[Sun Sep  6 10:34:22 2026]  smb3_init_transform_rq+0xf3/0x3f0 [cifs]
[Sun Sep  6 10:34:22 2026]  ? __blk_mq_alloc_requests+0x355/0x410
[Sun Sep  6 10:34:22 2026]  smb_send_rqst+0xe3/0x260 [cifs]
[Sun Sep  6 10:34:22 2026]  cifs_call_async+0x175/0x340 [cifs]
[Sun Sep  6 10:34:22 2026]  ? __pfx_smb2_writev_callback+0x10/0x10 [cifs]
[Sun Sep  6 10:34:22 2026]  smb2_async_writev+0x3bc/0x770 [cifs]
[Sun Sep  6 10:34:22 2026]  cifs_issue_write+0x83/0x250 [cifs]
[Sun Sep  6 10:34:22 2026]  ? cifs_issue_write+0x83/0x250 [cifs]
[Sun Sep  6 10:34:22 2026]  netfs_do_issue_write+0x3d/0xe0 [netfs]
[Sun Sep  6 10:34:22 2026]  netfs_advance_write+0x108/0x160 [netfs]
[Sun Sep  6 10:34:22 2026]  ? rolling_buffer_append+0x4d/0xf0 [netfs]
[Sun Sep  6 10:34:22 2026]  netfs_write_folio+0x293/0x890 [netfs]
[Sun Sep  6 10:34:22 2026]  netfs_writepages+0x11c/0x480 [netfs]
[Sun Sep  6 10:34:22 2026]  do_writepages+0xc4/0x180
[Sun Sep  6 10:34:22 2026]  __writeback_single_inode+0x45/0x340
[Sun Sep  6 10:34:22 2026]  writeback_sb_inodes+0x26b/0x5f0
[Sun Sep  6 10:34:22 2026]  ? ttwu_do_activate+0x8a/0x1b0
[Sun Sep  6 10:34:22 2026]  __writeback_inodes_wb+0x54/0x100
[Sun Sep  6 10:34:22 2026]  ? queue_io+0x86/0x170
[Sun Sep  6 10:34:22 2026]  wb_writeback+0x1a1/0x360
[Sun Sep  6 10:34:22 2026]  wb_workfn+0x38a/0x450
[Sun Sep  6 10:34:22 2026]  process_one_work+0x1a9/0x3c0
[Sun Sep  6 10:34:22 2026]  worker_thread+0x1b8/0x360
[Sun Sep  6 10:34:22 2026]  ? __pfx_worker_thread+0x10/0x10
[Sun Sep  6 10:34:22 2026]  kthread+0xf7/0x130
[Sun Sep  6 10:34:22 2026]  ? __pfx_kthread+0x10/0x10
[Sun Sep  6 10:34:22 2026]  ret_from_fork+0x2da/0x3a0
[Sun Sep  6 10:34:22 2026]  ? __pfx_kthread+0x10/0x10
[Sun Sep  6 10:34:22 2026]  ret_from_fork_asm+0x1a/0x30
[Sun Sep  6 10:34:22 2026]  </TASK>
[Sun Sep  6 10:34:22 2026] Mem-Info:
[Sun Sep  6 10:34:22 2026] active_anon:8539871 inactive_anon:177795 isolated_anon:0
                            active_file:36160 inactive_file:19598049 isolated_file:0
                            unevictable:38521 dirty:1224862 writeback:1024
                            slab_reclaimable:353929 slab_unreclaimable:273652
                            mapped:54658 shmem:21231 pagetables:20758
                            sec_pagetables:8135 bounce:0
                            kernel_misc_reclaimable:0
                            free:337020 free_pcp:1788 free_cma:12165
[Sun Sep  6 10:34:22 2026] Node 0 active_anon:13698512kB inactive_anon:189996kB active_file:123368kB inactive_file:42422544kB unevictable:13204kB isolated(anon):0kB isolated(file):0kB mapped:112684kB dirty:3283192kB writeback:4096kB shmem:40720kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:13277184kB kernel_stack:11824kB pagetables:38072kB sec_pagetables:24416kB all_unreclaimable? no Balloon:0kB
[Sun Sep  6 10:34:22 2026] Node 1 active_anon:20461940kB inactive_anon:521184kB active_file:21272kB inactive_file:35970924kB unevictable:140880kB isolated(anon):0kB isolated(file):0kB mapped:105948kB dirty:1617544kB writeback:0kB shmem:44204kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:20269056kB kernel_stack:7392kB pagetables:44960kB sec_pagetables:8124kB all_unreclaimable? no Balloon:0kB
[Sun Sep  6 10:34:22 2026] Node 0 DMA free:11264kB boost:0kB min:8kB low:20kB high:32kB reserved_highatomic:0KB free_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB zspages:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sun Sep  6 10:34:22 2026] lowmem_reserve[]: 0 1480 63895 63895 63895
[Sun Sep  6 10:34:22 2026] Node 0 DMA32 free:251116kB boost:0kB min:824kB low:1944kB high:3064kB reserved_highatomic:0KB free_highatomic:0KB active_anon:2048kB inactive_anon:0kB active_file:0kB inactive_file:1256016kB unevictable:0kB writepending:128kB zspages:0kB present:1583100kB managed:1515672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:45856kB
[Sun Sep  6 10:34:22 2026] lowmem_reserve[]: 0 0 62415 62415 62415
[Sun Sep  6 10:34:22 2026] Node 0 Normal free:429540kB boost:0kB min:45344kB low:106860kB high:168376kB reserved_highatomic:0KB free_highatomic:0KB active_anon:13695464kB inactive_anon:189996kB active_file:123372kB inactive_file:41169672kB unevictable:13204kB writepending:3290340kB zspages:0kB present:65011712kB managed:63912980kB mlocked:10132kB bounce:0kB free_pcp:26956kB local_pcp:0kB free_cma:0kB
[Sun Sep  6 10:34:22 2026] lowmem_reserve[]: 0 0 0 0 0
[Sun Sep  6 10:34:22 2026] Node 1 Normal free:736448kB boost:0kB min:43928kB low:103520kB high:163112kB reserved_highatomic:0KB free_highatomic:0KB active_anon:20461940kB inactive_anon:521184kB active_file:21268kB inactive_file:35948872kB unevictable:140880kB writepending:1617584kB zspages:0kB present:67108864kB managed:66046972kB mlocked:140880kB bounce:0kB free_pcp:304kB local_pcp:0kB free_cma:2804kB
[Sun Sep  6 10:34:22 2026] lowmem_reserve[]: 0 0 0 0 0
[Sun Sep  6 10:34:22 2026] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
[Sun Sep  6 10:34:22 2026] Node 0 DMA32: 79*4kB (UME) 78*8kB (UMEC) 78*16kB (MEC) 99*32kB (UMEC) 92*64kB (EC) 86*128kB (UEC) 78*256kB (UE) 64*512kB (EC) 48*1024kB (UEC) 28*2048kB (EC) 17*4096kB (UEC) = 251116kB
[Sun Sep  6 10:34:22 2026] Node 0 Normal: 30511*4kB (UME) 7392*8kB (UME) 4010*16kB (UME) 1695*32kB (UM) 267*64kB (UME) 165*128kB (ME) 93*256kB (M) 96*512kB (M) 5*1024kB (M) 0*2048kB 0*4096kB = 415868kB
[Sun Sep  6 10:34:22 2026] Node 1 Normal: 36373*4kB (UMEC) 20829*8kB (UMEC) 16472*16kB (UME) 3747*32kB (UME) 333*64kB (UME) 4*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 717404kB
[Sun Sep  6 10:34:22 2026] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Sep  6 10:34:22 2026] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Sep  6 10:34:22 2026] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Sep  6 10:34:22 2026] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Sep  6 10:34:22 2026] 19654912 total pagecache pages
[Sun Sep  6 10:34:22 2026] 0 pages in swap cache
[Sun Sep  6 10:34:22 2026] Free swap  = 0kB
[Sun Sep  6 10:34:22 2026] Total swap = 0kB
[Sun Sep  6 10:34:22 2026] 33429918 pages RAM
[Sun Sep  6 10:34:22 2026] 0 pages HighMem/MovableOnly
[Sun Sep  6 10:34:22 2026] 557172 pages reserved
[Sun Sep  6 10:34:22 2026] 0 pages cma reserved
[Sun Sep  6 10:34:22 2026] 0 pages hwpoisoned
[Sun Sep  6 10:34:22 2026] Memory cgroup min protection 0kB -- low protection 0kB

The final "broken pipe" occured at 10:41:08. Seems like the RAM fills up with dirty pages. I set the CIFS cache to none to test if that helps:

Bash: 
pvesm set <storage-name> --options cache=none
systemctl restart pvestatd

Make sure that the mount option was applied by issuing a mount-command and checking the output. Default setting is cache=strict.
 
Last edited:
I want to add that I am having the exact same issue since updating to Proxmox VE 9. As far as I know, backups on version 8 were nearly always successful. My backup destination is a Windows Server 2019 SMB share. It seems very random which VMs fail. Different VMs fail each time, occasionally all are successful. Separating the job so each VM backup runs one at a time does not seem to help.
 
You must log in or register to reply here.
Top Bottom
Back