Super slow, timeout, and VM stuck while backing up, after updated to PVE 9.1.1 and PBS 4.0.20

GE_Admin said:
So the only thing we have in common is 10 Gbps... Can you test it on a 1 Gbps?
Click to expand...

1 Gbps backup:



INFO: starting new backup job: vzdump 201 --remove 0 --node stige --storage pbs01 --notes-template '{{guestname}}' --notification-mode auto --mode snapshot
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2025-12-02 14:52:30
INFO: status = running
INFO: VM Name: SMIPRINTER
INFO: include disk 'scsi1' 'local-lvm:vm-201-disk-1' 32G
INFO: include disk 'efidisk0' 'local-lvm:vm-201-disk-0' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/201/2025-12-02T13:52:30Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'a5f1509f-cec7-4fed-982c-5a6e83682760'
INFO: resuming VM again
INFO: efidisk0: dirty-bitmap status: OK (drive clean)
INFO: scsi1: dirty-bitmap status: OK (11.7 GiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 11.7 GiB dirty of 32.0 GiB total
INFO: 1% (228.0 MiB of 11.7 GiB) in 3s, read: 76.0 MiB/s, write: 68.0 MiB/s
INFO: 3% (408.0 MiB of 11.7 GiB) in 6s, read: 60.0 MiB/s, write: 52.0 MiB/s
INFO: 4% (572.0 MiB of 11.7 GiB) in 9s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 6% (736.0 MiB of 11.7 GiB) in 12s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 7% (900.0 MiB of 11.7 GiB) in 15s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 8% (1.0 GiB of 11.7 GiB) in 18s, read: 56.0 MiB/s, write: 52.0 MiB/s
INFO: 10% (1.2 GiB of 11.7 GiB) in 21s, read: 53.3 MiB/s, write: 50.7 MiB/s
INFO: 11% (1.4 GiB of 11.7 GiB) in 24s, read: 57.3 MiB/s, write: 57.3 MiB/s
INFO: 13% (1.6 GiB of 11.7 GiB) in 27s, read: 64.0 MiB/s, write: 61.3 MiB/s
INFO: 14% (1.7 GiB of 11.7 GiB) in 30s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: 16% (1.9 GiB of 11.7 GiB) in 33s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 17% (2.1 GiB of 11.7 GiB) in 36s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 19% (2.2 GiB of 11.7 GiB) in 39s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO: 20% (2.4 GiB of 11.7 GiB) in 42s, read: 61.3 MiB/s, write: 60.0 MiB/s
INFO: 22% (2.6 GiB of 11.7 GiB) in 45s, read: 62.7 MiB/s, write: 58.7 MiB/s
INFO: 23% (2.8 GiB of 11.7 GiB) in 48s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 24% (2.9 GiB of 11.7 GiB) in 51s, read: 57.3 MiB/s, write: 53.3 MiB/s
INFO: 26% (3.1 GiB of 11.7 GiB) in 54s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 27% (3.2 GiB of 11.7 GiB) in 57s, read: 58.7 MiB/s, write: 56.0 MiB/s
INFO: 29% (3.4 GiB of 11.7 GiB) in 1m, read: 58.7 MiB/s, write: 53.3 MiB/s
INFO: 30% (3.6 GiB of 11.7 GiB) in 1m 3s, read: 60.0 MiB/s, write: 54.7 MiB/s
INFO: 32% (3.8 GiB of 11.7 GiB) in 1m 6s, read: 56.0 MiB/s, write: 50.7 MiB/s
INFO: 33% (3.9 GiB of 11.7 GiB) in 1m 9s, read: 57.3 MiB/s, write: 56.0 MiB/s
INFO: 35% (4.1 GiB of 11.7 GiB) in 1m 12s, read: 58.7 MiB/s, write: 53.3 MiB/s
INFO: 36% (4.3 GiB of 11.7 GiB) in 1m 16s, read: 43.0 MiB/s, write: 43.0 MiB/s
INFO: 37% (4.4 GiB of 11.7 GiB) in 1m 19s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 39% (4.6 GiB of 11.7 GiB) in 1m 22s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 40% (4.8 GiB of 11.7 GiB) in 1m 25s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 42% (5.0 GiB of 11.7 GiB) in 1m 28s, read: 68.0 MiB/s, write: 68.0 MiB/s
INFO: 43% (5.1 GiB of 11.7 GiB) in 1m 31s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: 45% (5.3 GiB of 11.7 GiB) in 1m 34s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 46% (5.5 GiB of 11.7 GiB) in 1m 37s, read: 57.3 MiB/s, write: 56.0 MiB/s
INFO: 48% (5.6 GiB of 11.7 GiB) in 1m 40s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 49% (5.8 GiB of 11.7 GiB) in 1m 43s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 50% (5.9 GiB of 11.7 GiB) in 1m 46s, read: 52.0 MiB/s, write: 50.7 MiB/s
INFO: 52% (6.1 GiB of 11.7 GiB) in 1m 49s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 53% (6.3 GiB of 11.7 GiB) in 1m 52s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 54% (6.4 GiB of 11.7 GiB) in 1m 55s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 56% (6.6 GiB of 11.7 GiB) in 1m 58s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 57% (6.7 GiB of 11.7 GiB) in 2m 1s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 59% (6.9 GiB of 11.7 GiB) in 2m 4s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 60% (7.1 GiB of 11.7 GiB) in 2m 7s, read: 50.7 MiB/s, write: 50.7 MiB/s
INFO: 61% (7.2 GiB of 11.7 GiB) in 2m 10s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 63% (7.4 GiB of 11.7 GiB) in 2m 13s, read: 57.3 MiB/s, write: 57.3 MiB/s
INFO: 64% (7.6 GiB of 11.7 GiB) in 2m 16s, read: 68.0 MiB/s, write: 62.7 MiB/s
INFO: 66% (7.8 GiB of 11.7 GiB) in 2m 19s, read: 64.0 MiB/s, write: 58.7 MiB/s
INFO: 68% (7.9 GiB of 11.7 GiB) in 2m 22s, read: 60.0 MiB/s, write: 58.7 MiB/s
INFO: 69% (8.1 GiB of 11.7 GiB) in 2m 25s, read: 53.3 MiB/s, write: 52.0 MiB/s
INFO: 70% (8.3 GiB of 11.7 GiB) in 2m 28s, read: 53.3 MiB/s, write: 52.0 MiB/s
INFO: 72% (8.4 GiB of 11.7 GiB) in 2m 31s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 73% (8.6 GiB of 11.7 GiB) in 2m 34s, read: 58.7 MiB/s, write: 54.7 MiB/s
INFO: 74% (8.7 GiB of 11.7 GiB) in 2m 37s, read: 52.0 MiB/s, write: 49.3 MiB/s
INFO: 76% (8.9 GiB of 11.7 GiB) in 2m 40s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 77% (9.1 GiB of 11.7 GiB) in 2m 43s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 79% (9.2 GiB of 11.7 GiB) in 2m 46s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO: 80% (9.4 GiB of 11.7 GiB) in 2m 49s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 81% (9.5 GiB of 11.7 GiB) in 2m 52s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 83% (9.7 GiB of 11.7 GiB) in 2m 55s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 84% (9.9 GiB of 11.7 GiB) in 2m 58s, read: 52.0 MiB/s, write: 50.7 MiB/s
INFO: 85% (10.0 GiB of 11.7 GiB) in 3m 1s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 87% (10.2 GiB of 11.7 GiB) in 3m 4s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO: 88% (10.4 GiB of 11.7 GiB) in 3m 7s, read: 57.3 MiB/s, write: 56.0 MiB/s
INFO: 90% (10.5 GiB of 11.7 GiB) in 3m 10s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 91% (10.7 GiB of 11.7 GiB) in 3m 13s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 92% (10.8 GiB of 11.7 GiB) in 3m 16s, read: 50.7 MiB/s, write: 50.7 MiB/s
INFO: 94% (11.0 GiB of 11.7 GiB) in 3m 19s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 95% (11.1 GiB of 11.7 GiB) in 3m 22s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 96% (11.3 GiB of 11.7 GiB) in 3m 25s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 98% (11.5 GiB of 11.7 GiB) in 3m 28s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 99% (11.6 GiB of 11.7 GiB) in 3m 31s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 100% (11.7 GiB of 11.7 GiB) in 3m 34s, read: 25.3 MiB/s, write: 25.3 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup was done incrementally, reused 20.69 GiB (64%)
INFO: transferred 11.68 GiB in 225 seconds (53.2 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 201 (00:03:51)
INFO: Backup finished at 2025-12-02 14:56:21
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`
TASK OK
 
pfornara said:
1 Gbps backup:



INFO: starting new backup job: vzdump 201 --remove 0 --node stige --storage pbs01 --notes-template '{{guestname}}' --notification-mode auto --mode snapshot
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2025-12-02 14:52:30
INFO: status = running
INFO: VM Name: SMIPRINTER
INFO: include disk 'scsi1' 'local-lvm:vm-201-disk-1' 32G
INFO: include disk 'efidisk0' 'local-lvm:vm-201-disk-0' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/201/2025-12-02T13:52:30Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'a5f1509f-cec7-4fed-982c-5a6e83682760'
INFO: resuming VM again
INFO: efidisk0: dirty-bitmap status: OK (drive clean)
INFO: scsi1: dirty-bitmap status: OK (11.7 GiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 11.7 GiB dirty of 32.0 GiB total
INFO: 1% (228.0 MiB of 11.7 GiB) in 3s, read: 76.0 MiB/s, write: 68.0 MiB/s
INFO: 3% (408.0 MiB of 11.7 GiB) in 6s, read: 60.0 MiB/s, write: 52.0 MiB/s
INFO: 4% (572.0 MiB of 11.7 GiB) in 9s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 6% (736.0 MiB of 11.7 GiB) in 12s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 7% (900.0 MiB of 11.7 GiB) in 15s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 8% (1.0 GiB of 11.7 GiB) in 18s, read: 56.0 MiB/s, write: 52.0 MiB/s
INFO: 10% (1.2 GiB of 11.7 GiB) in 21s, read: 53.3 MiB/s, write: 50.7 MiB/s
INFO: 11% (1.4 GiB of 11.7 GiB) in 24s, read: 57.3 MiB/s, write: 57.3 MiB/s
INFO: 13% (1.6 GiB of 11.7 GiB) in 27s, read: 64.0 MiB/s, write: 61.3 MiB/s
INFO: 14% (1.7 GiB of 11.7 GiB) in 30s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: 16% (1.9 GiB of 11.7 GiB) in 33s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 17% (2.1 GiB of 11.7 GiB) in 36s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 19% (2.2 GiB of 11.7 GiB) in 39s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO: 20% (2.4 GiB of 11.7 GiB) in 42s, read: 61.3 MiB/s, write: 60.0 MiB/s
INFO: 22% (2.6 GiB of 11.7 GiB) in 45s, read: 62.7 MiB/s, write: 58.7 MiB/s
INFO: 23% (2.8 GiB of 11.7 GiB) in 48s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 24% (2.9 GiB of 11.7 GiB) in 51s, read: 57.3 MiB/s, write: 53.3 MiB/s
INFO: 26% (3.1 GiB of 11.7 GiB) in 54s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 27% (3.2 GiB of 11.7 GiB) in 57s, read: 58.7 MiB/s, write: 56.0 MiB/s
INFO: 29% (3.4 GiB of 11.7 GiB) in 1m, read: 58.7 MiB/s, write: 53.3 MiB/s
INFO: 30% (3.6 GiB of 11.7 GiB) in 1m 3s, read: 60.0 MiB/s, write: 54.7 MiB/s
INFO: 32% (3.8 GiB of 11.7 GiB) in 1m 6s, read: 56.0 MiB/s, write: 50.7 MiB/s
INFO: 33% (3.9 GiB of 11.7 GiB) in 1m 9s, read: 57.3 MiB/s, write: 56.0 MiB/s
INFO: 35% (4.1 GiB of 11.7 GiB) in 1m 12s, read: 58.7 MiB/s, write: 53.3 MiB/s
INFO: 36% (4.3 GiB of 11.7 GiB) in 1m 16s, read: 43.0 MiB/s, write: 43.0 MiB/s
INFO: 37% (4.4 GiB of 11.7 GiB) in 1m 19s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 39% (4.6 GiB of 11.7 GiB) in 1m 22s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 40% (4.8 GiB of 11.7 GiB) in 1m 25s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 42% (5.0 GiB of 11.7 GiB) in 1m 28s, read: 68.0 MiB/s, write: 68.0 MiB/s
INFO: 43% (5.1 GiB of 11.7 GiB) in 1m 31s, read: 60.0 MiB/s, write: 60.0 MiB/s
INFO: 45% (5.3 GiB of 11.7 GiB) in 1m 34s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 46% (5.5 GiB of 11.7 GiB) in 1m 37s, read: 57.3 MiB/s, write: 56.0 MiB/s
INFO: 48% (5.6 GiB of 11.7 GiB) in 1m 40s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 49% (5.8 GiB of 11.7 GiB) in 1m 43s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 50% (5.9 GiB of 11.7 GiB) in 1m 46s, read: 52.0 MiB/s, write: 50.7 MiB/s
INFO: 52% (6.1 GiB of 11.7 GiB) in 1m 49s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 53% (6.3 GiB of 11.7 GiB) in 1m 52s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 54% (6.4 GiB of 11.7 GiB) in 1m 55s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 56% (6.6 GiB of 11.7 GiB) in 1m 58s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 57% (6.7 GiB of 11.7 GiB) in 2m 1s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 59% (6.9 GiB of 11.7 GiB) in 2m 4s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 60% (7.1 GiB of 11.7 GiB) in 2m 7s, read: 50.7 MiB/s, write: 50.7 MiB/s
INFO: 61% (7.2 GiB of 11.7 GiB) in 2m 10s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 63% (7.4 GiB of 11.7 GiB) in 2m 13s, read: 57.3 MiB/s, write: 57.3 MiB/s
INFO: 64% (7.6 GiB of 11.7 GiB) in 2m 16s, read: 68.0 MiB/s, write: 62.7 MiB/s
INFO: 66% (7.8 GiB of 11.7 GiB) in 2m 19s, read: 64.0 MiB/s, write: 58.7 MiB/s
INFO: 68% (7.9 GiB of 11.7 GiB) in 2m 22s, read: 60.0 MiB/s, write: 58.7 MiB/s
INFO: 69% (8.1 GiB of 11.7 GiB) in 2m 25s, read: 53.3 MiB/s, write: 52.0 MiB/s
INFO: 70% (8.3 GiB of 11.7 GiB) in 2m 28s, read: 53.3 MiB/s, write: 52.0 MiB/s
INFO: 72% (8.4 GiB of 11.7 GiB) in 2m 31s, read: 53.3 MiB/s, write: 53.3 MiB/s
INFO: 73% (8.6 GiB of 11.7 GiB) in 2m 34s, read: 58.7 MiB/s, write: 54.7 MiB/s
INFO: 74% (8.7 GiB of 11.7 GiB) in 2m 37s, read: 52.0 MiB/s, write: 49.3 MiB/s
INFO: 76% (8.9 GiB of 11.7 GiB) in 2m 40s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 77% (9.1 GiB of 11.7 GiB) in 2m 43s, read: 56.0 MiB/s, write: 54.7 MiB/s
INFO: 79% (9.2 GiB of 11.7 GiB) in 2m 46s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO: 80% (9.4 GiB of 11.7 GiB) in 2m 49s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 81% (9.5 GiB of 11.7 GiB) in 2m 52s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 83% (9.7 GiB of 11.7 GiB) in 2m 55s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 84% (9.9 GiB of 11.7 GiB) in 2m 58s, read: 52.0 MiB/s, write: 50.7 MiB/s
INFO: 85% (10.0 GiB of 11.7 GiB) in 3m 1s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO: 87% (10.2 GiB of 11.7 GiB) in 3m 4s, read: 57.3 MiB/s, write: 54.7 MiB/s
INFO: 88% (10.4 GiB of 11.7 GiB) in 3m 7s, read: 57.3 MiB/s, write: 56.0 MiB/s
INFO: 90% (10.5 GiB of 11.7 GiB) in 3m 10s, read: 54.7 MiB/s, write: 53.3 MiB/s
INFO: 91% (10.7 GiB of 11.7 GiB) in 3m 13s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 92% (10.8 GiB of 11.7 GiB) in 3m 16s, read: 50.7 MiB/s, write: 50.7 MiB/s
INFO: 94% (11.0 GiB of 11.7 GiB) in 3m 19s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 95% (11.1 GiB of 11.7 GiB) in 3m 22s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 96% (11.3 GiB of 11.7 GiB) in 3m 25s, read: 54.7 MiB/s, write: 52.0 MiB/s
INFO: 98% (11.5 GiB of 11.7 GiB) in 3m 28s, read: 54.7 MiB/s, write: 54.7 MiB/s
INFO: 99% (11.6 GiB of 11.7 GiB) in 3m 31s, read: 52.0 MiB/s, write: 52.0 MiB/s
INFO: 100% (11.7 GiB of 11.7 GiB) in 3m 34s, read: 25.3 MiB/s, write: 25.3 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup was done incrementally, reused 20.69 GiB (64%)
INFO: transferred 11.68 GiB in 225 seconds (53.2 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 201 (00:03:51)
INFO: Backup finished at 2025-12-02 14:56:21
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`
TASK OK
Click to expand...
Slow but constant... Is this test done with the latest kernel, right?
 
pfornara said:
nope ... with 6.14
Click to expand...
Okay,
I've done a test with latest kernel and 2 x 1 Gbps LACP, 1500 MTU, no VLan and it's working!!!

INFO: starting new backup job: vzdump 110 --node pve-gom-003 --mode snapshot --remove 0 --storage pbs-gom-001_man --notes-template '{{guestname}}' --notification-mode auto
INFO: Starting Backup of VM 110 (qemu)
INFO: Backup started at 2025-12-02 15:43:22
INFO: status = running
INFO: VM Name: SRV-TEST
INFO: include disk 'scsi0' 'Pool_1:vm-110-disk-0' 66112M
INFO: include disk 'scsi1' 'Pool_1:vm-110-disk-1' 262400M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/110/2025-12-02T14:43:22Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '73eec0a9-f33e-48cc-b86d-c0a7a79df42d'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (9.0 GiB of 64.6 GiB dirty)
INFO: scsi1: dirty-bitmap status: OK (1.8 GiB of 256.2 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 10.8 GiB dirty of 320.8 GiB total
INFO: 11% (1.2 GiB of 10.8 GiB) in 3s, read: 405.3 MiB/s, write: 361.3 MiB/s
INFO: 19% (2.1 GiB of 10.8 GiB) in 6s, read: 296.0 MiB/s, write: 269.3 MiB/s
INFO: 24% (2.7 GiB of 10.8 GiB) in 9s, read: 214.7 MiB/s, write: 204.0 MiB/s
INFO: 32% (3.5 GiB of 10.8 GiB) in 12s, read: 276.0 MiB/s, write: 261.3 MiB/s
INFO: 39% (4.2 GiB of 10.8 GiB) in 15s, read: 249.3 MiB/s, write: 234.7 MiB/s
INFO: 45% (4.9 GiB of 10.8 GiB) in 18s, read: 221.3 MiB/s, write: 205.3 MiB/s
INFO: 51% (5.6 GiB of 10.8 GiB) in 21s, read: 237.3 MiB/s, write: 232.0 MiB/s
INFO: 57% (6.2 GiB of 10.8 GiB) in 24s, read: 216.0 MiB/s, write: 210.7 MiB/s
INFO: 63% (6.9 GiB of 10.8 GiB) in 27s, read: 224.0 MiB/s, write: 214.7 MiB/s
INFO: 69% (7.5 GiB of 10.8 GiB) in 30s, read: 226.7 MiB/s, write: 218.7 MiB/s
INFO: 77% (8.3 GiB of 10.8 GiB) in 33s, read: 272.0 MiB/s, write: 252.0 MiB/s
INFO: 84% (9.1 GiB of 10.8 GiB) in 36s, read: 272.0 MiB/s, write: 248.0 MiB/s
INFO: 93% (10.1 GiB of 10.8 GiB) in 39s, read: 342.7 MiB/s, write: 244.0 MiB/s
INFO: 100% (10.8 GiB of 10.8 GiB) in 42s, read: 226.7 MiB/s, write: 169.3 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup is sparse: 384.00 MiB (3%) total zero data
INFO: backup was done incrementally, reused 311.07 GiB (96%)
INFO: transferred 10.78 GiB in 45 seconds (245.3 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 110 (00:00:48)
INFO: Backup finished at 2025-12-02 15:44:10
INFO: Backup job finished successfully
INFO: skipping disabled matcher 'default-matcher'
TASK OK

So, on my PBS, the problem arise only on 2 x 10 Gbps LACP, MTU 9000 and MLAG switches... On Gigabit network is working fine.
 
Yet another reminder: people with the issue who do not use ZFS, if you can find the opportunity, please test with mainline kernel builds. As long as we do not have a reproducer for the issue, all we can do is search for kernel commits that seem suspicious. But you could help us a lot to narrow it down by testing:
fiona said:
The ZFS kernel module is the same version 2.3.4+pve1 in both kernel 6.14 and 6.17, so the likely cause of the issue is in the rest of the kernel code. Unfortunately, the difference between 6.14 and 6.17 is very big. If anybody is not using ZFS and still affected by the issue at hand, you could test mainline builds to help narrow it down:
https://kernel.ubuntu.com/mainline/v6.15/
https://kernel.ubuntu.com/mainline/v6.16/
(the amd64/linux-image... and amd64/linux-modules... packages need to be installed).
Click to expand...
Testing with the following would also be appreciated, the issue might very well be already fixed in a newer release:
https://kernel.ubuntu.com/mainline/v6.17.10/
https://kernel.ubuntu.com/mainline/v6.18/
 
pfornara said:
yep you're right, but its write speed very strange on a 10Gbit link
Click to expand...
well, it reads a lot but doesn't need to write a lot, since most of what it reads can be re-used without uploading (==writing), so that is totally normal and expected in this case :)
 
@ Staff.

I see the 6.17 kernel is still in the enterprise repository for PBS.. with these problems resulting in broken VM disks i'd have expected that to be pulled for now until a fix is available. Right now if I, or anyone would update in a simular situation their env blows up.
 
fabian said:
those numbers seem fine? note that the bitmap was cleared, so the backup had to read the full disk contents (and did so at roughly 400MB/s!)
Click to expand...
Yes I agree with fabian here, if you look at your 'read' speed that is stable, and it is scanning the full disk to find the blocks to backup which can result in parts of the backup actually writing 0 bytes and in nearly all cases itll write less then it will read. This looks normal.
 
Michiel_1afa said:
@ Staff.

I see the 6.17 kernel is still in the enterprise repository for PBS.. with these problems resulting in broken VM disks i'd have expected that to be pulled for now until a fix is available. Right now if I, or anyone would update in a simular situation their env blows up.
Click to expand...
Hi, in my case I can confirm that one Debian VM, stuck on backup, got a corrupted filesystem. Luckily it was a non so vital VM and I restored it to the previous backup made with the old kernel...
 
Michiel_1afa said:
@ Staff.

I see the 6.17 kernel is still in the enterprise repository for PBS.. with these problems resulting in broken VM disks i'd have expected that to be pulled for now until a fix is available. Right now if I, or anyone would update in a simular situation their env blows up.
Click to expand...
totally agree with Michiel_1afa

@ Staff

Consider to remove 6.17 kernel from enterprise repo until fix
 
we've already added it to the known issues of the 4.1 release notes. it's not possible to just pull kernel 6.17. we've managed to reproduce the issue once so far and will update this thread once we know more (and of course, roll out a fix quickly once a culprit is identified).
 
  • Like
Reactions: Heracleos, quanto11 and DjWulf
Same issue here with pbs 4.1 and kernel 6.17, i've tried abl and rr bond, both cause network stuck during backup. Detach and attach network cable, restore network functionality. I've downgrate to 6.14 and all works fine.
Network card is I226-V, no issue with another pbs and I225-V card and lacp bond
 
Last edited:
Michiel_1afa said:
@ Staff.

I see the 6.17 kernel is still in the enterprise repository for PBS.. with these problems resulting in broken VM disks i'd have expected that to be pulled for now until a fix is available. Right now if I, or anyone would update in a simular situation their env blows up.
Click to expand...

Just updated today a 3 node pve cluster from 8.4.1 to 9.1.1 and PBS from 3.x to 4.1.0 to find some vm crashed this evening and came here after some hours of scraching my head. All with enterpriese licenses.

reverted the PBS kernel from 6.17 to 6.14.11-4-pve and rebooted the PBS. Started a new backup job backing up all vm from all 3 hosts and now run without problem.

So the problem is the 6.17 Kernel on PBS

PVE HOSTS:
  • 3 node CEPH Hyperconverged
  • LACP 4x25GbE over 2 MLAG Switches (2 nic on every switch).
  • MTU 9000.
  • Management IP on a linux Vlan.
PBS HOST:
  • LACP 2x10GbE over the same MLAG Switches.
  • MTU 9000.
  • ip assigned directly to BOND, no vlan on PBS side, the switch ports are in access on the same management vlan of the pve hosts.
  • ZFS with 4 HDD in RAID 10.

IPERF from PVE node to PBS node:
# iperf -c 192.168.253.131 -t 120 -i 10 -P 4
[SUM] 0.0000-94.8412 sec 219 GBytes 19.8 Gbits/sec -> saturating 2x10 GbE links.
 
Last edited:
I wonder if there were any sudden slowdowns on the PBS located on the same node as the PVE.
If so, then network settings probably aren't to blame.
 
@Chris managed to reproduce an issue that matches the symptoms in this thread. a newer 6.17.4 kernel is available on pbs-test that seems to fix that particular reproducer. testing and feedback would be highly appreciated!
 
  • Like
Reactions: Zlobniy Shurik, Chris and Heracleos
Zlobniy Shurik said:
I wonder if there were any sudden slowdowns on the PBS located on the same node as the PVE.
If so, then network settings probably aren't to blame.
Click to expand...
In my case, yes, PBS runs as a VM on a PVE node. But some others here have PBS as a physical.
Anyway, reading through all the messages, the only thing we all seem to have in common with this problem is the PBS or PVE network, with at least 2 NICs configured in Lacp Bond, and PBS running with kernel 6.17-1 / 6.17-2
 
Heracleos said:
In my case, yes, PBS runs as a VM on a PVE node. But some others here have PBS as a physical.
Anyway, reading through all the messages, the only thing we all seem to have in common with this problem is the PBS or PVE network, with at least 2 NICs configured in Lacp Bond, and PBS running with kernel 6.17-1 / 6.17-2
Click to expand...

and MTU 9000
 
Heracleos said:
In my case, yes, PBS runs as a VM on a PVE node. But some others here have PBS as a physical.
Anyway, reading through all the messages, the only thing we all seem to have in common with this problem is the PBS or PVE network, with at least 2 NICs configured in Lacp Bond, and PBS running with kernel 6.17-1 / 6.17-2
Click to expand...

Got same issues with 1 NIC@10Gbit, MTU 9000 and kernel 6.17-*. on PBS
PBS : 4.1
PVEs: 8.4.14 running on 6.18.12-13

Reverted to 6.14 on PBS all backups working well
 
Last edited:
You must log in or register to reply here.
Top Bottom