Hi All!
I am experience a issue i hope somebody can shed some light on.
Running a 3 node cluster, ceph as storage, one(1) out of 37 vm machines failes while backing up to PBS.
ERROR: backup close image failed: command error: fixed writer 'drive-scsi0.img.fidx' close failed - unexpected chunk count (65536 != 65537)
All other vms backup perfectly.
Furthermore, the vm in question, can successfully be backuped to a nfs mount, as a compressed archive.
Anybody has any ideas on what could cause this ?
All the best
Ras
----full log below --- FAIL JOB
INFO: starting new backup job: vzdump 503 504 506 507 509 510 511 512 500 501 502 505 508 513 514 516 517 518 521 520 519 526 525 524 523 522 532 530 529 528 527 533 539 538 --mailto it@xxx.xxx--mailnotification failure --mode snapshot --notes-template '{{guestname}}' --storage fra-vault01-clust01 --quiet 1
INFO: skip external VMs: 500, 501, 502, 503, 505, 506, 509, 510, 511, 512, 514, 517, 519, 520, 521, 522, 523, 524, 525, 528, 538
<moved succesfull jobs from log>
INFO: Starting Backup of VM 539 (qemu)
INFO: Backup started at 2023-08-26 21:10:06
INFO: status = running
INFO: VM Name: VSPHERE-UCS-2-DC-Main-IP.65
INFO: include disk 'scsi0' 'fra-ceph01:vm-539-disk-0'
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/539/2023-08-26T19:10:06Z'
INFO: started backup task 'bde907b6-c283-4265-924b-278a10ff0b6b'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (636.0 MiB of 256.0 GiB) in 3s, read: 212.0 MiB/s, write: 177.3 MiB/s
INFO: 1% (2.8 GiB of 256.0 GiB) in 19s, read: 139.0 MiB/s, write: 129.5 MiB/s
INFO: 4% (11.5 GiB of 256.0 GiB) in 31s, read: 740.0 MiB/s, write: 123.7 MiB/s
INFO: 11% (29.9 GiB of 256.0 GiB) in 34s, read: 6.1 GiB/s, write: 16.0 MiB/s
INFO: 17% (43.6 GiB of 256.0 GiB) in 37s, read: 4.6 GiB/s, write: 9.3 MiB/s
INFO: 26% (67.1 GiB of 256.0 GiB) in 40s, read: 7.8 GiB/s, write: 9.3 MiB/s
INFO: 35% (90.4 GiB of 256.0 GiB) in 43s, read: 7.8 GiB/s, write: 12.0 MiB/s
INFO: 45% (115.3 GiB of 256.0 GiB) in 46s, read: 8.3 GiB/s, write: 16.0 MiB/s
INFO: 46% (119.4 GiB of 256.0 GiB) in 49s, read: 1.4 GiB/s, write: 192.0 MiB/s
INFO: 47% (120.9 GiB of 256.0 GiB) in 54s, read: 310.4 MiB/s, write: 112.0 MiB/s
INFO: 48% (123.0 GiB of 256.0 GiB) in 1m, read: 352.0 MiB/s, write: 134.0 MiB/s
INFO: 49% (127.0 GiB of 256.0 GiB) in 1m 3s, read: 1.3 GiB/s, write: 136.0 MiB/s
INFO: 50% (128.1 GiB of 256.0 GiB) in 1m 9s, read: 184.0 MiB/s, write: 152.7 MiB/s
INFO: 51% (130.7 GiB of 256.0 GiB) in 1m 12s, read: 868.0 MiB/s, write: 129.3 MiB/s
INFO: 52% (135.6 GiB of 256.0 GiB) in 1m 15s, read: 1.6 GiB/s, write: 129.3 MiB/s
INFO: 53% (137.5 GiB of 256.0 GiB) in 1m 18s, read: 649.3 MiB/s, write: 148.0 MiB/s
INFO: 55% (143.1 GiB of 256.0 GiB) in 1m 21s, read: 1.9 GiB/s, write: 132.0 MiB/s
INFO: 56% (143.6 GiB of 256.0 GiB) in 1m 24s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 57% (146.0 GiB of 256.0 GiB) in 1m 36s, read: 200.3 MiB/s, write: 195.3 MiB/s
INFO: 58% (148.5 GiB of 256.0 GiB) in 1m 48s, read: 212.7 MiB/s, write: 206.7 MiB/s
INFO: 59% (151.2 GiB of 256.0 GiB) in 2m 1s, read: 216.0 MiB/s, write: 206.8 MiB/s
INFO: 60% (153.6 GiB of 256.0 GiB) in 2m 13s, read: 206.0 MiB/s, write: 200.7 MiB/s
INFO: 61% (156.3 GiB of 256.0 GiB) in 2m 27s, read: 195.7 MiB/s, write: 190.9 MiB/s
INFO: 62% (158.9 GiB of 256.0 GiB) in 2m 40s, read: 202.2 MiB/s, write: 192.0 MiB/s
INFO: 63% (161.3 GiB of 256.0 GiB) in 2m 53s, read: 191.1 MiB/s, write: 177.5 MiB/s
INFO: 64% (163.9 GiB of 256.0 GiB) in 3m 8s, read: 177.9 MiB/s, write: 156.3 MiB/s
INFO: 65% (166.5 GiB of 256.0 GiB) in 3m 21s, read: 200.9 MiB/s, write: 187.7 MiB/s
INFO: 66% (169.2 GiB of 256.0 GiB) in 3m 33s, read: 230.3 MiB/s, write: 220.0 MiB/s
INFO: 67% (171.6 GiB of 256.0 GiB) in 3m 45s, read: 205.0 MiB/s, write: 194.3 MiB/s
INFO: 68% (174.1 GiB of 256.0 GiB) in 3m 58s, read: 199.1 MiB/s, write: 189.8 MiB/s
INFO: 69% (176.8 GiB of 256.0 GiB) in 4m 12s, read: 201.1 MiB/s, write: 181.4 MiB/s
INFO: 70% (179.2 GiB of 256.0 GiB) in 4m 24s, read: 204.0 MiB/s, write: 194.0 MiB/s
INFO: 71% (181.9 GiB of 256.0 GiB) in 4m 37s, read: 212.9 MiB/s, write: 208.3 MiB/s
INFO: 72% (184.4 GiB of 256.0 GiB) in 4m 50s, read: 194.8 MiB/s, write: 190.2 MiB/s
INFO: 73% (186.9 GiB of 256.0 GiB) in 5m 4s, read: 181.4 MiB/s, write: 170.9 MiB/s
INFO: 74% (189.5 GiB of 256.0 GiB) in 5m 17s, read: 204.0 MiB/s, write: 199.7 MiB/s
INFO: 75% (192.0 GiB of 256.0 GiB) in 5m 31s, read: 187.4 MiB/s, write: 182.0 MiB/s
INFO: 76% (194.6 GiB of 256.0 GiB) in 5m 44s, read: 205.2 MiB/s, write: 200.6 MiB/s
INFO: 77% (197.2 GiB of 256.0 GiB) in 5m 57s, read: 200.0 MiB/s, write: 190.8 MiB/s
INFO: 78% (199.8 GiB of 256.0 GiB) in 6m 14s, read: 159.8 MiB/s, write: 156.2 MiB/s
INFO: 79% (202.3 GiB of 256.0 GiB) in 6m 26s, read: 208.7 MiB/s, write: 203.7 MiB/s
INFO: 80% (204.8 GiB of 256.0 GiB) in 6m 38s, read: 216.3 MiB/s, write: 206.3 MiB/s
INFO: 81% (207.5 GiB of 256.0 GiB) in 6m 51s, read: 209.5 MiB/s, write: 204.9 MiB/s
INFO: 82% (210.1 GiB of 256.0 GiB) in 7m 4s, read: 203.7 MiB/s, write: 199.1 MiB/s
INFO: 83% (212.7 GiB of 256.0 GiB) in 7m 16s, read: 223.0 MiB/s, write: 213.3 MiB/s
INFO: 84% (215.1 GiB of 256.0 GiB) in 7m 28s, read: 204.0 MiB/s, write: 199.0 MiB/s
INFO: 85% (217.8 GiB of 256.0 GiB) in 7m 40s, read: 229.7 MiB/s, write: 224.7 MiB/s
INFO: 86% (220.3 GiB of 256.0 GiB) in 7m 51s, read: 233.8 MiB/s, write: 228.4 MiB/s
INFO: 87% (222.9 GiB of 256.0 GiB) in 8m 2s, read: 241.1 MiB/s, write: 230.2 MiB/s
INFO: 88% (225.3 GiB of 256.0 GiB) in 8m 13s, read: 231.6 MiB/s, write: 226.2 MiB/s
INFO: 89% (228.1 GiB of 256.0 GiB) in 8m 25s, read: 233.0 MiB/s, write: 227.3 MiB/s
INFO: 90% (230.7 GiB of 256.0 GiB) in 8m 36s, read: 242.2 MiB/s, write: 231.6 MiB/s
INFO: 91% (233.1 GiB of 256.0 GiB) in 8m 47s, read: 223.6 MiB/s, write: 217.8 MiB/s
INFO: 92% (235.6 GiB of 256.0 GiB) in 8m 59s, read: 214.3 MiB/s, write: 208.7 MiB/s
INFO: 93% (238.2 GiB of 256.0 GiB) in 9m 13s, read: 192.3 MiB/s, write: 182.9 MiB/s
INFO: 94% (240.8 GiB of 256.0 GiB) in 9m 28s, read: 173.3 MiB/s, write: 153.6 MiB/s
INFO: 95% (243.3 GiB of 256.0 GiB) in 9m 46s, read: 141.8 MiB/s, write: 141.8 MiB/s
INFO: 96% (245.8 GiB of 256.0 GiB) in 10m 5s, read: 138.7 MiB/s, write: 138.7 MiB/s
INFO: 97% (248.4 GiB of 256.0 GiB) in 10m 24s, read: 139.8 MiB/s, write: 139.8 MiB/s
INFO: 98% (250.9 GiB of 256.0 GiB) in 10m 43s, read: 136.2 MiB/s, write: 136.2 MiB/s
INFO: 99% (253.6 GiB of 256.0 GiB) in 11m 3s, read: 134.2 MiB/s, write: 134.2 MiB/s
INFO: 100% (256.0 GiB of 256.0 GiB) in 11m 21s, read: 138.4 MiB/s, write: 138.4 MiB/s
ERROR: backup close image failed: command error: fixed writer 'drive-scsi0.img.fidx' close failed - unexpected chunk count (65536 != 65537)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 539 failed - backup close image failed: command error: fixed writer 'drive-scsi0.img.fidx' close failed - unexpected chunk count (65536 != 65537)
INFO: Failed at 2023-08-26 21:21:27
INFO: Backup job finished with errors
TASK ERROR: job errors
--- LOG --- SUCCESS dump to nfs/compressed archive
INFO: starting new backup job: vzdump 539 --remove 0 --compress zstd --notes-template '{{guestname}}' --node fra-proxnode03 --storage fra-vault01-proxmox --mode snapshot
INFO: Starting Backup of VM 539 (qemu)
INFO: Backup started at 2023-08-27 09:39:48
INFO: status = running
INFO: VM Name: VSPHERE-UCS-2-DC-Main-IP.65
INFO: include disk 'scsi0' 'fra-ceph01:vm-539-disk-0'
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/fra-vault01-proxmox/dump/vzdump-qemu-539-2023_08_27-09_39_48.vma.zst'
INFO: started backup task '592f26db-e487-4c67-80e5-dc54719a6b77'
INFO: resuming VM again
INFO: 0% (710.5 MiB of 256.0 GiB) in 3s, read: 236.8 MiB/s, write: 172.0 MiB/s
INFO: 1% (2.7 GiB of 256.0 GiB) in 17s, read: 145.5 MiB/s, write: 114.1 MiB/s
INFO: 5% (14.0 GiB of 256.0 GiB) in 28s, read: 1.0 GiB/s, write: 106.8 MiB/s
INFO: 16% (42.9 GiB of 256.0 GiB) in 31s, read: 9.6 GiB/s, write: 5.2 MiB/s
INFO: 28% (72.2 GiB of 256.0 GiB) in 34s, read: 9.8 GiB/s, write: 189.3 KiB/s
INFO: 39% (100.8 GiB of 256.0 GiB) in 37s, read: 9.5 GiB/s, write: 1.7 MiB/s
INFO: 46% (118.9 GiB of 256.0 GiB) in 40s, read: 6.0 GiB/s, write: 84.3 MiB/s
INFO: 47% (120.7 GiB of 256.0 GiB) in 44s, read: 473.1 MiB/s, write: 221.5 MiB/s
INFO: 48% (124.7 GiB of 256.0 GiB) in 50s, read: 685.7 MiB/s, write: 150.3 MiB/s
INFO: 49% (127.3 GiB of 256.0 GiB) in 53s, read: 865.2 MiB/s, write: 176.7 MiB/s
INFO: 50% (128.7 GiB of 256.0 GiB) in 58s, read: 287.1 MiB/s, write: 117.5 MiB/s
INFO: 51% (132.6 GiB of 256.0 GiB) in 1m 1s, read: 1.3 GiB/s, write: 146.3 MiB/s
INFO: 53% (136.9 GiB of 256.0 GiB) in 1m 4s, read: 1.4 GiB/s, write: 154.6 MiB/s
INFO: 55% (142.8 GiB of 256.0 GiB) in 1m 7s, read: 2.0 GiB/s, write: 128.2 MiB/s
INFO: 56% (143.5 GiB of 256.0 GiB) in 1m 11s, read: 180.3 MiB/s, write: 179.2 MiB/s
INFO: 57% (145.9 GiB of 256.0 GiB) in 1m 23s, read: 211.5 MiB/s, write: 206.1 MiB/s
INFO: 58% (148.6 GiB of 256.0 GiB) in 1m 35s, read: 229.0 MiB/s, write: 216.7 MiB/s
INFO: 59% (151.1 GiB of 256.0 GiB) in 1m 46s, read: 229.4 MiB/s, write: 220.0 MiB/s
INFO: 60% (153.7 GiB of 256.0 GiB) in 1m 58s, read: 223.2 MiB/s, write: 214.7 MiB/s
INFO: 61% (156.3 GiB of 256.0 GiB) in 2m 10s, read: 219.6 MiB/s, write: 212.2 MiB/s
INFO: 62% (158.9 GiB of 256.0 GiB) in 2m 22s, read: 222.3 MiB/s, write: 209.0 MiB/s
INFO: 63% (161.4 GiB of 256.0 GiB) in 2m 35s, read: 195.8 MiB/s, write: 185.9 MiB/s
INFO: 64% (163.9 GiB of 256.0 GiB) in 2m 49s, read: 186.2 MiB/s, write: 175.3 MiB/s
INFO: 65% (166.4 GiB of 256.0 GiB) in 3m 1s, read: 213.0 MiB/s, write: 201.4 MiB/s
INFO: 66% (169.0 GiB of 256.0 GiB) in 3m 12s, read: 243.5 MiB/s, write: 229.4 MiB/s
INFO: 67% (171.5 GiB of 256.0 GiB) in 3m 23s, read: 233.3 MiB/s, write: 215.0 MiB/s
INFO: 68% (174.3 GiB of 256.0 GiB) in 3m 35s, read: 235.3 MiB/s, write: 214.5 MiB/s
INFO: 69% (176.8 GiB of 256.0 GiB) in 3m 47s, read: 212.1 MiB/s, write: 190.3 MiB/s
INFO: 70% (179.2 GiB of 256.0 GiB) in 3m 58s, read: 229.7 MiB/s, write: 217.6 MiB/s
INFO: 71% (181.9 GiB of 256.0 GiB) in 4m 10s, read: 231.4 MiB/s, write: 225.9 MiB/s
INFO: 72% (184.4 GiB of 256.0 GiB) in 4m 22s, read: 207.5 MiB/s, write: 202.0 MiB/s
INFO: 73% (187.1 GiB of 256.0 GiB) in 4m 35s, read: 213.6 MiB/s, write: 197.8 MiB/s
INFO: 74% (189.4 GiB of 256.0 GiB) in 4m 46s, read: 219.5 MiB/s, write: 212.6 MiB/s
INFO: 75% (192.2 GiB of 256.0 GiB) in 5m, read: 199.6 MiB/s, write: 188.9 MiB/s
INFO: 76% (194.7 GiB of 256.0 GiB) in 5m 11s, read: 230.6 MiB/s, write: 218.7 MiB/s
INFO: 77% (197.2 GiB of 256.0 GiB) in 5m 23s, read: 215.5 MiB/s, write: 209.8 MiB/s
INFO: 78% (199.8 GiB of 256.0 GiB) in 5m 38s, read: 181.7 MiB/s, write: 173.2 MiB/s
INFO: 79% (202.3 GiB of 256.0 GiB) in 5m 49s, read: 226.7 MiB/s, write: 220.2 MiB/s
INFO: 80% (205.0 GiB of 256.0 GiB) in 6m 1s, read: 234.4 MiB/s, write: 223.5 MiB/s
INFO: 81% (207.5 GiB of 256.0 GiB) in 6m 13s, read: 211.8 MiB/s, write: 206.2 MiB/s
INFO: 82% (210.0 GiB of 256.0 GiB) in 6m 24s, read: 232.0 MiB/s, write: 226.0 MiB/s
INFO: 83% (212.5 GiB of 256.0 GiB) in 6m 35s, read: 233.6 MiB/s, write: 227.6 MiB/s
INFO: 84% (215.2 GiB of 256.0 GiB) in 6m 47s, read: 226.3 MiB/s, write: 215.3 MiB/s
INFO: 85% (217.7 GiB of 256.0 GiB) in 6m 58s, read: 240.6 MiB/s, write: 234.3 MiB/s
INFO: 86% (220.4 GiB of 256.0 GiB) in 7m 9s, read: 244.2 MiB/s, write: 237.9 MiB/s
INFO: 87% (222.8 GiB of 256.0 GiB) in 7m 19s, read: 249.4 MiB/s, write: 235.9 MiB/s
INFO: 88% (225.4 GiB of 256.0 GiB) in 7m 30s, read: 238.6 MiB/s, write: 232.6 MiB/s
INFO: 89% (227.9 GiB of 256.0 GiB) in 7m 41s, read: 233.3 MiB/s, write: 227.0 MiB/s
INFO: 90% (230.5 GiB of 256.0 GiB) in 7m 52s, read: 243.2 MiB/s, write: 237.3 MiB/s
INFO: 91% (233.0 GiB of 256.0 GiB) in 8m 3s, read: 236.0 MiB/s, write: 223.4 MiB/s
INFO: 92% (235.6 GiB of 256.0 GiB) in 8m 15s, read: 219.0 MiB/s, write: 211.6 MiB/s
INFO: 93% (238.1 GiB of 256.0 GiB) in 8m 28s, read: 196.9 MiB/s, write: 188.0 MiB/s
INFO: 94% (240.6 GiB of 256.0 GiB) in 8m 42s, read: 187.5 MiB/s, write: 171.3 MiB/s
INFO: 95% (243.3 GiB of 256.0 GiB) in 8m 57s, read: 184.4 MiB/s, write: 179.3 MiB/s
INFO: 96% (245.8 GiB of 256.0 GiB) in 9m 11s, read: 180.0 MiB/s, write: 177.9 MiB/s
INFO: 97% (248.4 GiB of 256.0 GiB) in 9m 26s, read: 174.5 MiB/s, write: 173.0 MiB/s
INFO: 98% (250.9 GiB of 256.0 GiB) in 9m 41s, read: 173.8 MiB/s, write: 173.6 MiB/s
INFO: 99% (253.5 GiB of 256.0 GiB) in 9m 56s, read: 176.4 MiB/s, write: 175.9 MiB/s
INFO: 100% (256.0 GiB of 256.0 GiB) in 10m 11s, read: 171.0 MiB/s, write: 170.1 MiB/s
INFO: backup is sparse: 139.78 GiB (54%) total zero data
INFO: transferred 256.00 GiB in 611 seconds (429.0 MiB/s)
INFO: archive file size: 25.87GB
INFO: adding notes to backup
INFO: Finished Backup of VM 539 (00:10:16)
INFO: Backup finished at 2023-08-27 09:50:04
INFO: Backup job finished successfully
TASK OK
I am experience a issue i hope somebody can shed some light on.
Running a 3 node cluster, ceph as storage, one(1) out of 37 vm machines failes while backing up to PBS.
ERROR: backup close image failed: command error: fixed writer 'drive-scsi0.img.fidx' close failed - unexpected chunk count (65536 != 65537)
All other vms backup perfectly.
Furthermore, the vm in question, can successfully be backuped to a nfs mount, as a compressed archive.
Anybody has any ideas on what could cause this ?
All the best
Ras
----full log below --- FAIL JOB
INFO: starting new backup job: vzdump 503 504 506 507 509 510 511 512 500 501 502 505 508 513 514 516 517 518 521 520 519 526 525 524 523 522 532 530 529 528 527 533 539 538 --mailto it@xxx.xxx--mailnotification failure --mode snapshot --notes-template '{{guestname}}' --storage fra-vault01-clust01 --quiet 1
INFO: skip external VMs: 500, 501, 502, 503, 505, 506, 509, 510, 511, 512, 514, 517, 519, 520, 521, 522, 523, 524, 525, 528, 538
<moved succesfull jobs from log>
INFO: Starting Backup of VM 539 (qemu)
INFO: Backup started at 2023-08-26 21:10:06
INFO: status = running
INFO: VM Name: VSPHERE-UCS-2-DC-Main-IP.65
INFO: include disk 'scsi0' 'fra-ceph01:vm-539-disk-0'
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/539/2023-08-26T19:10:06Z'
INFO: started backup task 'bde907b6-c283-4265-924b-278a10ff0b6b'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (636.0 MiB of 256.0 GiB) in 3s, read: 212.0 MiB/s, write: 177.3 MiB/s
INFO: 1% (2.8 GiB of 256.0 GiB) in 19s, read: 139.0 MiB/s, write: 129.5 MiB/s
INFO: 4% (11.5 GiB of 256.0 GiB) in 31s, read: 740.0 MiB/s, write: 123.7 MiB/s
INFO: 11% (29.9 GiB of 256.0 GiB) in 34s, read: 6.1 GiB/s, write: 16.0 MiB/s
INFO: 17% (43.6 GiB of 256.0 GiB) in 37s, read: 4.6 GiB/s, write: 9.3 MiB/s
INFO: 26% (67.1 GiB of 256.0 GiB) in 40s, read: 7.8 GiB/s, write: 9.3 MiB/s
INFO: 35% (90.4 GiB of 256.0 GiB) in 43s, read: 7.8 GiB/s, write: 12.0 MiB/s
INFO: 45% (115.3 GiB of 256.0 GiB) in 46s, read: 8.3 GiB/s, write: 16.0 MiB/s
INFO: 46% (119.4 GiB of 256.0 GiB) in 49s, read: 1.4 GiB/s, write: 192.0 MiB/s
INFO: 47% (120.9 GiB of 256.0 GiB) in 54s, read: 310.4 MiB/s, write: 112.0 MiB/s
INFO: 48% (123.0 GiB of 256.0 GiB) in 1m, read: 352.0 MiB/s, write: 134.0 MiB/s
INFO: 49% (127.0 GiB of 256.0 GiB) in 1m 3s, read: 1.3 GiB/s, write: 136.0 MiB/s
INFO: 50% (128.1 GiB of 256.0 GiB) in 1m 9s, read: 184.0 MiB/s, write: 152.7 MiB/s
INFO: 51% (130.7 GiB of 256.0 GiB) in 1m 12s, read: 868.0 MiB/s, write: 129.3 MiB/s
INFO: 52% (135.6 GiB of 256.0 GiB) in 1m 15s, read: 1.6 GiB/s, write: 129.3 MiB/s
INFO: 53% (137.5 GiB of 256.0 GiB) in 1m 18s, read: 649.3 MiB/s, write: 148.0 MiB/s
INFO: 55% (143.1 GiB of 256.0 GiB) in 1m 21s, read: 1.9 GiB/s, write: 132.0 MiB/s
INFO: 56% (143.6 GiB of 256.0 GiB) in 1m 24s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 57% (146.0 GiB of 256.0 GiB) in 1m 36s, read: 200.3 MiB/s, write: 195.3 MiB/s
INFO: 58% (148.5 GiB of 256.0 GiB) in 1m 48s, read: 212.7 MiB/s, write: 206.7 MiB/s
INFO: 59% (151.2 GiB of 256.0 GiB) in 2m 1s, read: 216.0 MiB/s, write: 206.8 MiB/s
INFO: 60% (153.6 GiB of 256.0 GiB) in 2m 13s, read: 206.0 MiB/s, write: 200.7 MiB/s
INFO: 61% (156.3 GiB of 256.0 GiB) in 2m 27s, read: 195.7 MiB/s, write: 190.9 MiB/s
INFO: 62% (158.9 GiB of 256.0 GiB) in 2m 40s, read: 202.2 MiB/s, write: 192.0 MiB/s
INFO: 63% (161.3 GiB of 256.0 GiB) in 2m 53s, read: 191.1 MiB/s, write: 177.5 MiB/s
INFO: 64% (163.9 GiB of 256.0 GiB) in 3m 8s, read: 177.9 MiB/s, write: 156.3 MiB/s
INFO: 65% (166.5 GiB of 256.0 GiB) in 3m 21s, read: 200.9 MiB/s, write: 187.7 MiB/s
INFO: 66% (169.2 GiB of 256.0 GiB) in 3m 33s, read: 230.3 MiB/s, write: 220.0 MiB/s
INFO: 67% (171.6 GiB of 256.0 GiB) in 3m 45s, read: 205.0 MiB/s, write: 194.3 MiB/s
INFO: 68% (174.1 GiB of 256.0 GiB) in 3m 58s, read: 199.1 MiB/s, write: 189.8 MiB/s
INFO: 69% (176.8 GiB of 256.0 GiB) in 4m 12s, read: 201.1 MiB/s, write: 181.4 MiB/s
INFO: 70% (179.2 GiB of 256.0 GiB) in 4m 24s, read: 204.0 MiB/s, write: 194.0 MiB/s
INFO: 71% (181.9 GiB of 256.0 GiB) in 4m 37s, read: 212.9 MiB/s, write: 208.3 MiB/s
INFO: 72% (184.4 GiB of 256.0 GiB) in 4m 50s, read: 194.8 MiB/s, write: 190.2 MiB/s
INFO: 73% (186.9 GiB of 256.0 GiB) in 5m 4s, read: 181.4 MiB/s, write: 170.9 MiB/s
INFO: 74% (189.5 GiB of 256.0 GiB) in 5m 17s, read: 204.0 MiB/s, write: 199.7 MiB/s
INFO: 75% (192.0 GiB of 256.0 GiB) in 5m 31s, read: 187.4 MiB/s, write: 182.0 MiB/s
INFO: 76% (194.6 GiB of 256.0 GiB) in 5m 44s, read: 205.2 MiB/s, write: 200.6 MiB/s
INFO: 77% (197.2 GiB of 256.0 GiB) in 5m 57s, read: 200.0 MiB/s, write: 190.8 MiB/s
INFO: 78% (199.8 GiB of 256.0 GiB) in 6m 14s, read: 159.8 MiB/s, write: 156.2 MiB/s
INFO: 79% (202.3 GiB of 256.0 GiB) in 6m 26s, read: 208.7 MiB/s, write: 203.7 MiB/s
INFO: 80% (204.8 GiB of 256.0 GiB) in 6m 38s, read: 216.3 MiB/s, write: 206.3 MiB/s
INFO: 81% (207.5 GiB of 256.0 GiB) in 6m 51s, read: 209.5 MiB/s, write: 204.9 MiB/s
INFO: 82% (210.1 GiB of 256.0 GiB) in 7m 4s, read: 203.7 MiB/s, write: 199.1 MiB/s
INFO: 83% (212.7 GiB of 256.0 GiB) in 7m 16s, read: 223.0 MiB/s, write: 213.3 MiB/s
INFO: 84% (215.1 GiB of 256.0 GiB) in 7m 28s, read: 204.0 MiB/s, write: 199.0 MiB/s
INFO: 85% (217.8 GiB of 256.0 GiB) in 7m 40s, read: 229.7 MiB/s, write: 224.7 MiB/s
INFO: 86% (220.3 GiB of 256.0 GiB) in 7m 51s, read: 233.8 MiB/s, write: 228.4 MiB/s
INFO: 87% (222.9 GiB of 256.0 GiB) in 8m 2s, read: 241.1 MiB/s, write: 230.2 MiB/s
INFO: 88% (225.3 GiB of 256.0 GiB) in 8m 13s, read: 231.6 MiB/s, write: 226.2 MiB/s
INFO: 89% (228.1 GiB of 256.0 GiB) in 8m 25s, read: 233.0 MiB/s, write: 227.3 MiB/s
INFO: 90% (230.7 GiB of 256.0 GiB) in 8m 36s, read: 242.2 MiB/s, write: 231.6 MiB/s
INFO: 91% (233.1 GiB of 256.0 GiB) in 8m 47s, read: 223.6 MiB/s, write: 217.8 MiB/s
INFO: 92% (235.6 GiB of 256.0 GiB) in 8m 59s, read: 214.3 MiB/s, write: 208.7 MiB/s
INFO: 93% (238.2 GiB of 256.0 GiB) in 9m 13s, read: 192.3 MiB/s, write: 182.9 MiB/s
INFO: 94% (240.8 GiB of 256.0 GiB) in 9m 28s, read: 173.3 MiB/s, write: 153.6 MiB/s
INFO: 95% (243.3 GiB of 256.0 GiB) in 9m 46s, read: 141.8 MiB/s, write: 141.8 MiB/s
INFO: 96% (245.8 GiB of 256.0 GiB) in 10m 5s, read: 138.7 MiB/s, write: 138.7 MiB/s
INFO: 97% (248.4 GiB of 256.0 GiB) in 10m 24s, read: 139.8 MiB/s, write: 139.8 MiB/s
INFO: 98% (250.9 GiB of 256.0 GiB) in 10m 43s, read: 136.2 MiB/s, write: 136.2 MiB/s
INFO: 99% (253.6 GiB of 256.0 GiB) in 11m 3s, read: 134.2 MiB/s, write: 134.2 MiB/s
INFO: 100% (256.0 GiB of 256.0 GiB) in 11m 21s, read: 138.4 MiB/s, write: 138.4 MiB/s
ERROR: backup close image failed: command error: fixed writer 'drive-scsi0.img.fidx' close failed - unexpected chunk count (65536 != 65537)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 539 failed - backup close image failed: command error: fixed writer 'drive-scsi0.img.fidx' close failed - unexpected chunk count (65536 != 65537)
INFO: Failed at 2023-08-26 21:21:27
INFO: Backup job finished with errors
TASK ERROR: job errors
--- LOG --- SUCCESS dump to nfs/compressed archive
INFO: starting new backup job: vzdump 539 --remove 0 --compress zstd --notes-template '{{guestname}}' --node fra-proxnode03 --storage fra-vault01-proxmox --mode snapshot
INFO: Starting Backup of VM 539 (qemu)
INFO: Backup started at 2023-08-27 09:39:48
INFO: status = running
INFO: VM Name: VSPHERE-UCS-2-DC-Main-IP.65
INFO: include disk 'scsi0' 'fra-ceph01:vm-539-disk-0'
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/fra-vault01-proxmox/dump/vzdump-qemu-539-2023_08_27-09_39_48.vma.zst'
INFO: started backup task '592f26db-e487-4c67-80e5-dc54719a6b77'
INFO: resuming VM again
INFO: 0% (710.5 MiB of 256.0 GiB) in 3s, read: 236.8 MiB/s, write: 172.0 MiB/s
INFO: 1% (2.7 GiB of 256.0 GiB) in 17s, read: 145.5 MiB/s, write: 114.1 MiB/s
INFO: 5% (14.0 GiB of 256.0 GiB) in 28s, read: 1.0 GiB/s, write: 106.8 MiB/s
INFO: 16% (42.9 GiB of 256.0 GiB) in 31s, read: 9.6 GiB/s, write: 5.2 MiB/s
INFO: 28% (72.2 GiB of 256.0 GiB) in 34s, read: 9.8 GiB/s, write: 189.3 KiB/s
INFO: 39% (100.8 GiB of 256.0 GiB) in 37s, read: 9.5 GiB/s, write: 1.7 MiB/s
INFO: 46% (118.9 GiB of 256.0 GiB) in 40s, read: 6.0 GiB/s, write: 84.3 MiB/s
INFO: 47% (120.7 GiB of 256.0 GiB) in 44s, read: 473.1 MiB/s, write: 221.5 MiB/s
INFO: 48% (124.7 GiB of 256.0 GiB) in 50s, read: 685.7 MiB/s, write: 150.3 MiB/s
INFO: 49% (127.3 GiB of 256.0 GiB) in 53s, read: 865.2 MiB/s, write: 176.7 MiB/s
INFO: 50% (128.7 GiB of 256.0 GiB) in 58s, read: 287.1 MiB/s, write: 117.5 MiB/s
INFO: 51% (132.6 GiB of 256.0 GiB) in 1m 1s, read: 1.3 GiB/s, write: 146.3 MiB/s
INFO: 53% (136.9 GiB of 256.0 GiB) in 1m 4s, read: 1.4 GiB/s, write: 154.6 MiB/s
INFO: 55% (142.8 GiB of 256.0 GiB) in 1m 7s, read: 2.0 GiB/s, write: 128.2 MiB/s
INFO: 56% (143.5 GiB of 256.0 GiB) in 1m 11s, read: 180.3 MiB/s, write: 179.2 MiB/s
INFO: 57% (145.9 GiB of 256.0 GiB) in 1m 23s, read: 211.5 MiB/s, write: 206.1 MiB/s
INFO: 58% (148.6 GiB of 256.0 GiB) in 1m 35s, read: 229.0 MiB/s, write: 216.7 MiB/s
INFO: 59% (151.1 GiB of 256.0 GiB) in 1m 46s, read: 229.4 MiB/s, write: 220.0 MiB/s
INFO: 60% (153.7 GiB of 256.0 GiB) in 1m 58s, read: 223.2 MiB/s, write: 214.7 MiB/s
INFO: 61% (156.3 GiB of 256.0 GiB) in 2m 10s, read: 219.6 MiB/s, write: 212.2 MiB/s
INFO: 62% (158.9 GiB of 256.0 GiB) in 2m 22s, read: 222.3 MiB/s, write: 209.0 MiB/s
INFO: 63% (161.4 GiB of 256.0 GiB) in 2m 35s, read: 195.8 MiB/s, write: 185.9 MiB/s
INFO: 64% (163.9 GiB of 256.0 GiB) in 2m 49s, read: 186.2 MiB/s, write: 175.3 MiB/s
INFO: 65% (166.4 GiB of 256.0 GiB) in 3m 1s, read: 213.0 MiB/s, write: 201.4 MiB/s
INFO: 66% (169.0 GiB of 256.0 GiB) in 3m 12s, read: 243.5 MiB/s, write: 229.4 MiB/s
INFO: 67% (171.5 GiB of 256.0 GiB) in 3m 23s, read: 233.3 MiB/s, write: 215.0 MiB/s
INFO: 68% (174.3 GiB of 256.0 GiB) in 3m 35s, read: 235.3 MiB/s, write: 214.5 MiB/s
INFO: 69% (176.8 GiB of 256.0 GiB) in 3m 47s, read: 212.1 MiB/s, write: 190.3 MiB/s
INFO: 70% (179.2 GiB of 256.0 GiB) in 3m 58s, read: 229.7 MiB/s, write: 217.6 MiB/s
INFO: 71% (181.9 GiB of 256.0 GiB) in 4m 10s, read: 231.4 MiB/s, write: 225.9 MiB/s
INFO: 72% (184.4 GiB of 256.0 GiB) in 4m 22s, read: 207.5 MiB/s, write: 202.0 MiB/s
INFO: 73% (187.1 GiB of 256.0 GiB) in 4m 35s, read: 213.6 MiB/s, write: 197.8 MiB/s
INFO: 74% (189.4 GiB of 256.0 GiB) in 4m 46s, read: 219.5 MiB/s, write: 212.6 MiB/s
INFO: 75% (192.2 GiB of 256.0 GiB) in 5m, read: 199.6 MiB/s, write: 188.9 MiB/s
INFO: 76% (194.7 GiB of 256.0 GiB) in 5m 11s, read: 230.6 MiB/s, write: 218.7 MiB/s
INFO: 77% (197.2 GiB of 256.0 GiB) in 5m 23s, read: 215.5 MiB/s, write: 209.8 MiB/s
INFO: 78% (199.8 GiB of 256.0 GiB) in 5m 38s, read: 181.7 MiB/s, write: 173.2 MiB/s
INFO: 79% (202.3 GiB of 256.0 GiB) in 5m 49s, read: 226.7 MiB/s, write: 220.2 MiB/s
INFO: 80% (205.0 GiB of 256.0 GiB) in 6m 1s, read: 234.4 MiB/s, write: 223.5 MiB/s
INFO: 81% (207.5 GiB of 256.0 GiB) in 6m 13s, read: 211.8 MiB/s, write: 206.2 MiB/s
INFO: 82% (210.0 GiB of 256.0 GiB) in 6m 24s, read: 232.0 MiB/s, write: 226.0 MiB/s
INFO: 83% (212.5 GiB of 256.0 GiB) in 6m 35s, read: 233.6 MiB/s, write: 227.6 MiB/s
INFO: 84% (215.2 GiB of 256.0 GiB) in 6m 47s, read: 226.3 MiB/s, write: 215.3 MiB/s
INFO: 85% (217.7 GiB of 256.0 GiB) in 6m 58s, read: 240.6 MiB/s, write: 234.3 MiB/s
INFO: 86% (220.4 GiB of 256.0 GiB) in 7m 9s, read: 244.2 MiB/s, write: 237.9 MiB/s
INFO: 87% (222.8 GiB of 256.0 GiB) in 7m 19s, read: 249.4 MiB/s, write: 235.9 MiB/s
INFO: 88% (225.4 GiB of 256.0 GiB) in 7m 30s, read: 238.6 MiB/s, write: 232.6 MiB/s
INFO: 89% (227.9 GiB of 256.0 GiB) in 7m 41s, read: 233.3 MiB/s, write: 227.0 MiB/s
INFO: 90% (230.5 GiB of 256.0 GiB) in 7m 52s, read: 243.2 MiB/s, write: 237.3 MiB/s
INFO: 91% (233.0 GiB of 256.0 GiB) in 8m 3s, read: 236.0 MiB/s, write: 223.4 MiB/s
INFO: 92% (235.6 GiB of 256.0 GiB) in 8m 15s, read: 219.0 MiB/s, write: 211.6 MiB/s
INFO: 93% (238.1 GiB of 256.0 GiB) in 8m 28s, read: 196.9 MiB/s, write: 188.0 MiB/s
INFO: 94% (240.6 GiB of 256.0 GiB) in 8m 42s, read: 187.5 MiB/s, write: 171.3 MiB/s
INFO: 95% (243.3 GiB of 256.0 GiB) in 8m 57s, read: 184.4 MiB/s, write: 179.3 MiB/s
INFO: 96% (245.8 GiB of 256.0 GiB) in 9m 11s, read: 180.0 MiB/s, write: 177.9 MiB/s
INFO: 97% (248.4 GiB of 256.0 GiB) in 9m 26s, read: 174.5 MiB/s, write: 173.0 MiB/s
INFO: 98% (250.9 GiB of 256.0 GiB) in 9m 41s, read: 173.8 MiB/s, write: 173.6 MiB/s
INFO: 99% (253.5 GiB of 256.0 GiB) in 9m 56s, read: 176.4 MiB/s, write: 175.9 MiB/s
INFO: 100% (256.0 GiB of 256.0 GiB) in 10m 11s, read: 171.0 MiB/s, write: 170.1 MiB/s
INFO: backup is sparse: 139.78 GiB (54%) total zero data
INFO: transferred 256.00 GiB in 611 seconds (429.0 MiB/s)
INFO: archive file size: 25.87GB
INFO: adding notes to backup
INFO: Finished Backup of VM 539 (00:10:16)
INFO: Backup finished at 2023-08-27 09:50:04
INFO: Backup job finished successfully
TASK OK
