Linux 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100)
pve-manager/6.3-6/2184247e
Over the last weeks I have noticed the following behaviour. When running a
scheduled or manual backup of my VMs there is a high change that the proxmox server itself will completly freeze. The only resolution would be to force a reboot of proxmox. The issue occurs both when using USB attached storage as backup target or even local storage.
Anything specific in regards to configuration,.. I can look at to resolve this issue?
**full backups logs can be found attached**
--------------------------------------RUN 1--------------------------------------
Status: stopped: unexpected status
Task type: vzdump
Start Time: 2021-04-24 03:00:01
End Time: 2021-04-24 08:53:38
INFO: starting new backup job: vzdump --exclude 101 --mailnotification failure --quiet 1 --all 1 --compress zstd --mode snapshot --storage external-backup
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2021-04-24 03:00:01
INFO: status = running
INFO: VM Name: rootsop
INFO: include disk 'scsi1' 'local-lvm:vm-102-disk-0' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-102-2021_04_24-03_00_01.vma.zst'
INFO: started backup task '674be1f2-7973-4ad8-b553-f4da5b5785ad'
INFO: resuming VM again
INFO: 2% (1.7 GiB of 60.0 GiB) in 3s, read: 575.2 MiB/s, write: 155.7 MiB/s
INFO: 4% (3.0 GiB of 60.0 GiB) in 6s, read: 447.6 MiB/s, write: 158.2 MiB/s
INFO: 6% (3.6 GiB of 60.0 GiB) in 9s, read: 218.1 MiB/s, write: 136.8 MiB/s
INFO: 7% (4.4 GiB of 60.0 GiB) in 13s, read: 189.4 MiB/s, write: 174.3 MiB/s
INFO: 8% (4.8 GiB of 60.0 GiB) in 16s, read: 159.5 MiB/s, write: 148.3 MiB/s
INFO: 10% (6.3 GiB of 60.0 GiB) in 20s, read: 362.4 MiB/s, write: 120.7 MiB/s
INFO: 15% (9.1 GiB of 60.0 GiB) in 23s, read: 970.4 MiB/s, write: 7.5 MiB/s
INFO: 25% (15.1 GiB of 60.0 GiB) in 26s, read: 2.0 GiB/s, write: 45.1 MiB/s
INFO: 26% (16.0 GiB of 60.0 GiB) in 29s, read: 298.1 MiB/s, write: 208.7 MiB/s
INFO: 27% (16.6 GiB of 60.0 GiB) in 32s, read: 219.9 MiB/s, write: 217.2 MiB/s
INFO: 28% (17.4 GiB of 60.0 GiB) in 35s, read: 270.7 MiB/s, write: 204.5 MiB/s
INFO: 30% (18.1 GiB of 60.0 GiB) in 38s, read: 229.6 MiB/s, write: 227.3 MiB/s
INFO: 32% (19.7 GiB of 60.0 GiB) in 41s, read: 554.6 MiB/s, write: 125.9 MiB/s
INFO: 35% (21.3 GiB of 60.0 GiB) in 44s, read: 540.3 MiB/s, write: 120.5 MiB/s
INFO: 36% (22.0 GiB of 60.0 GiB) in 47s, read: 247.5 MiB/s, write: 220.0 MiB/s
INFO: 39% (23.6 GiB of 60.0 GiB) in 50s, read: 552.2 MiB/s, write: 132.6 MiB/s
INFO: 40% (24.3 GiB of 60.0 GiB) in 53s, read: 227.2 MiB/s, write: 226.2 MiB/s
INFO: 41% (25.0 GiB of 60.0 GiB) in 56s, read: 231.8 MiB/s, write: 231.6 MiB/s
INFO: 42% (25.7 GiB of 60.0 GiB) in 59s, read: 244.3 MiB/s, write: 200.3 MiB/s
INFO: 43% (26.3 GiB of 60.0 GiB) in 1m 2s, read: 205.4 MiB/s, write: 204.7 MiB/s
INFO: 47% (28.8 GiB of 60.0 GiB) in 1m 5s, read: 856.3 MiB/s, write: 39.4 MiB/s
INFO: 52% (31.4 GiB of 60.0 GiB) in 1m 8s, read: 903.0 MiB/s, write: 39.5 MiB/s
INFO: 53% (32.1 GiB of 60.0 GiB) in 1m 11s, read: 209.0 MiB/s, write: 207.4 MiB/s
INFO: 55% (33.4 GiB of 60.0 GiB) in 1m 14s, read: 460.1 MiB/s, write: 54.8 MiB/s
INFO: 57% (34.3 GiB of 60.0 GiB) in 1m 17s, read: 311.3 MiB/s, write: 132.1 MiB/s
INFO: 60% (36.5 GiB of 60.0 GiB) in 1m 20s, read: 759.1 MiB/s, write: 61.4 MiB/s
INFO: 64% (38.7 GiB of 60.0 GiB) in 1m 23s, read: 737.3 MiB/s, write: 85.2 MiB/s
INFO: 66% (39.8 GiB of 60.0 GiB) in 1m 26s, read: 382.6 MiB/s, write: 165.9 MiB/s
INFO: 67% (40.4 GiB of 60.0 GiB) in 1m 29s, read: 213.3 MiB/s, write: 212.5 MiB/s
INFO: 69% (41.7 GiB of 60.0 GiB) in 1m 32s, read: 415.2 MiB/s, write: 174.2 MiB/s
INFO: 71% (43.1 GiB of 60.0 GiB) in 1m 35s, read: 495.5 MiB/s, write: 139.6 MiB/s
INFO: 76% (45.7 GiB of 60.0 GiB) in 1m 38s, read: 885.3 MiB/s, write: 43.7 MiB/s
INFO: 79% (47.6 GiB of 60.0 GiB) in 1m 41s, read: 659.2 MiB/s, write: 102.7 MiB/s
INFO: 84% (50.6 GiB of 60.0 GiB) in 1m 44s, read: 995.2 MiB/s, write: 7.2 MiB/s
INFO: 87% (52.8 GiB of 60.0 GiB) in 1m 47s, read: 757.2 MiB/s, write: 74.3 MiB/s
INFO: 92% (55.8 GiB of 60.0 GiB) in 1m 50s, read: 1.0 GiB/s, write: 457.3 KiB/s
INFO: 96% (57.6 GiB of 60.0 GiB) in 1m 53s, read: 628.8 MiB/s, write: 98.9 MiB/s
INFO: 99% (59.7 GiB of 60.0 GiB) in 1m 56s, read: 695.1 MiB/s, write: 77.5 MiB/s
INFO: 100% (60.0 GiB of 60.0 GiB) in 1m 57s, read: 352.5 MiB/s, write: 0 B/s
INFO: backup is sparse: 45.11 GiB (75%) total zero data
INFO: transferred 60.00 GiB in 117 seconds (525.1 MiB/s)
INFO: archive file size: 6.76GB
INFO: removing backup 'external-backup:backup/vzdump-qemu-102-2021_04_12-00_00_02.vma.zst'
INFO: Finished Backup of VM 102 (00:01:57)
INFO: Backup finished at 2021-04-24 03:01:58
....
INFO: Starting Backup of VM 109 (qemu)
INFO: Backup started at 2021-04-24 03:06:19
INFO: status = running
INFO: VM Name: rootutil01
INFO: include disk 'scsi0' 'local-lvm:vm-109-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-109-2021_04_24-03_06_19.vma.zst'
INFO: started backup task 'e36bae52-8bbf-4771-a6b1-ab97b15bda82'
INFO: resuming VM again
INFO: 3% (1.2 GiB of 32.0 GiB) in 3s, read: 400.0 MiB/s, write: 139.8 MiB/s
INFO: 5% (1.6 GiB of 32.0 GiB) in 6s, read: 153.5 MiB/s, write: 150.4 MiB/s
INFO: 7% (2.3 GiB of 32.0 GiB) in 9s, read: 219.9 MiB/s, write: 156.1 MiB/s
INFO: 9% (2.9 GiB of 32.0 GiB) in 12s, read: 215.1 MiB/s, write: 157.8 MiB/s
INFO: 10% (3.5 GiB of 32.0 GiB) in 15s, read: 199.4 MiB/s, write: 155.6 MiB/s
INFO: 12% (4.0 GiB of 32.0 GiB) in 18s, read: 172.8 MiB/s, write: 171.1 MiB/s
INFO: 13% (4.5 GiB of 32.0 GiB) in 21s, read: 164.3 MiB/s, write: 157.3 MiB/s
INFO: 15% (5.0 GiB of 32.0 GiB) in 24s, read: 171.6 MiB/s, write: 169.1 MiB/s
INFO: 17% (5.6 GiB of 32.0 GiB) in 27s, read: 206.6 MiB/s, write: 169.9 MiB/s
INFO: 18% (6.0 GiB of 32.0 GiB) in 30s, read: 153.5 MiB/s, write: 151.7 MiB/s
INFO: 20% (6.6 GiB of 32.0 GiB) in 33s, read: 195.8 MiB/s, write: 193.4 MiB/s
INFO: 22% (7.2 GiB of 32.0 GiB) in 36s, read: 205.4 MiB/s, write: 183.8 MiB/s
INFO: 23% (7.7 GiB of 32.0 GiB) in 39s, read: 158.3 MiB/s, write: 157.7 MiB/s
--------------------------------------RUN 2--------------------------------------
Status: stopped: unexpected status
Task type: vzdump
Start Time: 2021-04-21 07:40:54
End Time: 2021-04-21 07:54:17
INFO: starting new backup job: vzdump --storage external-backup --all 1 --mailnotification failure --compress zstd --quiet 1 --node rootmox --exclude 100,101,106,107,103 --mode snapshot
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2021-04-21 07:40:54
INFO: status = running
INFO: VM Name: rootsop
INFO: include disk 'scsi1' 'local-lvm:vm-102-disk-0' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-102-2021_04_21-07_40_54.vma.zst'
INFO: started backup task 'cbc7cf87-a462-4ddf-9a7a-e76c5df24920'
INFO: resuming VM again
INFO: 2% (1.7 GiB of 60.0 GiB) in 3s, read: 592.1 MiB/s, write: 172.6 MiB/s
INFO: 5% (3.2 GiB of 60.0 GiB) in 6s, read: 488.6 MiB/s, write: 148.2 MiB/s
INFO: 6% (3.6 GiB of 60.0 GiB) in 9s, read: 162.7 MiB/s, write: 132.3 MiB/s
INFO: 7% (4.3 GiB of 60.0 GiB) in 13s, read: 178.5 MiB/s, write: 165.5 MiB/s
INFO: 8% (4.8 GiB of 60.0 GiB) in 16s, read: 169.2 MiB/s, write: 160.5 MiB/s
INFO: 9% (5.6 GiB of 60.0 GiB) in 21s, read: 158.8 MiB/s, write: 131.7 MiB/s
INFO: 14% (8.5 GiB of 60.0 GiB) in 24s, read: 998.9 MiB/s, write: 3.5 MiB/s
INFO: 22% (13.4 GiB of 60.0 GiB) in 27s, read: 1.6 GiB/s, write: 50.1 MiB/s
INFO: 26% (16.0 GiB of 60.0 GiB) in 30s, read: 897.1 MiB/s, write: 154.1 MiB/s
INFO: 27% (16.7 GiB of 60.0 GiB) in 33s, read: 233.2 MiB/s, write: 229.1 MiB/s
INFO: 28% (17.3 GiB of 60.0 GiB) in 36s, read: 206.6 MiB/s, write: 206.6 MiB/s
INFO: 30% (18.1 GiB of 60.0 GiB) in 39s, read: 273.1 MiB/s, write: 206.2 MiB/s
INFO: 32% (19.7 GiB of 60.0 GiB) in 42s, read: 552.2 MiB/s, write: 121.6 MiB/s
INFO: 34% (20.7 GiB of 60.0 GiB) in 45s, read: 337.7 MiB/s, write: 185.0 MiB/s
INFO: 36% (22.0 GiB of 60.0 GiB) in 48s, read: 458.0 MiB/s, write: 171.2 MiB/s
INFO: 39% (23.6 GiB of 60.0 GiB) in 51s, read: 552.8 MiB/s, write: 125.3 MiB/s
INFO: 40% (24.3 GiB of 60.0 GiB) in 54s, read: 236.8 MiB/s, write: 235.8 MiB/s
INFO: 41% (25.0 GiB of 60.0 GiB) in 57s, read: 222.3 MiB/s, write: 222.1 MiB/s
INFO: 42% (25.8 GiB of 60.0 GiB) in 1m, read: 277.9 MiB/s, write: 233.9 MiB/s
INFO: 44% (26.5 GiB of 60.0 GiB) in 1m 3s, read: 222.3 MiB/s, write: 221.8 MiB/s
INFO: 46% (28.0 GiB of 60.0 GiB) in 1m 6s, read: 537.9 MiB/s, write: 46.0 MiB/s
INFO: 51% (31.0 GiB of 60.0 GiB) in 1m 9s, read: 1013.2 MiB/s, write: 2.4 MiB/s
INFO: 53% (32.0 GiB of 60.0 GiB) in 1m 12s, read: 354.5 MiB/s, write: 176.4 MiB/s
INFO: 55% (33.6 GiB of 60.0 GiB) in 1m 15s, read: 529.2 MiB/s, write: 122.6 MiB/s
INFO: 59% (35.4 GiB of 60.0 GiB) in 1m 18s, read: 626.8 MiB/s, write: 127.2 MiB/s
INFO: 62% (37.6 GiB of 60.0 GiB) in 1m 21s, read: 736.8 MiB/s, write: 88.2 MiB/s
INFO: 66% (39.6 GiB of 60.0 GiB) in 1m 24s, read: 693.0 MiB/s, write: 95.5 MiB/s
INFO: 67% (40.4 GiB of 60.0 GiB) in 1m 28s, read: 199.2 MiB/s, write: 199.2 MiB/s
INFO: 69% (41.6 GiB of 60.0 GiB) in 1m 31s, read: 407.5 MiB/s, write: 165.7 MiB/s
INFO: 71% (42.7 GiB of 60.0 GiB) in 1m 34s, read: 366.3 MiB/s, write: 165.9 MiB/s
INFO: 75% (45.2 GiB of 60.0 GiB) in 1m 37s, read: 860.2 MiB/s, write: 43.7 MiB/s
INFO: 79% (47.4 GiB of 60.0 GiB) in 1m 40s, read: 774.0 MiB/s, write: 38.4 MiB/s
INFO: 82% (49.4 GiB of 60.0 GiB) in 1m 43s, read: 659.7 MiB/s, write: 71.2 MiB/s
INFO: 85% (51.5 GiB of 60.0 GiB) in 1m 46s, read: 730.4 MiB/s, write: 63.0 MiB/s
INFO: 90% (54.1 GiB of 60.0 GiB) in 1m 49s, read: 877.0 MiB/s, write: 11.7 MiB/s
INFO: 94% (56.7 GiB of 60.0 GiB) in 1m 52s, read: 899.8 MiB/s, write: 30.7 KiB/s
INFO: 96% (57.7 GiB of 60.0 GiB) in 1m 55s, read: 351.1 MiB/s, write: 142.6 MiB/s
INFO: 99% (59.9 GiB of 60.0 GiB) in 1m 58s, read: 737.9 MiB/s, write: 33.7 MiB/s
INFO: 100% (60.0 GiB of 60.0 GiB) in 1m 59s, read: 91.6 MiB/s, write: 0 B/s
INFO: backup is sparse: 45.12 GiB (75%) total zero data
INFO: transferred 60.00 GiB in 119 seconds (516.3 MiB/s)
INFO: archive file size: 6.39GB
INFO: removing backup 'external-backup:backup/vzdump-qemu-102-2021_04_21-00_00_02.vma.zst'
INFO: Finished Backup of VM 102 (00:01:59)
INFO: VM Name: rootutil02
INFO: include disk 'scsi0' 'external-hdd-img:110/vm-110-disk-0.qcow2' 32G
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-110-2021_04_21-07_50_11.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '4b504b29-39fa-48b2-ac38-30b126011c5a'
INFO: 3% (1.2 GiB of 32.0 GiB) in 3s, read: 409.6 MiB/s, write: 64.1 MiB/s
INFO: 4% (1.4 GiB of 32.0 GiB) in 6s, read: 81.0 MiB/s, write: 76.9 MiB/s
INFO: 5% (1.7 GiB of 32.0 GiB) in 9s, read: 76.9 MiB/s, write: 73.0 MiB/s
INFO: 6% (2.0 GiB of 32.0 GiB) in 18s, read: 36.4 MiB/s, write: 28.8 MiB/s
INFO: 7% (2.3 GiB of 32.0 GiB) in 24s, read: 51.4 MiB/s, write: 31.6 MiB/s
INFO: 8% (2.6 GiB of 32.0 GiB) in 27s, read: 102.7 MiB/s, write: 56.3 MiB/s
INFO: 10% (3.4 GiB of 32.0 GiB) in 30s, read: 275.1 MiB/s, write: 64.1 MiB/s
INFO: 16% (5.1 GiB of 32.0 GiB) in 33s, read: 593.9 MiB/s, write: 15.3 MiB/s
INFO: 17% (5.5 GiB of 32.0 GiB) in 38s, read: 70.4 MiB/s, write: 69.4 MiB/s
INFO: 18% (5.8 GiB of 32.0 GiB) in 44s, read: 49.3 MiB/s, write: 47.2 MiB/s
INFO: 19% (6.1 GiB of 32.0 GiB) in 49s, read: 75.6 MiB/s, write: 71.2 MiB/s
INFO: 20% (6.4 GiB of 32.0 GiB) in 53s, read: 75.0 MiB/s, write: 71.9 MiB/s
INFO: 21% (6.7 GiB of 32.0 GiB) in 1m 3s, read: 31.6 MiB/s, write: 30.0 MiB/s
INFO: 22% (7.2 GiB of 32.0 GiB) in 1m 7s, read: 112.4 MiB/s, write: 76.3 MiB/s
INFO: 23% (7.4 GiB of 32.0 GiB) in 1m 10s, read: 76.4 MiB/s, write: 74.5 MiB/s
INFO: 24% (7.7 GiB of 32.0 GiB) in 1m 14s, read: 74.1 MiB/s, write: 70.4 MiB/s
INFO: 25% (8.0 GiB of 32.0 GiB) in 1m 19s, read: 66.0 MiB/s, write: 62.7 MiB/s
pve-manager/6.3-6/2184247e
Over the last weeks I have noticed the following behaviour. When running a
scheduled or manual backup of my VMs there is a high change that the proxmox server itself will completly freeze. The only resolution would be to force a reboot of proxmox. The issue occurs both when using USB attached storage as backup target or even local storage.
Anything specific in regards to configuration,.. I can look at to resolve this issue?
**full backups logs can be found attached**
--------------------------------------RUN 1--------------------------------------
Status: stopped: unexpected status
Task type: vzdump
Start Time: 2021-04-24 03:00:01
End Time: 2021-04-24 08:53:38
INFO: starting new backup job: vzdump --exclude 101 --mailnotification failure --quiet 1 --all 1 --compress zstd --mode snapshot --storage external-backup
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2021-04-24 03:00:01
INFO: status = running
INFO: VM Name: rootsop
INFO: include disk 'scsi1' 'local-lvm:vm-102-disk-0' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-102-2021_04_24-03_00_01.vma.zst'
INFO: started backup task '674be1f2-7973-4ad8-b553-f4da5b5785ad'
INFO: resuming VM again
INFO: 2% (1.7 GiB of 60.0 GiB) in 3s, read: 575.2 MiB/s, write: 155.7 MiB/s
INFO: 4% (3.0 GiB of 60.0 GiB) in 6s, read: 447.6 MiB/s, write: 158.2 MiB/s
INFO: 6% (3.6 GiB of 60.0 GiB) in 9s, read: 218.1 MiB/s, write: 136.8 MiB/s
INFO: 7% (4.4 GiB of 60.0 GiB) in 13s, read: 189.4 MiB/s, write: 174.3 MiB/s
INFO: 8% (4.8 GiB of 60.0 GiB) in 16s, read: 159.5 MiB/s, write: 148.3 MiB/s
INFO: 10% (6.3 GiB of 60.0 GiB) in 20s, read: 362.4 MiB/s, write: 120.7 MiB/s
INFO: 15% (9.1 GiB of 60.0 GiB) in 23s, read: 970.4 MiB/s, write: 7.5 MiB/s
INFO: 25% (15.1 GiB of 60.0 GiB) in 26s, read: 2.0 GiB/s, write: 45.1 MiB/s
INFO: 26% (16.0 GiB of 60.0 GiB) in 29s, read: 298.1 MiB/s, write: 208.7 MiB/s
INFO: 27% (16.6 GiB of 60.0 GiB) in 32s, read: 219.9 MiB/s, write: 217.2 MiB/s
INFO: 28% (17.4 GiB of 60.0 GiB) in 35s, read: 270.7 MiB/s, write: 204.5 MiB/s
INFO: 30% (18.1 GiB of 60.0 GiB) in 38s, read: 229.6 MiB/s, write: 227.3 MiB/s
INFO: 32% (19.7 GiB of 60.0 GiB) in 41s, read: 554.6 MiB/s, write: 125.9 MiB/s
INFO: 35% (21.3 GiB of 60.0 GiB) in 44s, read: 540.3 MiB/s, write: 120.5 MiB/s
INFO: 36% (22.0 GiB of 60.0 GiB) in 47s, read: 247.5 MiB/s, write: 220.0 MiB/s
INFO: 39% (23.6 GiB of 60.0 GiB) in 50s, read: 552.2 MiB/s, write: 132.6 MiB/s
INFO: 40% (24.3 GiB of 60.0 GiB) in 53s, read: 227.2 MiB/s, write: 226.2 MiB/s
INFO: 41% (25.0 GiB of 60.0 GiB) in 56s, read: 231.8 MiB/s, write: 231.6 MiB/s
INFO: 42% (25.7 GiB of 60.0 GiB) in 59s, read: 244.3 MiB/s, write: 200.3 MiB/s
INFO: 43% (26.3 GiB of 60.0 GiB) in 1m 2s, read: 205.4 MiB/s, write: 204.7 MiB/s
INFO: 47% (28.8 GiB of 60.0 GiB) in 1m 5s, read: 856.3 MiB/s, write: 39.4 MiB/s
INFO: 52% (31.4 GiB of 60.0 GiB) in 1m 8s, read: 903.0 MiB/s, write: 39.5 MiB/s
INFO: 53% (32.1 GiB of 60.0 GiB) in 1m 11s, read: 209.0 MiB/s, write: 207.4 MiB/s
INFO: 55% (33.4 GiB of 60.0 GiB) in 1m 14s, read: 460.1 MiB/s, write: 54.8 MiB/s
INFO: 57% (34.3 GiB of 60.0 GiB) in 1m 17s, read: 311.3 MiB/s, write: 132.1 MiB/s
INFO: 60% (36.5 GiB of 60.0 GiB) in 1m 20s, read: 759.1 MiB/s, write: 61.4 MiB/s
INFO: 64% (38.7 GiB of 60.0 GiB) in 1m 23s, read: 737.3 MiB/s, write: 85.2 MiB/s
INFO: 66% (39.8 GiB of 60.0 GiB) in 1m 26s, read: 382.6 MiB/s, write: 165.9 MiB/s
INFO: 67% (40.4 GiB of 60.0 GiB) in 1m 29s, read: 213.3 MiB/s, write: 212.5 MiB/s
INFO: 69% (41.7 GiB of 60.0 GiB) in 1m 32s, read: 415.2 MiB/s, write: 174.2 MiB/s
INFO: 71% (43.1 GiB of 60.0 GiB) in 1m 35s, read: 495.5 MiB/s, write: 139.6 MiB/s
INFO: 76% (45.7 GiB of 60.0 GiB) in 1m 38s, read: 885.3 MiB/s, write: 43.7 MiB/s
INFO: 79% (47.6 GiB of 60.0 GiB) in 1m 41s, read: 659.2 MiB/s, write: 102.7 MiB/s
INFO: 84% (50.6 GiB of 60.0 GiB) in 1m 44s, read: 995.2 MiB/s, write: 7.2 MiB/s
INFO: 87% (52.8 GiB of 60.0 GiB) in 1m 47s, read: 757.2 MiB/s, write: 74.3 MiB/s
INFO: 92% (55.8 GiB of 60.0 GiB) in 1m 50s, read: 1.0 GiB/s, write: 457.3 KiB/s
INFO: 96% (57.6 GiB of 60.0 GiB) in 1m 53s, read: 628.8 MiB/s, write: 98.9 MiB/s
INFO: 99% (59.7 GiB of 60.0 GiB) in 1m 56s, read: 695.1 MiB/s, write: 77.5 MiB/s
INFO: 100% (60.0 GiB of 60.0 GiB) in 1m 57s, read: 352.5 MiB/s, write: 0 B/s
INFO: backup is sparse: 45.11 GiB (75%) total zero data
INFO: transferred 60.00 GiB in 117 seconds (525.1 MiB/s)
INFO: archive file size: 6.76GB
INFO: removing backup 'external-backup:backup/vzdump-qemu-102-2021_04_12-00_00_02.vma.zst'
INFO: Finished Backup of VM 102 (00:01:57)
INFO: Backup finished at 2021-04-24 03:01:58
....
INFO: Starting Backup of VM 109 (qemu)
INFO: Backup started at 2021-04-24 03:06:19
INFO: status = running
INFO: VM Name: rootutil01
INFO: include disk 'scsi0' 'local-lvm:vm-109-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-109-2021_04_24-03_06_19.vma.zst'
INFO: started backup task 'e36bae52-8bbf-4771-a6b1-ab97b15bda82'
INFO: resuming VM again
INFO: 3% (1.2 GiB of 32.0 GiB) in 3s, read: 400.0 MiB/s, write: 139.8 MiB/s
INFO: 5% (1.6 GiB of 32.0 GiB) in 6s, read: 153.5 MiB/s, write: 150.4 MiB/s
INFO: 7% (2.3 GiB of 32.0 GiB) in 9s, read: 219.9 MiB/s, write: 156.1 MiB/s
INFO: 9% (2.9 GiB of 32.0 GiB) in 12s, read: 215.1 MiB/s, write: 157.8 MiB/s
INFO: 10% (3.5 GiB of 32.0 GiB) in 15s, read: 199.4 MiB/s, write: 155.6 MiB/s
INFO: 12% (4.0 GiB of 32.0 GiB) in 18s, read: 172.8 MiB/s, write: 171.1 MiB/s
INFO: 13% (4.5 GiB of 32.0 GiB) in 21s, read: 164.3 MiB/s, write: 157.3 MiB/s
INFO: 15% (5.0 GiB of 32.0 GiB) in 24s, read: 171.6 MiB/s, write: 169.1 MiB/s
INFO: 17% (5.6 GiB of 32.0 GiB) in 27s, read: 206.6 MiB/s, write: 169.9 MiB/s
INFO: 18% (6.0 GiB of 32.0 GiB) in 30s, read: 153.5 MiB/s, write: 151.7 MiB/s
INFO: 20% (6.6 GiB of 32.0 GiB) in 33s, read: 195.8 MiB/s, write: 193.4 MiB/s
INFO: 22% (7.2 GiB of 32.0 GiB) in 36s, read: 205.4 MiB/s, write: 183.8 MiB/s
INFO: 23% (7.7 GiB of 32.0 GiB) in 39s, read: 158.3 MiB/s, write: 157.7 MiB/s
--------------------------------------RUN 2--------------------------------------
Status: stopped: unexpected status
Task type: vzdump
Start Time: 2021-04-21 07:40:54
End Time: 2021-04-21 07:54:17
INFO: starting new backup job: vzdump --storage external-backup --all 1 --mailnotification failure --compress zstd --quiet 1 --node rootmox --exclude 100,101,106,107,103 --mode snapshot
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2021-04-21 07:40:54
INFO: status = running
INFO: VM Name: rootsop
INFO: include disk 'scsi1' 'local-lvm:vm-102-disk-0' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-102-2021_04_21-07_40_54.vma.zst'
INFO: started backup task 'cbc7cf87-a462-4ddf-9a7a-e76c5df24920'
INFO: resuming VM again
INFO: 2% (1.7 GiB of 60.0 GiB) in 3s, read: 592.1 MiB/s, write: 172.6 MiB/s
INFO: 5% (3.2 GiB of 60.0 GiB) in 6s, read: 488.6 MiB/s, write: 148.2 MiB/s
INFO: 6% (3.6 GiB of 60.0 GiB) in 9s, read: 162.7 MiB/s, write: 132.3 MiB/s
INFO: 7% (4.3 GiB of 60.0 GiB) in 13s, read: 178.5 MiB/s, write: 165.5 MiB/s
INFO: 8% (4.8 GiB of 60.0 GiB) in 16s, read: 169.2 MiB/s, write: 160.5 MiB/s
INFO: 9% (5.6 GiB of 60.0 GiB) in 21s, read: 158.8 MiB/s, write: 131.7 MiB/s
INFO: 14% (8.5 GiB of 60.0 GiB) in 24s, read: 998.9 MiB/s, write: 3.5 MiB/s
INFO: 22% (13.4 GiB of 60.0 GiB) in 27s, read: 1.6 GiB/s, write: 50.1 MiB/s
INFO: 26% (16.0 GiB of 60.0 GiB) in 30s, read: 897.1 MiB/s, write: 154.1 MiB/s
INFO: 27% (16.7 GiB of 60.0 GiB) in 33s, read: 233.2 MiB/s, write: 229.1 MiB/s
INFO: 28% (17.3 GiB of 60.0 GiB) in 36s, read: 206.6 MiB/s, write: 206.6 MiB/s
INFO: 30% (18.1 GiB of 60.0 GiB) in 39s, read: 273.1 MiB/s, write: 206.2 MiB/s
INFO: 32% (19.7 GiB of 60.0 GiB) in 42s, read: 552.2 MiB/s, write: 121.6 MiB/s
INFO: 34% (20.7 GiB of 60.0 GiB) in 45s, read: 337.7 MiB/s, write: 185.0 MiB/s
INFO: 36% (22.0 GiB of 60.0 GiB) in 48s, read: 458.0 MiB/s, write: 171.2 MiB/s
INFO: 39% (23.6 GiB of 60.0 GiB) in 51s, read: 552.8 MiB/s, write: 125.3 MiB/s
INFO: 40% (24.3 GiB of 60.0 GiB) in 54s, read: 236.8 MiB/s, write: 235.8 MiB/s
INFO: 41% (25.0 GiB of 60.0 GiB) in 57s, read: 222.3 MiB/s, write: 222.1 MiB/s
INFO: 42% (25.8 GiB of 60.0 GiB) in 1m, read: 277.9 MiB/s, write: 233.9 MiB/s
INFO: 44% (26.5 GiB of 60.0 GiB) in 1m 3s, read: 222.3 MiB/s, write: 221.8 MiB/s
INFO: 46% (28.0 GiB of 60.0 GiB) in 1m 6s, read: 537.9 MiB/s, write: 46.0 MiB/s
INFO: 51% (31.0 GiB of 60.0 GiB) in 1m 9s, read: 1013.2 MiB/s, write: 2.4 MiB/s
INFO: 53% (32.0 GiB of 60.0 GiB) in 1m 12s, read: 354.5 MiB/s, write: 176.4 MiB/s
INFO: 55% (33.6 GiB of 60.0 GiB) in 1m 15s, read: 529.2 MiB/s, write: 122.6 MiB/s
INFO: 59% (35.4 GiB of 60.0 GiB) in 1m 18s, read: 626.8 MiB/s, write: 127.2 MiB/s
INFO: 62% (37.6 GiB of 60.0 GiB) in 1m 21s, read: 736.8 MiB/s, write: 88.2 MiB/s
INFO: 66% (39.6 GiB of 60.0 GiB) in 1m 24s, read: 693.0 MiB/s, write: 95.5 MiB/s
INFO: 67% (40.4 GiB of 60.0 GiB) in 1m 28s, read: 199.2 MiB/s, write: 199.2 MiB/s
INFO: 69% (41.6 GiB of 60.0 GiB) in 1m 31s, read: 407.5 MiB/s, write: 165.7 MiB/s
INFO: 71% (42.7 GiB of 60.0 GiB) in 1m 34s, read: 366.3 MiB/s, write: 165.9 MiB/s
INFO: 75% (45.2 GiB of 60.0 GiB) in 1m 37s, read: 860.2 MiB/s, write: 43.7 MiB/s
INFO: 79% (47.4 GiB of 60.0 GiB) in 1m 40s, read: 774.0 MiB/s, write: 38.4 MiB/s
INFO: 82% (49.4 GiB of 60.0 GiB) in 1m 43s, read: 659.7 MiB/s, write: 71.2 MiB/s
INFO: 85% (51.5 GiB of 60.0 GiB) in 1m 46s, read: 730.4 MiB/s, write: 63.0 MiB/s
INFO: 90% (54.1 GiB of 60.0 GiB) in 1m 49s, read: 877.0 MiB/s, write: 11.7 MiB/s
INFO: 94% (56.7 GiB of 60.0 GiB) in 1m 52s, read: 899.8 MiB/s, write: 30.7 KiB/s
INFO: 96% (57.7 GiB of 60.0 GiB) in 1m 55s, read: 351.1 MiB/s, write: 142.6 MiB/s
INFO: 99% (59.9 GiB of 60.0 GiB) in 1m 58s, read: 737.9 MiB/s, write: 33.7 MiB/s
INFO: 100% (60.0 GiB of 60.0 GiB) in 1m 59s, read: 91.6 MiB/s, write: 0 B/s
INFO: backup is sparse: 45.12 GiB (75%) total zero data
INFO: transferred 60.00 GiB in 119 seconds (516.3 MiB/s)
INFO: archive file size: 6.39GB
INFO: removing backup 'external-backup:backup/vzdump-qemu-102-2021_04_21-00_00_02.vma.zst'
INFO: Finished Backup of VM 102 (00:01:59)
INFO: VM Name: rootutil02
INFO: include disk 'scsi0' 'external-hdd-img:110/vm-110-disk-0.qcow2' 32G
INFO: creating vzdump archive '/mnt/data/backup/dump/vzdump-qemu-110-2021_04_21-07_50_11.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '4b504b29-39fa-48b2-ac38-30b126011c5a'
INFO: 3% (1.2 GiB of 32.0 GiB) in 3s, read: 409.6 MiB/s, write: 64.1 MiB/s
INFO: 4% (1.4 GiB of 32.0 GiB) in 6s, read: 81.0 MiB/s, write: 76.9 MiB/s
INFO: 5% (1.7 GiB of 32.0 GiB) in 9s, read: 76.9 MiB/s, write: 73.0 MiB/s
INFO: 6% (2.0 GiB of 32.0 GiB) in 18s, read: 36.4 MiB/s, write: 28.8 MiB/s
INFO: 7% (2.3 GiB of 32.0 GiB) in 24s, read: 51.4 MiB/s, write: 31.6 MiB/s
INFO: 8% (2.6 GiB of 32.0 GiB) in 27s, read: 102.7 MiB/s, write: 56.3 MiB/s
INFO: 10% (3.4 GiB of 32.0 GiB) in 30s, read: 275.1 MiB/s, write: 64.1 MiB/s
INFO: 16% (5.1 GiB of 32.0 GiB) in 33s, read: 593.9 MiB/s, write: 15.3 MiB/s
INFO: 17% (5.5 GiB of 32.0 GiB) in 38s, read: 70.4 MiB/s, write: 69.4 MiB/s
INFO: 18% (5.8 GiB of 32.0 GiB) in 44s, read: 49.3 MiB/s, write: 47.2 MiB/s
INFO: 19% (6.1 GiB of 32.0 GiB) in 49s, read: 75.6 MiB/s, write: 71.2 MiB/s
INFO: 20% (6.4 GiB of 32.0 GiB) in 53s, read: 75.0 MiB/s, write: 71.9 MiB/s
INFO: 21% (6.7 GiB of 32.0 GiB) in 1m 3s, read: 31.6 MiB/s, write: 30.0 MiB/s
INFO: 22% (7.2 GiB of 32.0 GiB) in 1m 7s, read: 112.4 MiB/s, write: 76.3 MiB/s
INFO: 23% (7.4 GiB of 32.0 GiB) in 1m 10s, read: 76.4 MiB/s, write: 74.5 MiB/s
INFO: 24% (7.7 GiB of 32.0 GiB) in 1m 14s, read: 74.1 MiB/s, write: 70.4 MiB/s
INFO: 25% (8.0 GiB of 32.0 GiB) in 1m 19s, read: 66.0 MiB/s, write: 62.7 MiB/s
Last edited:
