Hello, since a couple of days the backup of a vm hosted by proxmox VE started to have performance issues.
Everything worked fine since last week, but in the last days the backup of a single vm after about 50% hangs (as you can see from the attached log the i/o from the virtual disk become less than 10 mb/sec compared to the 1000 mb/sec of the first half of backup). Even if i stop the backup the vm remain almost unresponsive because of the slow i/o and the only way is to stop and start the vm, even a hard reset is unable to solve.
As said before nothing changed in proxmox ve and proxmox backup and backup of other machines runs fine.
Any idea?
Thank you in advance
Luca
INFO: starting new backup job: vzdump 152 --node vm05 --remove 0 --mailto l.spallarossa@arte.ge.it --storage proxmox_backup_gelim8 --mode snapshot
INFO: Starting Backup of VM 152 (qemu)
INFO: Backup started at 2022-02-08 19:39:15
INFO: status = running
INFO: VM Name: servera-gcs-new05
INFO: include disk 'virtio0' 'data:152/vm-152-disk-0.qcow2' 150G
INFO: include disk 'virtio1' 'data:152/vm-152-disk-1.qcow2' 1500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/152/2022-02-08T18:39:15Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '2d75739a-d7ac-4a41-a776-011f4ed0269a'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: created new
INFO: virtio1: dirty-bitmap status: created new
INFO: 0% (1.8 GiB of 1.6 TiB) in 3s, read: 614.7 MiB/s, write: 9.3 MiB/s
INFO: 1% (16.7 GiB of 1.6 TiB) in 29s, read: 588.2 MiB/s, write: 94.9 MiB/s
INFO: 2% (33.4 GiB of 1.6 TiB) in 1m 2s, read: 517.8 MiB/s, write: 65.8 MiB/s
INFO: 3% (49.8 GiB of 1.6 TiB) in 2m 17s, read: 223.5 MiB/s, write: 124.1 MiB/s
INFO: 4% (66.4 GiB of 1.6 TiB) in 3m 9s, read: 327.8 MiB/s, write: 121.7 MiB/s
INFO: 5% (82.5 GiB of 1.6 TiB) in 4m 6s, read: 289.0 MiB/s, write: 124.2 MiB/s
INFO: 6% (99.3 GiB of 1.6 TiB) in 5m 11s, read: 263.6 MiB/s, write: 121.2 MiB/s
INFO: 7% (116.0 GiB of 1.6 TiB) in 5m 51s, read: 429.8 MiB/s, write: 105.1 MiB/s
INFO: 8% (132.4 GiB of 1.6 TiB) in 6m 12s, read: 797.1 MiB/s, write: 0 B/s
INFO: 9% (148.9 GiB of 1.6 TiB) in 6m 30s, read: 939.3 MiB/s, write: 1.1 MiB/s
INFO: 10% (165.6 GiB of 1.6 TiB) in 6m 48s, read: 951.8 MiB/s, write: 0 B/s
INFO: 11% (181.9 GiB of 1.6 TiB) in 7m 4s, read: 1.0 GiB/s, write: 0 B/s
INFO: 12% (198.4 GiB of 1.6 TiB) in 7m 22s, read: 937.3 MiB/s, write: 227.6 KiB/s
INFO: 13% (214.6 GiB of 1.6 TiB) in 7m 38s, read: 1.0 GiB/s, write: 512.0 KiB/s
INFO: 14% (231.8 GiB of 1.6 TiB) in 7m 58s, read: 879.4 MiB/s, write: 25.2 MiB/s
INFO: 15% (247.8 GiB of 1.6 TiB) in 8m 16s, read: 912.9 MiB/s, write: 0 B/s
INFO: 16% (264.5 GiB of 1.6 TiB) in 8m 33s, read: 1004.2 MiB/s, write: 963.8 KiB/s
INFO: 17% (281.4 GiB of 1.6 TiB) in 8m 53s, read: 865.2 MiB/s, write: 614.4 KiB/s
INFO: 18% (297.0 GiB of 1.6 TiB) in 9m 9s, read: 1002.5 MiB/s, write: 1.2 MiB/s
INFO: 19% (313.6 GiB of 1.6 TiB) in 9m 28s, read: 894.1 MiB/s, write: 0 B/s
INFO: 20% (330.4 GiB of 1.6 TiB) in 9m 47s, read: 906.7 MiB/s, write: 0 B/s
INFO: 21% (347.0 GiB of 1.6 TiB) in 10m 5s, read: 940.9 MiB/s, write: 455.1 KiB/s
INFO: 22% (363.4 GiB of 1.6 TiB) in 10m 22s, read: 988.7 MiB/s, write: 963.8 KiB/s
INFO: 23% (380.1 GiB of 1.6 TiB) in 10m 40s, read: 948.2 MiB/s, write: 227.6 KiB/s
INFO: 24% (396.3 GiB of 1.6 TiB) in 10m 57s, read: 979.3 MiB/s, write: 2.1 MiB/s
INFO: 25% (413.4 GiB of 1.6 TiB) in 11m 16s, read: 919.2 MiB/s, write: 3.6 MiB/s
INFO: 26% (429.9 GiB of 1.6 TiB) in 11m 34s, read: 941.8 MiB/s, write: 3.6 MiB/s
INFO: 27% (446.0 GiB of 1.6 TiB) in 11m 50s, read: 1.0 GiB/s, write: 3.0 MiB/s
INFO: 28% (462.5 GiB of 1.6 TiB) in 12m 15s, read: 675.7 MiB/s, write: 327.7 KiB/s
INFO: 29% (479.2 GiB of 1.6 TiB) in 12m 32s, read: 1004.9 MiB/s, write: 4.2 MiB/s
INFO: 30% (495.1 GiB of 1.6 TiB) in 12m 48s, read: 1019.5 MiB/s, write: 3.2 MiB/s
INFO: 31% (512.0 GiB of 1.6 TiB) in 13m 4s, read: 1.1 GiB/s, write: 2.8 MiB/s
INFO: 32% (528.2 GiB of 1.6 TiB) in 13m 21s, read: 973.9 MiB/s, write: 240.9 KiB/s
INFO: 33% (545.3 GiB of 1.6 TiB) in 13m 38s, read: 1.0 GiB/s, write: 5.4 MiB/s
INFO: 34% (561.4 GiB of 1.6 TiB) in 13m 54s, read: 1.0 GiB/s, write: 2.0 MiB/s
INFO: 35% (577.9 GiB of 1.6 TiB) in 14m 11s, read: 994.6 MiB/s, write: 240.9 KiB/s
INFO: 36% (594.9 GiB of 1.6 TiB) in 14m 29s, read: 963.3 MiB/s, write: 0 B/s
INFO: 37% (610.6 GiB of 1.6 TiB) in 14m 54s, read: 644.5 MiB/s, write: 76.2 MiB/s
INFO: 38% (627.7 GiB of 1.6 TiB) in 15m 12s, read: 970.7 MiB/s, write: 455.1 KiB/s
INFO: 39% (644.4 GiB of 1.6 TiB) in 15m 30s, read: 951.1 MiB/s, write: 0 B/s
INFO: 40% (660.8 GiB of 1.6 TiB) in 15m 46s, read: 1.0 GiB/s, write: 256.0 KiB/s
INFO: 41% (676.6 GiB of 1.6 TiB) in 16m 3s, read: 955.5 MiB/s, write: 240.9 KiB/s
INFO: 42% (693.7 GiB of 1.6 TiB) in 16m 21s, read: 973.3 MiB/s, write: 0 B/s
INFO: 43% (709.6 GiB of 1.6 TiB) in 16m 37s, read: 1014.8 MiB/s, write: 0 B/s
INFO: 44% (726.0 GiB of 1.6 TiB) in 16m 54s, read: 989.9 MiB/s, write: 240.9 KiB/s
INFO: 45% (742.9 GiB of 1.6 TiB) in 17m 11s, read: 1017.6 MiB/s, write: 240.9 KiB/s
INFO: 46% (759.1 GiB of 1.6 TiB) in 17m 28s, read: 976.5 MiB/s, write: 240.9 KiB/s
INFO: 47% (776.0 GiB of 1.6 TiB) in 17m 46s, read: 962.0 MiB/s, write: 455.1 KiB/s
INFO: 48% (792.8 GiB of 1.6 TiB) in 18m 5s, read: 901.5 MiB/s, write: 0 B/s
INFO: 49% (808.6 GiB of 1.6 TiB) in 39m 27s, read: 12.6 MiB/s, write: 12.8 KiB/s
INFO: 50% (825.0 GiB of 1.6 TiB) in 1h 9m 28s, read: 9.4 MiB/s, write: 4.5 KiB/s
INFO: 51% (841.6 GiB of 1.6 TiB) in 1h 39m 33s, read: 9.4 MiB/s, write: 0 B/s
Everything worked fine since last week, but in the last days the backup of a single vm after about 50% hangs (as you can see from the attached log the i/o from the virtual disk become less than 10 mb/sec compared to the 1000 mb/sec of the first half of backup). Even if i stop the backup the vm remain almost unresponsive because of the slow i/o and the only way is to stop and start the vm, even a hard reset is unable to solve.
As said before nothing changed in proxmox ve and proxmox backup and backup of other machines runs fine.
Any idea?
Thank you in advance
Luca
INFO: starting new backup job: vzdump 152 --node vm05 --remove 0 --mailto l.spallarossa@arte.ge.it --storage proxmox_backup_gelim8 --mode snapshot
INFO: Starting Backup of VM 152 (qemu)
INFO: Backup started at 2022-02-08 19:39:15
INFO: status = running
INFO: VM Name: servera-gcs-new05
INFO: include disk 'virtio0' 'data:152/vm-152-disk-0.qcow2' 150G
INFO: include disk 'virtio1' 'data:152/vm-152-disk-1.qcow2' 1500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/152/2022-02-08T18:39:15Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '2d75739a-d7ac-4a41-a776-011f4ed0269a'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: created new
INFO: virtio1: dirty-bitmap status: created new
INFO: 0% (1.8 GiB of 1.6 TiB) in 3s, read: 614.7 MiB/s, write: 9.3 MiB/s
INFO: 1% (16.7 GiB of 1.6 TiB) in 29s, read: 588.2 MiB/s, write: 94.9 MiB/s
INFO: 2% (33.4 GiB of 1.6 TiB) in 1m 2s, read: 517.8 MiB/s, write: 65.8 MiB/s
INFO: 3% (49.8 GiB of 1.6 TiB) in 2m 17s, read: 223.5 MiB/s, write: 124.1 MiB/s
INFO: 4% (66.4 GiB of 1.6 TiB) in 3m 9s, read: 327.8 MiB/s, write: 121.7 MiB/s
INFO: 5% (82.5 GiB of 1.6 TiB) in 4m 6s, read: 289.0 MiB/s, write: 124.2 MiB/s
INFO: 6% (99.3 GiB of 1.6 TiB) in 5m 11s, read: 263.6 MiB/s, write: 121.2 MiB/s
INFO: 7% (116.0 GiB of 1.6 TiB) in 5m 51s, read: 429.8 MiB/s, write: 105.1 MiB/s
INFO: 8% (132.4 GiB of 1.6 TiB) in 6m 12s, read: 797.1 MiB/s, write: 0 B/s
INFO: 9% (148.9 GiB of 1.6 TiB) in 6m 30s, read: 939.3 MiB/s, write: 1.1 MiB/s
INFO: 10% (165.6 GiB of 1.6 TiB) in 6m 48s, read: 951.8 MiB/s, write: 0 B/s
INFO: 11% (181.9 GiB of 1.6 TiB) in 7m 4s, read: 1.0 GiB/s, write: 0 B/s
INFO: 12% (198.4 GiB of 1.6 TiB) in 7m 22s, read: 937.3 MiB/s, write: 227.6 KiB/s
INFO: 13% (214.6 GiB of 1.6 TiB) in 7m 38s, read: 1.0 GiB/s, write: 512.0 KiB/s
INFO: 14% (231.8 GiB of 1.6 TiB) in 7m 58s, read: 879.4 MiB/s, write: 25.2 MiB/s
INFO: 15% (247.8 GiB of 1.6 TiB) in 8m 16s, read: 912.9 MiB/s, write: 0 B/s
INFO: 16% (264.5 GiB of 1.6 TiB) in 8m 33s, read: 1004.2 MiB/s, write: 963.8 KiB/s
INFO: 17% (281.4 GiB of 1.6 TiB) in 8m 53s, read: 865.2 MiB/s, write: 614.4 KiB/s
INFO: 18% (297.0 GiB of 1.6 TiB) in 9m 9s, read: 1002.5 MiB/s, write: 1.2 MiB/s
INFO: 19% (313.6 GiB of 1.6 TiB) in 9m 28s, read: 894.1 MiB/s, write: 0 B/s
INFO: 20% (330.4 GiB of 1.6 TiB) in 9m 47s, read: 906.7 MiB/s, write: 0 B/s
INFO: 21% (347.0 GiB of 1.6 TiB) in 10m 5s, read: 940.9 MiB/s, write: 455.1 KiB/s
INFO: 22% (363.4 GiB of 1.6 TiB) in 10m 22s, read: 988.7 MiB/s, write: 963.8 KiB/s
INFO: 23% (380.1 GiB of 1.6 TiB) in 10m 40s, read: 948.2 MiB/s, write: 227.6 KiB/s
INFO: 24% (396.3 GiB of 1.6 TiB) in 10m 57s, read: 979.3 MiB/s, write: 2.1 MiB/s
INFO: 25% (413.4 GiB of 1.6 TiB) in 11m 16s, read: 919.2 MiB/s, write: 3.6 MiB/s
INFO: 26% (429.9 GiB of 1.6 TiB) in 11m 34s, read: 941.8 MiB/s, write: 3.6 MiB/s
INFO: 27% (446.0 GiB of 1.6 TiB) in 11m 50s, read: 1.0 GiB/s, write: 3.0 MiB/s
INFO: 28% (462.5 GiB of 1.6 TiB) in 12m 15s, read: 675.7 MiB/s, write: 327.7 KiB/s
INFO: 29% (479.2 GiB of 1.6 TiB) in 12m 32s, read: 1004.9 MiB/s, write: 4.2 MiB/s
INFO: 30% (495.1 GiB of 1.6 TiB) in 12m 48s, read: 1019.5 MiB/s, write: 3.2 MiB/s
INFO: 31% (512.0 GiB of 1.6 TiB) in 13m 4s, read: 1.1 GiB/s, write: 2.8 MiB/s
INFO: 32% (528.2 GiB of 1.6 TiB) in 13m 21s, read: 973.9 MiB/s, write: 240.9 KiB/s
INFO: 33% (545.3 GiB of 1.6 TiB) in 13m 38s, read: 1.0 GiB/s, write: 5.4 MiB/s
INFO: 34% (561.4 GiB of 1.6 TiB) in 13m 54s, read: 1.0 GiB/s, write: 2.0 MiB/s
INFO: 35% (577.9 GiB of 1.6 TiB) in 14m 11s, read: 994.6 MiB/s, write: 240.9 KiB/s
INFO: 36% (594.9 GiB of 1.6 TiB) in 14m 29s, read: 963.3 MiB/s, write: 0 B/s
INFO: 37% (610.6 GiB of 1.6 TiB) in 14m 54s, read: 644.5 MiB/s, write: 76.2 MiB/s
INFO: 38% (627.7 GiB of 1.6 TiB) in 15m 12s, read: 970.7 MiB/s, write: 455.1 KiB/s
INFO: 39% (644.4 GiB of 1.6 TiB) in 15m 30s, read: 951.1 MiB/s, write: 0 B/s
INFO: 40% (660.8 GiB of 1.6 TiB) in 15m 46s, read: 1.0 GiB/s, write: 256.0 KiB/s
INFO: 41% (676.6 GiB of 1.6 TiB) in 16m 3s, read: 955.5 MiB/s, write: 240.9 KiB/s
INFO: 42% (693.7 GiB of 1.6 TiB) in 16m 21s, read: 973.3 MiB/s, write: 0 B/s
INFO: 43% (709.6 GiB of 1.6 TiB) in 16m 37s, read: 1014.8 MiB/s, write: 0 B/s
INFO: 44% (726.0 GiB of 1.6 TiB) in 16m 54s, read: 989.9 MiB/s, write: 240.9 KiB/s
INFO: 45% (742.9 GiB of 1.6 TiB) in 17m 11s, read: 1017.6 MiB/s, write: 240.9 KiB/s
INFO: 46% (759.1 GiB of 1.6 TiB) in 17m 28s, read: 976.5 MiB/s, write: 240.9 KiB/s
INFO: 47% (776.0 GiB of 1.6 TiB) in 17m 46s, read: 962.0 MiB/s, write: 455.1 KiB/s
INFO: 48% (792.8 GiB of 1.6 TiB) in 18m 5s, read: 901.5 MiB/s, write: 0 B/s
INFO: 49% (808.6 GiB of 1.6 TiB) in 39m 27s, read: 12.6 MiB/s, write: 12.8 KiB/s
INFO: 50% (825.0 GiB of 1.6 TiB) in 1h 9m 28s, read: 9.4 MiB/s, write: 4.5 KiB/s
INFO: 51% (841.6 GiB of 1.6 TiB) in 1h 39m 33s, read: 9.4 MiB/s, write: 0 B/s