hello,
i 've a cluster with 4 node proxmox ve 6.2-10
nodes shares NFS storage (2 nas, 1 fileserver)
since about 1 month i have been having problems with backups of almost all vm . vm have their disk on nfs storage. backup run usually on other nfs storage.
Backups starts regularly, even if one with poor performance, and then has timeouts of hours.
This happens regardless of source and destination storage. Occurs most often with snapshot backup mode but also with "stop" mode .
I don't think it's a network problem. Virtual machines are fast and responsive when running. The cluster synchronizes correctly and I have no problem migrating and making the HA work.
What is going on?
i 've a cluster with 4 node proxmox ve 6.2-10
nodes shares NFS storage (2 nas, 1 fileserver)
since about 1 month i have been having problems with backups of almost all vm . vm have their disk on nfs storage. backup run usually on other nfs storage.
Backups starts regularly, even if one with poor performance, and then has timeouts of hours.
This happens regardless of source and destination storage. Occurs most often with snapshot backup mode but also with "stop" mode .
I don't think it's a network problem. Virtual machines are fast and responsive when running. The cluster synchronizes correctly and I have no problem migrating and making the HA work.
What is going on?
Code:
()
INFO: starting new backup job: vzdump 104 --remove 0 --storage fileserver --compress zstd --mode snapshot --node tourronde
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2020-07-29 10:57:08
INFO: status = running
INFO: VM Name: varie-remapp
INFO: include disk 'scsi1' 'annapurna:104/vm-104-disk-0.raw' 121G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/fileserver/dump/vzdump-qemu-104-2020_07_29-10_57_08.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '8dee07e9-927e-4eed-be4d-a5a347f4acf2'
INFO: resuming VM again
INFO: status: 0% (127.4 MiB of 121.0 GiB), duration 4, read: 31.9 MiB/s, write: 15.8 MiB/s
INFO: status: 1% (1.2 GiB of 121.0 GiB), duration 24, read: 56.0 MiB/s, write: 49.8 MiB/s
INFO: status: 2% (2.5 GiB of 121.0 GiB), duration 49, read: 51.9 MiB/s, write: 50.0 MiB/s
INFO: status: 3% (3.7 GiB of 121.0 GiB), duration 69, read: 60.2 MiB/s, write: 59.5 MiB/s
INFO: status: 4% (4.9 GiB of 121.0 GiB), duration 89, read: 62.2 MiB/s, write: 61.6 MiB/s
INFO: status: 5% (6.1 GiB of 121.0 GiB), duration 109, read: 63.1 MiB/s, write: 62.7 MiB/s
INFO: status: 6% (7.3 GiB of 121.0 GiB), duration 129, read: 60.6 MiB/s, write: 58.7 MiB/s
INFO: status: 7% (8.5 GiB of 121.0 GiB), duration 149, read: 62.9 MiB/s, write: 61.3 MiB/s
INFO: status: 8% (9.7 GiB of 121.0 GiB), duration 171, read: 55.8 MiB/s, write: 55.1 MiB/s
INFO: status: 9% (10.9 GiB of 121.0 GiB), duration 186, read: 81.0 MiB/s, write: 76.2 MiB/s
INFO: status: 10% (12.1 GiB of 121.0 GiB), duration 201, read: 81.5 MiB/s, write: 78.6 MiB/s
INFO: status: 11% (13.3 GiB of 121.0 GiB), duration 224, read: 55.0 MiB/s, write: 54.9 MiB/s
INFO: status: 12% (14.6 GiB of 121.0 GiB), duration 244, read: 62.5 MiB/s, write: 61.0 MiB/s
INFO: status: 13% (15.7 GiB of 121.0 GiB), duration 263, read: 63.7 MiB/s, write: 63.3 MiB/s
INFO: status: 14% (17.0 GiB of 121.0 GiB), duration 285, read: 58.5 MiB/s, write: 57.9 MiB/s
INFO: status: 15% (18.2 GiB of 121.0 GiB), duration 8312, read: 151.6 KiB/s, write: 6.2 KiB/s
INFO: status: 16% (19.4 GiB of 121.0 GiB), duration 14221, read: 215.2 KiB/s, write: 297.0 B/s
INFO: status: 17% (20.6 GiB of 121.0 GiB), duration 16972, read: 459.8 KiB/s, write: 14.4 KiB/s
INFO: status: 18% (21.8 GiB of 121.0 GiB), duration 18830, read: 680.3 KiB/s, write: 138.8 KiB/s
INFO: status: 19% (23.0 GiB of 121.0 GiB), duration 20666, read: 705.8 KiB/s, write: 356.0 B/s
INFO: status: 20% (24.2 GiB of 121.0 GiB), duration 20684, read: 68.9 MiB/s, write: 67.7 MiB/s
INFO: status: 21% (25.4 GiB of 121.0 GiB), duration 20704, read: 62.5 MiB/s, write: 62.1 MiB/s
INFO: status: 22% (26.6 GiB of 121.0 GiB), duration 20726, read: 55.9 MiB/s, write: 55.7 MiB/s
INFO: status: 23% (27.9 GiB of 121.0 GiB), duration 20750, read: 52.7 MiB/s, write: 50.0 MiB/s
INFO: status: 24% (29.1 GiB of 121.0 GiB), duration 20766, read: 75.5 MiB/s, write: 75.2 MiB/s
INFO: status: 25% (30.3 GiB of 121.0 GiB), duration 20791, read: 52.2 MiB/s, write: 52.0 MiB/s
INFO: status: 26% (31.5 GiB of 121.0 GiB), duration 20814, read: 50.4 MiB/s, write: 48.4 MiB/s
INFO: status: 27% (32.7 GiB of 121.0 GiB), duration 20830, read: 77.8 MiB/s, write: 70.8 MiB/s
INFO: status: 28% (34.0 GiB of 121.0 GiB), duration 20853, read: 56.8 MiB/s, write: 52.8 MiB/s
INFO: status: 29% (35.2 GiB of 121.0 GiB), duration 20876, read: 53.5 MiB/s, write: 51.4 MiB/s
INFO: status: 30% (36.3 GiB of 121.0 GiB), duration 20893, read: 69.0 MiB/s, write: 65.8 MiB/s
INFO: status: 31% (37.6 GiB of 121.0 GiB), duration 20917, read: 53.0 MiB/s, write: 49.5 MiB/s
INFO: status: 32% (38.8 GiB of 121.0 GiB), duration 20940, read: 54.2 MiB/s, write: 52.3 MiB/s
INFO: status: 33% (39.9 GiB of 121.0 GiB), duration 20976, read: 33.3 MiB/s, write: 31.8 MiB/s
INFO: status: 34% (41.2 GiB of 121.0 GiB), duration 21005, read: 43.4 MiB/s, write: 39.3 MiB/s
INFO: status: 35% (42.4 GiB of 121.0 GiB), duration 21026, read: 57.8 MiB/s, write: 52.8 MiB/s
INFO: status: 36% (43.6 GiB of 121.0 GiB), duration 21057, read: 40.1 MiB/s, write: 31.5 MiB/s
INFO: status: 37% (44.8 GiB of 121.0 GiB), duration 22639, read: 800.2 KiB/s, write: 36.4 KiB/s
INFO: status: 38% (46.0 GiB of 121.0 GiB), duration 22758, read: 10.8 MiB/s, write: 8.3 MiB/s
INFO: status: 39% (47.2 GiB of 121.0 GiB), duration 22779, read: 58.1 MiB/s, write: 58.1 MiB/s
INFO: status: 40% (48.4 GiB of 121.0 GiB), duration 22803, read: 51.1 MiB/s, write: 51.1 MiB/s
INFO: status: 41% (49.6 GiB of 121.0 GiB), duration 23583, read: 1.6 MiB/s, write: 1.1 MiB/s
INFO: status: 42% (50.9 GiB of 121.0 GiB), duration 24273, read: 1.9 MiB/s, write: 612.8 KiB/s
INFO: status: 43% (52.0 GiB of 121.0 GiB), duration 24293, read: 60.0 MiB/s, write: 57.7 MiB/s
INFO: status: 44% (53.2 GiB of 121.0 GiB), duration 31623, read: 171.5 KiB/s, write: 42.9 KiB/s
INFO: status: 45% (54.5 GiB of 121.0 GiB), duration 39328, read: 164.5 KiB/s, write: 8.0 B/s
INFO: status: 46% (55.7 GiB of 121.0 GiB), duration 43249, read: 323.1 KiB/s, write: 0 B/s
INFO: status: 47% (56.9 GiB of 121.0 GiB), duration 44140, read: 1.4 MiB/s, write: 507.3 KiB/s
INFO: status: 48% (58.1 GiB of 121.0 GiB), duration 48077, read: 322.4 KiB/s, write: 16.0 B/s
INFO: status: 49% (59.3 GiB of 121.0 GiB), duration 48877, read: 1.5 MiB/s, write: 564.1 KiB/s
INFO: status: 50% (60.5 GiB of 121.0 GiB), duration 50341, read: 871.9 KiB/s, write: 12.8 KiB/s
INFO: status: 51% (61.7 GiB of 121.0 GiB), duration 54242, read: 324.0 KiB/s, write: 108.2 KiB/s
INFO: status: 52% (62.9 GiB of 121.0 GiB), duration 74764, read: 61.9 KiB/s, write: 3.0 B/s
#here i stopped backup process
ERROR: interrupted by signal
INFO: aborting backup job