My linux VMs back up fine, but the windows ones do not. I was using zstd and the backups kept freezing (I couldn't stop the process without restarting the node), so I am trying gzip now. This time, the process at least ends without hanging, but there are still errors on the windows VMs.
I'm backing up to a TrueNAS server. Here are the logs:
I'm backing up to a TrueNAS server. Here are the logs:
Code:
INFO: starting new backup job: vzdump 105 101 100 107 --prune-backups 'keep-last=3,keep-weekly=2' --mode snapshot --quiet 1 --storage proxmox-backups --compress gzip --mailnotification failure --notes-template '{{guestname}}' --mailto sshank@gmail.com
Code:
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2023-11-09 02:00:03
INFO: status = running
INFO: VM Name: ubuntu-server
INFO: include disk 'scsi0' 'local-btrfs:100/vm-100-disk-0.raw' 42G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox-backups/dump/vzdump-qemu-100-2023_11_09-02_00_02.vma.gz'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '092cb646-59e5-4160-98f0-c24d0858e90b'
INFO: resuming VM again
INFO: 0% (210.2 MiB of 42.0 GiB) in 3s, read: 70.1 MiB/s, write: 26.2 MiB/s
INFO: 1% (435.0 MiB of 42.0 GiB) in 12s, read: 25.0 MiB/s, write: 23.6 MiB/s
INFO: 2% (873.6 MiB of 42.0 GiB) in 30s, read: 24.4 MiB/s, write: 23.9 MiB/s
INFO: 4% (1.7 GiB of 42.0 GiB) in 37s, read: 121.7 MiB/s, write: 23.3 MiB/s
INFO: 5% (2.1 GiB of 42.0 GiB) in 40s, read: 151.0 MiB/s, write: 37.6 MiB/s
INFO: 6% (2.5 GiB of 42.0 GiB) in 55s, read: 28.0 MiB/s, write: 27.5 MiB/s
INFO: 7% (2.9 GiB of 42.0 GiB) in 1m 13s, read: 23.0 MiB/s, write: 22.4 MiB/s
INFO: 8% (3.4 GiB of 42.0 GiB) in 1m 32s, read: 24.0 MiB/s, write: 22.8 MiB/s
INFO: 9% (3.8 GiB of 42.0 GiB) in 1m 49s, read: 24.5 MiB/s, write: 22.5 MiB/s
INFO: 10% (4.2 GiB of 42.0 GiB) in 2m 5s, read: 27.9 MiB/s, write: 27.1 MiB/s
...
INFO: 90% (37.8 GiB of 42.0 GiB) in 29m 30s, read: 23.0 MiB/s, write: 22.8 MiB/s
INFO: 91% (38.2 GiB of 42.0 GiB) in 29m 45s, read: 29.7 MiB/s, write: 21.0 MiB/s
INFO: 92% (38.7 GiB of 42.0 GiB) in 30m 19s, read: 12.7 MiB/s, write: 12.6 MiB/s
INFO: 93% (39.1 GiB of 42.0 GiB) in 30m 39s, read: 20.8 MiB/s, write: 20.1 MiB/s
INFO: 94% (39.5 GiB of 42.0 GiB) in 30m 59s, read: 21.8 MiB/s, write: 21.7 MiB/s
INFO: 95% (39.9 GiB of 42.0 GiB) in 31m 47s, read: 9.2 MiB/s, write: 8.9 MiB/s
INFO: 96% (40.3 GiB of 42.0 GiB) in 31m 59s, read: 35.6 MiB/s, write: 22.2 MiB/s
INFO: 97% (40.8 GiB of 42.0 GiB) in 32m 16s, read: 24.9 MiB/s, write: 23.0 MiB/s
INFO: 98% (41.2 GiB of 42.0 GiB) in 32m 32s, read: 26.3 MiB/s, write: 22.4 MiB/s
INFO: 99% (41.6 GiB of 42.0 GiB) in 32m 53s, read: 20.7 MiB/s, write: 18.9 MiB/s
INFO: 100% (42.0 GiB of 42.0 GiB) in 33m 12s, read: 22.2 MiB/s, write: 19.8 MiB/s
INFO: backup is sparse: 5.21 GiB (12%) total zero data
INFO: transferred 42.00 GiB in 1992 seconds (21.6 MiB/s)
INFO: archive file size: 20.15GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=3, keep-weekly=2
INFO: removing backup 'proxmox-backups:backup/vzdump-qemu-100-2023_11_04-03_00_01.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 100 (00:36:09)
INFO: Backup finished at 2023-11-09 02:36:11
Code:
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-11-09 02:36:11
INFO: status = running
INFO: VM Name: DietPi
INFO: include disk 'scsi0' 'local-btrfs:101/vm-101-disk-0.raw' 8G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox-backups/dump/vzdump-qemu-101-2023_11_09-02_36_11.vma.gz'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '6f07b0c5-4020-43c7-a593-49b965abe9d6'
INFO: resuming VM again
INFO: 1% (97.9 MiB of 8.0 GiB) in 3s, read: 32.6 MiB/s, write: 28.2 MiB/s
INFO: 2% (181.2 MiB of 8.0 GiB) in 6s, read: 27.8 MiB/s, write: 22.4 MiB/s
INFO: 3% (257.4 MiB of 8.0 GiB) in 9s, read: 25.4 MiB/s, write: 21.5 MiB/s
INFO: 5% (416.9 MiB of 8.0 GiB) in 12s, read: 53.2 MiB/s, write: 31.6 MiB/s
INFO: 6% (514.8 MiB of 8.0 GiB) in 15s, read: 32.6 MiB/s, write: 23.7 MiB/s
INFO: 7% (590.9 MiB of 8.0 GiB) in 18s, read: 25.4 MiB/s, write: 21.8 MiB/s
INFO: 8% (674.2 MiB of 8.0 GiB) in 21s, read: 27.8 MiB/s, write: 22.6 MiB/s
INFO: 9% (746.8 MiB of 8.0 GiB) in 24s, read: 24.2 MiB/s, write: 21.8 MiB/s
INFO: 10% (822.9 MiB of 8.0 GiB) in 27s, read: 25.4 MiB/s, write: 22.3 MiB/s
INFO: 11% (902.6 MiB of 8.0 GiB) in 30s, read: 26.6 MiB/s, write: 21.8 MiB/s
INFO: 12% (1007.8 MiB of 8.0 GiB) in 34s, read: 26.3 MiB/s, write: 21.8 MiB/s
INFO: 14% (1.2 GiB of 8.0 GiB) in 37s, read: 71.3 MiB/s, write: 25.4 MiB/s
INFO: 15% (1.3 GiB of 8.0 GiB) in 40s, read: 21.8 MiB/s, write: 16.4 MiB/s
INFO: 16% (1.3 GiB of 8.0 GiB) in 43s, read: 25.4 MiB/s, write: 24.3 MiB/s
INFO: 17% (1.4 GiB of 8.0 GiB) in 46s, read: 27.8 MiB/s, write: 21.9 MiB/s
INFO: 19% (1.6 GiB of 8.0 GiB) in 49s, read: 50.8 MiB/s, write: 21.6 MiB/s
INFO: 20% (1.6 GiB of 8.0 GiB) in 52s, read: 21.8 MiB/s, write: 19.7 MiB/s
INFO: 21% (1.7 GiB of 8.0 GiB) in 55s, read: 21.8 MiB/s, write: 20.7 MiB/s
INFO: 22% (1.8 GiB of 8.0 GiB) in 59s, read: 19.9 MiB/s, write: 18.6 MiB/s
INFO: 23% (1.8 GiB of 8.0 GiB) in 1m 3s, read: 20.8 MiB/s, write: 19.4 MiB/s
INFO: 24% (1.9 GiB of 8.0 GiB) in 1m 7s, read: 20.8 MiB/s, write: 20.7 MiB/s
INFO: 25% (2.0 GiB of 8.0 GiB) in 1m 10s, read: 41.1 MiB/s, write: 26.4 MiB/s
INFO: 28% (2.2 GiB of 8.0 GiB) in 1m 13s, read: 65.2 MiB/s, write: 39.5 MiB/s
INFO: 29% (2.3 GiB of 8.0 GiB) in 1m 17s, read: 25.4 MiB/s, write: 24.6 MiB/s
INFO: 30% (2.4 GiB of 8.0 GiB) in 1m 20s, read: 25.4 MiB/s, write: 24.8 MiB/s
INFO: 31% (2.5 GiB of 8.0 GiB) in 1m 23s, read: 23.0 MiB/s, write: 20.3 MiB/s
INFO: 32% (2.6 GiB of 8.0 GiB) in 1m 28s, read: 20.3 MiB/s, write: 19.7 MiB/s
INFO: 33% (2.7 GiB of 8.0 GiB) in 1m 32s, read: 19.9 MiB/s, write: 19.1 MiB/s
INFO: 34% (2.7 GiB of 8.0 GiB) in 1m 36s, read: 19.9 MiB/s, write: 19.9 MiB/s
INFO: 35% (2.8 GiB of 8.0 GiB) in 1m 39s, read: 23.0 MiB/s, write: 19.4 MiB/s
INFO: 36% (2.9 GiB of 8.0 GiB) in 1m 42s, read: 26.6 MiB/s, write: 22.1 MiB/s
INFO: 37% (3.0 GiB of 8.0 GiB) in 1m 46s, read: 25.4 MiB/s, write: 20.4 MiB/s
INFO: 38% (3.1 GiB of 8.0 GiB) in 1m 49s, read: 25.4 MiB/s, write: 20.6 MiB/s
INFO: 39% (3.1 GiB of 8.0 GiB) in 1m 52s, read: 27.8 MiB/s, write: 22.8 MiB/s
INFO: 48% (3.9 GiB of 8.0 GiB) in 1m 56s, read: 190.3 MiB/s, write: 19.4 MiB/s
INFO: 100% (8.0 GiB of 8.0 GiB) in 1m 58s, read: 2.1 GiB/s, write: 6.0 MiB/s
INFO: backup is sparse: 5.46 GiB (68%) total zero data
INFO: transferred 8.00 GiB in 118 seconds (69.4 MiB/s)
INFO: archive file size: 1.01GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=3, keep-weekly=2
INFO: removing backup 'proxmox-backups:backup/vzdump-qemu-101-2023_10_21-04_56_16.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 101 (00:02:28)
INFO: Backup finished at 2023-11-09 02:38:39
Code:
INFO: Starting Backup of VM 105 (qemu)
INFO: Backup started at 2023-11-09 02:38:39
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: windows-qb-dan
INFO: include disk 'sata1' 'local-btrfs:105/vm-105-disk-6.raw' 50G
INFO: creating vzdump archive '/mnt/pve/proxmox-backups/dump/vzdump-qemu-105-2023_11_09-02_38_39.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task '34afc4bb-26ab-4c3f-a2af-ddc2041a8b1a'
INFO: 0% (406.0 MiB of 50.0 GiB) in 3s, read: 135.3 MiB/s, write: 25.8 MiB/s
INFO: 1% (529.2 MiB of 50.0 GiB) in 7s, read: 30.8 MiB/s, write: 29.0 MiB/s
INFO: 2% (1.0 GiB of 50.0 GiB) in 25s, read: 27.6 MiB/s, write: 26.6 MiB/s
INFO: 3% (1.5 GiB of 50.0 GiB) in 47s, read: 24.1 MiB/s, write: 23.5 MiB/s
INFO: 4% (2.0 GiB of 50.0 GiB) in 1m 7s, read: 25.0 MiB/s, write: 24.7 MiB/s
INFO: 5% (2.5 GiB of 50.0 GiB) in 1m 29s, read: 24.1 MiB/s, write: 23.6 MiB/s
INFO: 6% (3.0 GiB of 50.0 GiB) in 1m 50s, read: 23.5 MiB/s, write: 23.0 MiB/s
INFO: 7% (3.5 GiB of 50.0 GiB) in 2m 9s, read: 28.2 MiB/s, write: 27.1 MiB/s
INFO: 8% (4.0 GiB of 50.0 GiB) in 2m 29s, read: 24.5 MiB/s, write: 24.0 MiB/s
INFO: 9% (4.5 GiB of 50.0 GiB) in 2m 51s, read: 23.9 MiB/s, write: 23.7 MiB/s
INFO: 10% (5.0 GiB of 50.0 GiB) in 3m 12s, read: 24.0 MiB/s, write: 23.6 MiB/s
...
INFO: 90% (45.0 GiB of 50.0 GiB) in 1h 9m 13s, read: 16.9 MiB/s, write: 16.9 MiB/s
INFO: 91% (45.5 GiB of 50.0 GiB) in 1h 10m 40s, read: 6.0 MiB/s, write: 3.2 MiB/s
INFO: 92% (46.1 GiB of 50.0 GiB) in 1h 10m 50s, read: 55.5 MiB/s, write: 22.3 MiB/s
INFO: 93% (46.5 GiB of 50.0 GiB) in 1h 10m 56s, read: 77.3 MiB/s, write: 28.7 MiB/s
INFO: 94% (47.0 GiB of 50.0 GiB) in 1h 11m 24s, read: 18.1 MiB/s, write: 18.1 MiB/s
INFO: 95% (47.5 GiB of 50.0 GiB) in 1h 11m 46s, read: 23.6 MiB/s, write: 23.4 MiB/s
INFO: 96% (48.0 GiB of 50.0 GiB) in 1h 11m 55s, read: 56.0 MiB/s, write: 24.9 MiB/s
INFO: 97% (48.5 GiB of 50.0 GiB) in 1h 12m 42s, read: 11.1 MiB/s, write: 10.9 MiB/s
INFO: 98% (49.0 GiB of 50.0 GiB) in 1h 14m 9s, read: 5.8 MiB/s, write: 5.8 MiB/s
INFO: 99% (49.5 GiB of 50.0 GiB) in 1h 14m 17s, read: 65.7 MiB/s, write: 24.3 MiB/s
INFO: 100% (50.0 GiB of 50.0 GiB) in 1h 14m 34s, read: 28.8 MiB/s, write: 23.9 MiB/s
INFO: backup is sparse: 9.01 GiB (18%) total zero data
INFO: transferred 50.00 GiB in 4474 seconds (11.4 MiB/s)
gzip: Warning: unable to close filehandle GEN8707 properly: Host is down at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 844.
stdout: Host is down
INFO: stopping kvm after backup task
ERROR: Backup of VM 105 failed - gzip --rsyncable failed - wrong exit status 1
INFO: Failed at 2023-11-09 03:55:34
Code:
INFO: Starting Backup of VM 107 (qemu)
INFO: Backup started at 2023-11-09 03:55:34
INFO: status = running
INFO: VM Name: windows-11
INFO: include disk 'sata0' 'local-btrfs:107/vm-107-disk-3.raw' 121433M
INFO: include disk 'efidisk0' 'local-btrfs:107/vm-107-disk-2.raw' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/proxmox-backups/dump/vzdump-qemu-107-2023_11_09-03_55_34.vma.gz'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'cf2f696d-02c4-4489-a660-2d3f7966396f'
INFO: resuming VM again
INFO: 0% (90.6 MiB of 118.6 GiB) in 3s, read: 30.2 MiB/s, write: 27.5 MiB/s
INFO: 1% (1.2 GiB of 118.6 GiB) in 47s, read: 26.0 MiB/s, write: 23.1 MiB/s
INFO: 2% (2.4 GiB of 118.6 GiB) in 1m 44s, read: 21.2 MiB/s, write: 20.8 MiB/s
INFO: 3% (3.6 GiB of 118.6 GiB) in 2m 44s, read: 20.1 MiB/s, write: 20.0 MiB/s
INFO: 4% (4.8 GiB of 118.6 GiB) in 3m 38s, read: 22.6 MiB/s, write: 22.4 MiB/s
INFO: 5% (5.9 GiB of 118.6 GiB) in 4m 54s, read: 15.9 MiB/s, write: 14.0 MiB/s
INFO: 6% (7.2 GiB of 118.6 GiB) in 5m 18s, read: 55.0 MiB/s, write: 54.0 MiB/s
INFO: 7% (8.3 GiB of 118.6 GiB) in 5m 55s, read: 30.3 MiB/s, write: 29.2 MiB/s
INFO: 8% (9.5 GiB of 118.6 GiB) in 7m 3s, read: 17.8 MiB/s, write: 17.0 MiB/s
INFO: 9% (10.7 GiB of 118.6 GiB) in 9m 5s, read: 9.9 MiB/s, write: 9.3 MiB/s
INFO: 10% (11.9 GiB of 118.6 GiB) in 9m 54s, read: 24.7 MiB/s, write: 23.6 MiB/s
...
INFO: 90% (107.5 GiB of 118.6 GiB) in 1h 46m 54s, read: 61.0 MiB/s, write: 23.2 MiB/s
INFO: 91% (108.3 GiB of 118.6 GiB) in 1h 46m 57s, read: 269.5 MiB/s, write: 23.4 MiB/s
INFO: 92% (109.1 GiB of 118.6 GiB) in 1h 47m 35s, read: 21.4 MiB/s, write: 21.3 MiB/s
INFO: 93% (110.3 GiB of 118.6 GiB) in 1h 48m 25s, read: 24.4 MiB/s, write: 17.2 MiB/s
INFO: 94% (111.6 GiB of 118.6 GiB) in 1h 48m 31s, read: 222.3 MiB/s, write: 22.5 MiB/s
INFO: 95% (112.7 GiB of 118.6 GiB) in 1h 50m 8s, read: 11.1 MiB/s, write: 9.2 MiB/s
INFO: 96% (113.8 GiB of 118.6 GiB) in 1h 50m 55s, read: 25.8 MiB/s, write: 16.1 MiB/s
INFO: 97% (115.0 GiB of 118.6 GiB) in 1h 53m 31s, read: 7.8 MiB/s, write: 7.8 MiB/s
INFO: 98% (116.4 GiB of 118.6 GiB) in 1h 54m 20s, read: 28.5 MiB/s, write: 12.7 MiB/s
INFO: 99% (117.6 GiB of 118.6 GiB) in 1h 54m 58s, read: 33.7 MiB/s, write: 18.4 MiB/s
INFO: 100% (118.6 GiB of 118.6 GiB) in 1h 55m 24s, read: 36.9 MiB/s, write: 17.2 MiB/s
INFO: backup is sparse: 22.64 GiB (19%) total zero data
INFO: transferred 118.59 GiB in 6924 seconds (17.5 MiB/s)
gzip: stdout: Host is down
ERROR: Backup of VM 107 failed - gzip --rsyncable failed - wrong exit status 1
INFO: Failed at 2023-11-09 05:53:38
INFO: Backup job finished with errors
TASK ERROR: job errors