Hello, I am very new to Proxmox in general and am working to understand an issue we have run into for my company.
I have a server we recently installed with Proxmox 8 (attempted a lower version, but hardware didn't support it). We set up a Backup configuration by attaching SMB/CIFS storage to a VM and setting up an automatic schedule. It ran successfully the first couple of times with the following logs:
Then, seemingly out of the blue, the next time it runs it appears to stall when attempting to resume VM again:
During this time the VM becomes unavailable and the database I am running on it cannot be accessed as needed. I've tried to find others with a similar issue, but haven't found much related to it. I have unfortunately had to power cycle the system to get the VM up and running again, but after the second time I adjusted the backup automation to "--mode stop" which then just errors out completely (code below) and I no longer have to hard reboot.
I appreciate any help that can be offered and am not sure where to start in troubleshooting and fixing this issue.
I have a server we recently installed with Proxmox 8 (attempted a lower version, but hardware didn't support it). We set up a Backup configuration by attaching SMB/CIFS storage to a VM and setting up an automatic schedule. It ran successfully the first couple of times with the following logs:
Code:
INFO: starting new backup job: vzdump 101 --quiet 1 --node goodearthmarkets --mode snapshot --compress zstd --notes-template '{{vmid}}{{guestname}}' --prune-backups 'keep-last=6' --storage Backups --mailnotification always --mailto ##############
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-09-10 01:00:02
INFO: status = running
INFO: VM Name: ########
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-0' 400G
INFO: include disk 'scsi1' 'ServerData:vm-101-disk-0' 800G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/Backups/dump/vzdump-qemu-101-2023_09_10-01_00_02.vma.zst'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task 'cefc159f-9c8a-42ef-8a88-c1caf468b325'
INFO: resuming VM again
INFO: 0% (512.4 MiB of 1.2 TiB) in 3s, read: 170.8 MiB/s, write: 160.0 MiB/s
INFO: 1% (12.0 GiB of 1.2 TiB) in 1m 33s, read: 131.3 MiB/s, write: 74.5 MiB/s
INFO: 2% (24.0 GiB of 1.2 TiB) in 3m 24s, read: 110.4 MiB/s, write: 58.9 MiB/s
INFO: 3% (36.1 GiB of 1.2 TiB) in 5m 3s, read: 125.1 MiB/s, write: 64.5 MiB/s
|...
INFO: 99% (1.2 TiB of 1.2 TiB) in 18m 30s, read: 6.6 GiB/s, write: 0 B/s
INFO: 100% (1.2 TiB of 1.2 TiB) in 18m 32s, read: 298.8 MiB/s, write: 213.8 MiB/s
INFO: backup is sparse: 1.10 TiB (93%) total zero data
INFO: transferred 1.17 TiB in 1112 seconds (1.1 GiB/s)
INFO: archive file size: 60.30GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=6
INFO: pruned 0 backup(s)
INFO: Finished Backup of VM 101 (00:18:41)
INFO: Backup finished at 2023-09-10 01:18:43
INFO: Backup job finished successfully
TASK OKThen, seemingly out of the blue, the next time it runs it appears to stall when attempting to resume VM again:
Code:
INFO: starting new backup job: vzdump 101 --mailnotification always --mailto ############### --prune-backups 'keep-last=6' --storage Backups --notes-template '{{vmid}}{{guestname}}' --mode snapshot --quiet 1 --node goodearthmarkets --compress zstd
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-09-17 01:00:02
INFO: status = running
INFO: VM Name: ########
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-0' 400G
INFO: include disk 'scsi1' 'ServerData:vm-101-disk-0' 800G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/Backups/dump/vzdump-qemu-101-2023_09_17-01_00_02.vma.zst'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '51c93560-a72f-4a0f-87f2-d58d5d87ea39'
INFO: resuming VM again
INFO: 0% (635.6 MiB of 1.2 TiB) in 3s, read: 211.9 MiB/s, write: 201.1 MiB/sDuring this time the VM becomes unavailable and the database I am running on it cannot be accessed as needed. I've tried to find others with a similar issue, but haven't found much related to it. I have unfortunately had to power cycle the system to get the VM up and running again, but after the second time I adjusted the backup automation to "--mode stop" which then just errors out completely (code below) and I no longer have to hard reboot.
Code:
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-10-22 01:00:06
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ########
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-0' 400G
INFO: include disk 'scsi1' 'ServerData:vm-101-disk-0' 800G
INFO: stopping virtual guest
INFO: VM quit/powerdown failed
ERROR: Backup of VM 101 failed - command 'qm shutdown 101 --skiplock --keepActive --timeout 600' failed: exit code 255
INFO: Failed at 2023-10-22 01:10:06
INFO: Backup job finished with errors
TASK ERROR: job errorsI appreciate any help that can be offered and am not sure where to start in troubleshooting and fixing this issue.