I have created a PVE backup schedule for our production VMs to a network Windows based CIFS target. The backup schedule is in Stop Mode with ZSTD compression. Retention on the CIFS storage is 3 copies. All of our VMs back up successfully except one, VM 106. If I delete all of the VM 106 backup files on the CIFS storage I can get a successful backup. However, if I let the job continue to run the next day(s) I receive the following errors in the log. I have one backup job that backs all of the VMs up on the same schedule after hours. I have successful 3 copy backups of each of them without issue, except VM 106.
I have Googled the errors without finding a solution. I was hoping that someone could point me in the right direction as to why this VM 106 is giving me trouble. I have ample space on the CIFS target and on the DAS ceph where the VM(s) live. The cluster is on version 6.3-6.
Thank you for any information you can offer.
INFO: Starting Backup of VM 106 (qemu)
INFO: Backup started at 2021-04-13 01:12:34
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: TCC-FS
INFO: include disk 'virtio0' 'VMOS:vm-106-disk-0' 100G
INFO: include disk 'virtio1' 'VMOS:vm-106-disk-1' 250G
INFO: stopping vm
INFO: creating vzdump archive '/mnt/pve/CIFS/dump/vzdump-qemu-106-2021_04_13-01_12_34.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task 'b85d97d4-9101-4ea8-8687-fbbcc437e5f4'
INFO: resuming VM again after 20 seconds
INFO: 0% (435.0 MiB of 350.0 GiB) in 3s, read: 145.0 MiB/s, write: 130.7 MiB/s
INFO: 1% (3.6 GiB of 350.0 GiB) in 29s, read: 125.4 MiB/s, write: 120.5 MiB/s
INFO: 2% (7.0 GiB of 350.0 GiB) in 57s, read: 125.7 MiB/s, write: 125.3 MiB/s
INFO: 3% (10.5 GiB of 350.0 GiB) in 1m 26s, read: 122.5 MiB/s, write: 122.3 MiB/s
INFO: 4% (14.1 GiB of 350.0 GiB) in 1m 55s, read: 126.0 MiB/s, write: 125.8 MiB/s
INFO: 5% (17.6 GiB of 350.0 GiB) in 2m 23s, read: 128.7 MiB/s, write: 128.6 MiB/s
INFO: 6% (21.0 GiB of 350.0 GiB) in 2m 51s, read: 126.0 MiB/s, write: 125.8 MiB/s
INFO: 7% (24.6 GiB of 350.0 GiB) in 3m 22s, read: 117.4 MiB/s, write: 117.4 MiB/s
INFO: 8% (28.0 GiB of 350.0 GiB) in 3m 52s, read: 116.2 MiB/s, write: 116.2 MiB/s
INFO: 9% (31.6 GiB of 350.0 GiB) in 4m 24s, read: 114.4 MiB/s, write: 114.4 MiB/s
INFO: 10% (35.0 GiB of 350.0 GiB) in 4m 56s, read: 110.9 MiB/s, write: 110.9 MiB/s
...
INFO: 90% (316.1 GiB of 350.0 GiB) in 1h 20m 57s, read: 1.2 GiB/s, write: 0 B/s
INFO: 91% (319.7 GiB of 350.0 GiB) in 1h 21m, read: 1.2 GiB/s, write: 0 B/s
INFO: 92% (322.8 GiB of 350.0 GiB) in 1h 21m 3s, read: 1.1 GiB/s, write: 0 B/s
INFO: 93% (326.0 GiB of 350.0 GiB) in 1h 21m 6s, read: 1.1 GiB/s, write: 0 B/s
INFO: 94% (329.5 GiB of 350.0 GiB) in 1h 21m 9s, read: 1.2 GiB/s, write: 0 B/s
INFO: 95% (332.9 GiB of 350.0 GiB) in 1h 21m 12s, read: 1.1 GiB/s, write: 0 B/s
INFO: 96% (336.3 GiB of 350.0 GiB) in 1h 21m 15s, read: 1.1 GiB/s, write: 0 B/s
INFO: 97% (340.0 GiB of 350.0 GiB) in 1h 21m 18s, read: 1.2 GiB/s, write: 0 B/s
INFO: 98% (343.8 GiB of 350.0 GiB) in 1h 21m 22s, read: 982.0 MiB/s, write: 0 B/s
INFO: 99% (346.7 GiB of 350.0 GiB) in 1h 21m 25s, read: 996.0 MiB/s, write: 0 B/s
zstd: /*stdout*\: Input/output error
INFO: 100% (350.0 GiB of 350.0 GiB) in 1h 21m 29s, read: 833.0 MiB/s, write: 1.0 KiB/s
INFO: backup is sparse: 81.83 GiB (23%) total zero data
INFO: transferred 350.00 GiB in 4889 seconds (73.3 MiB/s)
ERROR: Backup of VM 106 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1
INFO: Failed at 2021-04-13 02:34:37
I have Googled the errors without finding a solution. I was hoping that someone could point me in the right direction as to why this VM 106 is giving me trouble. I have ample space on the CIFS target and on the DAS ceph where the VM(s) live. The cluster is on version 6.3-6.
Thank you for any information you can offer.
Last edited:
