ERROR: job failed with err -16 - Device or resource busy. Promox backup fails randomly every other week

[Geek]

For some reason a VM is not being backed up at random times. It fails towards the end. I have the latest proxmox (7.0-10).

Following is the status log:

()
INFO: starting new backup job: vzdump 100 --storage vm_backup --mailnotification always --mode snapshot --compress zstd --quiet 1
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-09-10 21:00:01
INFO: status = running
INFO: VM Name: Win-10-Pro
INFO: include disk 'scsi0' 'new_vm_disk:100/vm-100-disk-0.raw' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/vm_backup/dump/vzdump-qemu-100-2021_09_10-21_00_01.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '30c8ad81-cf60-4616-a048-5093a2396088'
INFO: resuming VM again
INFO:   0% (184.9 MiB of 120.0 GiB) in 3s, read: 61.6 MiB/s, write: 42.1 MiB/s
INFO:   1% (1.2 GiB of 120.0 GiB) in 25s, read: 48.8 MiB/s, write: 45.0 MiB/s
INFO:   2% (2.4 GiB of 120.0 GiB) in 57s, read: 37.9 MiB/s, write: 37.3 MiB/s
INFO:   3% (3.6 GiB of 120.0 GiB) in 1m 30s, read: 37.0 MiB/s, write: 36.5 MiB/s
INFO:   4% (4.8 GiB of 120.0 GiB) in 2m, read: 41.1 MiB/s, write: 40.5 MiB/s
INFO:   5% (6.0 GiB of 120.0 GiB) in 2m 29s, read: 43.1 MiB/s, write: 42.7 MiB/s
INFO:   6% (7.2 GiB of 120.0 GiB) in 2m 59s, read: 41.1 MiB/s, write: 40.7 MiB/s
INFO:   7% (8.4 GiB of 120.0 GiB) in 3m 28s, read: 41.2 MiB/s, write: 41.1 MiB/s
INFO:   8% (9.6 GiB of 120.0 GiB) in 3m 59s, read: 39.6 MiB/s, write: 39.5 MiB/s
INFO:   9% (10.8 GiB of 120.0 GiB) in 4m 27s, read: 44.0 MiB/s, write: 39.7 MiB/s
INFO:  10% (12.0 GiB of 120.0 GiB) in 4m 57s, read: 42.3 MiB/s, write: 42.1 MiB/s
INFO:  11% (13.2 GiB of 120.0 GiB) in 5m 25s, read: 42.7 MiB/s, write: 42.1 MiB/s
INFO:  12% (14.4 GiB of 120.0 GiB) in 5m 58s, read: 36.8 MiB/s, write: 36.8 MiB/s
INFO:  13% (15.8 GiB of 120.0 GiB) in 6m 28s, read: 47.4 MiB/s, write: 5.9 MiB/s
INFO:  14% (17.0 GiB of 120.0 GiB) in 6m 55s, read: 47.3 MiB/s, write: 2.4 KiB/s
INFO:  15% (18.1 GiB of 120.0 GiB) in 7m 19s, read: 47.4 MiB/s, write: 8.0 KiB/s
INFO:  16% (19.3 GiB of 120.0 GiB) in 7m 39s, read: 57.6 MiB/s, write: 25.6 KiB/s
INFO:  17% (20.4 GiB of 120.0 GiB) in 8m 2s, read: 50.6 MiB/s, write: 50.3 MiB/s
INFO:  18% (21.6 GiB of 120.0 GiB) in 8m 32s, read: 41.4 MiB/s, write: 41.4 MiB/s
INFO:  19% (22.8 GiB of 120.0 GiB) in 9m 3s, read: 40.5 MiB/s, write: 40.4 MiB/s
INFO:  20% (24.0 GiB of 120.0 GiB) in 9m 32s, read: 41.8 MiB/s, write: 41.6 MiB/s
INFO:  21% (25.2 GiB of 120.0 GiB) in 10m 1s, read: 42.2 MiB/s, write: 42.1 MiB/s
INFO:  22% (26.4 GiB of 120.0 GiB) in 10m 32s, read: 40.0 MiB/s, write: 40.0 MiB/s
INFO:  23% (27.7 GiB of 120.0 GiB) in 10m 53s, read: 59.9 MiB/s, write: 59.9 MiB/s
INFO:  24% (28.8 GiB of 120.0 GiB) in 11m 18s, read: 47.8 MiB/s, write: 47.8 MiB/s
INFO:  25% (30.0 GiB of 120.0 GiB) in 11m 47s, read: 42.5 MiB/s, write: 42.5 MiB/s
INFO:  26% (31.2 GiB of 120.0 GiB) in 12m 16s, read: 41.9 MiB/s, write: 41.9 MiB/s
INFO:  27% (32.5 GiB of 120.0 GiB) in 12m 38s, read: 57.2 MiB/s, write: 57.2 MiB/s
INFO:  28% (33.6 GiB of 120.0 GiB) in 13m 1s, read: 51.4 MiB/s, write: 51.4 MiB/s
INFO:  29% (34.8 GiB of 120.0 GiB) in 13m 30s, read: 42.9 MiB/s, write: 42.8 MiB/s
INFO:  30% (36.0 GiB of 120.0 GiB) in 13m 56s, read: 47.4 MiB/s, write: 47.4 MiB/s
INFO:  31% (37.2 GiB of 120.0 GiB) in 14m 17s, read: 58.0 MiB/s, write: 57.9 MiB/s
INFO:  32% (38.4 GiB of 120.0 GiB) in 14m 46s, read: 42.5 MiB/s, write: 42.5 MiB/s
INFO:  33% (39.6 GiB of 120.0 GiB) in 15m 14s, read: 44.7 MiB/s, write: 44.7 MiB/s
INFO:  34% (40.8 GiB of 120.0 GiB) in 15m 35s, read: 57.5 MiB/s, write: 57.5 MiB/s
INFO:  35% (42.0 GiB of 120.0 GiB) in 16m, read: 49.3 MiB/s, write: 49.3 MiB/s
INFO:  36% (43.2 GiB of 120.0 GiB) in 16m 22s, read: 56.0 MiB/s, write: 56.0 MiB/s
INFO:  37% (44.4 GiB of 120.0 GiB) in 16m 45s, read: 53.6 MiB/s, write: 53.6 MiB/s
INFO:  38% (45.6 GiB of 120.0 GiB) in 17m 5s, read: 62.5 MiB/s, write: 62.5 MiB/s
INFO:  39% (46.8 GiB of 120.0 GiB) in 17m 30s, read: 48.6 MiB/s, write: 48.5 MiB/s
INFO:  40% (48.0 GiB of 120.0 GiB) in 17m 49s, read: 63.9 MiB/s, write: 63.9 MiB/s
INFO:  41% (49.3 GiB of 120.0 GiB) in 18m 14s, read: 50.8 MiB/s, write: 50.7 MiB/s
INFO:  42% (50.5 GiB of 120.0 GiB) in 18m 35s, read: 58.5 MiB/s, write: 58.5 MiB/s
INFO:  43% (51.7 GiB of 120.0 GiB) in 19m 3s, read: 44.0 MiB/s, write: 43.9 MiB/s
INFO:  44% (52.8 GiB of 120.0 GiB) in 19m 27s, read: 49.7 MiB/s, write: 49.4 MiB/s
INFO:  45% (54.0 GiB of 120.0 GiB) in 19m 59s, read: 38.5 MiB/s, write: 37.8 MiB/s
INFO:  46% (55.2 GiB of 120.0 GiB) in 20m 21s, read: 55.9 MiB/s, write: 55.0 MiB/s
INFO:  47% (56.4 GiB of 120.0 GiB) in 20m 53s, read: 38.1 MiB/s, write: 38.1 MiB/s
INFO:  48% (57.7 GiB of 120.0 GiB) in 21m 22s, read: 43.8 MiB/s, write: 43.7 MiB/s
INFO:  49% (58.8 GiB of 120.0 GiB) in 21m 43s, read: 56.4 MiB/s, write: 56.4 MiB/s
INFO:  50% (60.0 GiB of 120.0 GiB) in 22m 14s, read: 40.3 MiB/s, write: 40.3 MiB/s
INFO:  51% (61.2 GiB of 120.0 GiB) in 22m 45s, read: 39.3 MiB/s, write: 37.1 MiB/s
INFO:  52% (62.4 GiB of 120.0 GiB) in 23m 9s, read: 51.7 MiB/s, write: 51.6 MiB/s
INFO:  53% (63.6 GiB of 120.0 GiB) in 23m 35s, read: 46.1 MiB/s, write: 46.1 MiB/s
INFO:  54% (64.8 GiB of 120.0 GiB) in 24m 15s, read: 31.3 MiB/s, write: 31.3 MiB/s
INFO:  55% (66.1 GiB of 120.0 GiB) in 24m 47s, read: 39.0 MiB/s, write: 39.0 MiB/s
INFO:  56% (67.2 GiB of 120.0 GiB) in 25m 12s, read: 47.7 MiB/s, write: 47.7 MiB/s
INFO:  57% (68.4 GiB of 120.0 GiB) in 25m 49s, read: 33.4 MiB/s, write: 33.4 MiB/s
INFO:  58% (69.6 GiB of 120.0 GiB) in 26m 11s, read: 55.7 MiB/s, write: 55.7 MiB/s
INFO:  59% (70.8 GiB of 120.0 GiB) in 26m 44s, read: 38.1 MiB/s, write: 38.1 MiB/s
INFO:  60% (72.0 GiB of 120.0 GiB) in 27m 8s, read: 50.6 MiB/s, write: 50.6 MiB/s
INFO:  61% (73.2 GiB of 120.0 GiB) in 27m 39s, read: 39.2 MiB/s, write: 39.2 MiB/s
INFO:  62% (74.4 GiB of 120.0 GiB) in 27m 59s, read: 61.6 MiB/s, write: 61.6 MiB/s
INFO:  63% (75.6 GiB of 120.0 GiB) in 28m 27s, read: 43.5 MiB/s, write: 43.4 MiB/s
INFO:  64% (76.9 GiB of 120.0 GiB) in 28m 57s, read: 42.3 MiB/s, write: 42.1 MiB/s
INFO:  65% (78.1 GiB of 120.0 GiB) in 29m 20s, read: 53.7 MiB/s, write: 53.5 MiB/s
INFO:  66% (79.2 GiB of 120.0 GiB) in 29m 55s, read: 34.6 MiB/s, write: 34.6 MiB/s
INFO:  67% (80.4 GiB of 120.0 GiB) in 30m 22s, read: 44.2 MiB/s, write: 44.2 MiB/s
INFO:  68% (81.6 GiB of 120.0 GiB) in 31m 1s, read: 31.8 MiB/s, write: 31.8 MiB/s
INFO:  69% (82.8 GiB of 120.0 GiB) in 31m 26s, read: 48.7 MiB/s, write: 48.7 MiB/s
INFO:  70% (84.0 GiB of 120.0 GiB) in 31m 57s, read: 40.9 MiB/s, write: 40.9 MiB/s
INFO:  71% (85.2 GiB of 120.0 GiB) in 32m 18s, read: 58.5 MiB/s, write: 58.5 MiB/s
INFO:  72% (86.4 GiB of 120.0 GiB) in 32m 51s, read: 35.9 MiB/s, write: 35.9 MiB/s
INFO:  73% (87.6 GiB of 120.0 GiB) in 33m 13s, read: 57.0 MiB/s, write: 57.0 MiB/s
INFO:  74% (88.8 GiB of 120.0 GiB) in 33m 37s, read: 51.8 MiB/s, write: 51.8 MiB/s
INFO:  75% (90.0 GiB of 120.0 GiB) in 33m 58s, read: 58.7 MiB/s, write: 58.7 MiB/s
INFO:  76% (91.2 GiB of 120.0 GiB) in 34m 22s, read: 50.4 MiB/s, write: 50.4 MiB/s
INFO:  77% (92.5 GiB of 120.0 GiB) in 34m 47s, read: 50.8 MiB/s, write: 50.8 MiB/s
INFO:  78% (93.7 GiB of 120.0 GiB) in 35m 9s, read: 55.9 MiB/s, write: 55.9 MiB/s
INFO:  79% (94.9 GiB of 120.0 GiB) in 35m 34s, read: 49.3 MiB/s, write: 49.3 MiB/s
INFO:  80% (96.0 GiB of 120.0 GiB) in 35m 58s, read: 49.1 MiB/s, write: 49.1 MiB/s
INFO:  81% (97.2 GiB of 120.0 GiB) in 36m 23s, read: 48.6 MiB/s, write: 48.6 MiB/s
INFO:  82% (98.4 GiB of 120.0 GiB) in 36m 43s, read: 63.4 MiB/s, write: 28.7 MiB/s
INFO:  83% (99.6 GiB of 120.0 GiB) in 37m 6s, read: 51.9 MiB/s, write: 51.7 MiB/s
INFO:  84% (100.8 GiB of 120.0 GiB) in 37m 38s, read: 39.1 MiB/s, write: 39.1 MiB/s
INFO:  84% (101.0 GiB of 120.0 GiB) in 37m 46s, read: 24.8 MiB/s, write: 24.8 MiB/s
ERROR: job failed with err -16 - Device or resource busy
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - job failed with err -16 - Device or resource busy
INFO: Failed at 2021-09-10 21:37:59
INFO: Backup job finished with errors

TASK ERROR: job errors

All other backup failures have either occurred at 84%, 13% or 15% with the exact same error "ERROR: job failed with err -16 - Device or resource busy"
It is a windows 10 VM