VM turn off during backup

[jbukowskipl]

Thats my first time problem like bellow.. I have no idea how to debug this problem.

Guest:
Server Essentials 2019, qemu-agent, common cpu (host cpu causes error on boot)

VE:
CPU(s) 24 x Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (2 Sockets)
Kernel Version Linux 5.13.19-2-pve #1 SMP PVE 5.13.19-4 (Mon, 29 Nov 2021 12:10:09 +0100)
PVE Manager Version pve-manager/7.1-7/df5740ad

INFO: starting new backup job: vzdump 101 --mailto jacek@gmail.com --storage Synology --mailnotification failure --compress zstd --mode snapshot --quiet 1 --prune-backups 'keep-daily=7,keep-weekly=2'
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2022-03-22 21:30:13
INFO: status = running
INFO: VM Name: DTX
INFO: include disk 'virtio0' 'local-lvm:vm-101-disk-0' 1025G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/Synology/dump/vzdump-qemu-101-2022_03_22-21_30_13.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '6b8aabc3-43bc-4f8c-864e-2192224c1674'
INFO: resuming VM again
INFO: 0% (416.9 MiB of 1.0 TiB) in 3s, read: 139.0 MiB/s, write: 98.2 MiB/s
INFO: 1% (10.3 GiB of 1.0 TiB) in 2m 46s, read: 62.3 MiB/s, write: 60.9 MiB/s
INFO: 2% (20.6 GiB of 1.0 TiB) in 5m 12s, read: 71.9 MiB/s, write: 60.6 MiB/s
INFO: 3% (30.8 GiB of 1.0 TiB) in 7m 30s, read: 75.8 MiB/s, write: 74.9 MiB/s
INFO: 4% (41.0 GiB of 1.0 TiB) in 11m 4s, read: 48.8 MiB/s, write: 48.8 MiB/s
INFO: 5% (52.1 GiB of 1.0 TiB) in 14m 4s, read: 63.0 MiB/s, write: 42.7 MiB/s
INFO: 6% (61.9 GiB of 1.0 TiB) in 14m 12s, read: 1.2 GiB/s, write: 0 B/s
INFO: 7% (72.6 GiB of 1.0 TiB) in 14m 21s, read: 1.2 GiB/s, write: 0 B/s
INFO: 8% (82.7 GiB of 1.0 TiB) in 14m 29s, read: 1.3 GiB/s, write: 0 B/s
INFO: 9% (93.5 GiB of 1.0 TiB) in 14m 37s, read: 1.3 GiB/s, write: 0 B/s
INFO: 10% (102.5 GiB of 1.0 TiB) in 14m 44s, read: 1.3 GiB/s, write: 0 B/s
INFO: 11% (113.5 GiB of 1.0 TiB) in 14m 52s, read: 1.4 GiB/s, write: 0 B/s
INFO: 12% (123.3 GiB of 1.0 TiB) in 14m 58s, read: 1.6 GiB/s, write: 0 B/s
INFO: 13% (134.4 GiB of 1.0 TiB) in 15m 5s, read: 1.6 GiB/s, write: 0 B/s
INFO: 14% (144.0 GiB of 1.0 TiB) in 15m 12s, read: 1.4 GiB/s, write: 0 B/s
INFO: 15% (155.2 GiB of 1.0 TiB) in 15m 19s, read: 1.6 GiB/s, write: 0 B/s
INFO: 16% (165.2 GiB of 1.0 TiB) in 15m 26s, read: 1.4 GiB/s, write: 0 B/s
INFO: 17% (175.1 GiB of 1.0 TiB) in 15m 34s, read: 1.2 GiB/s, write: 0 B/s
INFO: 18% (185.2 GiB of 1.0 TiB) in 15m 42s, read: 1.3 GiB/s, write: 0 B/s
INFO: 19% (194.8 GiB of 1.0 TiB) in 15m 49s, read: 1.4 GiB/s, write: 0 B/s
INFO: 20% (206.3 GiB of 1.0 TiB) in 15m 59s, read: 1.1 GiB/s, write: 0 B/s
INFO: 21% (215.5 GiB of 1.0 TiB) in 16m 6s, read: 1.3 GiB/s, write: 0 B/s
INFO: 22% (226.7 GiB of 1.0 TiB) in 16m 15s, read: 1.2 GiB/s, write: 0 B/s
INFO: 23% (236.7 GiB of 1.0 TiB) in 16m 23s, read: 1.3 GiB/s, write: 0 B/s
INFO: 24% (246.6 GiB of 1.0 TiB) in 16m 31s, read: 1.2 GiB/s, write: 0 B/s
INFO: 25% (256.6 GiB of 1.0 TiB) in 16m 39s, read: 1.3 GiB/s, write: 0 B/s
INFO: 26% (266.7 GiB of 1.0 TiB) in 16m 48s, read: 1.1 GiB/s, write: 0 B/s
INFO: 27% (277.9 GiB of 1.0 TiB) in 16m 57s, read: 1.2 GiB/s, write: 0 B/s
INFO: 28% (287.2 GiB of 1.0 TiB) in 17m 3s, read: 1.5 GiB/s, write: 0 B/s
INFO: 29% (298.2 GiB of 1.0 TiB) in 17m 10s, read: 1.6 GiB/s, write: 0 B/s
INFO: 30% (307.8 GiB of 1.0 TiB) in 17m 16s, read: 1.6 GiB/s, write: 0 B/s
INFO: 31% (318.6 GiB of 1.0 TiB) in 17m 23s, read: 1.5 GiB/s, write: 0 B/s
INFO: 32% (329.3 GiB of 1.0 TiB) in 17m 30s, read: 1.5 GiB/s, write: 0 B/s
INFO: 33% (338.4 GiB of 1.0 TiB) in 17m 36s, read: 1.5 GiB/s, write: 0 B/s
INFO: 34% (349.0 GiB of 1.0 TiB) in 17m 43s, read: 1.5 GiB/s, write: 0 B/s
INFO: 35% (359.7 GiB of 1.0 TiB) in 17m 51s, read: 1.3 GiB/s, write: 0 B/s
INFO: 36% (370.1 GiB of 1.0 TiB) in 17m 58s, read: 1.5 GiB/s, write: 0 B/s
INFO: 37% (380.4 GiB of 1.0 TiB) in 18m 5s, read: 1.5 GiB/s, write: 0 B/s
INFO: 38% (390.8 GiB of 1.0 TiB) in 18m 12s, read: 1.5 GiB/s, write: 0 B/s
INFO: 39% (399.8 GiB of 1.0 TiB) in 18m 18s, read: 1.5 GiB/s, write: 0 B/s
INFO: 40% (410.2 GiB of 1.0 TiB) in 18m 25s, read: 1.5 GiB/s, write: 0 B/s
INFO: 41% (420.5 GiB of 1.0 TiB) in 18m 32s, read: 1.5 GiB/s, write: 0 B/s
INFO: 42% (430.8 GiB of 1.0 TiB) in 18m 39s, read: 1.5 GiB/s, write: 0 B/s
INFO: 43% (441.2 GiB of 1.0 TiB) in 18m 47s, read: 1.3 GiB/s, write: 0 B/s
INFO: 44% (451.1 GiB of 1.0 TiB) in 18m 54s, read: 1.4 GiB/s, write: 0 B/s
INFO: 45% (462.4 GiB of 1.0 TiB) in 19m 2s, read: 1.4 GiB/s, write: 0 B/s
INFO: 46% (472.2 GiB of 1.0 TiB) in 19m 9s, read: 1.4 GiB/s, write: 0 B/s
INFO: 47% (483.1 GiB of 1.0 TiB) in 19m 16s, read: 1.6 GiB/s, write: 0 B/s
INFO: 48% (492.6 GiB of 1.0 TiB) in 19m 22s, read: 1.6 GiB/s, write: 0 B/s
ERROR: VM 101 not running
INFO: aborting backup job
ERROR: VM 101 not running
INFO: resuming VM again
ERROR: Backup of VM 101 failed - VM 101 not running
INFO: Failed at 2022-03-22 21:49:48
INFO: Backup job finished with errors
TASK ERROR: job errors

Windows logs:
IVssWriterCallback. hr = 0x80070005, Access is denied.

vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'QEMU Guest Agent VSS Provider'
Provider type: Software
Provider Id: {3629d4ed-ee09-4e0e-9a5c-6d8ba2872aef}
Version: 102.10.0

Provider name: 'Microsoft File Share Shadow Copy provider'
Provider type: Fileshare
Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

 

[dietmar]

Most likely a problem with the underlying storage. Do you use a qcow2 format? If so, please check if that file has errors ("qemu-img check")

 

[jbukowskipl]

no, i use local-lvm (format raw), bellow is raid 10 @ 4x SSD new one unboxed 1 month ago.

 

[dietmar]

Any hint is syslog?

 

[jbukowskipl]

In my syslog i have smartd error, but i read that is normal when smartd is enable on hw raid controler?

Mar 24 14:51:28 pve pvedaemon[2431703]: <root@pam> successful auth for user 'root@pam'
Mar 24 14:56:05 pve kernel: [798112.073757] megaraid_sas 0000:01:00.0: 239547 (701445361s/0x0001/FATAL) - VD bad block table on VD 02/2 is full; unable to log block 32e8054e8 (on PD 07(e0x20/s7) at 32e8054e8)
Mar 24 14:56:11 pve kernel: [798118.283396] megaraid_sas 0000:01:00.0: 239549 (701445368s/0x0001/FATAL) - VD bad block table on VD 02/2 is full; unable to log block 32e805568 (on PD 07(e0x20/s7) at 32e805568)
Mar 24 14:56:17 pve kernel: [798124.362182] megaraid_sas 0000:01:00.0: 239551 (701445374s/0x0001/FATAL) - VD bad block table on VD 02/2 is full; unable to log block 32e8055f0 (on PD 07(e0x20/s7) at 32e8055f0)