Snapshots on LMV-thin stopped working after failed backup job

[shalak]

I have a VM (id=100) that runs OpenMediaVault - NAS server with CIFS shares. For the past 117 days everything was working properly, the backups were done every night with no errors. I initiated manual backup of this VM - and it failed. Here's the event log:

INFO: starting new backup job: vzdump 100 --node zoltan --storage zoltan-backups --compress zstd --mode snapshot --remove 0
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-01-12 22:52:16
INFO: status = running
INFO: VM Name: yarpen
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 32G
INFO: exclude disk 'virtio5' '/dev/disk/by-id/scsi-3600508b1001c902b8b9549d0a8090372' (backup=no)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/zoltan-backups/dump/vzdump-qemu-100-2022_01_12-22_52_16.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
closing with write buffer at /usr/share/perl5/IO/Multiplex.pm line 928.
INFO: issuing guest-agent 'fs-thaw' command
ERROR: got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - got timeout
INFO: Failed at 2022-01-12 22:55:23
INFO: Backup job finished with errors
TASK ERROR: job errors

I think it's important to note that the VM most likely was under heavy usage from some SMB client.

The backup storage is mounted over CIFS and comes from this exact VM. So basically this VM is backed up to itself. I never had issues with such backups (they were using snapshot mode)

Now, I'm unable to create snapshots for this VM, it says: "The current guest configuration does not support taking new snapshots" (it wasn't like this before, all other VMs still work properly and are able to create snapshots)

Here's the VM and storage info:

[23:29:33][root@zoltan]:~# qm config 100
agent: 1
boot: order=ide2;scsi0;net0
cores: 8
ide2: none,media=cdrom
memory: 8192
name: yarpen
net0: virtio=C6:8B:1B:2C:B6:0E,bridge=vmbr0,firewall=1
net1: virtio=92:81:AF:BE:BC:AE,bridge=vmbr1,firewall=1,link_down=1,tag=101
net2: virtio=4A:B5:D5:F1:25:42,bridge=vmbr1,firewall=1,link_down=1,tag=102
net3: virtio=E6:55:32:D6:63:F5,bridge=vmbr1,firewall=1,link_down=1,tag=103
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=55a275f7-42fc-4e11-aade-d26adb2296b4
sockets: 1
vcpus: 8
virtio5: /dev/disk/by-id/scsi-3600508b1001c902b8b9549d0a8090372,backup=0,size=15627943000K
vmgenid: 7542d62c-783b-4f6f-af09-ff71db320c8a

[23:29:34][root@zoltan]:~# cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content vztmpl,iso
    shared 0

lvmthin: local-lvm
    thinpool data
    vgname pve
    content images,rootdir

cifs: zoltan-backups
    path /mnt/pve/zoltan-backups
    server yarpen.MY_DOMAIN
    share zoltan-backups
    content backup
    prune-backups keep-daily=1,keep-last=4,keep-monthly=1,keep-weekly=1
    username zoltan

How do I unblock it? I tried rebooting the PVE host, didn't help. From the logs I understand that the PVE never confirmed that the fs-thaw happened, so I guess I have to trigger it manually?

 

[shalak]

More info from PVE:

[00:06:57][root@zoltan]:~# qm guest cmd 100 fsfreeze-status
thawed

On the VM itself:

[00:10:21][root@yarpen]:~# uname -a
Linux yarpen 5.10.0-0.bpo.9-amd64 #1 SMP Debian 5.10.70-1~bpo10+1 (2021-10-10) x86_64 GNU/Linux

 

[fabian]

I have a VM (id=100) that runs OpenMediaVault - NAS server with CIFS shares. For the past 117 days everything was working properly, the backups were done every night with no errors. I initiated manual backup of this VM - and it failed. Here's the event log:

INFO: starting new backup job: vzdump 100 --node zoltan --storage zoltan-backups --compress zstd --mode snapshot --remove 0
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-01-12 22:52:16
INFO: status = running
INFO: VM Name: yarpen
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 32G
INFO: exclude disk 'virtio5' '/dev/disk/by-id/scsi-3600508b1001c902b8b9549d0a8090372' (backup=no)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/zoltan-backups/dump/vzdump-qemu-100-2022_01_12-22_52_16.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
closing with write buffer at /usr/share/perl5/IO/Multiplex.pm line 928.
INFO: issuing guest-agent 'fs-thaw' command
ERROR: got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - got timeout
INFO: Failed at 2022-01-12 22:55:23
INFO: Backup job finished with errors
TASK ERROR: job errors

I think it's important to note that the VM most likely was under heavy usage from some SMB client.

it kind of reads like the freeze failed cause of load, but it's hard to tell from the log.. there was some issue communicating with the guest though for sure..

The backup storage is mounted over CIFS and comes from this exact VM. So basically this VM is backed up to itself. I never had issues with such backups (they were using snapshot mode)

I assume the CIFS exports are located on the pass-through virtio5 disk? else I can't see how that ever worked..

Now, I'm unable to create snapshots for this VM, it says: "The current guest configuration does not support taking new snapshots" (it wasn't like this before, all other VMs still work properly and are able to create snapshots)

snapshot mode for VM backups and storage snapshots are two very different things. the latter requires support from the storage layer for all disks - you have a virtio disk that is passed-through from a physical disk, so this VM is not snapshotable. a snapshot mode backup uses a qemu-internal snapshotting mechanism that sits between the guest and the volumes/disks backing the guest's disks and doesn't require support from the latter.

How do I unblock it? I tried rebooting the PVE host, didn't help. From the logs I understand that the PVE never confirmed that the fs-thaw happened, so I guess I have to trigger it manually?

unblock what? fs-freeze doesn't have any lasting effect (a thaw returns it to normal, but stopping and starting the guest again has the same effect)..

 

[shalak]

I assume the CIFS exports are located on the pass-through virtio5 disk? else I can't see how that ever worked..

Of course

snapshot mode for VM backups and storage snapshots are two very different things. the latter requires support from the storage layer for all disks - you have a virtio disk that is passed-through from a physical disk, so this VM is not snapshotable. a snapshot mode backup uses a qemu-internal snapshotting mechanism that sits between the guest and the volumes/disks backing the guest's disks and doesn't require support from the latter.

Wait - what? So you say that this VM was never able to be snapshotted and I only noticed this now? I cannot be certain, but I think I would've noticed this before. So the VM backup-snapshot does not utilize the copy-on-write feature of LVM?


So this basically boils down to the "works as expected" type of situation... How can I confirm that this is indeed what's going on?

 

[fabian]

Wait - what? So you say that this VM was never able to be snapshotted and I only noticed this now? I cannot be certain, but I think I would've noticed this before. So the VM backup-snapshot does not utilize the copy-on-write feature of LVM?

yeah - a VM with a local disk passed-through cannot be snapshotted, and a VM 'snapshot-mode' backup doesn't use the storage layer snapshot feature at all, it's all within Qemu's block layer.

So this basically boils down to the "works as expected" type of situation... How can I confirm that this is indeed what's going on?

well.. you got the info straight from the horse's mouth so to speak if the freeze/thaw issue occurs again, we should probably take a closer look what's happening under the hood..

 

[shalak]

Thank you very much for your help