After backups run, vm's come back up with read-only filesystems

[cdukes]

# pveversion
pve-manager/6.1-3/37248ce6 (running kernel: 5.3.10-1-pve)

I can't seem to figure out why this is happening, but every time our backups run, the vm's that were backed up come back up with read-only filesystems and I have to restore form the backup to get them working again.
This is only happening on cloud-init based images, not on full os images.
Also, not sure if it's relevant, these systems are part of an HA cluster. So maybe when the backups run, the cluster manager is starting the image before backup completes? (just a theory)

 

[t.lamprecht]

Weird. That's only happening with cloudinit makes it even weirder. Do you use the QEMU guest agent in the VMs?

Also, nit sure if it's relevant, these systems are part of an HA cluster. So maybe when the backups run, the cluster manager is starting the image before backup completes? (just a theory)

Good for mentioning, but the VM is locked during the backup, so even if the HA manager tried to stop or start it, that operation would fail due to the lock. The only way the lock can be removed is once the backup was finished or if the node fails and gets fenced, then the HA removes some locks after recovering the VM/CT to another node, to ensure it can be started there again. As the whole node where the backup was made is fenced and the VM is started newly this should not cause this.


What backup method do you use? What's the underlying storage (for backup and VM)?

 

[cdukes]

Weird. That's only happening with cloudinit makes it even weirder. Do you use the QEMU guest agent in the VMs?

Yes, all vm's have it.

What backup method do you use? What's the underlying storage (for backup and VM)?

LZO compression and Snapshot mode

What's the underlying storage (for backup and VM)?

Underlying storage for the VM's are ZFS on 4k block NVMe drives. (see attached screenshots)

 

[cdukes]

Backups ran last night and it happened again. Can anyone please help?