Snapshot delete is stuck

Ahmet Bas

Active Member
Aug 3, 2018
51
0
26
31
Hello,

We are currently running Proxmox version 6.4-14 with ZFS. We get frequently that VMs get locked during snapshot deletion. When I check the VM task history I see this:
Code:
TASK ERROR: VM <id> qmp command 'blockdev-snapshot-delete-internal-sync' failed - got timeout

Currently the only way to unlock is to perform this:
Code:
qm unlock <id>

To delete the snapshot I do this:
Code:
qm listsnapshot <id>
qm delsnapshot <id> <name>

If the delete fails I do force:
Code:
qm delsnapshot <id> <name> --force
Output
Code:
VM <id> qmp command 'blockdev-snapshot-delete-internal-sync' failed - Snapshot with id 'null' and name '<name>' does not exist on device 'drive-scsi1'

How can I prevent that VMs getting in lock state during snapshot delete?
 

Ahmet Bas

Active Member
Aug 3, 2018
51
0
26
31
We were unable to resolve this issue yet. I hope someone can assist us on resolving this issue. The problems occur mostly in the snapshot delete process.
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,194
406
88
Hi,
there should be a 10 minute timeout for this operation. How big are your disks? Also, ZFS snapshots should be done on ZFS directly, so the qmp command shouldn't even be used then. Do you maybe use qcow2 disks in a directory on top of ZFS? If yes, that's bad from a performance perspective.
 

Ahmet Bas

Active Member
Aug 3, 2018
51
0
26
31
Hi,
there should be a 10 minute timeout for this operation. How big are your disks?
We have 24 disks of 4TB disks.
Also, ZFS snapshots should be done on ZFS directly, so the qmp command shouldn't even be used then.
Can you elaborate this? How can we achieve this?

Do you maybe use qcow2 disks in a directory on top of ZFS? If yes, that's bad from a performance perspective.
It's a Windows Server with qcow2, what would your suggestion if you only have a ZFS pool and you want to make snapshots?
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,194
406
88
We have 24 disks of 4TB disks.
Sorry, I meant how big the disks attached to the VM are.

Can you elaborate this? How can we achieve this?
By using zvols, i.e. creating the disks on the ZFS storage directly. You need to add the ZFS to Proxmox VE's storage configuration (Datacenter > Storage > Add > ZFS) if you didn't do so already.

It's a Windows Server with qcow2, what would your suggestion if you only have a ZFS pool and you want to make snapshots?
When using ZFS volumes, a rollback can only be done to the most recent snapshot (otherwise more recent ones need to be removed first). Please be aware of that.
 

Ahmet Bas

Active Member
Aug 3, 2018
51
0
26
31
Sorry, I meant how big the disks attached to the VM are.
It's a 400GB disk. The snapshot is made without RAM.

Sorry, I meant how big the disks attached to the VM are.


By using zvols, i.e. creating the disks on the ZFS storage directly. You need to add the ZFS to Proxmox VE's storage configuration (Datacenter > Storage > Add > ZFS) if you didn't do so already.
This is already done. The storage is added as a ZFS storage to the Proxmox cluster.
When using ZFS volumes, a rollback can only be done to the most recent snapshot (otherwise more recent ones need to be removed first). Please be aware of that.
I'm aware of that, but what would be your advice to do when a snapshot needs to be created when using ZFS. You mentioned that we should not use the qmp command what other options do we have?
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,194
406
88
It's a 400GB disk. The snapshot is made without RAM.


This is already done. The storage is added as a ZFS storage to the Proxmox cluster.
It's not possible to create a qcow2 image on a storage of type zfspool via PVE API.

I'm aware of that, but what would be your advice to do when a snapshot needs to be created when using ZFS. You mentioned that we should not use the qmp command what other options do we have?
I'd guess that the qcow2 images are on a storage of type dir. Please check the type with pvesm status and have a look at the documentation.

If the disks are on a storage of type zfspool, the qmp command should not be invoked by Proxmox VE. Instead it will use ZFS's snapshot feature.
 

Ahmet Bas

Active Member
Aug 3, 2018
51
0
26
31
It's not possible to create a qcow2 image on a storage of type zfspool via PVE API.
When running pvesm status is shows this:
Code:
data         nfs     active    xxx     xxx     xxx   xx%

If the disks are on a storage of type zfspool, the qmp command should not be invoked by Proxmox VE. Instead it will use ZFS's snapshot feature.
It's not but from what I can see in the logs this issue happens on server where we want to remove the snapshot. This goes wrong in most cases. Which command should we use instead of the qmp or through the API?
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,194
406
88
When running pvesm status is shows this:
Code:
data         nfs     active    xxx     xxx     xxx   xx%
Did you add it as an NFS storage, because you have a cluster and need to share the storage? If so, adding it as a zfspool which is local, is of course not an alternative.

It's not but from what I can see in the logs this issue happens on server where we want to remove the snapshot. This goes wrong in most cases. Which command should we use instead of the qmp or through the API?
You can't choose the command. Proxmox VE will choose the command depending on the underlying storage.
 

Ahmet Bas

Active Member
Aug 3, 2018
51
0
26
31
Hi,
there should be a 10 minute timeout for this operation. How big are your disks? Also, ZFS snapshots should be done on ZFS directly, so the qmp command shouldn't even be used then. Do you maybe use qcow2 disks in a directory on top of ZFS? If yes, that's bad from a performance perspective.
Is it expected behavior that a snapshot works very fast, but when we try to remove it takes up to 9 min? During these 9 minutes, the VM becomes unavailable.
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,194
406
88
AFAICT you are using qcow2 on top of NFS on top of ZFS, so you can't really expect the full performance.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!