Passthrough disks and snapshots

[otto001]

Hi,

one of my VMs is acting as nfs-server and so it has some disks of the host as passthrough disks.
The boot-disk is an ordinary qcow2-disk tough.
As this VM is also acting as nextcloud-server I would really like to take a snapshot before trying an update.
Any chance of snapshotting this VM without removing the disks before?
Sorry, if this was answered before, did not find anything...
Best regards and thanks in advance,
Otto

 

[LnxBil]

Any chance of snapshotting this VM without removing the disks before?

via the GUI, no. It does not make any sense.

You can however just snapshot the qcow2 file manually, but I recommend shutting the VM off before doing that just for the best consistency possible.

 

[otto001]

Ahm, why do you think this would not make sense?
I think this would make sense - like with backups. One can exclude disks from backup.
This would be a nice feature I think.
Unfortunately I have to use vmware in work - and there it is always a pain with dependent/independent disks.
I think proxmox could do better here...
Snapshots are (in my eyes) useful before changing the system (like updates). But if there are disks only with data connected to the VM, why should they be included in a backup? A better solution would be to let the user decide what should be part of a snapshot?

 

[LnxBil]

Unfortunately I have to use vmware in work - and there it is always a pain with dependent/independent disks.

Same problem and same solution. Only because you have a corner case where it could be no problem, there are a lot of cases in which it would be terrible. Therefore it is not possible, neither in VMware nor Proxmox VE.

A better solution would be to let the user decide what should be part of a snapshot?

Therefore you can do it yourself. PVE (and VMware) present the safe way of doing things and that is what is most useful for the vast majority of users.

 

[otto001]

hm. here I have a different point of view...
I still think that an option for excluding disks from snapshots in the gui would be useful for most admins.
will have a look on how to do this on shell tough....
thanks anyway!

 

[holckj]

My Proxmox host has a large disk that I use to host a mirror of a Linux distro. The disk is attached to the VM via sshfs. This may not be so fast, but it works, and it's easy to exclude the large disk from the daily backups of the VM. There's no reason to backup a mirror.

 

[rdhoore108]

It's indeed weird that one can exclude disks from backup but not from snapshot. And in order to make the backup, it needs to make a snapshot.

If an admin truly is not smart enough to decide for himself if a disk should be included or excluded in a snapshot, then a lot more will go wrong in his life.

 

[LnxBil]

And in order to make the backup, it needs to make a snapshot.

Technically true, but the two snapshots (disk and VM) are not the same, e.g. if you back up a VM on ZFS, there is no ZFS snapshot created, "only" a snapshot in the VM that is deleted after the backup is done.

If an admin truly is not smart enough to decide for himself if a disk should be included or excluded in a snapshot, then a lot more will go wrong in his life.

unfortunately that is not the real life

 

[rdhoore108]

unfortunately that is not the real life

I never expected such a philosophical reply here in this forum But it's absolutely true. Our real life is not in this material world.

Technically true, but the two snapshots (disk and VM) are not the same, e.g. if you back up a VM on ZFS, there is no ZFS snapshot created, "only" a snapshot in the VM that is deleted after the backup is done.

I just realized (by trying it out) that excluding a disk from backup indeed automatically excludes it from the temporary snapshot that is needed to create the backup. Now that's wonderful. Thank you very much for pointing this out. I had assumed the snapshot would fail, because manual snapshots are not possible for my virtual NAS with proxmox passed-through disks ("The current guest configuration does not support taking new snapshots"). But a backup is possible, thank God, even though it includes taking a snapshot of the running machine.

 

[fabian]

first, only containers do a storage snapshot when doing a 'snapshot' mode backup, VMs use a qemu-internal mechanism instead that doesn't require storage support.

second - the main reason why "skip volume when doing a snapshot" is not available as a feature is that handling of all sorts of related features quickly becomes tricky. for example rollback - is the expectation to leave the disk as it is? or to re-create it empty (like when restoring a backup with excluded mountpoints/volumes)? should it be configurable? what if the flag changed since the rollback target snapshot was taken (in either direction)? what about snapshots with RAM - those would no longer work (or rather, can no longer be allowed) when excluding disks. how should such disks be handled with regards to replication (and what about the implications for HA in that case, depending on the answer)?

the actual skipping part is not hard to implement obviously..

 

[rdhoore108]

first, only containers do a storage snapshot when doing a 'snapshot' mode backup, VMs use a qemu-internal mechanism instead that doesn't require storage support.

I see, that solves a lot of the puzzle indeed.

the main reason why "skip volume when doing a snapshot" is not available as a feature is that handling of all sorts of related features quickly becomes tricky.

Yes, it's absolutely true that the rollback part would become intricate. Thanks for pointing that out in detail. I never thought about HA as I'm not using that.

Anyhow, I'm perfectly happy with the fact that the pseudo snapshot mechanism exists in normal VM backup. Taking a backup before doing something risky is perfectly viable. I simply didn't know that it was possible. It really would have saved my day if I had known that the day before yesterday, when I upgraded my virtual NAS and it went all wrong, but well, better late than never .

 

[melroy89]

I actually missing this feature as well... I really would like to see this feature asap.

Of course you DO need to warn the user when creating snaphot using pass-through disks. Let them know it's not ideal and it can have unwanted side-effects if you have the wrong expectations.

@fabian But I think it would be possible. And here is how..

• Only allow snapshots without storing the RAM.
  • Maybe even only allow snapshots when the VM is turned off.
• Of course do not make the pass-through disk contents part of the snapshot or backup;
• Rollback will be done based on best effort (if you suddenly didn't include the disk or the snapshot didn't include the disk in fstab it's not your problem but the user his problem)
  • Basically you are only restoring the main VM os disk (excluding all the pass-through disks)
• All the questions regarding worrying about this disk during replication/HA can therefor also be ignored. You just ignore that the disk is even there during a snapshot or rollback.
  When using pass through disks it's up the user to solve those problems, that is why you need to show a warning. But no need to disable snapshots all together in Proxmox.
• Additional advance features regarding configuration can be done later, think Agile. Currently we have nothing in place and manually need to adapt config files in order to show the Snapshot button again . In the future we can then also think about maybe using qemu-guest-agent to help and improve live snapshots when using pass-through disks.

 

[saltarob]

via the GUI, no. It does not make any sense.

You can however just snapshot the qcow2 file manually, but I recommend shutting the VM off before doing that just for the best consistency possible.

How would this manual method work? Could you maybe provide an example. Thank you.

 

[LnxBil]

Could you maybe provide an example.

Sure: Create an image, list empty list of snapshots, create snapshot and list snapshot again:

$ qemu-img create -f qcow2 example.qcow2 4G
Formatting 'example.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 lazy_refcounts=off refcount_bits=16

$ qemu-img snapshot -l example.qcow2

$ qemu-img snapshot -c snap1 example.qcow2

$ qemu-img snapshot -l example.qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         snap1                 0 B 2023-08-19 10:43:50 00:00:00.000          0

 

[saltarob]

Sure: Create an image, list empty list of snapshots, create snapshot and list snapshot again:

$ qemu-img create -f qcow2 example.qcow2 4G
Formatting 'example.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 lazy_refcounts=off refcount_bits=16

$ qemu-img snapshot -l example.qcow2

$ qemu-img snapshot -c snap1 example.qcow2

$ qemu-img snapshot -l example.qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         snap1                 0 B 2023-08-19 10:43:50 00:00:00.000          0

Thank you so much for your answer. I think that my boot-disk is a raw disk image format and not an ordinary qcow2-disk.

root@pvemacpro:/dev/pve# cd /dev/pve
root@pvemacpro:/dev/pve# ls
root  vm-100-disk-0  vm-100-state-Backup_2023-08-16_HSVE-EFI  vm-103-disk-0  vm-104-disk-1  vm-104-disk-6
swap  vm-100-disk-1  vm-100-state-Test_2023-08-16             vm-103-disk-1  vm-104-disk-5  vm-104-state-CleanRestore_2023-08-18
root@pvemacpro:/dev/pve# qemu-img snapshot -c snap1 vm-104-disk-6
WARNING: Image format was not specified for 'vm-104-disk-6' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
qemu-img: Could not create snapshot 'snap1': -95 (Operation not supported)
root@pvemacpro:/dev/pve#

Can I use following command line?

qemu-img snapshot -c snap1 vm-104-disk-6.raw

My boot drive is VirtIO.

And how do I roll back? Like this:

qemu-img snapshot -d snap1

 

[LnxBil]

I think that my boot-disk is a raw disk image format and not an ordinary qcow2-disk.

RAW is not snapshottable.

 

[saltarob]

RAW is not snapshottable.

Oh, okay. How does then Proxmox make snapshots from that disk image? I made several shnapshots from the GUI before I passed through the disk. Does Proxmox not use qemu to make snapshots?

 

[LnxBil]

How does then Proxmox make snapshots from that disk image?

From raw? it can't do a storage snapshots - no software can. You need to create a derived qcow2 image from it, then it'll work for storage snapshots. If you mean snapshot backup, this is different and is done in QEMU for the sole purpose of creating a more consistent disk image for the backup.

 

[saltarob]

From raw? it can't do a storage snapshots - no software can. You need to create a derived qcow2 image from it, then it'll work for storage snapshots. If you mean snapshot backup, this is different and is done in QEMU for the sole purpose of creating a more consistent disk image for the backup.

Thank you. So I would have to convert the image to qcow2 and then generate a snapshot and to roll it back to convert it back to raw again...
Also vm-104-disk-6 is only a symlink to a dm-13 file.

I think first I would have to understand the underlying file structure of this VM boot disk and how Proxmox snapshots are generated to figure any workable solution.

Anyway, thank you so much.

 

[LnxBil]

Thank you. So I would have to convert the image to qcow2 and then generate a snapshot and to roll it back to convert it back to raw again...

Why do you want to have raw at all? Just stay with a storage system that supports snapshots.