VirtioFS and Snapshots?

[scyto]

I saw in the release notes that snapshots are not supported.

Is this a normal case of 'we havent tested the scenario'?

as a)snapshots without RAM (like for PBS) are supported and virtiofsd seems to be capable:

and https://gitlab.com/virtio-fs/virtiofsd/-/blob/main/doc/migration.md

So I assume it isn't a fundemental limitation of virtiofs? Or is this a design decision to do with the complexity of what happens to open files? as described in the doc above? my assumption is on a ceph based storage this shouldn't be an issue.... but assumptions are nevr good

 

[UdoB]

So I assume it isn't a fundemental limitation of virtiofs?

It mounts a folder, containing files, from the host. Stored on a filesystem of unknown type and possibly used by other processes simultaniously. How would it be possible to snapshot that?

It is my personal understanding that this is just impossible.

And while the VM contains state of those files it can not just ignore it either. What would happen on rollback with a different understanding of its content on the host and the VM? Parts of the state and/or the content may be in RAM inside of the guest.

 

[zodiac]

It's more a philosophical rather than a technical problem -- and I wished that Proxmox VE made this a decision that the user can configure.

If the shared state is considered part of the VM, then yes, snapshots don't make sense. If it is just a way to get a view onto a mutable remote resource, then snapshots should be OK. The connection is part of the snapshot, but the file contents wouldn't.

At that point, it boils down to the problem that any remote filesystem has; the state can change on the server, even if the client still caches old data. We have had this since the early days of NFS, and software simply has to deal with the fact that this can happen.

 

[alexskysilk]

I saw in the release notes that snapshots are not supported.

naturally. its a separate filesystem.

as a)snapshots without RAM (like for PBS) are supported and virtiofsd seems to be capable:

PBS doesnt use snapshots.

t's more a philosophical rather than a technical problem -- and I wished that Proxmox VE made this a decision that the user can configure.

You cant "configure" that which is not possible to do. virtiofs is like any other external mounts- nfs, iscsi, whatever- its outside the scope of the parent hypervisor.

 

[scyto]

I love all the folks commenting that it can’t do it without reading the links that clearly show it can. (I.e not in Proxmox but other situations involving qemu and virtiofs).

There are also a couple of great presentations on how it handles things like state / crashes.

Philosophically this is no different then the considerations I have to make when the docker swarm VM (and the swarms other members) are using glusterfs inside the vm or accessing ceph or nfs remotely.

I wouldn’t expect the snapshot to take any consideration of the file system - that would be for me and the application to figure out. Just like when I use a clustered file system.

Also PBS does you snapshot, I literally have if configured in snapshot mode. That is how it treats the disk. It doesn’t snapshot memory or cpu state, it does snapshot the disk.

 

[alexskysilk]

I love all the folks commenting that it can’t do it without reading the links that clearly show it can. (I.e not in Proxmox but other situations involving qemu and virtiofs).

The link you presented offers a discussion on how a hypervisor could use the provided glue to copy a presented virtiofs resource to a destination. In order for this to be useful, a product such as proxmox would have to make provisions to check if OTHER guests are connected to it, quiesce any pending writes, make a decision on what to do with the source upon completion; etc. All of this could be done, of course, but to what end? if you really wanted the data to be migratable without impacting other vms just make it an attached disk.

Philosophically this is no different then the considerations I have to make when the docker swarm VM (and the swarms other members) are using glusterfs inside the vm or accessing ceph or nfs remotely.

With a difference. all those other remote mounts have no host dependency. If thats what you're after, thats what you should use.

In all honesty, at this stage of development I would say virtiofs is more a technology preview then an actually useful tool, especially in light of its poor performance.

 

[scyto]

The link you presented offers a discussion on how a hypervisor could use the provided glue to copy a presented virtiofs resource to a destination. In order for this to be useful, a product such as proxmox would have to make provisions to check if OTHER guests are connected to it, quiesce any pending writes, make a decision on what to do with the source upon completion; etc. All of this could be done, of course, but to what end? if you really wanted the data to be migratable without impacting other vms just make it an attached disk.


With a difference. all those other remote mounts have no host dependency. If thats what you're after, thats what you should use.

In all honesty, at this stage of development I would say virtiofs is more a technology preview then an actually useful tool, especially in light of its poor performance.

I agree its a preview in proxmox - as i alluded to in my opening post

Virtiofsd absolutely supports qemu snapshots based on the documentation - as it clearly states the virtiofsd managed storage is NOT snapshotted and should not be included in the snapshot - you will note I didn't ask for it to be.

As for 'no host requirement' given the proxmox cluster is also the ceph storage cluster whether I use virtioFS to expose the cephFS into the VM, i used the ceph fuse client, or a CSI docker volume plugin the host requirement remains the same.

Also any considerations of how apps running on the VMs manage their access to that storage also doesn't change - the consideration of what an app does when the storage goes away / how multiple nodes access the same data remains the same. Be it glusterFS running in the VMs, the VMs accessing CephFS via any number of mechanisms, or virtiosfs passing trhough a cephFS volume (again a scenario virtiofs was explicitly inteded for_

I suspect the issue is you are applying *your* scenarios of what you want virtiofsd to and applying your conclusions of why that 'won't work' to my scenario which is a) quite different and b) totally the design target of virtioFS - speed is not the issue, nor is concurrent access as i have to manage that issue no matter what i use.

would I use this to provide a generalized volume to VM to treat like block storage - absolutely not, that is not what virtiofs was ever intended to be

tl;dr proxmox already presents the issues you refer to - it snapshots VMs that have processes running data on network storage like iSCIS / SMB / NFS / CEPH / Gluster / etc - you argument would naturally lead to a conclusion that any VM using network storage should not be snapshotted. Should we ask the devs for that feature to prevent users from hanging themselves?

 

[zodiac]

proxmox already presents the issues you refer to - it snapshots VMs that have processes running data on network storage like iSCIS / SMB / NFS / CEPH / Gluster / etc - you argument would naturally lead to a conclusion that any VM using network storage should not be snapshotted. Should we ask the devs for that feature to prevent users from hanging themselves?

Containers have always had this issue. The equivalent to virtiofsd for containers are bind mounts. If you create bind mount using the Proxmox GUI, you immediately lose the ability to create snapshots or to migrate a container. If you instead use NFS, SMB, or even just bind mounts that side-step Proxmox VE, you retain the ability to create snapshots. Of course, if you then tried to migrate a container from one node to another, you are on your own. This might work, or it could fail, depending on whether the same resources are available on the new node.

I am not particularly happy with this design choice. Automatically disallowing features such as snapshotting works well for some use cases, and it is entirely counter-productive for others. Proxmox VE enforces a policy decision that only works for some of its users. I would much rather like to see a configuration options, where the user can tell Proxmox VE how to treat shared remote storage.

And by extensions, I would like the same UI choices to be made for VMs. There should be a well-documented and fully exposed option to choose the desired behavior, as there are very legitimate use cases for both.

 

[scyto]

@zodiac totally agree

if folks think this is a net-new issue they probably haven't worked with shared storage in any meaningful way - its been like this for over 25+ years (yeah my windows clustering goes that far back.....) it just more complicated once we got vms, containers, multiple file systems - keeps us all in jobs, rofl.

i take great care in my swarm architecture to ensure containers in the docker swarm VMs use shared storage correctly - generally one replica of a container is running in the swarm at any one time that accesses shared storage, no database file can ever be accessed by more than one container etc (i wish more pre-made images would allow use of external databases)

snapshots are not make or break for me - they would be useful when I am doing maintenance to drain a node of containers, snapshot it, modify it, and then roll back if something catastrophic happens

in reality it would only take me about 30 mins to reconstitute a swarm node from first principle

esp once i have eliminated glusterfs from my setup and replaced with virtioFS or ceph directly (i just moved my first container to the virtioFS backed by CephFS approach)

for the latter, i am watching this thread with anticipation https://github.com/ceph/ceph-csi/issues/3769 monamin seems to be making progress.... at least for using a ceph RBD....

this is my rough notes of my progress implementing the virtiofs scenario
https://gist.github.com/scyto/1b526c38b9c7f7dca58ca71052653820

this is the swarm notes https://gist.github.com/scyto/f4624361c4e8c3be2aad9b3f0073c7f9 for the version with gluster