Large number (+100) VM snapshot impact

[GabrieleV]

Hy,
I have two scenarios:

• PVE with CEPH storage
• PVE with NFS on ZFS storage

VMs disks varies from single disk 31GB to multiple striped disk of 512GB

How will PVE performs on these two scenarios if i put in place an automated VM snapshot procedure to maintain always more than 100 VMs snapshots ?

Will performance be impacted ?

Will reliability will be impacted ?



Thank you

 

[bbgeek17]

Its not quiet clear what "NFS on ZFS" is. Its also unclear if you want to keep 100 snapshots for each disk of each VM, or 100 total.
Regardless - excessive number of snapshots will always have an impact on storage performance in some way. What type of impact really depends on underlying hardware and software implementation. Not all ZFS and NFS servers are equal. Performance of Raspberry PI would be quiet different than Zen3.

The best approach is for you test each and find out what works best for you.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[aaron]

Additionally, keep in mind that you can create snapshots via the Proxmox VE API that contain the state of the VM, or just snapshot the disk images. Snapshots that include the VM state take quite a bit longer to create (depending on the storage) as the RAM and state of the VM needs to be written to the storage.

Each snapshot also stores the config of the VM at the time of the snapshot, thus increasing the size of the VM config file. In current Proxmox VE 7 installations, the limit for a file in /etc/pve is 1 Mbyte. Looking at the config files I have at hand, they are roughly always around 0.5 kbyte for VMs without snapshots. So even 100 snapshots per VM should not increase the size of that config file. For VMs with long comments, this could not be true anymore though!

 

[GabrieleV]

Its not quiet clear what "NFS on ZFS" is.

I have a ZPOOL shared as NFS. VMs disks so are qcow2

Its also unclear if you want to keep 100 snapshots for each disk of each VM, or 100 total.

100 snapshot per VM

The best approach is for you test each and find out what works best for you.

It would be nice to exchange experiences. If someone has already do this, please comment...

 

[GabrieleV]

Additionally, keep in mind that you can create snapshots via the Proxmox VE API that contain the state of the VM, or just snapshot the disk images.

I want disk only snapshots. Do not want to keep RAM status too.
So for CEPH RBD, only native RBD snapshots, for NFS, only qcow2 snapshots

Each snapshot also stores the config of the VM at the time of the snapshot, thus increasing the size of the VM config file. In current Proxmox VE 7 installations, the limit for a file in /etc/pve is 1 Mbyte.

This is a new for me. I Have to take care of this. Is there a check in PVE fot managing this limit ?

 

[aaron]

This is a new for me. I Have to take care of this. Is there a check in PVE fot managing this limit ?

Once the limit is reached, you should get an error message when you want to create additional snapshots. Though with the 1Mbyte / file limit, you should be quite fine, unless you want to store books in the comment for the guest

 

[bbgeek17]

It would be nice to exchange experiences. If someone has already do this, please comment...

I'd be interested in that as well, but my guess is that such use cases are very rare. One reason being - 100 snapshot retention is not practical.
With a 10GB volume doing a daily snapshot and 10% per day data churn, by day 10 you will need 100% more space for snapshot storage. By 100th snapshot your 10GB are now taking 110GB on disk.
It all depends on dataset - plex VM with 1TB of videos where only new data is added, i.e. zeros are saved to snapshot, is a good candidate for 100 snapshot protection.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Dunuin]

PBS might also be an alternative. Then you got backups and not snapshots but capacity used should be similar becauses of the deduplication. Especially useful in case you got cheaps HDDs for the PBS and expensive SSDs for the PVE. Snapshots would be always stored on the expensive SSDs (except for the RAM dump) while backups can be stored on cheap HDDs.
So might be way cheaper to use PBS instead of snapshots when you want to alot of backups/snapshots and but it doesn't has to be that fast (because you for example just want daily and not hourly backups/snapshots).

I switched from ZFS snapshots + backups to PBS backups only and are now saving alot of space while still having the same retention and granularity.

 

[guletz]

I switched from ZFS snapshots + backups to PBS backups only and are now saving alot of space while still having the same retention and granularity.

Hi,

My preference is somehow in the middle: keep last 3 snapshots on the PMX host on the zfs local storage for any VM, but also keep much more backups on the PBS. Especially for big data VMs.
In case of problems, it is a higher probabiltity to need to revert a local zfs snapshot(fast lightning), and is a lower probability to need to restore from PBS(who is not so fast compared with zfs snapshot) - 1Gbit storage network.
In case of small size vDisks, time restore from PBS is OK.

I have a ZPOOL shared as NFS. VMs disks so are qcow2

Not so good. COW(qcow2) over another COW(zfs).

Good luck / Bafta!

 

[GabrieleV]

Not so good. COW(qcow2) over another COW(zfs).

What other shared storage do you recommend ?

 

[guletz]

What other shared storage do you recommend ?

RAW

 

[GabrieleV]

RAW

But if use RAW on NFS over ZFS, i loose snapshot support in PVE ...

 

[guletz]

But if use RAW on NFS over ZFS, i loose snapshot support in PVE ...

Indeed! But you also have zfs snapshots!