Proxmox, FreeNAS, NFS and snapshot per dataset

[saphirblanc]

Hello,

I'm looking into building a FreeNAS server which will host my VMs through a NFS share.
As Proxmox does not support snapshot on raw image hosted by an NFS server, I'm trying to find a way to do this in an "easy" way.

I was thinking about creating a dataset per VM which will allow me to set some quota, and create a snapshot per VM instead for the whole VMs, and therefore let me restore the VM storage only. However, after that, I'll have to configure quite a lot of NFS storage on Proxmox and this is certainly wrong.

What would be the correct way to, without changing the core of Proxmox (no patch) to host VMs in a cluster way (I've got 2 nodes) which will be connected to the same Freenas server with the ability of doing snapshots.

Thanks for your reply!

 

[fortechitsolutions]

can you clarify your 'requirements' you are attempting to address with this,
"snapshot on raw image hosted by an NFS server"

- you want regular backups (not difficult on stock proxmox)
- you want something else I am guessing?
- I am a big fan of 'keep it simple' to make operations very stable, easy to fix / hard to break. So nice vanilla proxmox with multiple proxmox nodes, and a shared NFS server, is definitely a good thing.

I just don't understand what you hope to achieve as your end point.

- possibly doing snapshots at NFS Filer level, below the level at which proxmox will see or care about, as a way to have point-in-time rollback for your VM images?
- possibly just simple NFS, and simple regular scheduled backups (ie, nightly) is actually pretty good (?) done at proxmox level.

proxmox can use ZFS storage, which might give you what you want? (I am not sure really what you want though).

end of the day, it may not be possible to have (Cheap, easy, reliable, good performance) all at the same time, and then you must make decisions about what is more important /and what you will choose to not do by intention/design.

Tim

 

[DynFi User]

What we have been doing since ages now with 0 problem is the following :

We have setup two FreeNAS server :

• first servers will be the one that you already have deployed, you'll put all your CT / KVM images in a specific ZFS mount point - I suggest
  • 1 MP for KVM
  • 1 MP for LXC
• second server (does not need to be hyper sophisticated) will receive the ZFS snapshots of the first FreeNAS server - I suggest
  • 1 ZFS replica (snapshot) for KVM
  • 1 ZFS replica for LXC
  • 1 eventually to do direct ZFS snap from your hypervisor (provided that you are using local ZFS storage)
    • in this last case, you can use pve-zsync which is very steady and works like a charm

Advantages of such solution :

• Fairly easy to setup
• Instant access to ZFS snapshot (you can clone the snapshot and content of the snapshots will be readily available)
• Fully automated (once replica between "first FreeNAS" and "second FreeNAS" is set - there is not much to do
• If you happen to have 10G network, It will be very fast to restore
• Rock solid (have used that since 6 years with 0 problem so far)

Advises :

• use separate dedicated networking for your PVE / NAS (this is part of the Proxmox requirement anyway)
• use aggregated bonds for network (like LAGG) for more stable and efficient dump / restore
• test your backup regularly (might want to check this for KVM with DB inside as it might not work due to transactional DB mechanism and corruption of DB…)
• use fast network (if available)
• configure your FreeNAS correctly (caching ZIL / Slog, …)

 

[saphirblanc]

What we have been doing since ages now with 0 problem is the following :

We have setup two FreeNAS server :

• first servers will be the one that you already have deployed, you'll put all your CT / KVM images in a specific ZFS mount point - I suggest
  • 1 MP for KVM
  • 1 MP for LXC
• second server (does not need to be hyper sophisticated) will receive the ZFS snapshots of the first FreeNAS server - I suggest
  • 1 ZFS replica (snapshot) for KVM
  • 1 ZFS replica for LXC
  • 1 eventually to do direct ZFS snap from your hypervisor (provided that you are using local ZFS storage)
    • in this last case, you can use pve-zsync which is very steady and works like a charm

Advantages of such solution :

• Fairly easy to setup
• Instant access to ZFS snapshot (you can clone the snapshot and content of the snapshots will be readily available)
• Fully automated (once replica between "first FreeNAS" and "second FreeNAS" is set - there is not much to do
• If you happen to have 10G network, It will be very fast to restore
• Rock solid (have used that since 6 years with 0 problem so far)

Advises :

• use separate dedicated networking for your PVE / NAS (this is part of the Proxmox requirement anyway)
• use aggregated bonds for network (like LAGG) for more stable and efficient dump / restore
• test your backup regularly (might want to check this for KVM with DB inside as it might not work due to transactional DB mechanism and corruption of DB…)
• use fast network (if available)
• configure your FreeNAS correctly (caching ZIL / Slog, …)

Hi, thank you very much for your reply, very interesting as I was thinking about doing the same.

Except DB, you did not experienced big issues when switching to a cloned snapshot from the second NAS ? Like read only VM or corrupted VM ?

How do you manage the updates from
FreeNAS on your both FreeNAS?

Thanks.

 

[DynFi User]

Hi, thank you very much for your reply, very interesting as I was thinking about doing the same.

Except DB, you did not experienced big issues when switching to a cloned snapshot from the second NAS ? Like read only VM or corrupted VM ?

How do you manage the updates from
FreeNAS on your both FreeNAS?

Thanks.

No, I had 0 problem so far.
I do have experienced problem when increasing the size of a VM hosted on an NFS share (highlighteed in a thread from 2016).
But this is most probably due to a very outdated version of FreeNAS that we are using (9.1 - ouch !).

I will move to latest FreeNAS + NVMe soon…
Not sure if I'll use ZFS over iSCSi as highlighted in the GrandWazoo post or stick to more classical approach…