QCOW2 on ZFS, but over NFS?

[mambojuice]

We all know that a COW disk on a COW filesystem is bad, but what if the underlying storage is over NFS?

I haven't been able to find a definitive answer, but I would imagine that QCOW2 disk images on NFS storage should be safe from write amplification even if the underlying NFS storage is ZFS.

Wouldn't the QCOW2 copy-on-write occur on the PVE host, while the raw storage copy-on-write happens on the ZFS box? Or am I way off in my assumption?

 

[alexskysilk]

nfs is just the transport. no special magic occurs because thats what you are using for connectivity.

 

[IsThisThingOn]

We all know that a COW disk on a COW filesystem is bad, but what if the underlying storage is over NFS?

That makes it even worse by introducing another abstraction layer.

but I would imagine that QCOW2 disk images on NFS storage should be safe from write amplification even if the underlying NFS storage is ZFS.

No. Why should that be the case?

Wouldn't the QCOW2 copy-on-write occur on the PVE host, while the raw storage copy-on-write happens on the ZFS box? Or am I way off in my assumption?

So you want to move of the storage from PVE to a ZFS box and access it over NFS?
So then the no copy on write happens on your PVE host but you still have copy on write twice, plus you added NFS, you just moved the problem to another box.

 

[Mom]

So rather use RAW instead of QCOW2 ? ( ZFS > NFS > RAW )

 

[IsThisThingOn]

This is just my opinion:

You should use RAW on top of ZFS. All local, no network shenanigans.
You should not use (and I don't think it is even possible) use RAW over NFS, where the NFS is on top of ZFS.
You should not use RAW over iSCSI, where the iSCSI is on top of ZFS.
You should not use QCOW2 over NFS, where the NFS is on top of ZFS.

In general, you should not use QCOW2 in combination with ZFS.
ZFS does all you need already, even better than QCOW. Adding QCOW is just an abstraction layer of stuff that could go wrong, without you gaining anything.

 

[Mom]

Ok, I understand your point there.

What would you recommend when making vDisk-storage available over network is needed ? (For example for VM live migration to another Proxmox server.)

Scenario being:
Proxmox-1 <= ZFS <= network share mounted <= vDisks
Proxmox-2 ........... <= network share mounted <= vDisks

 

[IsThisThingOn]

Since we have no idea about your use case, it is hard to tell.

I would seriously reconsider if you actually need shared storage.
Because IMHO in most cases it is not worth the downsides.
If migration does not have to be real time, you can also simply create a backup to a NFS destination and then reimport it on another host.

 

[Johannes S]

For migration between Proxmox servers or clusters you could use the qm/pct remote-migrate commands or the Proxmox datacenter Manager.

 

[Johannes S]

If you want the NFS storage just for migration it's fine, the procedure for vmware to ProxmoxVE is described in wiki:

https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Attach_Disk_&_Move_Disk_(minimal_downtime)

It should be no problem to adapt this to your needs although imho the native tools for PVE to PVE migrations are far superior.

With pve-zsync you an also replicate vms and their settings to another cluster or single-node: https://pve.proxmox.com/wiki/PVE-zsync

 

[Mom]

Thanks, I will have a look into it.