NVME over Fabric

[ran]

Hi

We are using our Proxmox with a storage that connects over fabric using Mellanox driver

We also use Nvmesh as our storage management software therefore we can see the volume as local

and we are using it as a LOCAL ZFS file system , the problem is that proxmox doesn't see it as shared volume even though I can use it from two different physical servers.

is there anything we can do about it so Proxmox will see it as a shared volume (just like ZFS over ISCSI for example)?

are there any plans to support NVME over Fabric in the future?

Thanks.

 

[spirit]

Hi,
I don't see how you can share a local filesytem like zfs between 2 hosts ? (active-active)
Until you use a cluster filesystem like ocfs2 or gfs.
Or you need some kind of zfs mount/umount failover between 2 hosts.

with zfs over iscsi, the zfs filesystem is only mounted once, on the iscsi target server.

 

[spirit]

(even with zfs zvol block devices, I think that the main zpool can't be attached to 2 nodes at the same time)

 

[ran]

Hi

This is what Nvmesh technology offers (and much more) , I can see the same physical raid volume (nvme) as local to each
Proxmox node , and then I can force import the ZFS volume mount and it can run on two separate servers.

Thanks

 

[ertanerbek]

Hi

We are using our Proxmox with a storage that connects over fabric using Mellanox driver

We also use Nvmesh as our storage management software therefore we can see the volume as local

and we are using it as a LOCAL ZFS file system , the problem is that proxmox doesn't see it as shared volume even though I can use it from two different physical servers.

is there anything we can do about it so Proxmox will see it as a shared volume (just like ZFS over ISCSI for example)?

are there any plans to support NVME over Fabric in the future?

Thanks.

ZFS is not Cluster aware system, you can use normal LUN but this time you should be have thin provisioning on sotorage side because Proxmox also not support Thin provisioning system on shared system. Other point, ZFS for local disk not for ISCSI/FC connection..

I want another question to you, why you want use ZFS on your NVME based shared storage ?

 

[ran]

Because ZFS provides a really stable reliable backup solutions using snapshots , which is a lot better than proxmox built-in backup
for large and multiple VMS

 

[ertanerbek]

If I am not wrong, ZFS not provaide Aplication Avare backup maybe I am wrong but in past I never see any client application for SQL or other aplication ... If I am wrong please fix me.. So you using NVME over FC san Storage it should be that storage snapshoot system better than ZFS backup/sync, for cold backup you should be need software based backup solition for Aplication Avare solition...

For aplication avare consistent backup I think you should be use software backup solition himself if you want free solioton, if not use Veeam or smaller software for real backup.

I think ZFS backup only interasting with DISK so ZFS can not be know how many transection is comming or data when be consistent on DISK or how many response giving from RAM or how many transection wait on RAM buffer etc... Only software can know that.

 

[spirit]

Hi

This is what Nvmesh technology offers (and much more) , I can see the same physical raid volume (nvme) as local to each
Proxmox node , and then I can force import the ZFS volume mount and it can run on two separate servers.

Thanks

yes, I understand that. This is the same shared a shared sas array.

You can use lvm without any problem for example (not lvm-thin because of metadatas not shareable), because each lvm volume can me mounted separatly.

but with zfs, you have the main ztank/pool with metadatas,logs,... and this can't be shared.

Another way could be to use ceph (you'll have snapshot, backup with rbd export|import,....) with osd on top of your nvme mesh.
(but maybe it's a little bit overkill to have a nvme mesh, you can install ceph directly locally on server where you have your nvme drives)