NVME over Fabric

ran

Member
Feb 13, 2019
18
1
6
21
Hi

We are using our Proxmox with a storage that connects over fabric using Mellanox driver

We also use Nvmesh as our storage management software therefore we can see the volume as local

and we are using it as a LOCAL ZFS file system , the problem is that proxmox doesn't see it as shared volume even though I can use it from two different physical servers.

is there anything we can do about it so Proxmox will see it as a shared volume (just like ZFS over ISCSI for example)?

are there any plans to support NVME over Fabric in the future?

Thanks.
 

spirit

Famous Member
Apr 2, 2010
5,865
702
133
www.odiso.com
Hi,
I don't see how you can share a local filesytem like zfs between 2 hosts ? (active-active)
Until you use a cluster filesystem like ocfs2 or gfs.
Or you need some kind of zfs mount/umount failover between 2 hosts.

with zfs over iscsi, the zfs filesystem is only mounted once, on the iscsi target server.
 

ran

Member
Feb 13, 2019
18
1
6
21
Hi

This is what Nvmesh technology offers (and much more) , I can see the same physical raid volume (nvme) as local to each
Proxmox node , and then I can force import the ZFS volume mount and it can run on two separate servers.

Thanks
 

ertanerbek

Member
Mar 29, 2019
91
6
13
41
Hi

We are using our Proxmox with a storage that connects over fabric using Mellanox driver

We also use Nvmesh as our storage management software therefore we can see the volume as local

and we are using it as a LOCAL ZFS file system , the problem is that proxmox doesn't see it as shared volume even though I can use it from two different physical servers.

is there anything we can do about it so Proxmox will see it as a shared volume (just like ZFS over ISCSI for example)?

are there any plans to support NVME over Fabric in the future?

Thanks.


ZFS is not Cluster aware system, you can use normal LUN but this time you should be have thin provisioning on sotorage side because Proxmox also not support Thin provisioning system on shared system. Other point, ZFS for local disk not for ISCSI/FC connection..

I want another question to you, why you want use ZFS on your NVME based shared storage ?
 

ran

Member
Feb 13, 2019
18
1
6
21
Because ZFS provides a really stable reliable backup solutions using snapshots , which is a lot better than proxmox built-in backup
for large and multiple VMS
 

ertanerbek

Member
Mar 29, 2019
91
6
13
41
If I am not wrong, ZFS not provaide Aplication Avare backup maybe I am wrong but in past I never see any client application for SQL or other aplication ... If I am wrong please fix me.. So you using NVME over FC san Storage it should be that storage snapshoot system better than ZFS backup/sync, for cold backup you should be need software based backup solition for Aplication Avare solition...

For aplication avare consistent backup I think you should be use software backup solition himself if you want free solioton, if not use Veeam or smaller software for real backup.

I think ZFS backup only interasting with DISK so ZFS can not be know how many transection is comming or data when be consistent on DISK or how many response giving from RAM or how many transection wait on RAM buffer etc... Only software can know that.
 

spirit

Famous Member
Apr 2, 2010
5,865
702
133
www.odiso.com
Hi

This is what Nvmesh technology offers (and much more) , I can see the same physical raid volume (nvme) as local to each
Proxmox node , and then I can force import the ZFS volume mount and it can run on two separate servers.

Thanks
yes, I understand that. This is the same shared a shared sas array.

You can use lvm without any problem for example (not lvm-thin because of metadatas not shareable), because each lvm volume can me mounted separatly.

but with zfs, you have the main ztank/pool with metadatas,logs,... and this can't be shared.

Another way could be to use ceph (you'll have snapshot, backup with rbd export|import,....) with osd on top of your nvme mesh.
(but maybe it's a little bit overkill to have a nvme mesh, you can install ceph directly locally on server where you have your nvme drives)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!