Infiniband, ZFS, NFS?

[Cup 'O Joe]

Hello all, I'm hoping you maybe able to help me as I seem a bit in over my head this time.

I have three physical servers that I am working to build into an HA cluster. Two of those servers will be the processing power (hereafter referred to as P1 and P2) of the cluster and one is the storage (hereafter referred to as S1).

On S1, I have two ZFS pools that I wish to connect the P1 and P2 servers to. I've connected both P1 and P2 to S1 via infiniband, and have even got as far as figuring out how to run NFS over RDMA. Here's my issue:

vm-100-disk-1.raw is located on S1 under the ZFS pool /flashpool.
For my plan to work, I have to be able to access that ZFS dataset at the file level from P1 and P2.

For example, on P1 I have the /flashpool pool mounted using NFS at /mnt/flashpool, however the "ls" command (on P1) does not display "vm-100-disk-1.raw" - under which case I cannot add the .raw image to a VM.

I hope I've been able to explain my issue clearly, and I appreciate any help you have to offer.

 

[Cup 'O Joe]

After a bit of digging around I solved the issue.

ZVols are block devices - file level access isn't possible. Instead what one must do if wishing to share a VM drive over NFS is to convert the zvol to a disk image, in my case I chose .raw. This can be accomplished using the dd utility:

dd if=/dev/zvol/pve/flashpool/vm-100-disk-1 of=/flashpool/vm-100-disk-1.raw

 

[guletz]

This can be accomplished using the dd utility:

dd if=/dev/zvol/pve/flashpool/vm-100-disk-1 of=/flashpool/vm-100-disk-1.raw

It is much easy to select your vHDD, and from PMX web interface you can CLICK on MOVE DISK button(so from zfs zvol storage -> NFS storage).