[TUTORIAL] NFSoRDMA

[efeu]

Hi,

(Basicaly you have to get RDMA working first. Look here: https://enterprise-support.nvidia.com/s/article/howto-configure-nfs-over-rdma--roce-x)

it took me a while to find a way how proxmox could use NFSoRDMA. First I tried to change the storage options in /etc/pve/storage.cfg from

nfs: NFSoRDMA
        export /testdir
        path /mnt/pve/NFSoRDMA
        server 192.168.40.10
        content images
        options vers=4.2
        prune-backups keep-all=1

to "options rdma,port=20049,vers=4.2" but it was still mounted with TCP. (Why?)

But there is a way, if you mount the NFS share to some other folder with RDMA option before the pvedaemon mounts the storage, it will also get mounted with RDMA instead of TCP as protocol.

mount -o rdma,port=20049,vers=4.2 192.168.40.10:/testdir /testdir

#mounts
192.168.40.10:/testdir on /testdir type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)
192.168.40.10:/testdir on /mnt/pve/NFSoRDMA type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)

@ ProxmoxDevs
It would be nice if this could be done more elegant in future.

 

[alpha754293]

Hi,

(Basicaly you have to get RDMA working first. Look here: https://enterprise-support.nvidia.com/s/article/howto-configure-nfs-over-rdma--roce-x)

it took me a while to find a way how proxmox could use NFSoRDMA. First I tried to change the storage options in /etc/pve/storage.cfg from

nfs: NFSoRDMA
        export /testdir
        path /mnt/pve/NFSoRDMA
        server 192.168.40.10
        content images
        options vers=4.2
        prune-backups keep-all=1

to "options rdma,port=20049,vers=4.2" but it was still mounted with TCP. (Why?)

But there is a way, if you mount the NFS share to some other folder with RDMA option before the pvedaemon mounts the storage, it will also get mounted with RDMA instead of TCP as protocol.

mount -o rdma,port=20049,vers=4.2 192.168.40.10:/testdir /testdir

#mounts
192.168.40.10:/testdir on /testdir type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)
192.168.40.10:/testdir on /mnt/pve/NFSoRDMA type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)

@ ProxmoxDevs
It would be nice if this could be done more elegant in future.

Your NFSoRDMA server -- are you using Proxmox as the host as well, or are you using a RHEL-derived distro (like CentOS, as mentioned in the RDMA/RoCE link that you have provided above)?

 

[floh8]

@efeu: Die you configure the mountpoint as Dir Storage in Proxmox?
When yes, got you speed drops according to the direct Mount point?

 

[alpha754293]

Code:

nfs: NFSoRDMA
        export /testdir
        path /mnt/pve/NFSoRDMA
        server 192.168.40.10
        content images
        options vers=4.2
        prune-backups keep-all=1

@efeu: Die you configure the mountpoint as Dir Storage in Proxmox?
When yes, got you speed drops according to the direct Mount point?

From what it looks like above -- it would appear that the storage is defined as NFS.