[TUTORIAL] NFSoRDMA

E

efeu

Renowned Member
Proxmox Subscriber
Nov 6, 2015
92
10
73
Hi,

(Basicaly you have to get RDMA working first. Look here: https://enterprise-support.nvidia.com/s/article/howto-configure-nfs-over-rdma--roce-x)

it took me a while to find a way how proxmox could use NFSoRDMA. First I tried to change the storage options in /etc/pve/storage.cfg from

Code: 
nfs: NFSoRDMA
        export /testdir
        path /mnt/pve/NFSoRDMA
        server 192.168.40.10
        content images
        options vers=4.2
        prune-backups keep-all=1

to "options rdma,port=20049,vers=4.2" but it was still mounted with TCP. (Why?)

But there is a way, if you mount the NFS share to some other folder with RDMA option before the pvedaemon mounts the storage, it will also get mounted with RDMA instead of TCP as protocol.

Code: 
mount -o rdma,port=20049,vers=4.2 192.168.40.10:/testdir /testdir

Code: 
#mounts
192.168.40.10:/testdir on /testdir type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)
192.168.40.10:/testdir on /mnt/pve/NFSoRDMA type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)

@ ProxmoxDevs
It would be nice if this could be done more elegant in future.
 
Last edited:
  • Like
Reactions: esi_y and floh8
efeu said:
Hi,

(Basicaly you have to get RDMA working first. Look here: https://enterprise-support.nvidia.com/s/article/howto-configure-nfs-over-rdma--roce-x)

it took me a while to find a way how proxmox could use NFSoRDMA. First I tried to change the storage options in /etc/pve/storage.cfg from

Code: 
nfs: NFSoRDMA
        export /testdir
        path /mnt/pve/NFSoRDMA
        server 192.168.40.10
        content images
        options vers=4.2
        prune-backups keep-all=1

to "options rdma,port=20049,vers=4.2" but it was still mounted with TCP. (Why?)

But there is a way, if you mount the NFS share to some other folder with RDMA option before the pvedaemon mounts the storage, it will also get mounted with RDMA instead of TCP as protocol.

Code: 
mount -o rdma,port=20049,vers=4.2 192.168.40.10:/testdir /testdir

Code: 
#mounts
192.168.40.10:/testdir on /testdir type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)
192.168.40.10:/testdir on /mnt/pve/NFSoRDMA type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.40.11,local_lock=none,addr=192.168.40.10)

@ ProxmoxDevs
It would be nice if this could be done more elegant in future.
Click to expand...

Your NFSoRDMA server -- are you using Proxmox as the host as well, or are you using a RHEL-derived distro (like CentOS, as mentioned in the RDMA/RoCE link that you have provided above)?
 
@efeu: Die you configure the mountpoint as Dir Storage in Proxmox?
When yes, got you speed drops according to the direct Mount point?
 
efeu said:
Code:
Code: 
nfs: NFSoRDMA
        export /testdir
        path /mnt/pve/NFSoRDMA
        server 192.168.40.10
        content images
        options vers=4.2
        prune-backups keep-all=1
Click to expand...
floh8 said:
@efeu: Die you configure the mountpoint as Dir Storage in Proxmox?
When yes, got you speed drops according to the direct Mount point?
Click to expand...
From what it looks like above -- it would appear that the storage is defined as NFS.
 
Last edited:
You must log in or register to reply here.
Top Bottom