nfs "multipathing"

micush

Active Member
Jul 18, 2015
69
3
28
Seeing as how neither NFS session trunking or pNFS is supported in Proxmox, I was faced with an interesting question as to how to provide multiple parallel links to my NFS storage for both load distribution and path redundancy.

From experience, switch level port aggregation with or without LACP works okay in failover scenarios with NFS but is not ideal with load distribution, especially when using multiple switches as uplinks for redundancy purposes.

Being a network guy by trade, I set aside my "server glasses" and put on my "network glasses" and I looked at the issue a little differently. I came up with a solution that has to-date worked very well for me, so I thought I would share.

OSPF will, by default, load balance traffic on a per-flow basis across up to 4 links - more if you reconfigure it. So this is exactly what I did. I installed Quagga on my four Proxmox hosts and on my NFS host. I created a loopback adapter on my NFS host and advertised that into OSPF. The NFS server and four Proxmox hosts each have 4 NICs within the storage network, each on a different physical switch and each in a different "throw-away" subnet. I mount the exported NFS share in Proxmox by the NFS server loopback IP address learned via OSPF on the Proxmox hosts. I have enabled OSPF fast-hellos in Quagga so that sub-second convergence happens in case of a link failure.

Link failure detection and new data path selection happens within a couple hundred milliseconds, and I have tried everything I can think of to break it but have failed to do so every time. For me this has worked out very well, so I thought I would share. I run about 20 VMs across four hosts with this setup, but of course YMMV.
 

manu

Proxmox Staff Member
Mar 3, 2015
806
64
28
Thanks for the detailed information. Out of curisoity, If you have path redundancy up to the NFS server, do you have any kind of HA on the NFS server itself ?
 

spirit

Famous Member
Apr 2, 2010
4,588
348
103
www.odiso.com
pnfs is supported by kernel 3.10 (you need to mount with -o vers=4.1)


of course, your filer need to support it too.

I have done test last year, i Have had random client crash (when connecting to netapp pnfs). Maybe now it's more stable.

(I just don't care about it anymore, I'm going to migrate to ceph this year)

 

micush

Active Member
Jul 18, 2015
69
3
28
I actually have two NFS hosts set up like this, both have fully redundant internal everything. One host is used as a datastore, and the other to store vzdump backups on. The data stored in the cluster is not mission critical, so some type of HA setup between the two NFS hosts is not currently necessary. In the event of a failure I simply restore the backed up VMs to the vzdump NFS host and start the VMs.

However, if it were to become a requirement to have HA NFS, I could easily convert the vzdump NFS host to run DRDB/Heartbeat to accomplish such a thing.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!