Advise: shared vm storage using nfs

[Daxcor]

Hello,

I have been running a 4 node cluster for a few years now. I have changed my storage options a few times. the last change I made was from ceph rbd to local zfs raid 1. This means that I can't use the HA features of proxmox 9.2 effectively. This begs the question why don't you stay with ceph. I have borked ceph twice now. I am not skilled enough to reliably admin this kind of technology. I don't have the budget for the recommended hardware and paid consultants to make it work.

Local storage can and does work. However, I want to get the communities opinion on using nfs 4.2 as a shared storage device for my vms. This seems like it would be a good compromise for complexity and opening HA in proxmox. Looking for the good, the bad and the ugly of this path.

Thanks.

 

[bbgeek17]

Hi @Daxcor,

NFS is a good choice for your use case. It is used for shared storage by a great number of entities, from home users to multi-billion-dollar corporations.

NFS has been around for 42 years. The good, bad, and ugly have probably been covered millions of times by now.

The only question you need to answer is where you plan to run the NFS server. A separate appliance makes things clean and easy. Running it as a VM will introduce dependencies that may be perfectly acceptable for a homelab, but may not be acceptable for a business.

The great thing about it is that you don't need to pick one or the other. You can use local storage and NFS in parallel.

Cheers.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[louie1961]

NFS works fine, but it has limitations. Each read and write is a network hop. You will benefit from (and likely appreciate) at least a 10g network. NFS specifically benefits enormously from a proper SLOG, using an nvme drive or even better, an optane drive. You don't need a big drive for SLOG. Even 64 GB is enough. But you want power loss protection and high endurance. A dedicated, non routed storage VLAN or network also helps a lot. And setting up your NAS storage pool properly helps a lot too. In non ZFS systems, you'll benefit from raid 10. In TrueNAS, I use three mirrored vdevs. Raidz1 is not the answer

 

[Daxcor]

I am using a dual 100G lacp bonded network, with a zfs raid 10 on nvme drives on the nfs server, which is stand alone hardware. Based on your recommendations, I think i check the boxes.

Not sure if it matters, but my vms are for kubernetes nodes. They don't use local storage, they use nfs for the pod workloads.

Thanks

 

[louie1961]

Honestly in that case, attach a virtual disk to each worker node in the cluster and install longhorn. You will NOT be disappointed. You can replicate those disks to shared storage as backup. Longhorn will make your storage HA across all the worker nodes. You can create separate Longhorn nodes if you want, but it's not necessary. Pods will migrate between nodes seamlessly

 

[alexskysilk]

NFS specifically benefits enormously from a proper SLOG,

NFS doesnt care; certain write patterns can benefit.

a slog is useful when the following two things are true:
- The write pattern is large number of synchronous small IO (eg, backing a database)
- The backing store is an order of magnitude slower then the prospective slog device.

its important to note that outside the above slogs make almost no difference at all. Which leads to

I am using a dual 100G lacp bonded network, with a zfs raid 10 on nvme drives on the nfs server,

It may be worth considering that by concentrating your payload to a single host you create a spof (single point of failure.) you cluster members are happy but if the storage goes down so does your cluster. If this is ok for your use case, I might have suggested deploying the filer as a NVOF SAN but thats even more complicated to set up then ceph, especialy since pve doesnt have any inbuilt provisions for it.

 

[louie1961]

Not sure if it matters, but my vms are for kubernetes nodes. They don't use local storage, they use nfs for the pod workloads.

I misunderstood your setup, I missed that pod storage is already on NFS. Let me back up. If every VM in that cluster is a k8s node, I'd question whether you need Proxmox HA at all. Two different HA controllers (Proxmox and K8S) would be reacting to the same host failure, and K8S reschedules long before Proxmox finishes fencing. Run 3 control nodes on 3 separate hosts, spread workers across all four, and let K8S own availability. Going 4 physical to 3 physical nodes won't likely impact your performance.

And if you add Longhorn for pod storage, you don't need shared VM storage either. Longhorn replicates volumes across physical nodes, so the data survives a host loss on its own and the VM disk becomes a disposable OS root. Your local ZFS is fine as-is, and you get to treat the VMs like cattle.

I know you said you were done with distributed storage, but I'd argue Longhorn is a VERY different animal from Ceph. Much simpler to set up and maintain, K8S native, and less network-sensitive (and with your 100G bond that's a non-issue anyway). When Ceph goes sideways your VM disks go with it, whereas a Longhorn problem is usually one degraded volume rebuilding while everything else keeps running. The catch is that replicas should really sit on independent local disks all on separate physical nodes. Putting them on NFS-backed VM disks means you now have a single point of failure at the NFS storage server.

How I'd set up your storage using the NFS server:

Local ZFS: VM disks, plus a dedicated dataset or disk per node for Longhorn replicas.
Longhorn: RWO PVs, plus RWX where the data is write-sensitive, like databases, Prometheus data, queues, anything doing real fsyncs.
NFS: RWX PVs with bulk, sequential or shared data that isn't fsync-heavy. Longhorn backs up to NFS very easily too, so put all backups here.


Also FYI, moving existing PVs off NFS is real migration work, not a config change.