Advise: shared vm storage using nfs

D

Daxcor

Active Member
Oct 31, 2021
58
5
28
59
Hello,

I have been running a 4 node cluster for a few years now. I have changed my storage options a few times. the last change I made was from ceph rbd to local zfs raid 1. This means that I can't use the HA features of proxmox 9.2 effectively. This begs the question why don't you stay with ceph. I have borked ceph twice now. I am not skilled enough to reliably admin this kind of technology. I don't have the budget for the recommended hardware and paid consultants to make it work.

Local storage can and does work. However, I want to get the communities opinion on using nfs 4.2 as a shared storage device for my vms. This seems like it would be a good compromise for complexity and opening HA in proxmox. Looking for the good, the bad and the ugly of this path.

Thanks.
 
Hi @Daxcor,

NFS is a good choice for your use case. It is used for shared storage by a great number of entities, from home users to multi-billion-dollar corporations.

NFS has been around for 42 years. The good, bad, and ugly have probably been covered millions of times by now.

The only question you need to answer is where you plan to run the NFS server. A separate appliance makes things clean and easy. Running it as a VM will introduce dependencies that may be perfectly acceptable for a homelab, but may not be acceptable for a business.

The great thing about it is that you don't need to pick one or the other. You can use local storage and NFS in parallel.

Cheers.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Daxcor
NFS works fine, but it has limitations. Each read and write is a network hop. You will benefit from (and likely appreciate) at least a 10g network. NFS specifically benefits enormously from a proper SLOG, using an nvme drive or even better, an optane drive. You don't need a big drive for SLOG. Even 64 GB is enough. But you want power loss protection and high endurance. A dedicated, non routed storage VLAN or network also helps a lot. And setting up your NAS storage pool properly helps a lot too. In non ZFS systems, you'll benefit from raid 10. In TrueNAS, I use three mirrored vdevs. Raidz1 is not the answer
 
  • Like
Reactions: Daxcor
I am using a dual 100G lacp bonded network, with a zfs raid 10 on nvme drives on the nfs server, which is stand alone hardware. Based on your recommendations, I think i check the boxes.

Not sure if it matters, but my vms are for kubernetes nodes. They don't use local storage, they use nfs for the pod workloads.

Thanks
 
Honestly in that case, attach a virtual disk to each worker node in the cluster and install longhorn. You will NOT be disappointed. You can replicate those disks to shared storage as backup. Longhorn will make your storage HA across all the worker nodes. You can create separate Longhorn nodes if you want, but it's not necessary. Pods will migrate between nodes seamlessly
 
You must log in or register to reply here.
Top Bottom
Back