## Looking for feedback on a Proxmox + ZFS + VirtIO-SCSI + NFS HA storage architecture
Hello,
I am looking for some feedback on the feasibility of a storage architecture we are considering for a small startup environment.
We are trying to build a reasonably high-performance and highly available storage platform without going into the €200k–€400k enterprise SAN/storage-system territory.
The basic idea is to use **3 Proxmox VE nodes**, each with local NVMe storage configured with ZFS, and provide storage to our VM environment through an **NFS VM**.
The architecture we are considering is roughly:
**3× Proxmox VE nodes → ZFS → VirtIO-SCSI → NFS VM → NFS → VM/compute cluster**
The goal is to have the storage layer highly available while keeping the architecture relatively simple and using commodity hardware.
### Proposed architecture
Each Proxmox node would have its own NVMe drives configured as a ZFS mirror/RAID configuration.
The three nodes would be roughly equivalent:
* Proxmox Node 1 — ZFS storage
* Proxmox Node 2 — ZFS storage
* Proxmox Node 3 — ZFS storage
The storage would then be presented to an NFS VM through **VirtIO-SCSI**.
The NFS VM would provide the shared filesystem to the compute/engineering VMs.
We would also like the NFS service itself to be HA, so that failure of one Proxmox/storage node does not result in the loss of the storage service.
The simplified architecture is:
The main question is whether **VirtIO-SCSI is appropriate for this design**, or whether we are approaching the problem incorrectly.
I understand that VirtIO-SCSI is essentially a virtual SCSI interface between the hypervisor and the guest VM. It does not, by itself, provide shared storage or replication.
Therefore, the important question is what should sit underneath the NFS VM's virtual disks.
Would something like the following be a sensible architecture?
**ZFS → replicated/shared block storage → VirtIO-SCSI → NFS VM → NFS clients**
Or would it be better to avoid putting NFS inside a VM altogether and expose storage through another mechanism?
### Requirements
Our requirements are approximately:
* **60 TB+ usable capacity**
* ~**100 VMs**
* 24/7 operation
* High IOPS / low latency
* Workload consisting of many small files
* Large burst workloads, typically **100 GB–1 TB**
* Shared home directories
* NFS shares used for exchanging data between VMs
* **25 GbE networking**
* HA at the storage/service level
* Preferably no single storage node being a single point of failure
* Reasonable recovery/failover time
* As much performance as possible from local NVMe storage
The hardware we are currently considering is along the lines of:
* **Supermicro H14SSL-N / ASRock Rack GENOAD8X-2T/BCM**
* **AMD EPYC 9175F**
* **128 GB RAM initially**
* Enterprise/datacenter NVMe
* **Intel X710 2× SFP28 25 GbE**
### Questions
1. **Is this architecture actually feasible?**
2. Is using **VirtIO-SCSI for the NFS VM's storage** a reasonable approach for this workload?
3. What would be the recommended mechanism underneath VirtIO-SCSI to provide replication/HA between the three Proxmox nodes?
4. Would **LINSTOR/DRBD**, Ceph, or another solution make more sense than trying to build this around ZFS?
5. Can ZFS be sensibly combined with a replicated block-storage layer in this scenario, or would that introduce unnecessary complexity?
6. Is running the NFS server as a VM a reasonable design, assuming the underlying storage is properly replicated?
7. Would you instead recommend **three independent NFS servers with some form of HA/failover**, rather than one NFS VM that can migrate between Proxmox nodes?
8. Are there any major performance or reliability problems I am overlooking with this approach?
I am particularly interested in feedback from people who have actually deployed **Proxmox + ZFS + NFS + HA** in production.
The goal is not to build the most expensive or theoretically perfect storage system possible. We are trying to find a practical architecture that gives us **good NVMe performance, 25 GbE connectivity, HA, and ~60 TB+ of usable storage** while keeping the cost reasonable.
Any criticism of the architecture is welcome. I would rather find out that the design is fundamentally wrong now than after purchasing the hardware.
Hello,
I am looking for some feedback on the feasibility of a storage architecture we are considering for a small startup environment.
We are trying to build a reasonably high-performance and highly available storage platform without going into the €200k–€400k enterprise SAN/storage-system territory.
The basic idea is to use **3 Proxmox VE nodes**, each with local NVMe storage configured with ZFS, and provide storage to our VM environment through an **NFS VM**.
The architecture we are considering is roughly:
**3× Proxmox VE nodes → ZFS → VirtIO-SCSI → NFS VM → NFS → VM/compute cluster**
The goal is to have the storage layer highly available while keeping the architecture relatively simple and using commodity hardware.
### Proposed architecture
Each Proxmox node would have its own NVMe drives configured as a ZFS mirror/RAID configuration.
The three nodes would be roughly equivalent:
* Proxmox Node 1 — ZFS storage
* Proxmox Node 2 — ZFS storage
* Proxmox Node 3 — ZFS storage
The storage would then be presented to an NFS VM through **VirtIO-SCSI**.
The NFS VM would provide the shared filesystem to the compute/engineering VMs.
We would also like the NFS service itself to be HA, so that failure of one Proxmox/storage node does not result in the loss of the storage service.
The simplified architecture is:
The main question is whether **VirtIO-SCSI is appropriate for this design**, or whether we are approaching the problem incorrectly.
I understand that VirtIO-SCSI is essentially a virtual SCSI interface between the hypervisor and the guest VM. It does not, by itself, provide shared storage or replication.
Therefore, the important question is what should sit underneath the NFS VM's virtual disks.
Would something like the following be a sensible architecture?
**ZFS → replicated/shared block storage → VirtIO-SCSI → NFS VM → NFS clients**
Or would it be better to avoid putting NFS inside a VM altogether and expose storage through another mechanism?
### Requirements
Our requirements are approximately:
* **60 TB+ usable capacity**
* ~**100 VMs**
* 24/7 operation
* High IOPS / low latency
* Workload consisting of many small files
* Large burst workloads, typically **100 GB–1 TB**
* Shared home directories
* NFS shares used for exchanging data between VMs
* **25 GbE networking**
* HA at the storage/service level
* Preferably no single storage node being a single point of failure
* Reasonable recovery/failover time
* As much performance as possible from local NVMe storage
The hardware we are currently considering is along the lines of:
* **Supermicro H14SSL-N / ASRock Rack GENOAD8X-2T/BCM**
* **AMD EPYC 9175F**
* **128 GB RAM initially**
* Enterprise/datacenter NVMe
* **Intel X710 2× SFP28 25 GbE**
### Questions
1. **Is this architecture actually feasible?**
2. Is using **VirtIO-SCSI for the NFS VM's storage** a reasonable approach for this workload?
3. What would be the recommended mechanism underneath VirtIO-SCSI to provide replication/HA between the three Proxmox nodes?
4. Would **LINSTOR/DRBD**, Ceph, or another solution make more sense than trying to build this around ZFS?
5. Can ZFS be sensibly combined with a replicated block-storage layer in this scenario, or would that introduce unnecessary complexity?
6. Is running the NFS server as a VM a reasonable design, assuming the underlying storage is properly replicated?
7. Would you instead recommend **three independent NFS servers with some form of HA/failover**, rather than one NFS VM that can migrate between Proxmox nodes?
8. Are there any major performance or reliability problems I am overlooking with this approach?
I am particularly interested in feedback from people who have actually deployed **Proxmox + ZFS + NFS + HA** in production.
The goal is not to build the most expensive or theoretically perfect storage system possible. We are trying to find a practical architecture that gives us **good NVMe performance, 25 GbE connectivity, HA, and ~60 TB+ of usable storage** while keeping the cost reasonable.
Any criticism of the architecture is welcome. I would rather find out that the design is fundamentally wrong now than after purchasing the hardware.
