PVE Cluster with PfSense and TrueNAS in Ha

[Urbaman]

Hi,

I'm new to Proxmox, and before going throw hardware selection, I'd be very glad to hear for tips and suggestions on what I'm thinking about.

I'd like to build a home 3-node cluster, basically for VM experimentation and various case scenarios.
The two most important scenarios at the moment would be:

• pfSense for my home lan/wlan
• TrueNAS

Given the home-based scenario, and some guides I found around, I'd go with each node such as this:

CPU: not important for the thread
Memory: not important for the thread
GPU: not important for the thread, probably will add one for Plex or similar
HBA1 (nonRAID for PVE):

2x 256/500GB SSD, RAIDZ1 (actual size 256/500GB)​

3x 2TB SSD, RAIDZ1 (actual size 4TB)​

HBA2 (nonRAID passthrough to TrueNAS):

3x 5TB SSD RAIDZ1 (actual size 10TB)​

dual Port NIC (passthrough to PfSense)
2x Single/Dual Port NIC (1 dedicated PVE, 1 dedicated VMs)

I add two diagrams (network and storage) to show how I am imagining the setup:


Things I much appreciate suggestions on are:

• general suggestions on the setup
• zfs vs ceph for both local redundancy (RAIDZ configs) and cluster/HA management: pros and cons
• is the scenario ok for PfSense HA? I should just need to make sure NIC/vNIC names are the same on the three nodes, right?
• no idea how/if it's working: TrueNAS HA. If PVE's HA is working, TrueNAS' VM is in HA, but the dedicated storage, being outside of PVE, is not. Is there a good/alternative way to also get disks attached to HBA2 redundant over the cluster, to get TrueNAS actually finding the files when it switches on secondary node?
• Is there some other hardware requirement, apart the non-RAID setting on the HBAs, given the scenario?

Thank you very much.

 

[mokaz]

Hey there,

I'm really far from a PVE storage expert, hence I won't be of much use there.
I went for something similar though, based on 3 nodes and currently using NFS shares for shared storage, to my tastes it's perfect.
Performance wise, well it's 1GE shared storage backed up on spinning disks so, it's ok nothing ground breaking.

NFS has been extremely easy to setup, rock solid (had so many iSCSI issues on my former hypervisor setup using the same NAS storage...) and I've done so through a dedicated VLAN present on L2 devices + dedicated NAS NIC for that purpose.

Concerning the NGFW thinggy, I'm not sure i'd push hardware resources up to the pfsense box. Perhaps you've got a good reason to do so but here, I'm passing trunk bridges to my NGFW which manages all the vLANs + L2 Switch management. Hence through virtio. Here as well all happy, could do all I wanted to along some twisted stuff and HA across the PVE Nodes (through dedicated HA VLANs per appliances families that needed HA).

Let me know,
Cheers,
m.

 

[Urbaman]

Hi there,

I'm actually deciding if I need dedicated NICs for the TrueNAS cluster like I do in Proxmox (3 NICS: management, cluster, glusterfs).

I'm settling myself on this topology:
VLAN1: home vlan, proxmox management
VLAN2: VMs (pfsense management, truenas management, ...)
VLAN3: proxmox cluster (no internet)
VLAN4: proxmox glusterfs (no internet)
VLAN5?: truenas cluster (no internet)
VLAN6?: truenas glusterfs (no internet)

Thanx for your interest, every suggestion is well accepted, I'll try to keep this updated as I go on planning and/or building it.