I am somewhat new to clustering and best practices when setting up networks for clusters. I'm wanting to setup a proxmox cluster with ceph and preferably as budget friendly as possible. This will be used for homelab use but, will also be serving critical services like Plex. 
I have a Freenas box which will hold all of the "user" data and the cluster will only serve up the VM's and containers. I have the ability for 4 LFF drives and 2 SFF drives. I was wanting to start out with 1 SFF SSD as the OS boot drive, 1 SFF SSD for journal drive, and 2 LFF spinners for OSD's per node.
I have 3 identical nodes. The specs are as follows
Supermicro X8STi-3F boards
Each board has 2 on-board Gig nics
I also have 3 intel x520-da2 10gb cards
I do not have a 10gb switch and most are probably beyond the budget, so i was interested in doing a mesh network as described in the wiki.
The next option is to buy three 4 port network cards and use a gigabit switch for the related networks. I'm kind of at a loss on where to go with the network to maximize bandwidth and future upgrades using the hardware I currently have. Any advise would be greatly appreciated.
I have a Freenas box which will hold all of the "user" data and the cluster will only serve up the VM's and containers. I have the ability for 4 LFF drives and 2 SFF drives. I was wanting to start out with 1 SFF SSD as the OS boot drive, 1 SFF SSD for journal drive, and 2 LFF spinners for OSD's per node.I have 3 identical nodes. The specs are as follows
Supermicro X8STi-3F boards
Each board has 2 on-board Gig nics
I also have 3 intel x520-da2 10gb cards
I do not have a 10gb switch and most are probably beyond the budget, so i was interested in doing a mesh network as described in the wiki.
The next option is to buy three 4 port network cards and use a gigabit switch for the related networks. I'm kind of at a loss on where to go with the network to maximize bandwidth and future upgrades using the hardware I currently have. Any advise would be greatly appreciated.
