[SOLVED] Ceph and Storage Networking Configuration

[kameechewa]

I have a three node cluster. Each node is an HP 600 G6 DM with the built in 1GbE port and a 10GbE add on through the Flex IO port. Each nodes gigabit port is connected to my UniFi Dream Machine on the 10.0.2.0/24 (VLAN2) network. The three 10GbE connections are connected to a USW-Flex-XG. I plan to use the 10GbE for Ceph for VMs. I have created a Ceph network of 192.168.4.0/24 (VLAN4). The USW-Flex-XG has a 1gbit uplink the another switch and then back to the Dream Machine. Also connected to the USW-Flex-XG is my Unraid server that will be hosting mass storage for the time being. Unraid is currently on the 10.0.0.0/24 (VLAN1) network.

It is my understanding that traffic that crosses VLANs has to go back to the Dream Machine for rules to be processed. So I might need to move my Unraid server to the 10.0.2.0 network. What I want is Ceph and Unraid storage connections to stay on the USW-Flex-XG as it's 10Gig. Proxmox web interface, internet, and VMs, can all go through the 1Gig interface. My problem is that I don't know how to configure that in Proxmox or what changes may need made in UniFi.

Edit: I ended up figuring it out. I did the following:

1. Select a node
2. Click Network
3. Click Create
4. Click Linux Bridge
5. Name: vmbr4
6. Bridge Ports: enp3s0
7. IP: 192.168.4.7x/24
8. Comment: VLAN4
9. Click Create
10. Click Create
11. Click Linux Bridge
12. Name: vmbr1
13. Bridge Ports: enp3s0.1
14. IP: 10.0.0.7x/24
15. Comment: VLAN1
16. Click Create
17. Click Apply Configuration
18. Click Yes

On my switch I set the 10 gig ports for the three nodes to default to VLAN4 and allowed VLAN1 as well. iperf3 showed me getting 10Gig on Ceph interfaces and 2.5Gig to Unraid. Doing a radon benchmark gave me about 600-700MBps peak and 460MBps average. Each node has one 512GB NVMe used exclusively for Ceph, so only 3 OSDs.