In the process of putting together a plea for help as to how to get my cluster back together (copies of /etc/network/interfaces, /etc/hosts, /etc/corosync/corosync.conf for each of my 3 nodes) I found the mismatches and remembered to increment the config version up one. Now corosync is back. Ceph survived throughout. I have 3 nodes, each with a fast interface and a slow interface. All address are 192.168.x.x:
I added the 25g cards to try to make a 25gig ring network for ceph. The node with the four 2.5g ports is a router box that won't take a 25g card and will hopefully be replaced with full server soon. I'd consider putting the fast cards on their own VLAN but that's how I broke things to begin with. I also want to be able to access my ceph storage faster than 1 gigabit and I don't have any more ports. I could scrape together some old 10 gig cards and DACs to add those as a third link. Can I dual purpose the fast links as ceph and VM virtual NICs?
Whatever interface the VMs/CTs use, it has to have the same name across nodes so they can be migrated.
Is there a suggestion or best practice I can follow to get these two networks separated. Is there really any advantage to all this work right now or should I wait until I get the third node as a three NIC setup (Gbe management, SFP+ VM, SFP28 ceph)?
For context, I am using a Ubiquiti Hi-Capacity Aggregation switch for my VLANs. It seems pretty easy to set a port to a VLAN but then that seems to prevent me from running another virtual interface on the same port for a different VLAN. Documentation is scarce.
| Node | Fast | Slow |
| pve1 (want to change name to alpha) | vmbr1 from bond0 from eth1, eth2, eth3 (3x 2.5g LAG) 192.168.1.2 | vmbr0 from eth0 (2.5g running at 1gbe) at 192.168.3.2 |
| beta | vmbr1 from enp40s0f1np1 (25g) on 192.168.1.41 | vmbr0 from enp3s0 (1gbe) at 192.168.1.5 |
| gamma | enp4s0f1np1 (25g) on 192.168.1.42 | vmbr0 from eno1 (1gbe) at 192.168.1.6 |
Whatever interface the VMs/CTs use, it has to have the same name across nodes so they can be migrated.
Is there a suggestion or best practice I can follow to get these two networks separated. Is there really any advantage to all this work right now or should I wait until I get the third node as a three NIC setup (Gbe management, SFP+ VM, SFP28 ceph)?
For context, I am using a Ubiquiti Hi-Capacity Aggregation switch for my VLANs. It seems pretty easy to set a port to a VLAN but then that seems to prevent me from running another virtual interface on the same port for a different VLAN. Documentation is scarce.
