Question regarding network interfaces and gateway

[IroesStrongarm]

I'm at a clear loss in my understanding for hoping for some help here. I have a three node cluster. Each node has it's main NIC used for management and vmbr0, and then another just currently for corosync and migration. I'm adding a third NIC in that will be corosync only. I noticed yesterday that my router doesn't show any clients on the VLAN 192.168.2.x. There should be three clients there. Also, I can ping all clients from my laptop on 192.168.1.x and the logs seem to confirm the nodes are communicating over 192.168.2.x. Here's my config for a node:

auto lo
iface lo inet loopback

iface enp1s0f0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.22/24
        gateway 192.168.1.1
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0

iface enxca8b8b27e813 inet manual

auto enp1s0f1
iface enp1s0f1 inet static
        address 192.168.2.8/24

I realized I never added the gateway of 192.168.2.1 under enp1s0f1. I tried doing this in the proxmox GUI and it said that the gateway is already set through vmbr0. I fully admit my lack of understanding here of networking and what should and shouldn't be. Do I not need to add that gateway for the VLAN? As mentioned, the router doesn't acknowledge any clients on 192.168.2.x. Also, I can't access the management GUI by going to https://192.168.2.8:8006 as I understand I should be able to (my understanding here could just be wrong of course).

Any help here would be appreciated. Thank you.

 

[Folke]

First off, I think you are mixing up vlans and subnets as your config never touches anything related to vlans.

Not defining a gateway isn't necessarily wrong, this only means that only means that your devices don't know where to route their packages if the source address is 192.168.2.x/24 but the target is not in that network. Since this is only used for communication between the nodes, this will probably not be the case.

As for the router not displaying the devices from the 2.x subnet, the router can only display devices it knows about, be it via dhcp queries or because it receives packages from that device. Neither of those apply, because the nic's have a static ip and aren't even aware of the router (no gateway).
Additionally your router probably isn't even member of the 192.168.2.x/24 subnet, so all packets arriving there will be dropped.

I think you are not able to reach your servers via 192.168.2.x because, same as the router, you are not part of that subnet.
Your computer sends the packets to the router, because it doesn't know where they have to go, and it in turn isn't aware that the subnet exists and drops them, because from it's perspective they don't belong to your home net, but also can't be routed through the internet because they belong to a private ip range[0]

Thinking about it, if the proxmox instances have multiple physical nic's, have you connected them to same physical network
or have you created an additional physical for the nodes to speak to another?

[0] https://en.wikipedia.org/wiki/Reserved_IP_addresses

 

[IroesStrongarm]

Thanks for taking the time. It does sound like I'm confusing VLANs and Subnets so I should definitely read up more on that. I have created the subnets on my router, and I know that the clients that connect on those subnets do show up as they pull their addresses from the DHCP server.

While it's probably not important for them to show up, and after researching it looks like you aren't supposed to define other gateways in Proxmox, or a Linux host in general, for my own info, would defining the gateway with those relevant NICs make them show up to the router?

As for your last question, hoping I understand it correctly, I have the main bridge/management network connected to one switch, along with other devices. The migration NICs is connected to a separate, dedicated switch, that's connected back to my network. The ports are set to only connect to the migration subnet. The corosync NICs are connect to yet another dedicated switch, that like the above, only give out the corosync subnet.

 

[IroesStrongarm]

First off, I think you are mixing up vlans and subnets as your config never touches anything related to vlans.

Just as follow up, I looked it up, and specifically how it's done on my Unifi network. It looks like I have created them as VLANs. I followed setup guides awhile back for ensuring that my IoT network can't initiate conversation internally out of its VLAN, but communications can come from the main network. I didn't block traffic in both directions on these two new networks.

I will say, just because I've made them VLANs doesn't mean I understand it all, so hope I'm not coming off that way.

 

[IroesStrongarm]

Would I theoretically want to tell the switch to allow all networks on the physical port, and then change my config file as follows:

auto lo
iface lo inet loopback

iface enp1s0f0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.22/24
        gateway 192.168.1.1
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0

iface enxca8b8b27e813 inet manual

iface enp1s0f1 inet manual

auto enp1s0f1.2
iface enp1s0f1.2 inet static
        address 192.168.2.8/24