LXCs can see parent node but not gateway/internet

[Tydorius]

Solution:
Update for anyone else that comes here - I found the issue, it was a problem outside of Proxmox.

I had recently changed internet providers, but opted to keep my router. So instead of being LAN > Router > Cable modem, I was running LAN > Router > Fiber Modem / Gateway combo.

The gateway was within the 192.168.0.0/23 range. For some reason even when I set everything to /24 it still caused problems. But once I switched my router and LAN over to 10.0.0.0/16 it all simply started working with my existing configurations.

I don't know exactly why it caused trouble, on /24 it shouldn't have, but I think my router was trying to play nice with the gateway through some automated function and so it was not wanting to pass traffic.


I'll edit this into the original post as well and leave it. My new network is:

Fiber Gateway: 192.168.1.253/23
Router: 10.0.0.1/16

Proxmox: 10.0.1.11 - .15/16

LXCs: 10.0.1.21-.30/16


Everything else was simply kept the same.

Original post:

I am probably forgetting something really simple, but for the life of me I can't sort this out. I'll give as much information as I can. Hopefully someone can help me sort this out, I've been bashing my face against it for a few days now.

Environment:
5x Proxmox nodes
2x LXC per node, total of 10x LXCs.

Each LXC is able to ping only its parent node by hostname or IP.
Each node can ping its own LXCs and the other nodes, but not LXCs on other nodes. True for both IPs and hostnames.
Nodes can ping/resolve gateway and all external addresses.
LXCs can not ping or resolve the gateway or any external addresses.
Other machines on the network can ping both gateway and the cluster nodes, but can not ping the LXCs.

I will use the first node as the example, as they are all configured identically minus their names/ips.

Parent node for our example: 192.168.0.11 hl-pm-01.hls.cluster
LXC for our example: 192.168.0.21 dock-11.hls.cluster

Both parent nodes and LXCs have cluster details in their host files. I am not running a local DNS server at this time.

ip a output for parent node, hl-pm-01:

root@hl-pm-01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 6c:4b:90:a6:fa:f2 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3c:91:80:38:41:cb brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6c:4b:90:a6:fa:f2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.11/23 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::6e4b:90ff:fea6:faf2/64 scope link
       valid_lft forever preferred_lft forever
21: veth1002i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:a7:75:6b:24:51 brd ff:ff:ff:ff:ff:ff link-netnsid 1
22: veth1001i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:ca:c3:26:80:a7 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Node hl-pm-01 DNS settings:
Search domain: hls.cluster
DNS server: 192.168.0.1

Network on LXC:



Note that I am using /23 because I'll be using the 192.168.1.xx range in the future for an RPi cluster.

DNS on LXC is set to "use host settings" for both server and domain.

ip r on parent node:

default via 192.168.0.1 dev vmbr0 proto kernel onlink
192.168.0.0/23 dev vmbr0 proto kernel scope link src 192.168.0.11

ip r on LXC:

default via 192.168.0.1 dev eth0 proto static
192.168.0.0/23 dev eth0 proto kernel scope link src 192.168.0.21

Firewall is enabled with default settings on parent node. Disabling has no effect on network traffic.
Firewall is disabled on the LXC.

ip a on LXC:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:72:fb:b2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.21/23 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe72:fbb2/64 scope link
       valid_lft forever preferred_lft forever

I have static IPs assigned by my router based on the physical host MAC for each of the cluster nodes. It uses DHCP for everything else but I have confirmed there are no conflicts. My original plan was to have them connect, then tell the router to preserve those IPs for the LXCs. However, as far as my router is concerned, they are not on the network at all.

The LXCs are provisioned by Terraform. An example network block is below with variables replaced with static values:

  network {
        name = "eth0"
                bridge = "vmbr0"
                ip = "192.168.0.21/23"
                gw = 192.168.0.1
        }

 

[ceelight]

There's no gateway defined on the LXC example? Is that so?

 

[Tydorius]

There's no gateway defined on the LXC example? Is that so?

That was my mistake, I missed a line. It has a gateway as:

default via 192.168.0.1 dev eth0 proto static

I fixed it in the main post.

 

[Tydorius]

Update for anyone else that comes here - I found the issue, it was a problem outside of Proxmox.

I had recently changed internet providers, but opted to keep my router. So instead of being LAN > Router > Cable modem, I was running LAN > Router > Fiber Modem / Gateway combo.

The gateway was within the 192.168.0.0/23 range. For some reason even when I set everything to /24 it still caused problems. But once I switched my router and LAN over to 10.0.0.0/16 it all simply started working with my existing configurations.

I don't know exactly why it caused trouble, on /24 it shouldn't have, but I think my router was trying to play nice with the gateway through some automated function and so it was not wanting to pass traffic.

I'll edit this into the original post as well and leave it. My new network is:

Fiber Gateway: 192.168.1.253/23
Router: 10.0.0.1/16
Proxmox: 10.0.1.11 - .15/16
LXCs: 10.0.1.21-.30/16

Everything else was simply kept the same.