Network Design for OpenShift

[dejot]

Hello! My idea was to create home lab with OpenShift but I found issues with Proxmox Networking and Routing.
Setup looks following:
-----------
Home network: 10.10.10.[x]
Proxmox Server: 10.10.10.122
Home PC: 10.10.10.108
TP Link Router: 10.10.10.1
-----------
bastion: (jump host in two networks) 192.168.70.4 and 10.10.10.111
services: (DNS bind, haproxy) 192.168.70.5
master0: 192.168.70.11
master1: 192.168.70.12
master2: 192.168.70.13
worker1: 192.168.70.14
-----------

Goal: Access OpenShift Console Web Page from Home PC. How to achieve that?

What I tried so far?
Routing on the TP Link Router level.

Destination IP Address	Subnet Mask	Gateway	Status

192.168.70.0

255.255.255.0

10.10.10.111

Enabled

Edit

With the following routing setup I was able to ping only bastion. When I tried to ping other VMs like services or master0 I received timeout.

My network setup from proxmox server:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.122/24
        gateway 10.10.10.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.70.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '192.168.70.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '192.168.70.0/24' -o vmbr0 -j MASQUERADE

Any ideas?

 

[gurubert]

Why does the Proxmox host have interfaces in both networks when the bastion host is the router between them?
Have you enabled IP forwarding in the bastion host?
Do the other OpenShift hosts use the bastion host as default gateway?

 

[dejot]

Why does the Proxmox host have interfaces in both networks when the bastion host is the router between them?
Have you enabled IP forwarding in the bastion host?
Do the other OpenShift hosts use the bastion host as default gateway?

My idea was to enable internet traffic for this subnet 192.168.70.0/24 via NAT. I needed that, because OpenShift during installation fetching external sources for image download. That's why I created vmbr1 with NAT routing. Routing via bastion and TP Link Router is quite new idea, as I realized that access to OCP console via web browser on Home PC is needed for work with home lab.

Thank you for mentioning IP forwarding for the bastion and using bastion IP as a gateway. That helped and I am able to ping other VMs in the network.

Now the question is? How to enable internet traffic on hosts like master0, master1... when default gateway for them is now bastion IP. I presume additional bastion (router) configuration is needed.

 

[gurubert]

If your OpenShift VMs already have Internet access via the Proxmox host why don't you set the route to 192.168.70.0/24 via 10.10.10.122 in your TP link router?
You should change your MASQUERADE rules to exclude 10.10.10.0/24 and then your two networks are able to see each other.
Why did you introduce the bastion host?

 

[dejot]

Ok, I've set following rule on my TP Link Router and I am able to ping all 192.168.70.0/24 VMs from Home PC.

Destination IP Address	Subnet Mask	Gateway	Status	Edit

192.168.70.0

255.255.255.0

10.10.10.122

Enable

Edit

I left MASQUERADE rules as it was. I don't know how to set/prepare exclude rule. I will appreciate help here.

 

[gurubert]

Something like

iptables -t nat -A POSTROUTING -s '192.168.70.0/24' -d 10.10.10.0/24 -o vmbr0 -j RETURN

before the "real" masquerading rule.

 

[dejot]

I've added rule which you mentioned before, but network 192.168.70.0/24 can't see (ping) network 10.10.10.0/24.
Traffic from 10.10.10.0/24 to 192.168.70.0/24 works fine.

Now my Proxmox network config looks like this:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.122/24
        gateway 10.10.10.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.70.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        
        iptables -t nat -A POSTROUTING -s '192.168.70.0/24' -d 10.10.10.0/24 -o vmbr0 -j RETURN
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '192.168.70.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '192.168.70.0/24' -o vmbr0 -j MASQUERADE
        #post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
        #post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE