No internet access in VM through SDN VNet

[t.desmet]

Hi, first I want to thank everyone who answers on this thread and assists me with my issue!

GOAL
Have SDN VNet's for communication between VMs on different compute nodes and access the internet through SNAT on the VNet.

ISSUE
VMs configured with a VNet can't access the internet ping 8.8.8.8 fails.
Communication between VMs on the same VNet WORKS just fine, on both the same and different compute nodes. So the issue is only related to accessing the internet.
As you can see in the screenshot ping to 10.10.0.52 (VM on CN002 node) works. Any other ping fails.



INFO
I've tested with both VLAN and VXLAN SDN zone types, result is the same. The ideal bride/peers to use would be vmbr1 (peers 10.1.20.101...) but I've tested also with vmbr0 as I've notice that my vmbr1 for some reason cannot ping the internet too on the physcial node.



Network configuration on the nodes
Each now has the same network configuration only the IP's range from *.*.*.101 to *.*.*.104.

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4

# Management Network (VLAN 110)
auto bond0.110
iface bond0.110 inet static
        address 10.1.10.101/24
        gateway 10.1.10.1
        vlan-raw-device bond0

auto vmbr0
iface vmbr0 inet static
        address 10.1.10.101/24
        gateway 10.1.10.1
        bridge-ports bond0.110
        bridge-stp off
        bridge-fd 0

# Cluster Communication Network (VLAN 120)
auto bond0.120
iface bond0.120 inet static
        address 10.1.20.101/24
        vlan-raw-device bond0

auto vmbr1
iface vmbr1 inet static
        address 10.1.20.101/24
        bridge_ports bond0.120
        bridge_stp off
        bridge_fd 0

# Corosync Network (VLAN 130)
auto bond0.130
iface bond0.130 inet static
        address 10.1.30.101/24
        vlan-raw-device bond0

# Ceph Public Network (VLAN 140)
auto bond0.140
iface bond0.140 inet static
        address 10.1.40.101/24
        vlan-raw-device bond0

auto vmbr2
iface vmbr2 inet static
        address 10.1.40.101/24
        bridge_ports bond0.140
        bridge_stp off
        bridge_fd 0

# Ceph Internal Network (VLAN 150)
auto bond0.150
iface bond0.150 inet static
        address 10.1.50.101/24
        vlan-raw-device bond0

auto vmbr3
iface vmbr3 inet static
        address 10.1.50.101/24
        bridge_ports bond0.150
        bridge_stp off
        bridge_fd 0

# Public Network for VMs (VLAN 200)
auto bond0.200
iface bond0.200 inet manual
        vlan-raw-device bond0

auto vmbr4
iface vmbr4 inet manual
        bridge_ports bond0.200
        bridge_stp off
        bridge_fd 0

iface eno3 inet manual

iface eno4 inet manual


source /etc/network/interfaces.d/*

Network Configuration on the Fortigate

 

[sw-omit]

A few questions just to check:
Is the proxmox-server plugged directly into the fortigate or via a switch? (expecting the former but just making sure)
If you plug a laptop/PC directly into that port 17/18 on the router, and set both your IP-range and VLAN to one of the PVE-ranges, can you ping the router and get to the internet?
Can you ping the router from the VM?


Also, while reading through things a bit more, I might have found your issue:
Looks like you are trying to assign VLAN's to a bridge that already has VLAN's on it, meaning you're trying to put VLAN in VLAN.
Make a bridge port without any vlan on it, doesn't even need an IP on it, then put the SDN VLAN-zone on that, something like this.

auto vmbr5
iface vmbr5 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
#SDN-Bridge interface

 

[t.desmet]

On ports 17 and 18 on the Fortigate are 2 Cisco Nexus switches connected with an aggregated link for HA and failover.
We do have some other systems running through those switches all working perfectly. I'm also able to access the internet and ping the FortiGate/gateway from inside the physical Proxmox VE nodes. All that is working fine.

Also when configuring a VM with vmbr4 (public network) it works fine, I can access the internet from inside the VM, assign a public IP, and access it.
The only issue is when working with the SDN VNets. When creating an SDN simple zone it also works, I can access the internet from inside the VM, the only issue then is that a SIMPLE zone is internal node only, so it can't access the VMs running on other PVE nodes. When configuring a SDN zone with VLAN or VXLAN I can communicate between VMs on different nodes but then the internet access doesn't work anymore.

I've read online that Proxmox only takes care of virtual gateways for simple and EVPN types, and not for VLAN or VXLAN so maybe that could be an issue. But then there is the question of how to configure Proxmox so it works.

To clarify, I'm familiar with how OpenStack handles software-defined networks and I was hoping that Proxmox could achieve a similar functionality.

I need a Vnet so that Kubernetes masters and workers on different PVE nodes (for HA) can communicate with each other while still being able to access the internet for updates and accessing resources etc... Then I would configure some load balancing VMs with haproxy and a public IP to proxy traffic to the Kubernetes ingress controller and API.

An IP is configured on the vmbr1 interface as I expected that was required so that the physical nodes can reach other PVE nodes for SDN traffic.

 

[sw-omit]

The simple-on-vmbr1 and vmbr4-direct works, because then they just have that 1 VLAN set up
with vlan/xvlan-on-vmbr1 it does not work to get internet, as you're putting a VLAN inside of an already VLAN'd network, so running QinQ [1] / [2]
Because the Fortigate isn't expecting / looking for QinQ, and only sees the "outer" VLAN, it can't do anything with the "inner" VLAN.
Because the setups on the multiple proxmox-servers is the same, it does work there.

So like I said before, create a new bridge on the bond, WITHOUT a vlan tag, so that the VLAN-SDN is the one to add the tag, which in turn causes it to be only a single VLAN-tag going to the switch.

[1] https://info.support.huawei.com/info-finder/encyclopedia/en/QinQ.html
[2] https://en.wikipedia.org/wiki/IEEE_802.1ad

 

[t.desmet]

I've tried what u suggested. Created a vmbr5 bridge on bond0 (untagged) and configured it so that SDN places the tag but the same issue persists. Also, it doesn't seem the approach I'm looking for but was worth trying it out.

This approach would limit me to a single VNet over VLAN 120. I'm looking for a way that I can use VLAN 120 as a transport network for SDN traffic. In this example case I currently only configure a single VNet but in the end, this will be multiple VNet's. I'm not sure if the SDN VLAN type is even the right type for my use case. Doubting it.

 

[t.desmet]

Actually what in the background should happen is that Proxmox creates a gateway for every VNet that translates the outgoing requests from inside the VNet to the internet as one-way traffic. I've done this manually by creating a VM that has both an interface on the VNet and an interface on the public network. Then assign a public IP to it and enable IP forwarding. Then point the VMs that only have an interface on the VNet to the gateway VM. This way I'm able to ping external IPs from inside VMs that only have a private VNet. But of course, this is not an ideal setup. Also facing a small issue with this setup that the VMs can't resolve domains for some reason while they can ping to Google DNS servers.

But I would expect by enabling SNAT on the VNets that Proxmox would take care of a similar setup/configuration.