VM not connecting /pinging other hosts / Internet

[gggunk]

Hi, can someone help me figure out why my pfSense VM is not connecting to other Proxmox Hosts and to WAN, please?

Problem
VM (pfSense) on Proxmox VE 6 unable to connect to Internet, or ping any IP beyond the Proxmox Host on which it is running. My unskilled analysis suggests that the packets are not leaving the Proxmox Host. (But I could be wrong.)

Topology
Colo Provider > Router > 1G Core Switch > Proxmox Hosts (Mgmt Network for Cluster and Corosync)
10G Pvt Switch > Proxmox Hosts (Backbone Network for Proxmox) > OVS > VMs

Network status
VM pings Proxmox Host LAN and WAN IPs
VM does NOT ping directly connected L3 Switch or other Proxmox Hosts or connect to Internet
L3 Switch pings OVS IP addresses on Proxmox Host, but VM IPs NOT reachable

Proxmox host config
- 1G NIC dedicated to Proxmox Mgmt / Cluster - 192.168.15.X
- 4 x 10G NICs bonded to 40G Switch for backbone - 192.168.20.X
- OVS WAN IP - 192.168.20.248 (Gateway: 192.168.20.254)
- OVS VLAN101 for VM networking - 10.0.3.X

Proxmox host etc/hosts/interfaces file
allow-vmbr101 bond103
iface bond103 inet manual
ovs_bonds enp195s0f4 enp195s0f4d1 enp196s0f4 enp196s0f4d1
ovs_type OVSBond
ovs_bridge vmbr101
mtu 9214
ovs_options bond_mode=balance-tcp other-config:lacp-time=fast lacp=active
pre-up ( ip link set enp195s0f4 mtu 9214 && ip link set enp195s0f4d1 mtu 9214 && ip link set enp196s0f4 mtu 9214 && ip link set enp196s0f4d1 mtu 9214 )

auto lo
iface lo inet loopback

iface enp193s0f0 inet manual

allow-vmbr1 enp193s0f1
iface enp193s0f1 inet manual
ovs_type OVSPort
ovs_bridge vmbr1

iface enp193s0f1 inet manual

iface enp195s0f4 inet manual

iface enp195s0f4d1 inet manual

iface enp196s0f4 inet manual

iface enp196s0f4d1 inet manual

allow-vmbr101 vet103
iface vet103 inet static
address 192.168.20.103
netmask 24
ovs_type OVSIntPort
ovs_bridge vmbr101
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.20.0/24' -o vmbr101 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.20.0/24' -o vmbr101 -j MASQUERADE
#Interface VMs and Host

allow-vmbr101 vlan103
iface vlan103 inet static
address 10.0.3.250
netmask 24
ovs_type OVSIntPort
ovs_bridge vmbr101
ovs_options tag=103
#LAN Interface for Guest VMs and Containers

auto vmbr0
iface vmbr0 inet static
address 192.168.15.3
netmask 255.255.255.0
gateway 192.168.15.254
bridge-ports enp193s0f0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.15.4
netmask 24
ovs_type OVSBridge
ovs_ports enp193s0f1

auto vmbr101
iface vmbr101 inet static
address 192.168.20.3
netmask 24
ovs_type OVSBridge
ovs_ports bond103 vet103 vlan103
mtu 9214
#OVS Bridge

Proxmox Host IP Route result
root@host1:~# ip r
default via 192.168.15.254 dev vmbr0 onlink
10.0.3.0/24 dev vlan103 proto kernel scope link src 10.0.3.250
192.168.15.0/24 dev vmbr0 proto kernel scope link src 192.168.15.3
192.168.15.0/24 dev vmbr1 proto kernel scope link src 192.168.15.4
192.168.20.0/24 dev vet103 proto kernel scope link src 192.168.20.103
192.168.20.0/24 dev vmbr101 proto kernel scope link src 192.168.20.3

Please let me know if you need any more info to figure out the config issue.

 

[Alwin]

I hope you could resolve it by now. But if not, here are my two cents.

L3 Switch pings OVS IP addresses on Proxmox Host, but VM IPs NOT reachable

AFAIU, the VMs are behind NAT. The VMs need rules for forwarding, to be reachable from the outside.

- OVS WAN IP - 192.168.20.248 (Gateway: 192.168.20.254)

The gateway IP doesn't seem to be configured anywhere. Is there a part of the config missing?

192.168.20.0/24 dev vet103 proto kernel scope link src 192.168.20.103
192.168.20.0/24 dev vmbr101 proto kernel scope link src 192.168.20.3

There are two routes configured. One from the interface and the other on the bridge. Only one needs an IP.

In general, is there a need to use OVS? AFAIC, the network setup can be done with linux tools as well. The linux bridge is VLAN aware.


I suggest to start with a simplified setup first. Please see our docs for examples.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration