[SOLVED] SDN Simple zone: Not working between nodes

[snouhaud]

Hi All,

I try to test SDN with simple zone on 3 nodes:

Nodes:
1. pve
2. pve1
3. pve2

The subnet network on simple zone is 10.10.11.0/24 with gateway 10.10.11.1
On each node the network work well

from host i can ping and reach vm attached to meta100 vnet

But network don't work between node i connot ping


When i'm on one node VMs can access other vm on same node but not on other node

i have open traffic on port udp/4789 in firewall



Here my config:

SDN:
cat /etc/pve/sdn/*.cfg subnet: meta-10.10.11.0-24 vnet meta100 gateway 10.10.11.1 snat 1 vnet: meta100 zone meta simple: meta ipam pve

Proxmox version package
pveversion --verbose proxmox-ve: 7.2-1 (running kernel: 5.15.35-2-pve) pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc) pve-kernel-5.15: 7.2-4 pve-kernel-helper: 7.2-4 pve-kernel-5.4: 6.4-16 pve-kernel-5.15.35-2-pve: 5.15.35-5 pve-kernel-5.4.178-1-pve: 5.4.178-1 ceph-fuse: 15.2.16-pve1 corosync: 3.1.5-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: residual config ifupdown2: 3.1.0-1+pmx3 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.24-pve1 libproxmox-acme-perl: 1.4.2 libproxmox-backup-qemu0: 1.3.1-1 libpve-access-control: 7.2-2 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.2-2 libpve-guest-common-perl: 4.1-2 libpve-http-server-perl: 4.1-2 libpve-network-perl: 0.7.1 libpve-storage-perl: 7.2-4 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.12-1 lxcfs: 4.0.12-pve1 novnc-pve: 1.3.0-3 proxmox-backup-client: 2.2.3-1 proxmox-backup-file-restore: 2.2.3-1 proxmox-mini-journalreader: 1.3-1 proxmox-widget-toolkit: 3.5.1 pve-cluster: 7.2-1 pve-container: 4.2-1 pve-docs: 7.2-2 pve-edk2-firmware: 3.20210831-2 pve-firewall: 4.2-5 pve-firmware: 3.4-2 pve-ha-manager: 3.3-4 pve-i18n: 2.7-2 pve-qemu-kvm: 6.2.0-10 pve-xtermjs: 4.16.0-1 pve-zsync: 2.2.3 qemu-server: 7.2-3 smartmontools: 7.2-pve3 spiceterm: 3.2-2 swtpm: 0.7.1~bpo11+1 vncterm: 1.7-1 zfsutils-linux: 2.1.4-pve1



node 1 pve:

xxx.xxx.xxx.xx1 : public ip of node 1

network/interfaces
auto lo iface lo inet loopback iface lo inet6 loopback auto eno1 iface eno1 inet static address xxx.xxx.xxx.xx1/24 gateway xxx.xxx.xxx.254 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp auto vmbr1 iface vmbr1 inet static address 10.10.10.1/24 bridge-ports none bridge-stp off bridge-fd 0 bridge_maxwait 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 2 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 2 source /etc/network/interfaces.d/*

interfaces.d/sdn
auto meta100 iface meta100 address 10.10.11.1/24 post-up iptables -t nat -A POSTROUTING -s '10.10.11.0/24' -o eno1 -j SNAT --to-source xxx.xxx.xxx.xx1 post-down iptables -t nat -D POSTROUTING -s '10.10.11.0/24' -o eno1 -j SNAT --to-source xxx.xxx.xxx.xx1 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 bridge_ports none bridge_stp off bridge_fd 0 ip-forward on


node 2 pve2:

xxx.xxx.xxx.xx2 : public ip of node 2

network/interfaces
auto lo iface lo inet loopback iface lo inet6 loopback auto eth0 iface eth0 inet static address xxx.xxx.xxx.xx2/24 gateway xxx.xxx.xxx.254 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp iface eth1 inet manual auto vmbr1 iface vmbr1 inet static address 10.10.10.1/24 bridge-ports none bridge-stp off bridge-fd 0 hwaddress 8e:eb:9d:95:72:ac bridge_maxwait 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 2 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 2 source /etc/network/interfaces.d/*

interfaces.d/sdn
auto meta100 iface meta100 address 10.10.11.1/24 post-up iptables -t nat -A POSTROUTING -s '10.10.11.0/24' -o eth0 -j SNAT --to-source xxx.xxx.xxx.xx2 post-down iptables -t nat -D POSTROUTING -s '10.10.11.0/24' -o eth0 -j SNAT --to-source xxx.xxx.xxx.xx2 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 bridge_ports none bridge_stp off bridge_fd 0 ip-forward on


node 3 pve3:

xxx.xxx.xxx.xx3 : public ip of node 3

network/interfaces
auto lo iface lo inet loopback auto eno1 iface eno1 inet static address xxx.xxx.xxx.xx3/24 gateway xxx.xxx.xxx.254 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp iface eno2 inet manual auto vmbr1 iface vmbr1 inet static address 10.10.10.1/24 bridge-ports none bridge-stp off bridge-fd 0 bridge_maxwait 0 post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 2 post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 2 source /etc/network/interfaces.d/*

interfaces.d/sdn
auto meta100 iface meta100 address 10.10.11.1/24 post-up iptables -t nat -A POSTROUTING -s '10.10.11.0/24' -o eno1 -j SNAT --to-source xxx.xxx.xxx.xx3 post-down iptables -t nat -D POSTROUTING -s '10.10.11.0/24' -o eno1 -j SNAT --to-source xxx.xxx.xxx.xx3 post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 bridge_ports none bridge_stp off bridge_fd 0 ip-forward on


Any help/ideas welcome


Regards,

Stéphane NOUHAUD

 

[snouhaud]

Juste one more information the three proxmox 7 servers is in hosting provider OVH and it' s dedicated bare metal server

 

[spirit]

Simple zone are simple route zone or standalone zone for each host. Here at ovh, you can assign your public failover ips on simple zone. But you cant communicate with privates ips between vms. You need vxlan or evpn zones for this.

 

[snouhaud]

I misunderstand the doc. I try VXLAN zones Thanks for this clarification and your fantastic work !