Issue with SDN on 2 indepedent different system, 1 other independent system works fine

[fma965]

Hey guys, this is my first post here, i just registered to post this.

I have a similiar issue to this guy.
https://www.reddit.com/r/Proxmox/comments/1864usp/proxmox_81_sdn_feature_testing_testing_and/

I have 3 completely unlinked Proxmox Servers, all 3 are running 8.1.4, fully updated, have the required packages dnsmasq, libpve-network-perl, interfaces.d sources etc.

Server A works as intended


Server B and Server C are not working


IPAM is empty for both, i won't bother showing the config for both but Server C is my test system so here is the config from that.

/etc/network/interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.199/24
        gateway 192.168.1.1
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

/etc/network/interfaces.d/sdn

#version:33

auto test
iface test
        address 10.0.0.1/24
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 192.168.1.199
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to-source 192.168.1.199
        post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        mtu 1490
        ip-forward on

root@CY-HV2:~# service dnsmasq@dhcpsnat status
● dnsmasq@dhcpsnat.service - dnsmasq (dhcpsnat) - A lightweight DHCP and caching DNS server
     Loaded: loaded (/lib/systemd/system/dnsmasq@.service; disabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/dnsmasq@.service.d
             └─00-dnsmasq-after-networking.conf
     Active: active (running) since Tue 2024-02-06 19:19:17 GMT; 13min ago
    Process: 6087 ExecStartPre=/etc/init.d/dnsmasq checkconfig dhcpsnat (code=exited, status=0/SUCCESS)
    Process: 6094 ExecStart=/etc/init.d/dnsmasq systemd-exec dhcpsnat (code=exited, status=0/SUCCESS)
    Process: 6102 ExecStartPost=/etc/init.d/dnsmasq systemd-start-resolvconf dhcpsnat (code=exited, status=0/SUCCESS)
   Main PID: 6101 (dnsmasq)
      Tasks: 1 (limit: 18901)
     Memory: 732.0K
        CPU: 34ms
     CGroup: /system.slice/system-dnsmasq.slice/dnsmasq@dhcpsnat.service
             └─6101 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.dhcpsnat.pid -u dnsmasq --local-service --trust-anchor=.,20326,8,2,e06d44b8>

Feb 06 19:19:17 CY-HV2 systemd[1]: Starting dnsmasq@dhcpsnat.service - dnsmasq (dhcpsnat) - A lightweight DHCP and caching DNS server...
Feb 06 19:19:17 CY-HV2 dnsmasq[6101]: started, version 2.89 cachesize 150
Feb 06 19:19:17 CY-HV2 dnsmasq[6101]: DNS service limited to local subnets
Feb 06 19:19:17 CY-HV2 dnsmasq[6101]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipse>
Feb 06 19:19:17 CY-HV2 dnsmasq[6101]: reading /etc/resolv.conf
Feb 06 19:19:17 CY-HV2 dnsmasq[6101]: using nameserver 192.168.1.1#53
Feb 06 19:19:17 CY-HV2 dnsmasq[6101]: read /etc/hosts - 11 names
Feb 06 19:19:17 CY-HV2 systemd[1]: Started dnsmasq@dhcpsnat.service - dnsmasq (dhcpsnat) - A lightweight DHCP and caching DNS server.

Please let me know what else you need

 

[RMDrinan]

I am running into the same issue. I have a 4 node cluster - all up to date with no-enterprise repository. I was seeing an IP assigned via the IPAM screen but the VMs (ubuntu server and mint 21.3) both reported no wired connection. I switched the mint VM to vmbr0 and it connected fine.
I was wondering if the fact that I installed open vswitch was causing a problem - I am NOT mixing any native bridge/bonds, etc. with my vswitch - which is only for my internal ceph 18.2 10G DAC connections - 3 nodes working great.

 

[fma965]

Not quite the same issue but interesting to note, i am thinking it's something to do with a fresh install of 8.1, so i will try 7.x and upgrade later today.

 

[fma965]

So i just tried a fresh 7.4 install and updated it to 8.1.4 following all the required steps and i have the exact same issue still

 

[spirit]

the ipam is local to the cluster.

If you want a shared ipam, you need to use an external ipam; netbox or phpipam currnetly.

 

[shanreich]

Server B and Server C are not working

I assume you set up the Zone / Vnet the same as on Server A - and also created VMs with NICs on that bridge?
Can you send me the SDN configuration files located in `/etc/pve/sdn` ?

 

[fma965]

the ipam is local to the cluster.

If you want a shared ipam, you need to use an external ipam; netbox or phpipam currnetly.

As mentioned they are all SEPERATE instances, no clustering, no nothing, i have tried to repeat the same setup i have on Server A on my home server and on a test system, both fail, yesterday i reinstalled my test system to 7.4 and updated to 8.1, same issue.
I never once stated clustering or that i was trying to get it to show on multiple instances.

If you guys were to install Proxmox from scratch i'm confident you would have the exact same issue at this time, i have now tried it on 3 installs.

Again it's working perfectly fine on my web server.

 

[shanreich]

As mentioned they are all SEPERATE instances, no clustering, no nothing, i have tried to repeat the same setup i have on Server A on my home server and on a test system, both fail, yesterday i reinstalled my test system to 7.4 and updated to 8.1, same issue.
I never once stated clustering or that i was trying to get it to show on multiple instances.

If you guys were to install Proxmox from scratch i'm confident you would have the exact same issue at this time, i have now tried it on 3 installs.

Again it's working perfectly fine on my web server.

I've just tested it yesterday, albeit with a newer ISO, so no guarantees. Can you please post the following things:

• Contents of /etc/pve/sdn
• Output of systemctl status dnsmasq@*
• Contents of your journal that contains the time that you applied SDN config and created a VM on the VNet (e.g. journalctl --since '1 day ago')

 

[spirit]

As mentioned they are all SEPERATE instances, no clustering, no nothing, i have tried to repeat the same setup i have on Server A on my home server and on a test system, both fail, yesterday i reinstalled my test system to 7.4 and updated to 8.1, same issue.
I never once stated clustering or that i was trying to get it to show on multiple instances.

If you guys were to install Proxmox from scratch i'm confident you would have the exact same issue at this time, i have now tried it on 3 installs.

Again it's working perfectly fine on my web server.

do you have enable dhcp option in the simple zone ?

 

[fma965]

do you have enable dhcp option in the simple zone ?

Thanks, OK so on my test system i actually didn't have that checked oops.... but on my home prod system i did have it checked and the dnsmasq@zonename services were running as per the bash output on first post... and it still wasn't working.

However now i have just redone the SDN all from scratch once again on the Home Prod server and used different names etc and it's showing in IPAM and has DHCP etc. now the only difference i can think of is that this time i mapped the network to a VM and not a CT, could this be the reason?

 

[shanreich]

Did you check DHCP on the eth device of the container? It should work for containers just the same

 

[fma965]

Did you check DHCP on the eth device of the container? It should work for containers just the same

Yes it wasn't getting DHCP and it wasn't showing in IPAM, so strange but as i can't currently reproduce this i guess we may never know the issue, Although i did reference a reddit post of someone who seemed to have the exact same outcome also

 

[shanreich]

Okay, strange - if this ever comes up again and / or you can reliably reproduce it then please don't hesitate to post here again. I'll look into it

 

[fma965]

Okay, strange - if this ever comes up again and / or you can reliably reproduce it then please don't hesitate to post here again. I'll look into it

No problem, Thanks to everyone who assisted for the help, I am going to reinstall 8.1 fresh on my test system later today and replicate the same steps and see what happens, if it doesn't work again i will update you, although i think the test system was literally just me forgetting to tick the DHCP box so i expect it to work fine.

EDIT: I just reinstalled the test system and it's working as intended.

Does the dnsmasq@zonename service get created even if DHCP is not ticked? i'd assume not, which means i 100% ticked it for the home server but yet it still didn't work.... strange but ohwell. Thanks again!

 

[shanreich]

Does the dnsmasq@zonename service get created even if DHCP is not ticked? i'd assume not, which means i 100% ticked it for the home server but yet it still didn't work.... strange but ohwell. Thanks again!

No, it should only get created when DHCP is activated for the zone. It should also get removed as soon as you deactivate DHCP and apply the settings.