CT firewall off, datacenter firewall on - no WAN on CT

T

tomaski

New Member
Jan 9, 2025
3
0
1
Hello there,

after several days of googling, asking AI ;) and trying various configurations, I am ready to throw in the towel, hoping some wise person would help me from my misery :D

The setup:
dedicated server from OVH's Kimsufi range (so only single interface available)
IP/28 block ordered for CT's / VM's that need to be publicly accessible
every available IP has a vMAC assigned
test CT that uses public IP's, has that vMAC-IP combo configured

The goal:
Some machines will be publicly accessible (hence the IP/28 block) and some won't. Those that won't, will still need to be able to access the internet to fetch updates etc (so they'll be NAT'ted).

The problem:
With PVE datacenter firewall disabled the CT can reach (and be reached) internet.
As soon as I enable the datacenter firewall, all WAN connectivity for that CT stops even though the firewall for it is NOT enabled.
I can still access the node and reach the CT console via PVE dashboard, but any in/out CT traffic fails.
When I disable the PVE datacenter firewall, the CT still has no connectivity. Restarting it does not help. Restarting the node interfaces does not help as well. The only way to get it working back again is to reboot the PVE node.

The node config:
Code: 
auto lo
iface lo inet loopback

iface enp5s0f0 inet manual
#Public interface

auto vmbr0
iface vmbr0 inet static
# Main bridge for Proxmox host
        address 135.xx.xx.138/24
        gateway 135.xx.xx.254
        bridge-ports enp5s0f0
        bridge-stp off
        bridge-fd 0
        hwaddress aa:aa:aa:aa:aa:aa
        dns-nameservers 1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4


auto vmbr1
iface vmbr1 inet static
# Internal bridge, no WAN
        address 10.0.1.254/24
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        post-up echo 0 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -P FORWARD DROP


auto vmbr2
iface vmbr2 inet static
# Internal bridge, NAT
        address  10.0.2.254/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.0.2.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.2.0/24' -o vmbr0 -j MASQUERADE


source /etc/network/interfaces.d/*

The CT config:
Code: 
auto lo
iface lo inet loopback

allow-hotplug eth0
auto eth0
iface eth0 inet static
        address 217.xx.x.xx/32  # first available address from the /28 block
# --- BEGIN PVE ---
        post-up ip route add 135.xx.xx.254 dev eth0    # gateway address of the server
        post-up ip route add default via 135.xx.xx.254 dev eth0
        pre-down ip route del default via 135.xx.xx.254 dev eth0
        pre-down ip route del 135.xx.xx.254 dev eth0
# --- END PVE ---

allow-hotplug eth1
auto eth1
iface eth1 inet static
        address 10.0.1.201/24

I have also created a Debian12 VM, where I applied the exact same configuration and it won't connect even with firewall disabled, so that tells me my config must be botched.

What am I doing wrong?
 
Hi!
which rules do you have currently configured? Maybe you could try removing them? Also note the firewall drops all the incoming traffic by default (Datacenter -> Firewall -> Options).
 
Hey @ggoller!

I have 3 IP aliases:
ip-work-main
ip-work-failover
ip-home

I then made an an IPSet called home-work where I have added those 3 aliases.

Next, I created a Security Group management that has 2 rules:
accept incoming TCP on port 8006, from home-work
accept incoming SSH, from home-work

Finally, in firewall, I added that SG and set interface to vmbr0.

On the node itself I have no rules, as my understanding is, those will cascade down from datacenter, so unless something specific is required, it is not necessary to set node rules.

For the CT in question I had following rules:
accept incoming SSH, from home-work, on net0 interface
accept incoming TCP ports (Security Group with multiple ports), from everywhere, on net0 interface

but as that was not working, I changed the default input and output policy to accept
and since this wasn't working either, I explicitly added
accept in, from everywhere, on eth0
accept out, to everywhere, on eth0
and dragged those to the top of the list. Still no luck...

//edit:
I had also tried adding the CT specific Security Group to the datacenter firewall. Also without success.

No matter wheteher the CT firewall is disabled or enabled with accept all incoming and outgoing, all in/out traffic is dropped as soon as datacenter firewall is enabled.
 
Last edited:
Just to be sure, you also changed the default policy of the datacenter firewall right? Also do you use the iptables or the nftables-based firewall? Could you dump your rules (e.g. iptables -S, nft list ruleset)?
 
ggoller said:
Just to be sure, you also changed the default policy of the datacenter firewall right?
Click to expand...
no, with the previous testing I did not as I was under the assumption that those settings / rules cascade down only to the nodes and not VM/CT.
But I just did that, and with the datacenter firewall enabled, and default input policy set to accept, CT still has no connectivity. Disabling the datacenter firewall does not restore connectivity. pve-firewall stop also does not help.

ggoller said:
Also do you use the iptables or the nftables-based firewall?
Click to expand...
IPtables. the PVE installation is about 1 month old, and basically everything is vanilla.

here is the requested output of iptables -S

Code: 
root@pve01:~# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N GROUP-management-IN
-N GROUP-management-OUT
-N GROUP-teamspeak-IN
-N GROUP-teamspeak-OUT
-N PVEFW-Drop
-N PVEFW-DropBroadcast
-N PVEFW-FORWARD
-N PVEFW-FWBR-IN
-N PVEFW-FWBR-OUT
-N PVEFW-HOST-IN
-N PVEFW-HOST-OUT
-N PVEFW-INPUT
-N PVEFW-OUTPUT
-N PVEFW-Reject
-N PVEFW-SET-ACCEPT-MARK
-N PVEFW-logflags
-N PVEFW-reject
-N PVEFW-smurflog
-N PVEFW-smurfs
-N PVEFW-tcpflags
-N veth201i0-IN
-N veth201i0-OUT
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 --name DEFAULT --mask 255.255.255.255 --rsource -j DROP
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --mask 255.255.255.255 --rsource
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 --name DEFAULT --mask 255.255.255.255 --rsource -j DROP
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --mask 255.255.255.255 --rsource
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A GROUP-management-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-management-IN -p tcp -m set --match-set PVEFW-0-home-work-v4 src -m tcp --dport 8006 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -p tcp -m set --match-set PVEFW-0-home-work-v4 src -m tcp --dport 22 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -m comment --comment "PVESIG:rDOVqtz0DQIQvCEGIZc3LvNDtVw"
-A GROUP-management-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-management-OUT -m comment --comment "PVESIG:507jzQvmkj7pyZYZEiI0n/PJt4A"
-A GROUP-teamspeak-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-teamspeak-IN -p tcp -m tcp --dport 41144 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -p tcp -m tcp --dport 10443 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -p tcp -m tcp --dport 10080 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -p tcp -m tcp --dport 10022 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -p tcp -m tcp --dport 10011 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -p tcp -m tcp --dport 30033 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -p udp -m udp --dport 9987 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-teamspeak-IN -m comment --comment "PVESIG:Jmeq/Xo6d60sNWgeMe9fiLf/jw0"
-A GROUP-teamspeak-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-teamspeak-OUT -m comment --comment "PVESIG:QmB0eIT8dVxf4JsvqGZJ0fOlgeI"
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m physdev --physdev-out veth201i0 --physdev-is-bridged -j veth201i0-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:2yn3W/2cF11mA3JTt2WLkmY70b0"
-A PVEFW-FWBR-OUT -m physdev --physdev-in veth201i0 --physdev-is-bridged -j veth201i0-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:ZQtK/mHSVMsn143SEwJq0YEW0OE"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -i vmbr4 -j RETURN
-A PVEFW-HOST-IN -i vmbr0 -j GROUP-management-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i vmbr0 -p tcp -m set --match-set PVEFW-0-home-work-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -j RETURN
-A PVEFW-HOST-IN -m comment --comment "PVESIG:VxGGhtyfNDqR+ht8sLBbHyTvKvo"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -o vmbr4 -j RETURN
-A PVEFW-HOST-OUT -o vmbr0 -j GROUP-management-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -d 135.125.97.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 135.125.97.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 135.125.97.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 135.125.97.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:s+X1Gva0C+1RfplM6n4QAY3FjTA"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A veth201i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A veth201i0-IN -j ACCEPT
-A veth201i0-IN -p tcp -m set --match-set PVEFW-0-home-work-v4 src -m tcp --dport 22 -j ACCEPT
-A veth201i0-IN -j GROUP-teamspeak-IN
-A veth201i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
-A veth201i0-IN -j ACCEPT
-A veth201i0-IN -m comment --comment "PVESIG:pS+LLtLOhIP4EZlbYBKj1g2q0rc"
-A veth201i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A veth201i0-OUT -m mac ! --mac-source 02:00:00:89:ab:77 -j DROP
-A veth201i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A veth201i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A veth201i0-OUT -j GROUP-teamspeak-OUT
-A veth201i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A veth201i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A veth201i0-OUT -m comment --comment "PVESIG:IYMWBzYZ9rtPbOhv29QDEkRWUrw"
root@pve01:~#
 
You must log in or register to reply here.
Top Bottom