[SOLVED] The firewall is broken

Itay1778 · Oct 26, 2022

Hi everyone I have a serious problem with the firewall.
Today I wanted to activate the firewall in Proxmox to limit the access that things have to it on the network.
What I didn't know is that you need to activate the firewall in Datacenter to activate the firewall in node and VM/CTs - I learned something new.

And now for the problem, the second I activated it in Datacenter everything went crazy. I lost access to Proxmox both through the UI and through SSH and VM/CTs had no in or out access to the network.

So after tests regarding the loss of access to the host itself, I think it's because I accessed it from another subnet in my network and it sits in another subnet, but I could be wrong about that.

So after I disabled the firewall through the CLI, I regained access to Proxmox but not to CT/VMs... I tried to disable the firewall on a network device for each VM/CTs did not help, I checked that the firewall is turned off for each VM/CTs still did not help. At this point, the firewall was completely turned off. Nothing helped!
Until I rebooted the host, only then I regained access to the VM/CTs. I also tried to stop the pve-firewall service, but it didn't help.
Currently, I can't turn on the firewall on the host/Datacenter without causing the CT/VMs to completely lose access to the network.

I don't know what to do, I would appreciate your help.

Proxmox version: 7.2-11 everything is up-to-date

shrdlicka · Oct 28, 2022

Would it be possible to post your firewall rules when you can't access the VM/CTs. You can call iptables-save > firewall_rules.txt and attach the file here.

shrdlicka · Oct 28, 2022

Btw. you can see the default rules of the pve-firewall here [1]. Why the forwarding traffic doesn't get through is a bit weird

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pve_firewall_default_rules

Itay1778 · Oct 28, 2022

shrdlicka said:
Would it be possible to post your firewall rules when you can't access the VM/CTs. You can call iptables-save > firewall_rules.txt and attach the file here.

This is the output of iptables-save. It's easier to paste it here than to upload a file.
Note that there are docker rules because it is installed on my host for disk monitoring.

This is when the Input and output policies are set to ACCEPT
And the firewall is only on at the Datacenter level, everything else is turned off.

Code:

# Generated by iptables-save v1.8.7 on Fri Oct 28 17:38:06 2022
*nat
:PREROUTING ACCEPT [477104:122016591]
:INPUT ACCEPT [5209:715269]
:OUTPUT ACCEPT [56424:3760401]
:POSTROUTING ACCEPT [57972:3853281]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9001 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9001 -j DNAT --to-destination 172.17.0.2:9001
COMMIT
# Completed on Fri Oct 28 17:38:06 2022
# Generated by iptables-save v1.8.7 on Fri Oct 28 17:38:06 2022
*raw
:PREROUTING ACCEPT [39371676:54894634121]
:OUTPUT ACCEPT [12094316:132709196798]
COMMIT
# Completed on Fri Oct 28 17:38:06 2022
# Generated by iptables-save v1.8.7 on Fri Oct 28 17:38:06 2022
*filter
:INPUT ACCEPT [4448:686973]
:FORWARD DROP [2142:251093]
:OUTPUT ACCEPT [4383:878953]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9001 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-INPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-OUTPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT
# Completed on Fri Oct 28 17:38:06 2022

spirit · Oct 28, 2022

why do you have docker rules here ? you should really not run docker on the proxmox hypervisor. (not only iptables, but I'm not sure it's possible that docke push some routing rules, or other things not supported by proxmox)

Itay1778 · Oct 28, 2022

spirit said:
why do you have docker rules here ? you should really not run docker on the proxmox hypervisor. (not only iptables, but I'm not sure it's possible that docke push some routing rules, or other things not supported by proxmox)

I didn't add the docker rules here, docker added them by itself. And I don't think they should interfere with anything, these rules should only serve docker. In any case, I only run 2 docker CTs on the host itself. It is designed to monitor the disks that are connected to the system. My main docker is inside a VM.

Itay1778 · Oct 31, 2022

Hi, @shrdlicka I posted the iptables-save above. Can you see any problems that can make this happen?

shrdlicka · Nov 7, 2022

To quote this from the PVE FAQ wiki page [1] at the very bottom.

It is not recommended to run docker directly on your Proxmox VE host.

I would recommend you remove docker completely and see if the issue persists.

[1] https://pve.proxmox.com/wiki/FAQ

Itay1778 · Nov 7, 2022

shrdlicka said:
I would recommend you remove docker completely and see if the issue persists.

spirit said:
you should really not run docker on the proxmox hypervisor.

Ok, you are right. The problem was docker, after I remove it and its rules, the problem was solved.
Thanks

harold · Sep 5, 2023

You can run this on you PVE host:
iptables -P FORWARD ACCEPT

docker installed on PVE host would change the forward default policy to drop, which before was accept;

I don't know why or how, but after change the forward default policy to accept, firewall is ok

Search

Search

[SOLVED] The firewall is broken

Itay1778

Member

shrdlicka

Proxmox Retired Staff

shrdlicka

Proxmox Retired Staff

Itay1778

Member

spirit

Distinguished Member

Itay1778

Member

Itay1778

Member

shrdlicka

Proxmox Retired Staff

Itay1778

Member

harold

Member