Firewall stops vm/ct communication - also have to reboot to fix

moxmox · Nov 5, 2019

Hi,

I have just enabled the firewall on my proxmox machine and when I do all guests and vm's lose network connectivity? Any idea why this would be, from reading the datacenter/node level firewall should not affec the vm/ct's? (They all currently have their firewalls turned off)

Also after disabling the firewall at the dc/node level I have to reboot the machine to get it all working again which is strange?

It seems to be a similar issue to the one referenced here

https://forum.proxmox.com/threads/t...-stops-vm-lxc-connectivity.55634/#post-256939

here is my network config

Code:

auto lo
iface lo inet loopback

iface enp8s0 inet manual

iface enp0s25 inet manual

auto ens4
iface ens4 inet static
        address  10.0.0.2
        netmask  24
#10g direct link

iface enp11s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address  192.168.128.2
        netmask  255.255.255.0
        gateway  192.168.128.253
        bridge-ports enp0s25
        bridge-stp off
        bridge-fd 0

tukiyo3 · Nov 5, 2019

me too.

spirit · Nov 5, 2019

could be related to conntrack (which is global).
It's possible that already established connections before enabling firewall are dropped

can you try : sysctl -w net.netfilter.nf_conntrack_tcp_be_liberal=1

moxmox · Nov 5, 2019

thanks I will try that when I can have the server offline later

However I have restarted the vm after applying the firewall and it was still blocked so would that not suggest that its not anything to do with established connections?

spirit · Nov 5, 2019

moxmox said:
thanks I will try that when I can have the server offline later

However I have restarted the vm after applying the firewall and it was still blocked so would that not suggest that its not anything to do with established connections?

just to be sure if it's related, you can try to flush the conntrack

#conntrack -F

(need conntrack-tools package)

moxmox · Nov 5, 2019

I tested the conntrack -F but it didn't do anything I still had to reboot to fix it.

I have also tried leaving the dc/node level firewall turned on and then rebooting to see if that worked but the same situation.

Just done some more testing and when the firewall is turned on :

Inbound connections to the VM/CT work fine
Outbound connections from the VM/CT seem to be completely blocked.

I tested turning the firewall on at a VM level and disabling inbound and that did work and stopped me connecting in, but outbound connections from the VM were still not allowed (even though the VM level firewall output policy was set to Allow.

Do I need to set a node level rule to allow outbound from the VM/CTs for some reason?

moxmox · Nov 5, 2019

Interestingly I have another test proxmox system on the same exact version which should be setup in the same way but that one does not have this issue.

spirit · Nov 5, 2019

>>Do I need to set a node level rule to allow outbound from the VM/CTs for some reason?

no, datacenter/node level is for proxmox host management ips. (iptables INPUT|OUTPUT)

vm/ct rules are for iptables FORWARD.

can you send the result of "iptables-save" when vm outbound is blocked ?
do you see something in vm firewall logs ?

moxmox · Nov 5, 2019

yes thats what I thought.

here you go

Code:

# Generated by iptables-save v1.8.2 on Tue Nov  5 17:08:00 2019
*nat
:PREROUTING ACCEPT [112831:46427489]
:INPUT ACCEPT [987:87755]
:OUTPUT ACCEPT [10901:5706860]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 3306 -j DNAT --to-destination 192.168.128.95:3306
-A PREROUTING -p tcp -m tcp --dport 5001 -j DNAT --to-destination 192.168.128.69:5001
-A PREROUTING -p tcp -m tcp --dport 5002 -j DNAT --to-destination 192.168.128.69:5002
-A POSTROUTING -j MASQUERADE
COMMIT
# Completed on Tue Nov  5 17:08:00 2019
# Generated by iptables-save v1.8.2 on Tue Nov  5 17:08:00 2019
*filter
:INPUT ACCEPT [7:336]
:FORWARD ACCEPT [4424:1629527]
:OUTPUT ACCEPT [151:83302]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -d 192.168.128.95/32 -p tcp -m tcp --dport 3306 -j ACCEPT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -p udp -m udp --dport 2049 -j RETURN
-A PVEFW-HOST-IN -p udp -m udp --dport 111 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 111 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 2049 -j RETURN
-A PVEFW-HOST-IN -s 192.168.128.0/24 -p udp -m multiport --dports 135,445 -j RETURN
-A PVEFW-HOST-IN -s 192.168.128.0/24 -p udp -m udp --dport 137:139 -j RETURN
-A PVEFW-HOST-IN -s 192.168.128.0/24 -p udp -m udp --sport 137 --dport 1024:65535 -j RETURN
-A PVEFW-HOST-IN -s 192.168.128.0/24 -p tcp -m multiport --dports 135,139,445 -j RETURN
-A PVEFW-HOST-IN -s 192.168.128.0/24 -p tcp -m tcp --dport 3551 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -s 192.168.128.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:cnx89gfqPUGX7nyV1M9Dp3Vqz0w"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -d 192.168.128.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.128.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.128.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.128.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:2DqwG4wws2YfRFlLsKS5Pi/R/fc"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT
# Completed on Tue Nov  5 17:08:00 2019

moxmox · Nov 5, 2019

Code:

-A PREROUTING -p tcp -m tcp --dport 3306 -j DNAT --to-destination 192.168.128.95:3306
-A PREROUTING -p tcp -m tcp --dport 5001 -j DNAT --to-destination 192.168.128.69:5001
-A PREROUTING -p tcp -m tcp --dport 5002 -j DNAT --to-destination 192.168.128.69:5002
-A POSTROUTING -j MASQUERADE

-A FORWARD -d 192.168.128.95/32 -p tcp -m tcp --dport 3306 -j ACCEPT

I assume they are not causing an issue but these are some rules I added to port forward to containers. Its run a init script on startup.

I checked and there were no firewall logs.

spirit · Nov 5, 2019

mmm, if you have nat, I'm not sure if it'll work.
(I don't for for DNAT with port forwarding, but for snat/masquerade you can't firewall until you setup conntrack zones)
as you have masquerade for all outgoing traffic, that could explain why it's not working.

#iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1

should fix it.

(another thing to test: do you have tested to connect between 2vms/ct on the same host to see if outgoing is working ?)

aljaxus · Nov 5, 2019

spirit said:
mmm, if you have nat, I'm not sure if it'll work.
(I don't for for DNAT with port forwarding, but for snat/masquerade you can't firewall until you setup conntrack zones)
as you have masquerade for all outgoing traffic, that could explain why it's not working.

#iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1

should fix it.

(another thing to test: do you have tested to connect between 2vms/ct on the same host to see if outgoing is working ?)

HEY! This looks like a solution for my issue too! (If you have a minute or two, please take a fast look at it - thread link) Can you provide some simple explanation what all the parameters stand for?
Thank you in advance,
~ Aljaz S.

moxmox · Nov 5, 2019

spirit said:
mmm, if you have nat, I'm not sure if it'll work.
(I don't for for DNAT with port forwarding, but for snat/masquerade you can't firewall until you setup conntrack zones)
as you have masquerade for all outgoing traffic, that could explain why it's not working.

#iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1

should fix it.

(another thing to test: do you have tested to connect between 2vms/ct on the same host to see if outgoing is working ?)

yes your right from running the same init script on my test machine

this line

Code:

/sbin/iptables -t nat -A POSTROUTING  -j MASQUERADE

breaks everything!

Is there a preferred way to do port forwarding in this kind of scenario? (I assume its not something that can be done in the proxmox ui?)

Thanks for helping so far! Much appreciated! I am a developer so networking is not my strong point!

Edit - tested running your command above but it didn't seem to make any difference, the outbound is still blocked. (I have configured my test proxmox machine so its got the same issue so its easier to try different fixes)

spirit · Nov 6, 2019

About the conntrack zone, it's needed because if not, the packet are never going to postrouting for the masquerade. (because they already going through the postrouting of the vm bridge).

Think about conntrack zones like 2 different firewall, where rules from prerouting/postrouting will be applied twice.

Here a blog with good explain: (with s-nat, but this is the same than masquerade)
https://blog.lobraun.de/2019/05/19/prox/

Now, for port forwarding/DNAT, I really don't known

moxmox · Nov 6, 2019

Ok thanks I will have a read up on it. Now I know what the issue is though I may be able to work around it and remove the port forwarding as it seems less complicated from a firewall point of view.

thanks again for your help.

Search

Search

Firewall stops vm/ct communication - also have to reboot to fix

moxmox

Well-Known Member

tukiyo3

Well-Known Member

spirit

Distinguished Member

moxmox

Well-Known Member

spirit

Distinguished Member

moxmox

Well-Known Member

moxmox

Well-Known Member

spirit

Distinguished Member

moxmox

Well-Known Member

moxmox

Well-Known Member

spirit

Distinguished Member

aljaxus

Active Member

moxmox

Well-Known Member

spirit

Distinguished Member

moxmox

Well-Known Member