Firewall one way connection problem

Eris

Renowned Member
Dec 16, 2015
12
0
66
42
itronic.at
Hi,

I have a connection problem with public IPs between 2 vms.

First whats happening:

I have 2 VMs each with 2 network interfaces all bound to the same vmbr on the same host.
Firewall is disabled, at least on the node and on both vms. No other VM is on this node.

ipfilter-netX is renamed for both VMs
All Filters that are not disabled by the firewall are disabled (why the hell is a filter active if firewall is disabled?)

If ip ping from VM1001 to VM5102 I get a ICMP reply and it also travels the correct interfaces.

Now if I try to make a tcp request, for example wget I doesn't get a reply. The Syn-ack never comes back but it got sent by the VM5102.

Debugging starts, I followed the flow...

If I understand the flow correct it should flow the following way:

Syn:
VM1001-eth1
node31-tap1001i1
node31-fwbr1001i1
node31-fwln1001i1
node31-fwpr1001p1
node31-vmbr0
node31-fwpr5102p1
node31-fwln5102i1
node31-fwbr5102i1
node31-tap5102i1
VM5102-eth1

Syn-ack:
VM5102-eth1
node31-tap5102i1
node31-fwbr5102i1
node31-fwln5102i1
node31-fwpr5102p1
node31-vmbr0 - last time I see the package in tcpdump
node31-fwpr1001p1
node31-fwln1001i1
node31-fwbr1001i1
node31-tap1001i1
VM1001-eth1


Even if I add ACCEPT to all iptables Rules it doesn't work
# iptables -I INPUT -j ACCEPT
# iptables -I FORWARD -j ACCEPT
# iptables -I OUTPUT -j ACCEPT


It doesn't matter if I move one of the VMs to another node its the same but if I move the IP to another cluster it works without problems...


I tried the same IP von eth0 on both VMs but with no luck. If I use an internal 172.* I have no connection problems.

VMs are Debian, Node got updated today.

I would take any help...

--- IPtables with noting in it....
# Generated by iptables-save v1.6.0 on Fri Jul 27 19:17:54 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:180]
:pVEFW-Drop - [0:0]
:pVEFW-DropBroadcast - [0:0]
:pVEFW-FORWARD - [0:0]
:pVEFW-FWBR-IN - [0:0]
:pVEFW-FWBR-OUT - [0:0]
:pVEFW-INPUT - [0:0]
:pVEFW-OUTPUT - [0:0]
:pVEFW-Reject - [0:0]
:pVEFW-SET-ACCEPT-MARK - [0:0]
:pVEFW-logflags - [0:0]
:pVEFW-reject - [0:0]
:pVEFW-smurflog - [0:0]
:pVEFW-smurfs - [0:0]
:pVEFW-tcpflags - [0:0]
:tap1001i0-IN - [0:0]
:tap1001i0-OUT - [0:0]
:tap1001i1-IN - [0:0]
:tap1001i1-OUT - [0:0]
:tap5102i0-IN - [0:0]
:tap5102i0-OUT - [0:0]
:tap5102i1-IN - [0:0]
:tap5102i1-OUT - [0:0]
-A INPUT -j ACCEPT
-A INPUT -j PVEFW-INPUT
-A FORWARD -j ACCEPT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j ACCEPT
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m physdev --physdev-out tap1001i0 --physdev-is-bridged -j tap1001i0-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap1001i1 --physdev-is-bridged -j tap1001i1-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap5102i0 --physdev-is-bridged -j tap5102i0-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap5102i1 --physdev-is-bridged -j tap5102i1-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:xX/AgNcBbLheTc5eLDeU2DPBBD4"
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap1001i0 --physdev-is-bridged -j tap1001i0-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap1001i1 --physdev-is-bridged -j tap1001i1-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap5102i0 --physdev-is-bridged -j tap5102i0-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap5102i1 --physdev-is-bridged -j tap5102i1-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:KyOynIUH0H9dk2Al5lV0Lmshp8s"
-A PVEFW-INPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-OUTPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j NFLOG --nflog-prefix ":0:7:pVEFW-logflags: DROP: "
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:xxngynQ61gj3oDwdvenmOrWc1Z4"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j NFLOG --nflog-prefix ":0:7:pVEFW-smurflog: DROP: "
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:07iN6Ltw+eq1SF8lRxwoE+285nY"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A tap1001i0-IN -j ACCEPT
-A tap1001i0-IN -m comment --comment "PVESIG:6/i7HBviTdWE95tNd8maDixkN3E"
-A tap1001i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap1001i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap1001i0-OUT -m comment --comment "PVESIG:uemS0BGWAeqPZ3b3A4NgT9Wbbmw"
-A tap1001i1-IN -j ACCEPT
-A tap1001i1-IN -m comment --comment "PVESIG:qRCoc51l0vdkcryTXojKu9AK3/A"
-A tap1001i1-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap1001i1-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap1001i1-OUT -m comment --comment "PVESIG:0Y4PTnzdNh/JlSxU7/2ZL+OG0OU"
-A tap5102i0-IN -j ACCEPT
-A tap5102i0-IN -m comment --comment "PVESIG:E8T2Tx9IZCEDxqWyulmH6E4Ksl8"
-A tap5102i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap5102i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap5102i0-OUT -m comment --comment "PVESIG:j/ITcVK+fqo20H7rn5AVoJoC+IU"
-A tap5102i1-IN -j ACCEPT
-A tap5102i1-IN -m comment --comment "PVESIG:Y8RDkaJFL3JZcPSZ/273Keu2Vac"
-A tap5102i1-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap5102i1-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap5102i1-OUT -m comment --comment "PVESIG:hX3uhF/mmg9ndsWXFnbHyRctWKo"
COMMIT
# Completed on Fri Jul 27 19:17:54 2018

----



thx

Harald
 
ipfilter-netX is renamed for both VMs
All Filters that are not disabled by the firewall are disabled (why the hell is a filter active if firewall is disabled?)

Sound contradictory - in order to clarify the situation: post the pve firewall-setting, the easiest way to get is

Code:
cd /etc/pve
grep -r "" *.fw

If ip ping from VM1001 to VM5102 I get a ICMP reply and it also travels the correct interfaces.

Now if I try to make a tcp request, for example wget I doesn't get a reply. The Syn-ack never comes back but it got sent by the VM5102.

Debugging starts, I followed the flow...

If I understand the flow correct it should flow the following way:

Syn:
VM1001-eth1
node31-tap1001i1
node31-fwbr1001i1
node31-fwln1001i1
node31-fwpr1001p1
node31-vmbr0
node31-fwpr5102p1
node31-fwln5102i1
node31-fwbr5102i1
node31-tap5102i1
VM5102-eth1

Syn-ack:
VM5102-eth1
node31-tap5102i1
node31-fwbr5102i1
node31-fwln5102i1
node31-fwpr5102p1
node31-vmbr0 - last time I see the package in tcpdump
node31-fwpr1001p1
node31-fwln1001i1
node31-fwbr1001i1
node31-tap1001i1
VM1001-eth1


Even if I add ACCEPT to all iptables Rules it doesn't work
# iptables -I INPUT -j ACCEPT
# iptables -I FORWARD -j ACCEPT
# iptables -I OUTPUT -j ACCEPT


It doesn't matter if I move one of the VMs to another node its the same but if I move the IP to another cluster it works without problems...


I tried the same IP von eth0 on both VMs but with no luck. If I use an internal 172.* I have no connection problems.

I guess that's rather a routing problem than a firewall one. Verify routing tables and IP settings in VMs and host by

Code:
ip addr
ip route
 
I will sent the .fw configs later, but I'm not sure how it could be a routing problem in an bridged Environment.

Both VMs have the same subnet and the host doesn't have this subnet configured.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!