[SOLVED] Block outbound connection from VM not working

M

meepmeep

Renowned Member
Apr 14, 2013
51
8
73
Hi

I'm trying to block some simple outbound traffic from a specific VM.
Firewall is enable on the datacenter lever, on the node level and on the VM level (name "hub", ip "10.1.0.112").
I'm trying to block icmp to 1.1.1.1.

Code: 
root@marvin:/etc/pve/firewall# cat 112.fw
[OPTIONS]

ipfilter: 1
enable: 1
policy_out: ACCEPT
log_level_out: debug
policy_in: ACCEPT
log_level_in: debug

[ALIASES]

cloudflaredns 1.1.1.1

[RULES]

OUT DROP -source dc/hub -dest guest/cloudflaredns -p icmp -log debug -icmp-type any

root@marvin:/etc/pve/firewall# cat cluster.fw
[OPTIONS]

policy_in: ACCEPT
enable: 1
log_ratelimit: burst=5,enable=1,rate=1/second

[ALIASES]

hub 10.1.0.112
cloudflareDNS 1.1.1.1

[IPSET all_vm_except_fw] # All VM ip except FW

10.0.0.0/16 # Internal LAN
!10.0.0.254 # FW ip
10.1.0.2 # Marvin
10.1.0.253
10.7.0.0/24 # vpn client subnet

[group isolated_vm] # VM isolé (block vmbr1 except FW)

OUT DROP -dest +all_vm_except_fw -p icmp -log alert
OUT DROP -dest +all_vm_except_fw -p udp -log alert
OUT DROP -dest +all_vm_except_fw -p tcp -log alert

But nothing appears on the iptables chain "PVEFW-HOST-OUT".
I have only the "DROP line" if I add the rule on the node level :

Code: 
Chain PVEFW-HOST-OUT (1 references)
target     prot opt source               destination
ACCEPT     0    --  0.0.0.0/0            0.0.0.0/0
DROP       0    --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
ACCEPT     0    --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
RETURN     2    --  0.0.0.0/0            0.0.0.0/0
DROP       1    --  10.1.0.112           1.1.1.1              icmptype 255
RETURN     6    --  0.0.0.0/0            10.210.0.0/24        tcp dpt:8006
RETURN     6    --  0.0.0.0/0            10.210.0.0/24        tcp dpt:22
RETURN     6    --  0.0.0.0/0            10.210.0.0/24        tcp dpts:5900:5999
RETURN     6    --  0.0.0.0/0            10.210.0.0/24        tcp dpt:3128
RETURN     17   --  10.210.0.2           10.210.0.1           udp dpts:5404:5405
RETURN     0    --  0.0.0.0/0            0.0.0.0/0
           0    --  0.0.0.0/0            0.0.0.0/0            /* PVESIG:NmZbdGkH+OVDfAA839uPiAAtSQI */

With and without the rule in the node level, I can still ping 1.1.1.1 ...
What am I missing ?
 
Hi,

a minimal example for dropping outgoing icmp traffic to 1.1.1.1 for a VM looks like this and is to be placed in the VMs firewall config, /etc/pve/firewall/112.fw in your case:
Code: 
[OPTIONS]

enable: 1

[RULES]

OUT DROP -dest 1.1.1.1 -p icmp -log nolog -icmp-type any

The cluster wide firewall config located at /etc/pve/firewall/cluster.fw looks like this for it to work
Code: 
[OPTIONS]

enable: 1

Based on this you should be able to adapt the rule to your aliases.

Note that the PVEFW-HOST-OUT chain is not responsible for dropping the VM traffic, this is for host traffic. You should have a chain with tap112i0-OUT which should contain the rule.
 
Last edited:
thank you for the quick reply.

However 2 things :

  • I don't have chain named "tap112i0-OUT"
  • Even with your rules and a reboot of the vm, I can still ping 1.1.1.1

Could it be link to my ip forwarding rules on the node ? (vm has only a local ip, and everything is routed throught the node with this : )
/usr/sbin/iptables -t nat -A POSTROUTING -o enp5s0 -s 10.1.0.0/24 -j SNAT --to <public_ip_of_the_node>
 
well not so fast for me .. @Chris, sorry :)

Firewalled interface works but it also block all outbound traffic (output policy on the VM & Datacenter is set to ACCEPT)
In the log I can see the drop packet for icmp 1.1.1.1, but no log for other requests.
If I set my rule ACCEPT, i still can't ping, but log is still in the log as "ACCEPT" (but not going through)

112 7 tap112i0-OUT 02/Aug/2023:17:49:22 +0200 ACCEPT: IN=fwbr112i0 OUT=fwbr112i0 PHYSIN=tap112i0 PHYSOUT=fwln112i0 MAC=xx:xx:xx:xx:xx SRC=10.1.0.112 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=50455 DF PROTO=ICMP TYPE=8 CODE=0 ID=15812 SEQ=4

Does this means that my requests never match my POSTROUTING rules ?

edit :
(even with the default "output policy = accept", theses are the rules on the node :

Code: 
Chain tap112i0-OUT (1 references)
target     prot opt source               destination
PVEFW-SET-ACCEPT-MARK  17   --  0.0.0.0/0            0.0.0.0/0           [goto]  udp spt:68 dpt:67
DROP       0    --  0.0.0.0/0            0.0.0.0/0            MAC ! xx:vm:mac:xxx
MARK       0    --  0.0.0.0/0            0.0.0.0/0            MARK and 0x7fffffff
NFLOG      1    --  0.0.0.0/0            1.1.1.1              icmptype 255 limit: avg 1/sec burst 5 nflog-prefix ":112:7:tap112i0-OUT: DROP: "
DROP       1    --  0.0.0.0/0            1.1.1.1              icmptype 255
PVEFW-SET-ACCEPT-MARK  0    --  0.0.0.0/0            0.0.0.0/0           [goto]
           0    --  0.0.0.0/0            0.0.0.0/0            /* PVESIG:LjOy2qtZTiiFlpBZPiQthI8r4cg */

Is that normal to have no ACCEPT rule ?
 
Last edited:
meepmeep said:
well not so fast for me .. @Chris, sorry :)

Firewalled interface works but it also block all outbound traffic (output policy on the VM & Datacenter is set to ACCEPT)
In the log I can see the drop packet for icmp 1.1.1.1, but no log for other requests.
If I set my rule ACCEPT, i still can't ping, but log is still in the log as "ACCEPT" (but not going through)

112 7 tap112i0-OUT 02/Aug/2023:17:49:22 +0200 ACCEPT: IN=fwbr112i0 OUT=fwbr112i0 PHYSIN=tap112i0 PHYSOUT=fwln112i0 MAC=xx:xx:xx:xx:xx SRC=10.1.0.112 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=50455 DF PROTO=ICMP TYPE=8 CODE=0 ID=15812 SEQ=4

Does this means that my requests never match my POSTROUTING rules ?

edit :
(even with the default "output policy = accept", theses are the rules on the node :

Code: 
Chain tap112i0-OUT (1 references)
target     prot opt source               destination
PVEFW-SET-ACCEPT-MARK  17   --  0.0.0.0/0            0.0.0.0/0           [goto]  udp spt:68 dpt:67
DROP       0    --  0.0.0.0/0            0.0.0.0/0            MAC ! xx:vm:mac:xxx
MARK       0    --  0.0.0.0/0            0.0.0.0/0            MARK and 0x7fffffff
NFLOG      1    --  0.0.0.0/0            1.1.1.1              icmptype 255 limit: avg 1/sec burst 5 nflog-prefix ":112:7:tap112i0-OUT: DROP: "
DROP       1    --  0.0.0.0/0            1.1.1.1              icmptype 255
PVEFW-SET-ACCEPT-MARK  0    --  0.0.0.0/0            0.0.0.0/0           [goto]
           0    --  0.0.0.0/0            0.0.0.0/0            /* PVESIG:LjOy2qtZTiiFlpBZPiQthI8r4cg */

Is that normal to have no ACCEPT rule ?
Click to expand...
Well, there is no accept rule, as the packets will have to go trough other rules as well.

Please share the current firewall config from /etc/pve/firewall as well as the output of iptables-save, so we can see the actual rules you currently have.
 
112.fw :

Code: 
[OPTIONS]

enable: 1
policy_out: ACCEPT
policy_in: ACCEPT

[ALIASES]

cloudflaredns 1.1.1.1

[RULES]

OUT DROP -dest 1.1.1.1 -p icmp -log nolog -icmp-type any

cluster.fw
Code: 
[OPTIONS]

enable: 1
policy_in: ACCEPT
log_ratelimit: burst=5,enable=1,rate=1/second

[ALIASES]

cloudflareDNS 1.1.1.1
hub 10.1.0.112

[IPSET all_vm_except_fw] # All VM ip except FW

10.0.0.0/16 # Internal LAN
!10.0.0.254 # FW ip
10.1.0.2 # Marvin
10.1.0.253
10.7.0.0/24 # vpn client subnet

[group isolated_vm] # VM isolé (block vmbr1 except FW)

OUT DROP -dest +all_vm_except_fw -p icmp -log alert
OUT DROP -dest +all_vm_except_fw -p udp -log alert
OUT DROP -dest +all_vm_except_fw -p tcp -log alert


iptables-saves
Code: 
# Generated by iptables-save v1.8.9 on Thu Aug  3 09:17:24 2023
*raw
:PREROUTING ACCEPT [592304858:527188676860]
:OUTPUT ACCEPT [75569371:14798661673]
COMMIT
# Completed on Thu Aug  3 09:17:24 2023
# Generated by iptables-save v1.8.9 on Thu Aug  3 09:17:24 2023
*filter
:INPUT ACCEPT [2:80]
:FORWARD ACCEPT [36:3708]
:OUTPUT ACCEPT [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
:tap112i0-IN - [0:0]
:tap112i0-OUT - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m physdev --physdev-out tap112i0 --physdev-is-bridged -j tap112i0-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Mwu3OTiSWwPSTAVjBZ6Skny0gt8"
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap112i0 --physdev-is-bridged -j tap112i0-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:wxnGXb+tg9M+to8yI4GMXX5BXrk"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -s 10.210.0.1/32 -d 10.210.0.2/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j RETURN
-A PVEFW-HOST-IN -m comment --comment "PVESIG:7Qc5+IfLRg/qlUUYl1FGMhtbj1o"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -s 10.210.0.2/32 -d 10.210.0.1/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:u5rhVHdPpmYapXLzMKas3bMzrNc"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A tap112i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap112i0-IN -j ACCEPT
-A tap112i0-IN -m comment --comment "PVESIG:wMiFF9ebmPlzGLKPScD7xMM60E8"
-A tap112i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap112i0-OUT -m mac ! --mac-source 0a:d4:a0:ea:06:44 -j DROP
-A tap112i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap112i0-OUT -d 1.1.1.1/32 -p icmp -m icmp --icmp-type any -j DROP
-A tap112i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap112i0-OUT -m comment --comment "PVESIG:n7ENlEiKx0JmMoOFkF4deGY9jco"
COMMIT
# Completed on Thu Aug  3 09:17:24 2023
# Generated by iptables-save v1.8.9 on Thu Aug  3 09:17:24 2023
*nat
:PREROUTING ACCEPT [50168:3024500]
:INPUT ACCEPT [9904:505769]
:OUTPUT ACCEPT [209:12540]
:POSTROUTING ACCEPT [40462:2530219]
-A PREROUTING -d <public_ip>/32 -p tcp -m tcp --dport 2 -j DNAT --to-destination 10.1.0.112:22
-A POSTROUTING -s 10.1.0.0/24 -o enp5s0 -j MASQUERADE
COMMIT
# Completed on Thu Aug  3 09:17:24 2023

- 10.1.0.0/24 being the local subnet on vmbr1
- 10.210.0.0/24 the subnet dedicated to the cluster network

edit : In this configuration, I have no outbound traffic to internet from 10.1.0.112.
 
Last edited:
meepmeep said:
112.fw :

Code: 
[OPTIONS]

enable: 1
policy_out: ACCEPT
policy_in: ACCEPT

[ALIASES]

cloudflaredns 1.1.1.1

[RULES]

OUT DROP -dest 1.1.1.1 -p icmp -log nolog -icmp-type any

cluster.fw
Code: 
[OPTIONS]

enable: 1
policy_in: ACCEPT
log_ratelimit: burst=5,enable=1,rate=1/second

[ALIASES]

cloudflareDNS 1.1.1.1
hub 10.1.0.112

[IPSET all_vm_except_fw] # All VM ip except FW

10.0.0.0/16 # Internal LAN
!10.0.0.254 # FW ip
10.1.0.2 # Marvin
10.1.0.253
10.7.0.0/24 # vpn client subnet

[group isolated_vm] # VM isolé (block vmbr1 except FW)

OUT DROP -dest +all_vm_except_fw -p icmp -log alert
OUT DROP -dest +all_vm_except_fw -p udp -log alert
OUT DROP -dest +all_vm_except_fw -p tcp -log alert


iptables-saves
Code: 
# Generated by iptables-save v1.8.9 on Thu Aug  3 09:17:24 2023
*raw
:PREROUTING ACCEPT [592304858:527188676860]
:OUTPUT ACCEPT [75569371:14798661673]
COMMIT
# Completed on Thu Aug  3 09:17:24 2023
# Generated by iptables-save v1.8.9 on Thu Aug  3 09:17:24 2023
*filter
:INPUT ACCEPT [2:80]
:FORWARD ACCEPT [36:3708]
:OUTPUT ACCEPT [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
:tap112i0-IN - [0:0]
:tap112i0-OUT - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m physdev --physdev-out tap112i0 --physdev-is-bridged -j tap112i0-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Mwu3OTiSWwPSTAVjBZ6Skny0gt8"
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap112i0 --physdev-is-bridged -j tap112i0-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:wxnGXb+tg9M+to8yI4GMXX5BXrk"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -s 10.210.0.1/32 -d 10.210.0.2/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j RETURN
-A PVEFW-HOST-IN -m comment --comment "PVESIG:7Qc5+IfLRg/qlUUYl1FGMhtbj1o"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 10.210.0.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -s 10.210.0.2/32 -d 10.210.0.1/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:u5rhVHdPpmYapXLzMKas3bMzrNc"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A tap112i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap112i0-IN -j ACCEPT
-A tap112i0-IN -m comment --comment "PVESIG:wMiFF9ebmPlzGLKPScD7xMM60E8"
-A tap112i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap112i0-OUT -m mac ! --mac-source 0a:d4:a0:ea:06:44 -j DROP
-A tap112i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap112i0-OUT -d 1.1.1.1/32 -p icmp -m icmp --icmp-type any -j DROP
-A tap112i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap112i0-OUT -m comment --comment "PVESIG:n7ENlEiKx0JmMoOFkF4deGY9jco"
COMMIT
# Completed on Thu Aug  3 09:17:24 2023
# Generated by iptables-save v1.8.9 on Thu Aug  3 09:17:24 2023
*nat
:PREROUTING ACCEPT [50168:3024500]
:INPUT ACCEPT [9904:505769]
:OUTPUT ACCEPT [209:12540]
:POSTROUTING ACCEPT [40462:2530219]
-A PREROUTING -d <public_ip>/32 -p tcp -m tcp --dport 2 -j DNAT --to-destination 10.1.0.112:22
-A POSTROUTING -s 10.1.0.0/24 -o enp5s0 -j MASQUERADE
COMMIT
# Completed on Thu Aug  3 09:17:24 2023

- 10.1.0.0/24 being the local subnet on vmbr1
- 10.210.0.0/24 the subnet dedicated to the cluster network

edit : In this configuration, I have no outbound traffic to internet from 10.1.0.112.
Click to expand...
Please try to disabe the mac filtering for the VM firewall and check via tcpdump where the packages are going trough tcpdump -i vmbr1 icmp and tcpdump -i enp5s0 icmp. This should also show you if the masquerade works as expected.
 
With firewall enabled (and MAC filtering disabled).
For vmbr1 and enp5s0, results are the same :

ping 1.1.1.1 from VM

Code: 
tcpdump -i enp5s0 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp5s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:04:21.962372 IP 10.1.0.112 > dns.google: ICMP echo request, id 47540, seq 1, length 64
14:04:22.964483 IP 10.1.0.112 > dns.google: ICMP echo request, id 47540, seq 2, length 64
14:04:24.490536 IP 10.1.0.112 > one.one.one.one: ICMP echo request, id 56595, seq 1, length 64
14:04:25.492412 IP 10.1.0.112 > one.one.one.one: ICMP echo request, id 56595, seq 2, length 64
14:04:26.516469 IP 10.1.0.112 > one.one.one.one: ICMP echo request, id 56595, seq 3, length 64

ping 8.8.8.8 from VM
Code: 
14:05:52.179453 IP 10.1.0.112 > dns.google: ICMP echo request, id 62961, seq 1, length 64
14:05:53.204388 IP 10.1.0.112 > dns.google: ICMP echo request, id 62961, seq 2, length 64
14:05:54.228377 IP 10.1.0.112 > dns.google: ICMP echo request, id 62961, seq 3, length 64

No echo reply at all on the VM.
 
Strange thing, the same rule on the other node of my cluster works. I'm quite lost on this one ...

edit : and it's working on another vm on the same node as 112 ... o_O
 
Last edited:
meepmeep said:
With firewall enabled (and MAC filtering disabled).
For vmbr1 and enp5s0, results are the same :

ping 1.1.1.1 from VM

Code: 
tcpdump -i enp5s0 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp5s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:04:21.962372 IP 10.1.0.112 > dns.google: ICMP echo request, id 47540, seq 1, length 64
14:04:22.964483 IP 10.1.0.112 > dns.google: ICMP echo request, id 47540, seq 2, length 64
14:04:24.490536 IP 10.1.0.112 > one.one.one.one: ICMP echo request, id 56595, seq 1, length 64
14:04:25.492412 IP 10.1.0.112 > one.one.one.one: ICMP echo request, id 56595, seq 2, length 64
14:04:26.516469 IP 10.1.0.112 > one.one.one.one: ICMP echo request, id 56595, seq 3, length 64

ping 8.8.8.8 from VM
Code: 
14:05:52.179453 IP 10.1.0.112 > dns.google: ICMP echo request, id 62961, seq 1, length 64
14:05:53.204388 IP 10.1.0.112 > dns.google: ICMP echo request, id 62961, seq 2, length 64
14:05:54.228377 IP 10.1.0.112 > dns.google: ICMP echo request, id 62961, seq 3, length 64

No echo reply at all on the VM.
Click to expand...
Are these ping tests with the rule for dropping based on destination IP 1.1.1.1 enabled? It seems that both, icmp output filtering and NAT are not working as expected in this case. Which is in contradiction to the iptables rule which is definitely there
meepmeep said:
-A tap112i0-OUT -d 1.1.1.1/32 -p icmp -m icmp --icmp-type any -j DROP
Click to expand...
which can be seen here.

meepmeep said:
Strange thing, the same rule on the other node of my cluster works. I'm quite lost on this one ...
Click to expand...
What do you mean by that? By applying to a different VM or by migrating the VM to that node and retesting?
 
I must have messed up the test .. with the rule enabled I don't have any tcpdump trace when pinging 1.1.1.1.
For 8.8.8.8, I still only get icmp echo request and no reply.

Chris said:
What do you mean by that? By applying to a different VM or by migrating the VM to that node and retesting?
Click to expand...
It works for other vm on the same cluster even the same node.

I find the only difference between them and VM112.

  • It works with vm VM that have my opnsense as gateway (10.1.0.253).
    Code: 
    default via 10.1.0.253 dev ens18 onlink
10.1.0.0/24 dev ens18 proto kernel scope link src 10.1.0.112
  • it doesn't work (like VM112) when they use the node as gateway (10.1.0.2).
    Code: 
    default via 10.1.0.2 dev ens18 onlink
10.1.0.0/24 dev ens18 proto kernel scope link src 10.1.0.112

As soon as I switch vm112's gateway to opnsense, it works : rules apply correctly.
But I need this specific configuration (route directly to internet and not through opnsense)

Any idea what could be modify on the node ?
 
Last edited:
meepmeep said:
I must have messed up the test .. with the rule enabled I don't have any tcpdump trace when pinging 1.1.1.1.
For 8.8.8.8, I still only get icmp echo request and no reply.


It works for other vm on the same cluster even the same node.

I find the only difference between them and VM112.

  • It works with vm VM that have my opnsense as gateway (10.1.0.253).
    Code: 
    default via 10.1.0.253 dev ens18 onlink
10.1.0.0/24 dev ens18 proto kernel scope link src 10.1.0.112
  • it doesn't work (like VM112) when they use the node as gateway (10.1.0.2).
    Code: 
    default via 10.1.0.2 dev ens18 onlink
10.1.0.0/24 dev ens18 proto kernel scope link src 10.1.0.112

As soon as I switch vm112's gateway to opnsense, it works : rules apply correctly.
But I need this specific configuration (route directly to internet and not through opnsense)

Any idea what could be modify on the node ?
Click to expand...
So, your latest tcpdump shows that the packets are correctly leaving on interface enp5s0, but with the VMs IP address and not the masked host IP address. This means that the NAT rule is not working as expected.

Please share your current network configuration cat /etc/network/interfaces as well as the rules from the nat table iptables-save -t nat in order to investigate further.
 
on vm112

Code: 
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug ens18
iface ens18 inet static
        address 10.1.0.112
        netmask 255.255.255.0
        gateway 10.1.0.2

iface ens18 inet6 dhcp
An no iptables rules

On the node

Code: 
auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp5s0
iface enp5s0 inet static
        address <public_ip>/32
        gateway <public_gateway>

iface enp5s0 inet6 static
        address <public_ipv6>/128
        gateway fe80::1

auto enp5s0.4000
iface enp5s0.4000 inet static
        address 10.210.0.2/24

auto vmbr0
iface vmbr0 inet static
        address <public_ip>/32
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#WAN Bridge

iface vmbr0 inet6 static
        address <public_ipv6>/80

auto vmbr1
iface vmbr1 inet static
        address 10.1.0.2/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/16' ! -d '10.0.0.0/16' -o enp5s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/16' ! -d '10.0.0.0/16' -o enp5s0 -j MASQUERADE
#LAN Bridge

iptables nat
Code: 
# Generated by iptables-save v1.8.9 on Fri Aug  4 09:17:15 2023
*nat
:PREROUTING ACCEPT [62845:3749833]
:INPUT ACCEPT [15012:767887]
:OUTPUT ACCEPT [1698:101880]
:POSTROUTING ACCEPT [49422:3074684]
-A POSTROUTING -s 10.0.0.0/16 ! -d 10.0.0.0/16 -o enp5s0 -j MASQUERADE
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -o enp5s0 -j MASQUERADE
COMMIT
# Completed on Fri Aug  4 09:17:15 2023

(10.0.0.0/16 is the subnet of the other node, I don't know why it's deploy on this node)
 
Well there are a few things which you might improve upon. First of all I would recommend to only set the public IP address on the interface itself, not again on vmbr0 or rather attach the interface to the bridge as bridge port, if the IP is needed on the bridge.

Regarding your nat setup, then the rule shown on `post-up` does not match the network configuration for that bridge and should be corrected. Also, I would explicitly add the forward rule and also enable ipv4 forwarding as described in the docs [0].

Where does the
meepmeep said:
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -o enp5s0 -j MASQUERADE
Click to expand...
come from? Was this set manually? Also, you don't need the exclude for the destination subnet here, as packets for the same network are not routed.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_masquerading
 
Code: 
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -o enp5s0 -j MASQUERADE

.. is set by crontab at boot (i know, it's not very clean).
I modify the vmbr0 configuration by removing the public ip, removed the destination exclusion on iptables nat rules, and got the same result.

I re-read the doc and add the conntrack zone rules, just to try ... and it works !
Code: 
post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

Thank you !!
 
  • Like
Reactions: Chris
meepmeep said:
Code: 
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -o enp5s0 -j MASQUERADE

.. is set by crontab at boot (i know, it's not very clean).
I modify the vmbr0 configuration by removing the public ip, removed the destination exclusion on iptables nat rules, and got the same result.

I re-read the doc and add the conntrack zone rules, just to try ... and it works !
Code: 
post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

Thank you !!
Click to expand...
Glad it works, please mark the thread as solved.

On a side node, since I see you have partially also configured IPv6: Make sure to block IPv6 traffic for the VM if you don't want it to connect via it.

Also, instead of placing the NAT rule by a crontab entry, place it directly in the network configuration. This is the place where it should be, as it is directly related to your network setup.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom