[SOLVED] Proxmox 7.1-10 VM's and CT not able to exit internet or ping vmbrX interfaces (masquerade not applied)

P

Proxmox-Researcher

New Member
Mar 16, 2022
5
0
1
36
Hi,
I'm new to proxmox , and also this installation is fresh, I started with a 6.4 then upgraded manually to the 7.1-11.
I created next the first CT (using arch / debian / ubuntu ) templates and assigned manually the IP.
I did the same with also a VM installation and there's a very weird behavior and I'm dividing this in 2 problems , but first my network configuration:

1.png

Code: 
cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp41s0
iface enp41s0 inet static
    address xx.xx.xxx.67/26
    gateway xx.xx.xxx.65
    up route add -net xx.xx.xxx.64 netmask 255.255.255.192 gw xx.xx.xxx.65 dev enp41s0

iface enp41s0 inet6 static
    address ip:v:6:add:ress::2/64
    gateway fe80::1

auto enp41s0.4003
iface enp41s0.4003 inet static
    address 10.102.0.10/24
    mtu 1400
    up ip route add 10.0.0.0/8 via 10.102.0.2 dev enp41s0.4003
    down ip route del 10.0.0.0/8 via 10.102.0.2 dev enp41s0.4003

auto vmbr0
iface vmbr0 inet static
    address 10.102.10.10/23
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '10.102.10.0/23' -o enp41s0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.102.10.0/23' -o enp41s0 -j MASQUERADE
#MGMT - do NOT alter!!!

auto vmbr1
iface vmbr1 inet static
    address 192.168.0.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#Green

auto vmbr2
iface vmbr2 inet static
    address 192.168.1.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#Orange

auto vmbr3
iface vmbr3 inet static
    address 192.168.2.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#Blue

auto vmbr4
iface vmbr4 inet static
    address 192.168.111.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '192.168.111.0/24' -o enp41s0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.111.0/24' -o enp41s0 -j MASQUERADE
#WAN

And this is the current brctl show :

Code: 
brctl  show
bridge name    bridge id        STP enabled    interfaces
fwbr101i0        8000.be8ec2cdf3ee    no        fwln101i0
                                            veth101i0
vmbr0        8000.66037ed2247a    no      
vmbr1        8000.e6fbbddfd6d2    no      
vmbr2        8000.0a6711d745c3    no      
vmbr3        8000.8ad64d269cda    no      
vmbr4        8000.eeda74f994b7    no            fwpr101p0

=====================================
1 ) Unable to ping vmbr4 from CT
=====================================
I have a CT behind vmbr4 with IP
Code: 
[root@arch ~]# ip route
default via 192.168.111.1 dev eth0 proto static
192.168.111.0/24 dev eth0 proto kernel scope link src 192.168.111.2

I'm trying to ping 192.168.111.1 (vmbr4) but regardless I don't receive any reply.

Code: 
root@proxmox ~ # tcpdump -nnvvi any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
19:07:10.743970 veth101i0 P   IP (tos 0x0, ttl 64, id 43072, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 1, length 64
19:07:10.744001 fwln101i0 Out IP (tos 0x0, ttl 64, id 43072, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 1, length 64
19:07:10.744003 fwpr101p0 P   IP (tos 0x0, ttl 64, id 43072, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 1, length 64
19:07:10.744003 vmbr4 In  IP (tos 0x0, ttl 64, id 43072, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 1, length 64
19:07:11.769284 veth101i0 P   IP (tos 0x0, ttl 64, id 43256, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 2, length 64
19:07:11.769297 fwln101i0 Out IP (tos 0x0, ttl 64, id 43256, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 2, length 64
19:07:11.769297 fwpr101p0 P   IP (tos 0x0, ttl 64, id 43256, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 2, length 64
19:07:11.769297 vmbr4 In  IP (tos 0x0, ttl 64, id 43256, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 2, length 64
19:07:12.793282 veth101i0 P   IP (tos 0x0, ttl 64, id 43361, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 3, length 64
19:07:12.793295 fwln101i0 Out IP (tos 0x0, ttl 64, id 43361, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 3, length 64
19:07:12.793295 fwpr101p0 P   IP (tos 0x0, ttl 64, id 43361, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 3, length 64
19:07:12.793295 vmbr4 In  IP (tos 0x0, ttl 64, id 43361, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 3, length 64
19:07:13.817295 veth101i0 P   IP (tos 0x0, ttl 64, id 43404, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 4, length 64
19:07:13.817305 fwln101i0 Out IP (tos 0x0, ttl 64, id 43404, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 4, length 64
19:07:13.817306 fwpr101p0 P   IP (tos 0x0, ttl 64, id 43404, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.111.2 > 192.168.111.1: ICMP echo request, id 45363, seq 4, length 64

As you can see, vmbr4 receives the packet but for some reason there's no reply.


=====================================
2 ) Masquerade not matching in postrouting
=====================================
I noticed then another problem trying to reach 1.1.1.1 via ping or https, that the packet was exiting enp41s0 as 192.168.111.2 , so I said OK, let's masquerade maybe that's the problem.
I added the rule as you see from the config, but again it didn't match, and then I used -j TRACE with iptables to debug and see how the packet was hitting the nat table / POSTROUTING and I saw the following (while using tcpdump on enp41s0 which is the only physical interface)
Code: 
cat /var/log/pve-firewall.log | grep nat
0 5 - 16/Mar/2022:15:59:19 +0100 received terminate request (signal)
0 5 - 16/Mar/2022:16:00:32 +0100 received terminate request (signal)
0 6 - 16/Mar/2022:16:18:51 +0100 TRACE: nat:PREROUTING:policy:1 IN=fwbr101i0 PHYSIN=veth101i0 MAC=ee:da:74:f9:94:b7:ea:6d:1f:a7:e8:c1:08:00 SRC=192.168.111.2 DST=1.1.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55008 DF PROTO=TCP SPT=36962 DPT=443 SEQ=485556381 ACK=0 WINDOW=64240 SYN
0 6 - 16/Mar/2022:16:18:51 +0100 TRACE: nat:POSTROUTING:policy:3 OUT=fwbr101i0 PHYSOUT=fwln101i0 SRC=192.168.111.2 DST=1.1.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55008 DF PROTO=TCP SPT=36962 DPT=443 SEQ=485556381 ACK=0 WINDOW=64240 SYN

As you can see , the packet is actually hitting nat: POSTROUTING but with a random PHYSOUT that this time was fwln101l0....

I remember using v6 in the past (for few weeks) and I didn't remember to have this problems, as soon as I set up the vmbrX interface and put a CT or VM behind it I'll get to internet without additional configurations....
 
Last edited:
This is where the problem was :
1647613115348.png
The firewall was enabled, and when the firewall was enabled these bridges where created :

Code: 
bridge name    bridge id        STP enabled    interfaces
fwbr101i0        8000.be8ec2cdf3ee    no        fwln101i0
                            veth101i0
having the traffic go nowhere. Nothing changed on iptables level, so the fw block here seems to be handled entirely with the bridges.
I have no idea why this was a choice since the traffic will end up regardless in the default physical interface (it's not like it's being dropped) , the only thing is that it won't match the masquerade so the VM / CT will have no access to WAN but it will still have LAN access..
 
I did an additional test now, wondering if there's any other purpose to the Firewall : Yes settings, and there isn't...
If you configure CT's or VM's on different VMBRX's they'll still be able to ping each other , so there's no network isolation.
It might be my impression but I find the network configuration very lacking compared to something like virtualbox.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom