Proxmox FW shennanigans

J

Joris L.

Active Member
Proxmox Subscriber
May 16, 2020
278
17
38
51
Antwerp, Belgium
commandline.be
To my shame and dismay I must post to ask for feedback on this. I'm not the greatest documentation reader but I think I covered most if not all of it regarding.

To have the Proxmox firewall(s) working well is not proving 'easy' Not even with a simple set-up. To keep things manageable I started using only /etc/network/interfaces and the proxmox webui for firewall configurations. This because I'm using masquerading with DNAT.

issue 0 After hours of messing around leading to doubting my own sanity I've again 'partially fixed' the issue by disabling the DC and HOST FW.
  • I've not read any suggestion with masquerading there is a requirement for setting up rules in either or both DC and HOST firewalls so i did not.
  • Also, when attempting to review the firewall ruleset this is a bewildering maze of chains, implied rules and firewall rule logic. I'm just one guy :-D
  • because exit traffic filtering doesn't matter I allow out any firewall traffic, yet, this does not maintain state for the masqueraded traffic ?

Issue 1 firewall changes made in the web-UI are applied when ?
  • I don't understand really, since this is a server running multiple VM I prefer to not reboot on any change, just to make sure.
  • I've worked with iptables and nftables on other distro and never had to reboot to apply configuration.
issue 2 what the DC fw does vs what the HOST fw does ?
  • Running a one-system Proxmox instance leads me to think the DC=HOST in this case ?
  • Not clear to me, neither from the documentation. I'm sure it would make more sense once I run a cluster or an actual datacenter size
issue 3 firewall rules are not visible in the webUI and the webUI doesn't allow for all firewall rules ?
  • this may be obvious to people who have a dedicated role to maintaining proxmox, to me it is a bit of a struggle
  • when do I use the UI, should i only add extra rules in /etc/network/interfaces(.new) or can I add other rules and where so ?
 
Last edited:
Joris L. said:
issue 0 After hours of messing around leading to doubting my own sanity I've again 'partially fixed' the issue by disabling the DC and HOST FW.
Click to expand...
Disabling the firewall at DC level disables it completely, FWIW. So e.g. VM rules won't have effect.

Joris L. said:
Issue 1 firewall changes made in the web-UI are applied when ?
  • I don't understand really, since this is a server running multiple VM I prefer to not reboot on any change, just to make sure.
  • I've worked with iptables and nftables on other distro and never had to reboot to apply configuration.
Click to expand...
For the (old) iptables-firewall, all rules are reloaded every 10 seconds. The new nftables-based firewall reloads them every 5 seconds.
Thus you don't have to reboot to apply firewall changes with PVE too? What did give you the idea that this might be necessary?

Joris L. said:
issue 2 what the DC fw does vs what the HOST fw does ?
  • Running a one-system Proxmox instance leads me to think the DC=HOST in this case ?
  • Not clear to me, neither from the documentation. I'm sure it would make more sense once I run a cluster or an actual datacenter size
Click to expand...
Well, DC rules are applied on all nodes in a cluster, host rules only on the host they are created on. So yes, for single-node deployments there isn't really a difference.

Joris L. said:
issue 3 firewall rules are not visible in the webUI and the webUI doesn't allow for all firewall rules ?
  • this may be obvious to people who have a dedicated role to maintaining proxmox, to me it is a bit of a struggle
  • when do I use the UI, should i only add extra rules in /etc/network/interfaces(.new) or can I add other rules and where so ?
Click to expand...
What do you mean here exactly? What rules are not visible in the UI? How/where did you add them?
/etc/network/interfaces is only concerned with network interface setup and does not have anything do to with the firewall.

Please see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_configuration_files to see where settings & rules go, and https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_firewall_rules for the format of the rules.

Hope this helps :)
 
  • Like
Reactions: shanreich
cheiss said:
Disabling the firewall at DC level disables it completely, FWIW. So e.g. VM rules won't have effect.


For the (old) iptables-firewall, all rules are reloaded every 10 seconds. The new nftables-based firewall reloads them every 5 seconds.
Thus you don't have to reboot to apply firewall changes with PVE too? What did give you the idea that this might be necessary?


Well, DC rules are applied on all nodes in a cluster, host rules only on the host they are created on. So yes, for single-node deployments there isn't really a difference.


What do you mean here exactly? What rules are not visible in the UI? How/where did you add them?
/etc/network/interfaces is only concerned with network interface setup and does not have anything do to with the firewall.

Please see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_configuration_files to see where settings & rules go, and https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_firewall_rules for the format of the rules.

Hope this helps :)
Click to expand...
Thanks for the elaborate reply.

I'm still hurt with not being able to get IP masquerading to work with proxmox, regardless of anything I've tried.

The documentation mentions this should work with an adaptation such as below, yet this provably doesn't work. Traffic exits without being natted from one bridge named 'test' to the exit on vmbr0

auto vmbr0​
#real IP address​
iface vmbr0 inet static​
address 198.51.100.5/24​
gateway 198.51.100.1​


auto test​
#private sub network​
iface test inet static​
address 10.10.10.1/24​
bridge-ports none​
bridge-stp off​
bridge-fd 0​
​
post-up echo 1 > /proc/sys/net/ipv4/ip_forward​
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE​
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE​

This is not fun since it is blocking me from moving ahead with this proxmox POC
 
I assume you do not have activated the nftables-based firewall? As you are using iptables rules.

Is this the actual content of your /etc/network/interfaces? (next time, please also post such things in codetags, please, makes them readable!)
Seems off, e.g. vmbr0 is not connected to anything.

Can you post the whole contents of /etc/network/interfaces, verbatim, please?

Edit: Also, from the configuration above what you are trying to do (aka. https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_masquerading), you actually want SNAT, not DNAT? In this case, I'd suggest looking at SDN, where you can create a SNAT on a VNet via the web UI, without having to manually set up bridges and NAT rules.
 
Last edited:
thanks for the follow up.

The for mentioned is illustrative to the minimum documentatin provided by Proxmox, which does not work for NAT/Masquerading for traffic to internet from any VM. Not matter what was tried of configuration. To my knowledge this should be adequate.

Code: 
auto vmbr0

iface vmbr0 inet static

    address <mypublicip>/netmask

    gateway <mypublicipgatewayip>

    bridge-ports <interfacename>

    bridge-stp off

    bridge-fd 0

    hwaddress <mac-address>



auto exitnet

iface exitnet inet static

    address 10.1.1.1/24

    bridge-ports none

    bridge-stp off

    bridge-fd 0


        post-up iptables -t nat -A POSTROUTING -s 10.1.1.0/24 -o vmbr0 -j MASQUERADE

        post-down iptables -t nat -D POSTROUTING -s 10.1.1.0/24 -o vmbr0 -j MASQUERADE


output from a VM looks like this on the vmbr0 interface, no reply is observed, to me this indicates the NAT/Masquerading is not happening


1 0.000000000 10.1.1.10→ 4.4.4.4 ICMP 98 Echo (ping) request id=0x0f42, seq=21/5376, ttl=62
 
Last edited:
How does the network configuration look like inside the VM?
How does the VM configuration look like? (qm config <vmid>)

What's the output of iptables -t nat -L
 
thanks for the follow @shanreich

The documented example provably does not work, as others also reported in these forums.

for example:

Code: 
ping -I exitnet 1.1.1.1

should masquerade the traffic and show echo-reply, it does not, this also shows the same when using tshark -i vmbr0 icmp

Code: 
1 0.000000000  10.1.1.10→ 4.4.4.4 ICMP 98 Echo (ping) request  id=0x0f42, seq=21/5376, ttl=62

as this above line shows, the VM IP 10.1.1.10 is reaching the WAN vmbr0 interface which is assumed to exit since last seen here,
the return traffic though is not seen, this part of the configuration is missing from the documentation or something is broken in the example
 
You must log in or register to reply here.
Top Bottom