Question about ipsets and cluster fw

[holmis]

Hello.

We are in the process of migrating some physical servers to Proxmox vm (newbies on this...). In our current Linux servers we have a cron script collecting IP nubers/subnets from a number of sources and updating a blacklist iptables ipset every night.

Would it be possible, and if so how, to do this in the cluster firewall configuration instead, ie not having to do this in every single vm? The vm:s will be running Linux and Windows, if that matters.

Regards,
Åke

 

[aaron]

If I understand you correctly, you want to gather IP addresses of the guests on a regular basis, to use them to apply filter/whitelists on your firewalls?

If the guests all have the guest agent running and enabled in the Options of the VM, then you can gather the network info that way.

For example, via the CLI:
qm guest cmd {vmid} network-get-interfaces

Alternatively, you can use the API. The Proxmox VE wiki has an article in how to authenticate and use the API. I would recommend to use an API token, as that makes the authentication for some external tool/script a lot easier: https://pve.proxmox.com/wiki/Proxmox_VE_API

See the API viewer (also available locally on your Proxmox VE instance): https://pve.proxmox.com/pve-docs/api-viewer/index.html

The first step would be to query cluster/resources to get the current location of all the VMs. Then nodes/{node}/qemu/{vmid}/agent/network-get-interfaces to get the same info as with the CLI command from earlier.

 

[holmis]

If I understand you correctly, you want to gather IP addresses of the guests on a regular basis, to use them to apply filter/whitelists on your firewalls?

Sorry, almost the other way around. We are gathering external addresses that we want to put in an ipset and blacklist.

 

[aaron]

Ah okay, to apply to the internal Proxmox VE firewall? Then check the API documentation as well.
If you are not sure which endpoint to use, do it in the web UI manually. Keep the dev tools (network) in the browser open and check which API endpoints are called.

 

[holmis]

Thank You for the suggestion.

While this works in theory, having (?) to execute pvesh for every entry makes it extremly slow. On ur very old test machines (Xeon X5660@2.80GHz, SSD) every entry takes ~2 seconds. Considering we are talking about 25 - 30 000+ entrys, that will take 12+ hours on these machines. Remember, this is going to take place every night. Doing this directly with ipset and iptables takes about 1 second in total.

Is there any way to speed this up?

 

[aaron]

If you want to run it locally on the nodes, you could also take a look at the pve-firwall CLI tool, that might be faster.

The firewall settings are stored in /etc/pve/firewall/. Check out those files and how they are structured. If you set the configs in there directly, you circumvent any tooling that verifies the correctness of the settings. So be careful

 

[holmis]

If we manage the setting locally on each node, would it be a total disaster if we set up an empty cluster wide ipset in the gui and then manage the addresses in the ipset directly with the os ipset command locally on all of the nodes (like we are doing now in the non Proxmox machines)?

 

[aaron]

If the Proxmox VE firewall is enabled, it will regularly apply the settings from the config files. If you go and apply iptable rules yourself directly, they can be overwritten.
Even if it works, troubleshooting might be a lot harder unless you remember to mention that direct customization.
In the end, you are dealing with a Debian based distribution.

 

[holmis]

Ok, looks like the best option for us is to not use the proxmox firewall but iptables/ipset directly.