Zerotier network bypassing firewall. Add network adapter in VM to proxmox config?

[BrianVB]

Hello - this is my first time trying to use proxmox. I have a web development background, but extremely limited networking / IT knowledge. So this is all very new to me; I'm just saying that in case it helps formulate responses.

In my Proxmox setup, I have two VMs. One is a ubuntu server 22.04.3 for the sole purpose of self-hosting a zerotier controller running in. The other is another ubuntu server instance hosting web-based services. My hopes are to create a zerotier private network to allow members to access these services. It's volunteer work for a local spiritual / religious organization, so security isn't of utmost importance, but I want to use this as a good learning experience.

I followed instructions I could find to setting up the firewall in proxmox and applying it to each of the VMs. I am trying to make sure that the only access can be through either the local network, or the zerotier network, and that the only ports that can be accessed are the ones for the services on that VM. It took a bit of reading, but I was able to get everything to work with one gaping exception - accessing the devices using their IP address on the zerotier network completely bypasses the firewall. Once I turned the firewall on (before adding rules), it properly blocked me from using SSH or accessing the services on their local IPs (for example, 192.168.1.44:3777) but I could still SSH and access them if I used the zerotier IP (10.12.35.123:3777).

After doing quite a bit of reading, I found that the firewall needs to be enabled on the Network Device for the VM in order for it to work properly (in addition to the datacenter, node and VM). Zerotier creates its own network device, inside the VM. My best guess here is that I somehow need to make sure that the firewall is being applied to this network device as well.

Looking in the proxmox webgui, if I go to the VM > Hardware, I can see 'Network Device' there with the one it created, where I have enabled the firewall. When SSH'd into the VM, I can use ifconfig to see the network device that I see in the web gui, with the same MAC address. Also, using ifconfig, I can see the network device created by Zerotier, with its MAC address. I am wondering if I can somehow make the firewall apply to that network device. I see that I can add more network devices, but my gut tells me I can't just add a new one, using the MAC address that I see for the zerotier device in the VM, and have it work like I want. I haven't even tried that yet because I am concerned about creating some type of conflict.

Searching in the forum, I do see other people that are using Zerotier (for example), but from all of those converstaions it looks like they are using it directly to access the web gui. None of them mention the firewall issues I am seeing.

So, I am wondering how I can make sure that the firewall rules apply to the members of the zerotier network accessing the VM. That being said, I am a total neophyte to this and I realize that what I am trying to do might not be the best way to try to accomplish my goals. I've read stuff about bridging but honestly this is all still quite a bit over my head. If I'm not following best practices here, or my approach is just completely wrong, I'd be happy to be pointed in the right direction.

 

[BrianVB]

Okay, so this is a long shot, but based on the this topic I started looking around to see if I could figure out a way to do this for the VM instead of the proxmox installation. I did find what I believe to be the VM config file @ /etc/pve/nodes/proxmox/qemu-server/101.conf and I see a line:

net0: virtio=BC:24:11:7B:A5:6B,bridge=vmbr0,firewall=1

Seems like a potential lead on how to add in the network device that's existing in the VM but not showing in proxmox.

 

[BrianVB]

After creating a backup, I decided to try to add the network device on the VM using the MAC address of the one that already existed for the Zerotier network. I rebooted the machine, and it did not seem to be effective in applying the firewall.

 

[_gabriel]

host can't see software interface of the vm,
host just see packets from network card set in hardware of the vm.

 

[BrianVB]

Thanks for the follow up @_gabriel. So, I guess my question is, why isn't the host able to properly apply the firewall rules if it's all coming through the network card hardware? How is traffic coming through that software interface completely bypassing the firewall in proxmox? Since it's all coming through the hardware (ethernet) interface, why doesn't the firewall work on it?

 

[_gabriel]

what firewall rules do you want to apply ?

 

[BrianVB]

I am trying to block all access to the machine except for SSH and a few ports that are hosting web-based services (9443,81,5556) - and to only allow access to those from the LAN (192.168.1.0/24) and the Zerotier network (10.x.x.0/24).

The web-based services are all hosted in docker containers, which doesn't play nice with ufw. I think I have found a way to get ufw to work within the VM, but I'd much prefer to have it all centralized in proxmox.

 

[_gabriel]

I have found a way to get ufw to work within the VM, but I'd much prefer to have it all centralized in proxmox.

You can't centralize in the firewall of the proxmox host as it doesn't see the Zerotier network (10.x.x.0/24) because the Zerotier traffic is encapsulated into 192.168.1.x flow of the vm/container.
You must use "in-guest" firewall for Zerotier or any other vpn.

btw, no access from outside is possible without a NAT port redirection set in your internet gateway.

(sorry for my wording... )

 

[BrianVB]

Thanks for your responses. I understand what you mean, it seems like using ufw from within the VM is the way to go here.

 

[hutchwilco]

@BrianVB I realise this is some months later, but I think the answer here is to realise that ZeroTier is like a whole network of its own, overlaid above the VE LAN and interacting only with your VM, and only inside the VM. So imagine it only touches the VM, and never any of your VE host hardware at all.
So the controls you need to apply will have to be ZeroTier rules, which you'll need to apply in ZeroTier Central. This is only going to work against other clients in the ZeroTier network you have the VM on.
Have a look here (however I think these instructions are flawed): https://www.zerotier.com/blog/using-flow-rules-to-direct-users-to-services/

I believe what you're after is best achieved by using ZeroTier's tag rules, to tag the VM, and define only the destination node (your VM identity) as being allowed destport 22.
More on the tag method: https://discuss.zerotier.com/t/flow-rules-to-restrict-traffic-to-remote-desktop-only/12202

 

[BrianVB]

Thanks @hutchwilco - I understand now but definitely was new to the concept when I started.

I have set up some flow rules through ZeroTier to make sure that the servers I have are the only parts of the network that can be reached by clients. But I didn't go so far as to try to limit ports. I had made some attempts, but it wasn't working how I expected and there was no way to really troubleshoot it. So I ended up going the UFW route, which has actually been working quite nicely now that I have had a done it a few times!

Only problem now is that my Proxmox install consistently freezes and does a green-screen thing. Then the hardware gets REALLY hot. But, there are other forum topics about that on here and is a story for another day haha.