I’m looking to route my network traffic through my Proxmox host. The plan is to sit the server between the fiber modem and my mesh WiFi nodes to act as the edge router.
The Goal: I need granular control over the traffic passed to the WiFi clients, specifically:
DNS/Domain blocking (ad-blocking or parental controls).
MAC address blacklisting.
Time-based access scheduling for specific local IPs and MACs.
The Question: What is the best "out of the box" solution to virtualize on Proxmox for this? I'm open to full firewall distros or lighter-weight containerized solutions.
I'd recommend Opnsense. It fits all these requirements and it can be installed as a VM on Proxmox VE (just download the iso-vga installer) https://opnsense.org/
It's basically an open source firewall. Among other functions, you can create scheduled firewall rules (to block traffic at certain times) and use its integrated DNS resolver, which has built-in DNS blocklists for ad-blocking, parental control, etc.
Traffic blocking from specific MAC addresses can be done through firewall rules and/or using the integrated dhcp server (by not giving IPs to specific MACs).
Of course the best way would be to block the addresses directly on L2 (switch/AP level), so they cannot connect to the network. If you block them at firewall level, you're only blocking the traffic that goes through the firewall, not the local network.
A Debian box is quite capable of being a very decent router and you already have one: Proxmox. However, that is one for the likes of me to run up. What you probably need is something with a GUI and a reasonable learning curve. VyOS is superb but perhaps a bit involved for now (CLI only, unless the fabled web GUI has actually been written).
That leaves OPNSense and pfSense - either will do the job admirably. Both have a web GUI and have loads of features. OPNSense is an "aggravated fork" of pfSense that has settled down and has a passionate following. pfSense is what I use rather a lot but I am a massive fan of choice and I note that @carles89 endorses it.
Up to you but I suggest you poke around the forums of both OPNSense and pfSense and their Reddits first and decide which community you prefer because you will probably need to ask them stuff.