Proxmox Secure Communications Configuration

[Whitterquick]

This may end up being quite a loaded question but…

What would be the best/easiest way to setup SSL/secure communication for the WHOLE server including all VMs and Containers?

Is it possible to have local/LAN access ONLY, with remote access only available via a VPN/tunnel? Would it be possible to still allow updating and certain actions like fetching album/box art in Plex or similar apps?

Looking for something as close to being an offline server as possible, while still having the benefits of being internet connected (if that is even possible), and do it with FOSS and as little trust as possible. Hope someone can help or guide me to some nice tutorials/ideas. Thanks to all in advance.

 

[Dunuin]

If it is not a hosted server just don't port-forward anything from the internet through your router to the Proxmox server. For LXCs/VMs it is possible to create a OPNsense/pfsense VM that acts as another firewall/router/gateway between your PVE host and the VMs/LXCs.

 

[Whitterquick]

It it is not a hosted server just don't port-forward anything from the internet through your router to the Proxmox server. For LXCs/VMs it is possible to create a OPNsense/pfsense VM that acts as another firewall/router/gateway between your PVE host and the VMs/LXCs.

Thanks! You have so much knowledge and have given me so many tips! How do you document everything? Do you use Bookstack or something similar, or just notes/documents on a computer?

 

[Dunuin]

Actually its just a bunch of txt files. I've setup a local mediawiki to be able document stuff better ordered but I'm not often using it, because in case of a total server failure I wouldn't be able to access it to redo stuff.

 

[Whitterquick]

Actually its just a bunch of txt files. I've setup a local mediawiki to be able document stuff better ordered but I'm not often using it, because in case of a total server failure I wouldn't be able to access it to redo stuff.

Do you use a VLAN for your VMs and Containers? Any reason to (or not to?) in a small home network?

 

[Dunuin]

Do you use a VLAN for your VMs and Containers? Any reason to (or not to?) in a small home network?

Yes, I use 13 VLANs. Benefit is that you can easily separate subnets which will increase the security and allows you to use stuff like QoS to priorize specific VLANs so that for example VoIP will always work with a good latency even if a backup is running. The disadvantage is that you need to buy more expensive managed switches if you want all the features and that everything gets very complicated. 9 of these 13 VLANs for example need internet access, DHCP, DNS and so on so I need a router like a OPNsense that can route between all 9 VLANs, hosts 8 DHCP server instances and so on. Firewall rules can also get a bit tricky. With a filewall between all 9 VLANs I need to create rules for every VLAN what is allowed and what not for each other VLAN. With 9 VLANs that are 72 VLAN to VLAN combinations. So it is much more work compared to just a normal router you receive from your ISP where you just got WAN to LAN. And if you don't only use the OPNsense for internal server communication between hosts and guests, but use OPNsense as a firewall/gateway/router for your complete home network, you should think about what you will do if the VM or the complete server fails.
If just the VMs crashes, can you still access PVE to fix stuff and reboot the OPNsense VM? What will work and what not if you can't get the OPNsense VM running again. Without a running OPNsense VM nothing here at home will have any internet access except for the smartphone that can use the mobile internet connection. Also I wouldn't be able to reach hosts/VMs in other VLANs so access to selfhosted services like bookstack, running in a DMZ, would be limited. And if your server really dies and you got no OPNsense for weeks you can't just replace the OPNsense VM with a normal consumer router because such a router just can't handle VLANs and is only designed to use WAN and LAN. And you would need to redo hundrets or thousands of firewall rules from memory. So it might be faster to just wait for a replacement server.
So that really would be a deal breaker for me if there wouldn't be the high availability feature of OPNsense. I've got 2 OPNsense VMs running all the time and they are always in sync. One is running in a KVM VM on my TrueNAS server. The other one on my Proxmox server. If one OPNsense VM will fail the other one will take its place within seconds. Also nice because I'm still online if I need to reboot/shutdown one OPNsense VM or complete server for maintaince or if the daily backup kicks in and shuts down the VM for some minutes.

 

[Whitterquick]

Yes, I use 13 VLANs. Benefit is that you can easily separate subnets which will increase the security and allows you to use stuff like QoS to priorize specific VLANs so that for example VoIP will always work with a good latency even if a backup is running. The disadvantage is that you need to buy more expensive managed switches if you want all the features and that everything gets very complicated. 9 of these 13 VLANs for example need internet access, DHCP, DNS and so on so I need a router like a OPNsense that can route between all 9 VLANs, hosts 8 DHCP server instances and so on. Firewall rules can also get a bit tricky. With a filewall between all 9 VLANs I need to create rules for every VLAN what is allowed and what not for each other VLAN. With 9 VLANs that are 72 VLAN to VLAN combinations. So it is much more work compared to just a normal router you receive from your ISP where you just got WAN to LAN. And if you don't only use the OPNsense for internal server communication between hosts and guests, but use OPNsense as a firewall/gateway/router for your complete home network, you should think about what you will do if the VM or the complete server fails.
If just the VMs crashes, can you still access PVE to fix stuff and reboot the OPNsense VM? What will work and what not if you can't get the OPNsense VM running again. Without a running OPNsense VM nothing here at home will have any internet access except for the smartphone that can use the mobile internet connection. Also I wouldn't be able to reach hosts/VMs in other VLANs so access to selfhosted services like bookstack, running in a DMZ, would be limited. And if your server really dies and you got no OPNsense for weeks you can't just replace the OPNsense VM with a normal consumer router because such a router just can't handle VLANs and is only designed to use WAN and LAN. And you would need to redo hundrets or thousands of firewall rules from memory. So it might be faster to just wait for a replacement server.
So that really would be a deal breaker for me if there wouldn't be the high availability feature of OPNsense. I've got 2 OPNsense VMs running all the time and they are always in sync. One is running in a KVM VM on my TrueNAS server. The other one on my Proxmox server. If one OPNsense VM will fail the other one will take its place within seconds. Also nice because I'm still online if I need to reboot/shutdown one OPNsense VM or complete server for maintaince or if the daily backup kicks in and shuts down the VM for some minutes.

That is a very pro setup like we use at work. I don’t have a managed switch at this time so maybe something to look at more later. I am just experimenting with a lot of things right now, with SMB shares being the only actual use that doesn’t need tinkering/learning

Would you say OPNsense could have a place as a secondary router/firewall in a network (in a VM while also using the hardware router for this)? I also like your failover idea and was looking to setup something similar for my DNS/Pi-hole (not for router just yet though).

 

[Dunuin]

Would you say OPNsense could have a place as a secondary router/firewall in a network (in a VM while also using the hardware router for this)?

Sure, you don't need to use OPNsense for the complete network. If you just use OPNsense to route between some bridges or VLANs inside your server you still get most of the benefits but don't need managed switches. And failover isn't that important too, because a fail on the OPNsense VM would only affect the PVE server itself and not all hosts at home.

I also like your failover idea and was looking to setup something similar for my DNS/Pi-hole (not for router just yet though).

DNS got redundancy by design. You can always tell every OS to use atleast two DNS servers. It will randomly choose one of them and if it can't access it, it will try the other one. If you want to use 2 DNS servers just make sure to really use two pi-holes and don't mix a pi-hole with another normal nonblocking DNS server. Pi-hole is working by blocking DNS queries. So if a Domain gets blocked it might be possible that the clients OS will just ask the second DNS server again if it can finish the job. If it is another Pi-hole it will block it too and all is fine. If it is just a non-blocking DNS server it might stab the pi-hole in the back and just do the DNS resolution that the pi-hole was trying to prevent.

There is also a 3rd-party script "gravity sync" that hacks your two pi-holes and adds some syncing features. So you only need to admin one of the two pi-holes and the changes will be synced to the second pi-hole. But I'm not using it because I don't really trust 3rd-party hacks. So i'm just administrating both pi-holes individually so I'm everything twice.