separate IP (for PVE management) from VM/CT networks

[zyghom]

Hi,

My PVE has IP of: 192.168.1.3/24
Some of my VMs/CTs are in 192.168.1.x subnet, some are in 192.168.100.x subnet
Does it make sense to change the management IP of PVE from x.x.1.2 to i.e. totally different subnet i.e. 10.10.x.x. or it shall stay in 1.x subnet?
And if so - how to?
My questions is with relation to security only.

Btw PVE is behind opnsense (separate box) and it is not exposed to WAN (only exposed is nginx proxy on ports 80 and 443).

my /etc/network/interfaces:

auto lo
iface lo inet loopback
auto enp1s0
iface enp1s0 inet manual
auto vmbr0
iface vmbr0 inet static
    address 192.168.1.3/24
    gateway 192.168.1.1
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

thank you for advice

 

[Maksimus]

I join the question, but I have a slightly different situation. I have a cluster of N machines, and one of the nodes is in network A, and I want to transfer it to network B, won't changing the node ip break the cluster?

 

[LnxBil]

Does it make sense to change the management IP of PVE from x.x.1.2 to i.e. totally different subnet i.e. 10.10.x.x. or it shall stay in 1.x subnet?

Yes sure. Easiest method is to have separate vlans for everything and filter on your opnsense.
If you don't want to go that route, just configure a firewall on your PVE host to only allow connects to the hypervisor from your save-host list.

It is generally a VERY good idea to have PVE firewall for each VM/CT and not rely only on an external firewall. I have many different security groups e.g. for a virtual dmz that just blocks everything (in AND out) and only allows certain things like icmp echo request/reply, DNS to internal DNS-Server, NTP and access to the package mirrors to get their updates, DHCP if neccessary, etc.. Nothing will work unless I configure another security group for e.g. http/https and you can run a webserver.

 

[LnxBil]

I join the question, but I have a slightly different situation. I have a cluster of N machines, and one of the nodes is in network A, and I want to transfer it to network B, won't changing the node ip break the cluster?

You have a dedicated cluster network in place? Then it may work. Best is to have a working cluster and just configure your network on top of that in separate vlans in order to have a good separation. You can then use a transfer network to build a front page for your pve cluster with one ip/name that connects to one of the nodes randomly behind the scenes for administation.

 

[zyghom]

Yes sure. Easiest method is to have separate vlans for everything and filter on your opnsense.
If you don't want to go that route, just configure a firewall on your PVE host to only allow connects to the hypervisor from your save-host list.

It is generally a VERY good idea to have PVE firewall for each VM/CT and not rely only on an external firewall. I have many different security groups e.g. for a virtual dmz that just blocks everything (in AND out) and only allows certain things like icmp echo request/reply, DNS to internal DNS-Server, NTP and access to the package mirrors to get their updates, DHCP if neccessary, etc.. Nothing will work unless I configure another security group for e.g. http/https and you can run a webserver.

oh, so I did it this way:

• from opnsense box I have 2 LAN cables going to managed switch:
  • cable 1: vlan 100 - home network
  • cable 2: vlan 1 (that actually means no vlan, right?) - servers and IoTs
• 3rd cable from opnsense box goes directly to the ISP modem - not via switch
• to that switch there are 2 wireless APs connected:
  • 192.168.1.x - for VLAN 1
    • here as fixed IPs are my servers - all below 192.168.1.80:
      • they all have full access everywhere
      • all these servers are VMs/CTs on PVE (Home Assistant, NodeRed, MQTT etc.) plus PVE itself
      • PVE IP is also here
    • here as DHCP IPs are my all IoT devices - from 192.168.81-250
      • they have access ONLY to 192.168.1.0/24 - no internet at all
  • 192.168.100.x - for VLAN 100
    • here are my all home devices: laptops, tablets, phones - all are DHCP
    • all devices here have access everywhere
• DHCP is on opnsense box:
  • one for 192.168.1.x
  • one for 192.168.100.x
  • DHCP offers 2 DNS as below:
    • 192.168.1.23
    • 192.168.100.31
• there are 2 DNSs:
  • 192.168.1.23
  • 192.168.100.31
  • both are Pi-holes in containers
  • both Pi-holes use Unbound DNS on opnsense as their upstream DNS (192.168.1.1)

So I think I did some VLANs to protect the network.
But I am still thinking if I can do it better way.
Firewall in PVE is off completely.

 

[LnxBil]

Firewall in PVE is off completely.

That is very sad. It is much better than anthing external for protecting and jailing your machines.

from opnsense box I have 2 LAN cables going to managed switch:

• cable 1: vlan 100 - home network
• cable 2: vlan 1 (that actually means no vlan, right?) - servers and IoTs

VLAN can be tricky if you have untagged and tagged packages in the network.

 

[zyghom]

That is very sad. It is much better than anthing external for protecting and jailing your machines.


VLAN can be tricky if you have untagged and tagged packages in the network.

why would firewall on PVE be any better/worse than firewall on separate box?
are you talking inter VM/CT traffic?
but this traffic I am not "afraid" of - rather the traffic outside my network or incoming to my network - not the traffic ONLY inside the network
what am I missing?

 

[LnxBil]

why would firewall on PVE be any better/worse than firewall on separate box?
are you talking inter VM/CT traffic?
but this traffic I am not "afraid" of - rather the traffic outside my network or incoming to my network - not the traffic ONLY inside the network
what am I missing?

Because one VM an attack another VM and your external firewall will not see it. You know the statistics of attacks? Most of them come from the inside.

Best example is to download a Docker images inside of your network to try out the newest thing and it attacks your internal network ... and boom. Could be prevented with per-default security.

 

[zyghom]

Because one VM an attack another VM and your external firewall will not see it. You know the statistics of attacks? Most of them come from the inside.

Best example is to download a Docker images inside of your network to try out the newest thing and it attacks your internal network ... and boom. Could be prevented with per-default security.

you are right!
So you suggest 2 lines of defence:

• external firewall on the boundary of home LAN and
• internal firewall on every interface in PVE

if I understood you correctly.

By default PVE firewall has no rules, right?
I have not even seen anybody talking anywhere/describing the rules inside PVE firewall.

 

[LnxBil]

you are right!
So you suggest 2 lines of defence:

• external firewall on the boundary of home LAN and
• internal firewall on every interface in PVE

if I understood you correctly.

Yes, by "every interface" I mean not only the PVE hosts, but also on EVERY VM and container. Rules normally apply to all interfaces, so you do not have to set them explicitely in each rule.

By default PVE firewall has no rules, right?

None configured at the PVE GUI level, but there are some basic iptables rules defined for things like established traffic and a lot of chains/tables with references - the barebones.

I have not even seen anybody talking anywhere/describing the rules inside PVE firewall.

It's normal stateful firewalling (therefore you also need a deep inspection firewall at the network boundary), but the great advantage is the feature "security groups", which can be applied to each VM/container. In Datacenter -> Firewall -> Security Group, you can define a name and then add rules to the group.
In a VM/CT, you can add a security group to a VM and it inherits automatically all firewall rules you defined in the group on your datacenter level. This is optimally a rule that allows what you really need and rejects/drop (whatever you like) the rest. This should be added to each VM and should always be the last one (because of the reject/drop). Every service you're running is then actively activated. I have security groups for webservices, NFS (with fixed ports), Samba, netboot etc.

 

[zyghom]

Yes, by "every interface" I mean not only the PVE hosts, but also on EVERY VM and container. Rules normally apply to all interfaces, so you do not have to set them explicitely in each rule.


None configured at the PVE GUI level, but there are some basic iptables rules defined for things like established traffic and a lot of chains/tables with references - the barebones.


It's normal stateful firewalling (therefore you also need a deep inspection firewall at the network boundary), but the great advantage is the feature "security groups", which can be applied to each VM/container. In Datacenter -> Firewall -> Security Group, you can define a name and then add rules to the group.
In a VM/CT, you can add a security group to a VM and it inherits automatically all firewall rules you defined in the group on your datacenter level. This is optimally a rule that allows what you really need and rejects/drop (whatever you like) the rest. This should be added to each VM and should always be the last one (because of the reject/drop). Every service you're running is then actively activated. I have security groups for webservices, NFS (with fixed ports), Samba, netboot etc.

perfect - thank you for your explanation
but if I understood you:

1. common rules that are for all VM/CT - in the group
2. individual rules that are for specific VM/CT - in the individual setting for this VM/CT

because one VM might need port 80 open while another one 8123

right?

 

[LnxBil]

but if I understood you:

1. common rules that are for all VM/CT - in the group
2. individual rules that are for specific VM/CT - in the individual setting for this VM/CT

because one VM might need port 80 open while another one 8123

right?

Yes. That is possible. (Just that it is clear: you don't add VMs to the security group, you add the security group to the VM.)

I recommend to use default ports for web and use a reverse-proxy or iptables port redirection in order to have all werbserver on ports 80 and 443.

 

[zyghom]

Yes. That is possible. (Just that it is clear: you don't add VMs to the security group, you add the security group to the VM.)

I recommend to use default ports for web and use a reverse-proxy or iptables port redirection in order to have all werbserver on ports 80 and 443.

thanx man - you are opening my eyes ;-)

now question: I have a docker in VM, and there, 2 www servers, they are both on different IPs - I use macvlan so i.e. 1st www is 192.168.1.21 and 2nd www is 192.168.1.22
VM itself is another IP: 192.168.1.20
taks: restrict www1 to local addresses only while www2 let it be open
now I am a bit confused: do I use in PVE firewall in the rules: "destination IP" to distinguish them?

EDIT:
I was not able to do it so I split the www into 2 separate CTs - if I cannot do it this way, I will always find "another" way ;-)

 

[LnxBil]

now I am a bit confused: do I use in PVE firewall in the rules: "destination IP" to distinguish them?

Yes, this could work. In general, this is more complicated than just having everything reverse proxied through one IP that does the actual authentication and authorization for you. There are a lot of different middlewares to do this for you so that you can access all as you wish from a service standpoint, not from a stateful ip firewall.