how to secure proxmox installation

[benoitc]

Hi all,

So I have a proxmox cluster at home and I would like to make its installation more secure. I put the proxmox cluster in a DMZ, and have the Office network behind another firewall using NAT. What would be the best way to allow the access to the proxmox cluster resources securely and only expose the admin to the office network? Is there any possibility to route directly to the proxmox cluster instead of going through internet to access to it? Also not expose the admin ports to internet?

I have something like it:

           INTERNET
              |
              |
         CORE ROUTER
          |        \ (Fiber)
          |         \
        SWITCH     OFFICE Gateway (Nat)  --- LAN
          |
          |
   PROXMOX CLUSTER

 

[Astraea]

I have my Proxmox cluster behind a virtual firewall. the only way I can access local resources that are not explicitly exposed to the internet via either a mail gateway or a reverse proxy is through my remote VPN server.

I did not want to have my hypervisor be accessible directly same with other resources that contain sensitive information. My VPN server requires both a username and password to connect as well as a client certificate that is specific to each device and those devices are given a predetermined IP address that is allowed to move from the VPN network to the normal user network which allows access to specified resources. I can then VNC into a VM that has access to other resources that are on other networks with limited access to them from hosts outside that network.

 

[benoitc]

I have my Proxmox cluster behind a virtual firewall. the only way I can access local resources that are not explicitly exposed to the internet via either a mail gateway or a reverse proxy is through my remote VPN server.

I did not want to have my hypervisor be accessible directly same with other resources that contain sensitive information. My VPN server requires both a username and password to connect as well as a client certificate that is specific to each device and those devices are given a predetermined IP address that is allowed to move from the VPN network to the normal user network which allows access to specified resources. I can then VNC into a VM that has access to other resources that are on other networks with limited access to them from hosts outside that network.

did you put the cluster in a localsubnet accessible via the vpn only?

 

[Astraea]

The storage servers and the hypervisor servers are on their own VLAN, that only specific computers have access to at the administration level. and those computers can access from either the normal LAN and from the remote LAN

 

[apoc]

I have trouble understanding your current setup and what you plan to do.
But whatever you do: make sure that the proxmox servers (or any critical infrastructure) is accesible directly from the internet.
I am very concerned when I read "router" in your diagram. This should be a serious firewall with all ports closed / shut.
To access the pve infrastructure you simply should be able to add routes to their subnet. But thinking about this this shouldn't even be necessary if the NAT office firewall uses the router as a gatway.
it may help if you provide sample addresses for your setup.

 

[benoitc]

@tburger whel it is a dual firewall setup . the core router manage the borders. the DMZ is behind it, and the office office gateway (UDM Pro from ubiquiti) is connected to border router like it could be connected to an ISP. While in the DMZ i plan to use the proxmox firewall for the proxmox cluster and expose some other utility services for external people. There is a firewall on the core router (which is a mikrotik one). The main concern is to isolate anything in the office network (which is one bulding away) from the proxmox cluster. What I would like howerver is to limit the number of routes to connect to the proxmox cluster from the office network to reduce the latency and the overhead.

 

[Joris L.]

I have trouble understanding your current setup and what you plan to do.
But whatever you do: make sure that the proxmox servers (or any critical infrastructure) is accesible directly from the internet.
I am very concerned when I read "router" in your diagram. This should be a serious firewall with all ports closed / shut.
To access the pve infrastructure you simply should be able to add routes to their subnet. But thinking about this this shouldn't even be necessary if the NAT office firewall uses the router as a gatway.
it may help if you provide sample addresses for your setup.

to my knowledge, if a device is performing outbound NAT only then the inbound will fail per default if no NAT is configured to do so. Same result.

 

[Joris L.]

@tburger whel it is a dual firewall setup . the core router manage the borders. the DMZ is behind it, and the office office gateway (UDM Pro from ubiquiti) is connected to border router like it could be connected to an ISP. While in the DMZ i plan to use the proxmox firewall for the proxmox cluster and expose some other utility services for external people. There is a firewall on the core router (which is a mikrotik one). The main concern is to isolate anything in the office network (which is one bulding away) from the proxmox cluster. What I would like howerver is to limit the number of routes to connect to the proxmox cluster from the office network to reduce the latency and the overhead.

Don't take my word for it, here are a few considerations.

• be aware, very aware, if this proxmox cluster is running at home you are extending the office network to home and vice versa, you may be held accountable if anything goes wrong such as your home network is compromised. Unless you own the company, make sure you have discussed this and have approval from the responsible and accountable persons in the company/organisation you work for. If you are the owner, reconsider if what you are doing is creating more risks than it is supposed to mitigate.
  
  
• With the set-up above you do not route over internet, you route over the core router. I'm not sure what you are trying to avoid when you say you want to avoid routing over the internet. The mention of fiber suggests a dedicated line to the office builing/gateway ?
  
  
• Do you have visibility on bandwidth over the fiber ? Once people know there is a virtualisation host active they'll want to use it. While VM's typically put bandwidth requirements on the host it may put serious load on the fiber subscription.
  
  
• What do you mean with DMZ here ? Typically this term is used to suggest a device which requires to be continously reachable from the internet.
  
  
• I doubt the decision of putting the pmx cluster in the DMZ. Segment it with a VLAN and connect it to the office gateway on the inside could be an idea if the office gateway has access lists and vlan support. Also connect it with a vlan to the core router to assure segmentation of the internet connectivity.
  
  Just my 2cents.

 

[benoitc]

Don't take my word for it, here are a few considerations.

• be aware, very aware, if this proxmox cluster is running at home you are extending the office network to home and vice versa, you may be held accountable if anything goes wrong such as your home network is compromised. Unless you own the company, make sure you have discussed this and have approval from the responsible and accountable persons in the company/organisation you work for. If you are the owner, reconsider if what you are doing is creating more risks than it is supposed to mitigate.
  
  
• With the set-up above you do not route over internet, you route over the core router. I'm not sure what you are trying to avoid when you say you want to avoid routing over the internet. The mention of fiber suggests a dedicated line to the office builing/gateway ?

So it's all my network. And the office connection and proxmox are managed internally. What I'm trying to achieve is to ensure is that machine connection to the proxmox cluster via internet will not be able to reach the office network. the office network is connected via its own fiber to the core router and is mostly handled like a separate service. Ideally what I want is the following:

* Core router manage the connection to the internet and pass ips and connectivity to the office network and the proxmox cluster (or really any other machines).
* Machines on the office network can access to the machines on the proxmox cluster. I would like to skip rountrip outside to internet (office gateway -> Internet -> Proxmox Cluster) and instead have possibly direct connection to it (Office Gateway -> Proxmox cluster) without exposing the office network too much.
* some machines on the proxmox cluster will only have privates IPs. But I would like to be able to access to them via the office network and maybe a vpn.

• 
  
  
• Do you have visibility on bandwidth over the fiber ? Once people know there is a virtualisation host active they'll want to use it. While VM's typically put bandwidth requirements on the host it may put serious load on the fiber subscription.

yes. I can also put some QoS on it.

• 
  
• What do you mean with DMZ here ? Typically this term is used to suggest a device which requires to be continously reachable from the internet.
  
  
• I doubt the decision of putting the pmx cluster in the DMZ. Segment it with a VLAN and connect it to the office gateway on the inside could be an idea if the office gateway has access lists and vlan support. Also connect it with a vlan to the core router to assure segmentation of the internet connectivity.

Do you mean having one vlan for the UI and one vlan for internet access on the proxmox cluster?

• 
  
  Just my 2cents.

thanks for the hints, that helps a lot

 

[Joris L.]

yes, separate the network into functionality like (internet/mgmt/office) it will offer realistic segmentation and avoid mishaps

 

[guletz]

Do you mean having one vlan for the UI and one vlan for internet access on the proxmox cluster?

I think this will be a good ideea.

 

[guletz]

... and from what I understood from you, you will need that any VM/CT from PMX zone could not initiate NEW connections to the OFFICE zone or to Internet(maybe except for updates). For me, this is a DMZ zone. If you want a better setup(from security point of view only), you could put any other transparent-device(with a firewall on a bridge, without any IP on it - any Mikrotik can do this) between your Core Router and your OFFice zone. With no IP on it nobody can attack this device/change the firewall rules(so you can enforce your DMZ/OFFICE rules).

Good luck / Bafta !