structuring network - proxmox and pfsense

R

rafafell

Well-Known Member
Sep 24, 2016
61
2
48
37
Hi,

i am restructuring our proxmox cluster network and i have some doubts and requests for suggestions that i expose below

My scenario is as follows

(1) two public IP ports (untagged vlan) on different servers
(2) seven ports (tagged vlan)
(3) I don't have direct access to the switch of (1) and (2) - I have to ask for an eventual adjustment if necessary
(4) I am passing internet link through “masquerading (NAT) with iptables”
(5) I separated Cluster IPs into three distinct networks; Wan pfsense; IP VMs


Let the current design of the network be:rede-proxmox-pfsense.drawio.png


- I first mounted the cluster and then made a pfsense vm. is this the best way when using pfsense virtualized in proxmox?

- what is the most suitable way to provide internet link to cluster nodes?

- what is the most suitable way to let pfsense distribute ip to all VMs of the various nodes of the cluster?

- is the separation into three distinct networks a good thing? (Cluster IP - 192.168.0.0/23; VMs IPs - 192.168.4.0/23; Wan pfsense - 192.168.2.0/30)



best
 
rafafell said:
i am restructuring our proxmox cluster network and i have some doubts and requests for suggestions that i expose below
Click to expand...
So this is a proxmox cluster which is already up and running?
Do you have shared storage between the nodes?
 
bobmc said:
So this is a proxmox cluster which is already up and running?
Do you have shared storage between the nodes?
Click to expand...
the cluster is up and running (192.168.0.10-15), but I still haven't mounted the storage.

I'm using local storage to think about the network issue first

Storage must be zfs. And in a few months we'll switch to ceph with a dedicated switch
 
zfs storage is not cluster aware, each node can have local zfs storage and you can replicate pools or datasets between nodes, but it's not shared. It looks like you don't have enough network interfaces on the nodes to implement ceph storage successfully.

Do the VM's on the 192.168.4.0/23 network need to talk to each other or do they simply need to be able to reach the internet?
 
Thanks for the answer

All nodes have four NICs, but I can only connect to this switch the NICs of the above drawing (I could at most put one more NIC on this switch). These nodes are on different floors (192.168.0.10-12 on one floor and 192.168.0.13-15 on another) interconnected by a switch with 1gbit ports. I will switch to ceph within a few months of it has a separate switch.


Therefore, the VMs (192.168.4.0/23) would need to have access to the internet. Cluster nodes too (192.168.0.0/23)
 
Last edited:
rafafell said:
- I first mounted the cluster and then made a pfsense vm. is this the best way when using pfsense virtualized in proxmox?

- what is the most suitable way to provide internet link to cluster nodes?

- what is the most suitable way to let pfsense distribute ip to all VMs of the various nodes of the cluster?

- is the separation into three distinct networks a good thing? (Cluster IP - 192.168.0.0/23; VMs IPs - 192.168.4.0/23; Wan pfsense - 192.168.2.0/30)
Click to expand...
"best way" is always very subjective and a lot will depend on why (you're doing this), what (you have to work with) and where (do you want to achieve)

A lot of people will just use ip-tables on the host to allow internet traffic to their VM's and containers and Proxmox does have built-in firewall capabilities. However, some would feel that using a dedicated solution like pfSense is easier to manage and configure, and provides additional options.

Given you're already considering pfSense, you could use it to route your VM traffic to the internet and provide DHCP and DNS services to your cluster. I would be inclined to put your public IP(s) onto the WAN interface of pfSense - that way you won't need to use additional ip-tables rules in your hosts. The potential problem with this approach is that you need to be able access your host(s) on the management network while you're setting things up and you need to consider the situation if pfSense wasn't running - say during maintenance and updates. As far as the second point goes, you could setup pfSense in a failover configuration to address that concern - search "pfSense CARP"

Subnetting your traffic is a good idea, I would say you need one network for Management, and one for VM's. If you go for a failover setup, you'll need another for the Sync subnet. You can always add further subnets and VLANs later

Hope this helps you decide
 
  • Like
Reactions: rafafell
thank you @bobmc and @badji , I will advance here and post here soon how they are going

something else, in relation to network storage (ceph). I will have a limitation which is as follows:

- three nodes are on one floor
- three nodes are on another floor
- I have a maximum of 6 ports to interconnect the nodes of different floors on the same switch

my initial idea would be to create a ceph on each floor (they will have dedicated network cards and switches), do you think it's a good one?

Is there any way to connect these two storages?
 
  • Like
Reactions: badji
I have two LAN configured and apparently working fine. But I have the following problems:

LAN1 (192.168.4.0/23) provides correct IP for the vms of the local node, but I can't provide ips for vms of other nodes. I believe that if I linked this LAN to the physical interface of LAN2 it would solve this, but I am not able to do it

on LAN2 (192.168.0.0/23 - linked to the physical interface) I can provide ips to vms from any node in the cluster, but the internet link only passes to the vms and not to the cluster nodes (they are on this subnet)

Any idea?


proxmox01.jpegproxmox02.jpegproxmox03.jpeg
 
Last edited:
If LAN1 is local to that node (i.e it has no physical connection) then DHCP broadcast is not going to reach other nodes. I don't if or how it might be done via ip-tables. Given that the VM's are local to the node, the easiest way might be to run a dhcp server on each node - you should not need to worry about conflicting IP addresses as they will be local to the node.
 
  • Like
Reactions: rafafell
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back