Proxmox network setup, please help

[x86fantini]

Hello to all, after a week on proxmox setup, i'm ready to expose it to internet and place the servers into my rack. But i please need a hand to setup my network.

Currently, there are 2 servers, in cluster, with drbd/lvm and working correctly (replication, migration, fencing, etc..)

here is my nework setup right now:

# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

auto eth1
iface eth1 inet static
address 10.0.0.150
netmask 255.255.255.0

auto vmbr0
iface vmbr0 inet static
address 192.168.1.150
netmask 255.255.255.0
gateway 192.168.1.254
bridge_ports eth0
bridge_stp off
bridge_fd 0

eth1 is for DRBD, eth0 will be connected to primary internet cable.

on the server, i have those VMs:

1 x haproxy for loadbalance
1 x nginx webserver to serve php
1 x mysql server

and my questions are:

1) should i install any kind o software firewall directly on proxmox nodes or in a vm?
2) i need to expose on the public internet proxmox nodes + VM n°1 (haproxy) with a dedicated public ip address
3) all other VM do not need public visibility, since http traffic will be managed by haproxy to --> nginx webserver --> mysql
4) when node1 will die, and node2 will take care of the traffic, will my scenario work? (from a network point of view)

can you please help me to achive this network setup?

thank you
Simone

 

[cesarpk]

Since that only 1 machine can have one specific ip address, so the VM with haproxy will be the router and default gateway for the PVE Hosts, This mode is the more easy of configure for me, thus, the firewall will be on the VM with haproxy, and not in the PVE host

Other solution is doing the reverse, "shorewall" appears to be the appropriate firewall for the PVE Hosts, to be the simplest to use, but i don't have practice with this program.

I like that the firewall/router is on the VM, because this VM enjoy the features of PVE as "High Availability", "migration online", "backup online", etc., simply, for me it is easier to administer, also you can have haproxy and web servers on several VMs running simultaneously on several PVE Hosts.

A tip:
Always is recomended to have 2 NICs in bonding for each network connection, think what's up if suddenly a NIC die?

Best regards
Cesar

 

[x86fantini]

now, my network setup is like this:

# network interface settings
auto lo
iface lo inet loopback


iface eth0 inet manual

## only used for DRBD ##
auto eth1
iface eth1 inet static
address 10.0.0.150
netmask 255.255.255.0


auto vmbr0
iface vmbr0 inet static
address 192.168.1.150
netmask 255.255.255.0
gateway 192.168.1.254
bridge_ports eth0
bridge_stp off
bridge_fd 0


auto vmbr1
iface vmbr1 inet static
address 10.10.10.254
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE

If i assign VMBR1 to a kvm vm, i manually setup /etc/network/interfaces like this:

auto eth0
iface eth0 inet static
address 10.10.10.2
netmask 255.255.255.0
broadcast 10.10.10.255
gateway 10.10.10.254

and the results are:

1. i can not ping anywhere (/etc/resolv.conf is 8.8.8.8)
2. i can not apt-get
3. if i start a vm on VMBR0, those 2 vm can not see eachother

my goal is that only 1 vm should be faced on public internet (vmbr0) all others should be connected on vmbr1 and public vm passing all traffic to vmbr1 vm

can someone help me?
thank you
Simone

 

[x86fantini]

UPDATE:

just removed CSF firewall from my proxmox node, and now KVM VM, running on vmbr1 can ping internet and run apt-get so i'm 50% done.

i have started my kvm vm, running on vmbr0 ad vms can't see eatchother.

any solutions?

thx
Simone

 

[cesarpk]

UPDATE:

just removed CSF firewall from my proxmox node, and now KVM VM, running on vmbr1 can ping internet and run apt-get so i'm 50% done.

i have started my kvm vm, running on vmbr0 ad vms can't see eatchother.

any solutions?

thx
Simone

If the firewall/router is your VM, then this VM must have 2 Virtual NICs (for example 2 Virtual NICs bind to vmbr0). In this VM the first Virtual NIC is for exit to internet, the second Virtual NIC is for exit to the LAN, also this VM must have "ip_forward" enabled to work in Router mode, and also must have the typical firewall configured as "masquerade" enabled for that the machines into the LAN (Virtuals or physicals) can exit to Internet successfully.

After of this, the PVE Host(s) and the rest of machines (Virtuals or physicals) must have as default gateway the IP-LAN of the Virtual Machine that have the exit to Internet.

After of this, iptables must be configured as you need, for example the Firewall installed on the VM must forward the connection of Internet and from a IP specific on the port 8006 to a PVE Host, being well, "only you" will can connect since Internet to the PVE GUI (You will need to use different port numbers (chosen for you) if you want access to all Hosts PVE GUI without problems, that is to say, iptables can use a specific number of port for your connection (chosen for you), and after will connect to the specified PVE Hosts in the port 8006). For understand the How of iptables, Mr. google can show too much web links

For best security and less work over iptables, i always configure the firewall for use it with OpenVPN, the software Open Source OpenVPN lets me connect in mode peer-to-peer or peer-to-lan, and ideally OpenVPN server must be installed on the same VM running the firewall. I use OpenVPN with the configuration more secure and compressed for gain speed. Then when i want, since my PC workstation and utilizing the OpenVPN client, I can connect to remote LANs, and obviously i have iptables configured for accept all connections and ports since my IP of OpenVPN client. For understand the How of OpenVPN, his portal and Mr. google can show all about of this.

And don't forget it: Always is recomended to have 2 NICs in bonding for each network connection, think what's up if suddenly a NIC dies (for example for drbd or for vmbr0)?

Re-Edit: Ah! I forget it, for that the other VMs can see each other, all VMs must be in the same bridge, in principle, I would recommend putting all VMs in vmbr0 and also the PVE Host, considering that you only have one physical NIC for all your network connections, and obviously DRBD is exclusively in another physical NIC.

In regard to physicals NICs and avoid disconnections due to breakdown of one physical NIC, a excelent option is to use (configurable from PVE GUI and considering that a VM is the default gateway of all your LAN):
- 2 Ports for LAN connections in bonding active-backup mode (for vmbr0)
- 2 Ports for WAN connections in bonding active-backup mode (for vmbr1, and the PVE Host don't must have the IP configured)
- 2 Ports for DRBD connections in bonding balance-rr mode (only bond, not need to use vmbrX)

And for the other VMs and PVE Hosts can see each other (into the same LAN):
- 2 Ports for LAN connections in bonding active-backup mode (for vmbr0)
- And the physical LAN connections must be in the same NICs that have vmbr0 configured

And about of fencing with only two nodes, this is only possible if you have "manual_fence" and only 1 vote in the PVE cluster configuration, both properly configured, with physical fence devices you must have minimun three nodes to conserve quorum if a PVE Host dies. Otherwise fence not will work properly.
Have you tested your fence system?

Best regards
Cesar

 

[x86fantini]

great, thank you very much.

i'm working in this directions, and for HA/Cluster i think for me it's the best option..

will go on and hope no problems will come.

thx
Simone

 

[cesarpk]

Hello Simone

My previous post has been re-edited several times, please read it again

Best regards
Cesar

 

[x86fantini]

Hi Cesarpk, just went trhough and i have cleared my mind. perfect, thank you!

best regards
Simone

 

[cesarpk]

Hi Cesarpk, just went trhough and i have cleared my mind. perfect, thank you!

best regards
Simone

Hi Simone

I forget tell you some things:

1- If you want to use DRBD in bonding balance-rr:
1A- You must connect the NICs in mode NIC-to-NIC each one, and not in mode NIC-Switch-NIC for avoid problems of communication with the switch (sometimes the Switches changes the order of delivery of the packs of net and also avoid have a single point of failure)
1B- Preferably these NICs must be the same brand and model for avoid changes the order of delivery of the packs of net
1C- If you have DRBD in bonding balace-rr with 2 NICs, you will gain double speed in communication of net and avoid bottlenecks, this last due to that SATA Disks are more rapid that 1 Gb/s.

2- Don't use NICs Realtek, Realtek is garbage, in my personal experince NICs Realtek offers many problems (i see the text "Link down" on the log file and several times). Unless you want to try with the newest driver and check if it continues to lose the connection (that will mean compile it since the source).

3- A good option is to use INtel NICs, for example NICs Intel for Servers or Workstations. PVE have the latest version of all these Intel NICs accessible in his repository for download it, and i believe that the other Distros have too.

Best Regards
Cesar

 

[x86fantini]

Hi Cesar, thank you for your suggestion.

i'm actualy using supermicro servers, 2 identical servers both with Intel Gb nics, and working great .

i have a question: let's say i only have 2 nics, and actual situation is 1 nic for wan, i nic for drbd. Am i able to bond toghethere thise 2 nics BUT still be able to assign one for wan?
thank you
Simone

 

[cesarpk]

Hi Cesar, thank you for your suggestion.

i'm actualy using supermicro servers, 2 identical servers both with Intel Gb nics, and working great .

i have a question: let's say i only have 2 nics, and actual situation is 1 nic for wan, i nic for drbd. Am i able to bond toghethere thise 2 nics BUT still be able to assign one for wan?
thank you
Simone

The answer is no, for each bonding (with one or more NICs on the bonding) will be assigned only one IP address, then in your case you must have NICs independents and not use bonding.

My explications in my previous posts can be used only if the VM that have haproxy will be the unique and default router for all your LAN, but if you have now a router in your LAN, the that I said above does not make sense (because if this router dies, nobody will can exit to Internet and nobody can enter to the LAN.

New scenery - if you have 1 router and this isn't a VM:
If you have now a Router for all your LAN that is used for exit to Internet (For example Tplink, Cisco, kayomi, trendnet, ZTE, a PC, etc), then you can configure your router for forward the number of ports from Internet to the machine corresponding into the LAN.

For example:
1- For http/https (ports 80 and 443):
The router must be configured for forward from Internet these ports to the VM that have haproxy (But i believe that haproxy not works with https connections, in this case forward of 443 port number not will be necessary)

2- For access to the PVE GUI Host (port 8006):
The router must be configured for forward from a IP specific of Internet and from this port to the PVE GUI Host in the port 8006.

3- For access to the second Host PVE by GUI (use the port 8007 because exists the forward of the port 8006):
The router must be configured for forward from a IP specific of Internet and from the port 8007 to the second Host PVE at port 8006

4- For access to the third Host PVE by GUI (use the port 8008 because exists the forward of the ports 8006 and 8007):
The router must be configured for forward from a IP specific of Internet and from the port 8008 to the third Host PVE at port 8006

In conlusion with this scenery:
- Of this mode, if you have all PVE Hosts into the same LAN, all PVE Hosts only need to have connection to vmbr1, as well as his VMs too.

- And you don't need assign two virtual NICs for the VM that have the haproxy (only one virtual NIC will be necessary, unless "haproxy" says otherwise, that in this case you can link 2 virtual NICs to vmbr1 and the front-ends must have other range of IPs for get communicating successful to the "haproxy")

- All VMs on all PVE Hosts can exit by net using vmbr0

- Obviously all NICs that you have configured as vmbr0 will need connection physical to Switch LAN

- Obviously your present router will need connection physical to Switch LAN (besides the Internet output using other connection)

- All PVE hosts and his VMs must have as default gateway tha IP-LAN of your Router (The case of front-ends can be an exception depending on your general settings)

Hoping that my recommendations help you, I say see you soon

Best regards
Cesar

 

[x86fantini]

ok cesarpk, thank you very much. i think we can close this 3d

best regards to all community
Simone