[SOLVED] Set proxmox cluster on seperate NIC

[Maher Khalil]

Hello
I have 3 servers
first one with 1 NIC with 4 ports and second one with 2 NICs 2 ports each and third server same as first one
My goal is to make cluster on separate network as recommended by proxmox
What I did
on the first server: I have main bond for port 1 and 2 and bridge vmbr0 which works fine with virtual machine and static IPs, then I added second local network (for clustering) as mentioned below
================================> server 1 second network for cluster
auto eno4
iface eno4 inet static
address 10.10.1.10/24
=================================>
Then on the second server, I created another network for cluster as I did on server 1 and on the same subnet range
==================================> server 2 second network for cluster on the same range of server 1
auto enp65s0f1np1
iface enp65s0f1np1 inet static
address 10.10.1.20/24
===================================>
Then on the server 3, I followed same as 1
==================================> server 3 second network for cluster on the same range of server 1 and 2
auto eno4
iface eno4 inet static
address 10.10.1.30/24
===================================================================> the problem
the above 3 servers are connected to the same physical switch and all accessible on the static over IPs on the fisrt network but not on the cluster network where I cannot ping each other on cluster network with subnet 10.10.1.x/24.
also when I make cluster, It did not work
Any support

 

[vesalius]

Any vlans or a Managed switch used here?

 

[Maher Khalil]

No setting for vlan
We have juniper EX4000 switch but there is no special setting, only we did no shutdown for ports

 

[alexskysilk]

even without vlans, keep your nics on different subnets or all traffic will flow out of your default interface in your routing table, regardless of which one you want to use.

eg:
auto eno4
iface eno4 inet static
address 10.10.1.10/24

auto enp65s0f1np1
iface enp65s0f1np1 inet static
address 10.10.2.10/24

 

[Maher Khalil]

They are in different subnets
For each server there subnet for vm traffic + I created separate subnet 10.10.10.0/24 for all servers clustering

 

[alexskysilk]

if I understand you correctly, you have 6 nics on each server in total.

Assuming your switch is wide open, how certain are you that the nics you defined are the nics connected? use ethtool to verify.

I think I understand what you're asking. you want accessibility from the 10.10.1.0 subnet to another subnet- you need to define a route (usually at the router level. but can be at each individual host.)

edit 3- why do you want this anyway?

 

[vesalius]

No setting for vlan
We have juniper EX4000 switch but there is no special setting, only we did no shutdown for ports

Just trying to understand your network topology. so you have 2 same computers both using eno4 for this cluster network (10.10.1.10 and 10.10.1.30) and a third computer using enp65s0f1np1 (10.10.1.20). All 3 of these nics plug into your juniper - how are these 3 juniper switch ports configured ... default vlan, untagged or tagged vlans on these 3 ports?

Are those 3 switch ports configured differently from the switch ports connecting your VM subnet?

 

[Maher Khalil]

Hello
Yes you are right
there is no VLAN except default one
for each server I use bond of port 1 and 2 for VMs network
I attached the interface settings of server 1 and 2
Please note that firewall is off

 

[vesalius]

Nothing looks off with your proxmox node interfaces file. I assume the 10.10.1.10 interfaces file shown and the 10.10.1.30 file are exactly the same?

So on the juniper switch does the LACP bond you created there for the VM network also use the default vlan?

I see you have set the mtu to 9192 on the 10.10.1.10/24 Proxmox node vm network, but not the 10.10.1.20 Proxmox node. You probably want to make all of them the same there. Might that mean the juniper switch is set for jumbo frames on the default vlan?

can you ping any of the Proxmox node 10.10.1.0/24 addresses from the juniper switch gui and/or cli?

 

[Maher Khalil]

Hello
The bond network work without any issues and all VMs can access internet and also get accessed.
I did not set VLANs for any of my networks since I create separate subnet range for each server and also its IDRAC, I also separate VMs on separate subnets than main server IP subnet.'
My setup is as mentioned below
1. each server has it own subnet, and also VMs has separate subnet and that work without any issues since router knows about them
2. each IDRAC has its own subnet and also that does not have a problem
3. I created local subnet for clustering communication 10.10.1.0/24 and it will not go to internet. so that is local only, I assume switch will know about them from NIC MAC address and each IP can ping each other

 

[Maher Khalil]

Hello
It is working now
The problem was using wrong port enp65s0f1np1 instead enp65s0f1np0
Thank you for your help and support

 

[vesalius]

@alexskysilk nailed it up above then crossed it out from their second post.

Happy new year and glad it is working now.

 

[Maher Khalil]

I did already
Happy new year
Thank you

 

[alexskysilk]

@alexskysilk nailed it up above then crossed it out from their second post.

Happy new year and glad it is working now.

was having a hard time following what the issue was glad you got it figured out.