Private network across nodes in a cluster

[hanoon]

Hello Everyone.. This is my first post... I'm actually looking for the same request like a previous post..

I have searched forum for every relative thread and couldn't find an answer.

I have 3 nodes in a cluster... physical hosts.. I have

VM1 on Node1 VLAN ID 100
VM2 on Node2 VLAN ID 200
VM3 on Node3 VLAN ID 200
VM4 on Node3 VLAN ID 100


All I want is to have those VMs communicate over private network...

I have created eth3 on each Node and connected 3 ports to same Cisco switch and all of the ports are Trunked to Proxmox..

Could anyone provide me with a sample configuration.. Also how can we take advantage of the VM VLAN tagging option.. I would like to tag a VM1 , VM4 with the same tag to communicate then VM2 , VM3 with another VLAN tag ID...

Thanks in advance..

 

[hanoon]

I tested with physical switches are also routing the private VLAN and still not working..

Anyone can put some shade here...

Thanks

 

[mir]

Just create a bridge on your proxmox nodes and don't assign an interface to this bridge.

auto vmbr1
iface vmbr1 inet manual
bridge_ports none
bridge_stp off
bridge_fd 0

More examples here: https://gist.github.com/jpawlowski/3331593

 

[hanoon]

Thank's Mir for the advise... unfortunately it's not helping us :

We have a bit more advanced setup... posted some details at last post.

 

[mir]

I am not on skype so this is not possible.

Could you not present a diagram of what you want to achieve?

Also:
What have you been trying so far?
And what did not work with your setup?

 

[hanoon]

Issue resolved after upgrade.. Thanks

 

[mir]

What do you see from tcpdump on the VM's when you ping?

sudo tcpdump -t icmp -i eth1

You should see something like:
IP 10.1.1.1 > 10.1.1.2: ICMP echo request, id 13769, seq 1, length 64
IP 10.1.1.2 > 10.1.1.1: ICMP echo reply, id 13769, seq 1, length 64

 

[mir]

I think your problem is related to ESXi which in someway drops the traffic if packages are not tagged. To make your VM's vlan aware you could try loading the kernel module 8021q.
sudo modprobe 8021q in each VM.

 

[hanoon]

Ok, I have made some changes based on your recommendation...

1- Changed from VMware virtual machine to a physical host - latest version.
2- Connected host with 2 interfaces... eth0 and eth1 to a Cisco switch.
3- configured trunk ports on both switch ports.
4- configured vlan on eth0 and added it o LAN for easier configuration and GUI access.
5- Configured eth1 as follow:

/etc/network/interfaces
auto eth1
iface eth1 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0


6- Created a vlan 444 interface on my switch and confirmed it's up.

7- Create a Linux VM and I did ping tests and same results ... host unreachable

Installed a Windows MV on the host to avoid any kernel incompatibility..
8- both Linux/Windows VMs were put on vmbr0 and tagged to 444 on the vm hardware settings.

Ping still doesn't work... http://gyazo.com/9521f5a68cfe3fabd6c2939b151b4b11


9- Pings were done to and from the switch VLAN interface ip 10.1.1.10 to avoid troubleshooting on 2 hosts and a switch...



Please advise...

 

[mir]

Try this to see if you are able to communicate:

auto eth1.444
iface eth1.444 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge_ports eth1.444
bridge_stp off
bridge_fd 0

Why is this necessary? A linux bridge can only be an access port with or without vlan tags there is no such way as a bridge in trunk mode. To have a bridge carrying vlan tags you either need to configure its bridge_ports to a device carrying vlan tags like my example above or create a new bridge carrying vlan tags which is using the original bridge as brige_ports like:

auto vmbr0
iface vmbr0 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0

auto vmbr0.444
iface vmbr0.444 inet manual
bridge_ports vmbr0
bridge_stp off
bridge_fd 0

Read more here: http://blog.davidvassallo.me/2012/05/05/kvm-brctl-in-linux-bringing-vlans-to-the-guests/

 

[hanoon]

Thanks for prompt reply...

I know your example works cause that's how I have LAN "VLAN" assigned on the first NIC eth0 over the main trunk link,

My issue is:

I would like to take advantage of the VLAN tag setup on the VM level form hardware menu---> add VLAN tag...

The scenario would be... Client create private network so app assign a specific VLAN tag (Not in use or reserved) to this client... then assign that tag over API..

That way client has a truly isolated private network... and only VMs on that specific VLAN tag across one or multiple nodes can communicate...

Please advise on how to accomplish that...

 

[mir]

I have a hard time to see the difference between your wish and my example?

Creating a vlan tagged bridge and then assign this bridge only to VM's which should have access provides exactly the same protection.

Assigning a client directly to a device is, to the best of my knowledge, not supported by proxmox. You can choose between bridge, nat, and no nic.

 

[hanoon]

Thanks again for continues help... How can I take advantage of the VLAN tag setup on the VM level form hardware menu---> add VLAN tag?

a valid example of that working would help.... where I set the tag on the VM level not preset the VLANs on the host interface file.

Thanks

 

[mir]

I have not been using that so I have no example to show. But I think you should configure a plain bridge over eth1 and not use any kind of vlan tags on either eth1 or the bridge. Configure the nic assigned to the VM to use this bridge and then specify the vlan tag in the gui. This of course requires that eth1 is connected to a trunk port. I have not tested this.