[SOLVED] Interior Networking: Guests Communicating with Other Guests...

[epretorious]

I've created a VMware ESXi guest in PVE and cloned it twice to create a 3-node ESXi cluster on a single PVE host system. (Intel Core i7-7700K [4 cores @ 4.20GHz] with 64 GB of RAM)

I've added an Ubuntu "helper" VM to test network connectivity inside of PVE...

1. The ESXi Host Client interface of all three VM's are accessible from the browser installed on my desktop.
2. I was able to use the vCSA installer via my desktop and successfully install the VMware vSphere Client Server Appliance (vCSA) on the first ESXi guest.
3. The VMware Appliance Management Interface (i.e., vAMI) and the vSphere client are both accessible from my desktop.
4. Each of the ESXi hosts respond somewhat unreliably to ping requests from my desktop.
5. The Ubuntu "helper" VM that I've installed can ping all four addresses! i.e., The helper can ping each of the ESXi hosts and the vCSA!
6. But NONE of the ESXi hosts are able to ping each other!

i.e., Networking seems "reliable enough" from the from the perspective of my desktop and from the perspective of the "helper" VM.

1. But none of the ESXi host systems are able to ping each other.
2. And when I try to add ESXi hosts to the vCSA: I'm only able to add the ESXi host that currently hosts the vCSA. i.e., The vCSA is unable to reach the other two ESXi hosts!

What might be causing this behavior? Might my host system be overwhelmed by the three ESXi guests? Might networking be misconfigured in PVE?

TIA,
Eric Pretorious
Reno, Nevada

 

[epretorious]

I've verified that the PVE firewall is disabled on the vNIC of each ESXi host.

HTH,
Eric P.

 

[epretorious]

FWIW: All of the systems are on my simple, flat home network (i.e., 192.168.0.0/24).

HTH,
Eric P.

 

[Chris]

Hi,
do all of the cloned VMs have unique MAC addresses? Please monitor the icmp traffic on the bridge the VMs are attached to via tcpdump while pinging by running e.g. tcpdump -i vmbr0 icmp. Do you see the traffic there? What does and ip neigh tell you?

 

[mrwizardno2]

Hey Eric, check and see if you specified the /24 in the network settings on the guest. I had something similar where I used /32 and suddenly guests on the same host couldn't talk to each other, but could somehow talk to the gateway. Had me scratching my head!

 

[epretorious]

Hi,
do all of the cloned VMs have unique MAC addresses? Please monitor the icmp traffic on the bridge the VMs are attached to via tcpdump while pinging by running e.g. tcpdump -i vmbr0 icmp. Do you see the traffic there? What does and ip neigh tell you?

That's an interesting thought, Chris! I followed your directions and used tcpdump on the PVE host system to discover that the ESXi guests are all using duplicate MAC addresses!...

*** EDITED FOR CLARITY ***
16:47:41.522787 bc:24:11:f4:45:2b ... > bc:24:11:39:36:64 ... ubuntu-1.commcloudproj.local > 192.168.0.11: ICMP echo request, id 17 ...
16:47:41.522935 bc:24:11:39:36:64 ... > bc:24:11:f4:45:2b ... 192.168.0.11 > ubuntu-1.commcloudproj.local: ICMP echo reply, id 17 ...
16:47:41.526530 bc:24:11:f4:45:2b ... > bc:24:11:39:36:64 ... ubuntu-1.commcloudproj.local > 192.168.0.12: ICMP echo request, id 18 ...
16:47:41.526694 bc:24:11:39:36:64 ... > bc:24:11:f4:45:2b ... 192.168.0.12 > ubuntu-1.commcloudproj.local: ICMP echo reply, id 18 ...
16:47:41.530939 bc:24:11:f4:45:2b ... > bc:24:11:39:36:64 ... ubuntu-1.commcloudproj.local > 192.168.0.13: ICMP echo request, id 19 ...
16:47:41.531117 bc:24:11:39:36:64 ... > bc:24:11:f4:45:2b ... 192.168.0.13 > ubuntu-1.commcloudproj.local: ICMP echo reply, id 19 ...
16:47:41.534346 bc:24:11:f4:45:2b ... > 00:0c:29:95:98:27 ... ubuntu-1.commcloudproj.local > 192.168.0.20: ICMP echo request, id 20 ...
16:47:41.534795 00:0c:29:95:98:27 ... > bc:24:11:f4:45:2b ... 192.168.0.20 > ubuntu-1.commcloudproj.local: ICMP echo reply, id 20 ...

I'll try creating new ESXi guests from scratch and see if the problem goes away.

Eric P.

 

[epretorious]

I'll try creating new ESXi guests from scratch and see if the problem goes away.

Rather than go through the process of destroying the guests and creating them anew: I decided to simply reinstall ESXi. This had the effect of overwriting the hardware MAC address that was stored in the vmkernel settings with the VM's correct MAC address. And that solved the problem!

For more information about how to correctly clone ESXi guests: See this article...

https://williamlam.com/2013/12/how-to-properly-clone-nested-esxi-vm.html

Eric P.

 

[Chris]

Rather than go through the process of destroying the guests and creating them anew: I decided to simply reinstall ESXi. This had the effect of overwriting the hardware MAC address that was stored in the vmkernel settings with the VM's correct MAC address. And that solved the problem!

For more information about how to correctly clone ESXi guests: See this article...

https://williamlam.com/2013/12/how-to-properly-clone-nested-esxi-vm.html

Eric P.

Hi Eric,
I am glad you could solve the problem and shared your solution!

Please mark the thread as solved for others to find a solution more easily, thank you.

 

[epretorious]

Please mark the thread as solved for others to find a solution more easily, thank you.

How do I do that, Chris?

TIA,
Eric P.

 

[Chris]

How do I do that, Chris?

TIA,
Eric P.

You can add the solved prefix by editing the thread at the beginning, thank you.

 

[epretorious]

You can add the solved prefix by editing the thread at the beginning, thank you.

Done!

Thanks, again!
Eric P.