I have a strange problem that I'm hoping someone can help with.
I've set up 3 extra PM servers running under VMWare ESXi to use as HA in case our primary physical server goes down.
As a reference for this discussion, the 4 server names are:
pve0 = bare metal
pve1, 2 and 3: VMWare ESXi
The ESXi-based servers themselves work fine and can get out to the internet.
There are 2 VLANs on all servers:
VLAN 28 - Primary
VLAN10 - LAB
All servers can ping the 10 network and pve0's VM's work fine on it.
When I migrate a VM from pve0 to any of the ESXi-based servers, they can communicate fine to VLAN10.
But, if I reboot any of the VMs hosted the ESXI-based servers, they are unable to communicate to VLAN10.
So I have to migrate them back to pve0, then migrate them again to pve1, 2, or 3.
I have verified and re-verified our Cisco Switch and the ESXi vSwitch settings and can't find anything wrong.
Any help would be greatly appreciated.
Configs Follow
Cisco Switch:
pve0 (the bare metal server):
interface GigabitEthernet1/0/26
description PVE0-ETH5-Vlan10
switchport access vlan 10
switchport trunk native vlan 10
end
pve1, 2 and 3:
interface GigabitEthernet1/0/11
description Switch-Gi1/0/11->VMware ES vmnic2
switchport access vlan 10
switchport trunk native vlan 10
switchport mode trunk
end
Proxmox Network Config:
pve0:
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual
# VLAN 29 (Cluster comms)
auto eth4
iface eth4 inet static
address 172.168.29.11
netmask 255.255.255.0
iface enp6s0f1 inet manual
iface eth5 inet manual
auto vmbr0
iface vmbr0 inet static
address 172.168.28.11
netmask 255.255.255.0
gateway 172.168.28.1
bridge_ports eth0
bridge_stp off
bridge_fd 0
auto vmbr10
iface vmbr10 inet static
address 10.0.0.11
netmask 255.255.254.0
bridge_ports eth5
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes
#VLAN10 - LAB
pve1,2 and 3:
auto lo
iface lo inet loopback
auto ens224
iface ens224 inet static
address 172.168.29.220
netmask 255.255.255.0
iface ens192 inet manual
iface ens256 inet manual
auto vmbr0
iface vmbr0 inet static
address 172.168.28.220
netmask 255.255.255.0
gateway 172.168.28.1
bridge-ports ens256
bridge-stp off
bridge-fd 0
#VLAN 28
auto vmbr10
iface vmbr10 inet static
address 10.0.0.220
netmask 255.255.254.0
bridge-ports ens192
bridge-stp off
bridge-fd 0
bridge_vlan_aware yes
#VLAN 10
I've set up 3 extra PM servers running under VMWare ESXi to use as HA in case our primary physical server goes down.
As a reference for this discussion, the 4 server names are:
pve0 = bare metal
pve1, 2 and 3: VMWare ESXi
The ESXi-based servers themselves work fine and can get out to the internet.
There are 2 VLANs on all servers:
VLAN 28 - Primary
VLAN10 - LAB
All servers can ping the 10 network and pve0's VM's work fine on it.
When I migrate a VM from pve0 to any of the ESXi-based servers, they can communicate fine to VLAN10.
But, if I reboot any of the VMs hosted the ESXI-based servers, they are unable to communicate to VLAN10.
So I have to migrate them back to pve0, then migrate them again to pve1, 2, or 3.
I have verified and re-verified our Cisco Switch and the ESXi vSwitch settings and can't find anything wrong.
Any help would be greatly appreciated.
Configs Follow
Cisco Switch:
pve0 (the bare metal server):
interface GigabitEthernet1/0/26
description PVE0-ETH5-Vlan10
switchport access vlan 10
switchport trunk native vlan 10
end
pve1, 2 and 3:
interface GigabitEthernet1/0/11
description Switch-Gi1/0/11->VMware ES vmnic2
switchport access vlan 10
switchport trunk native vlan 10
switchport mode trunk
end
Proxmox Network Config:
pve0:
auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual
# VLAN 29 (Cluster comms)
auto eth4
iface eth4 inet static
address 172.168.29.11
netmask 255.255.255.0
iface enp6s0f1 inet manual
iface eth5 inet manual
auto vmbr0
iface vmbr0 inet static
address 172.168.28.11
netmask 255.255.255.0
gateway 172.168.28.1
bridge_ports eth0
bridge_stp off
bridge_fd 0
auto vmbr10
iface vmbr10 inet static
address 10.0.0.11
netmask 255.255.254.0
bridge_ports eth5
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes
#VLAN10 - LAB
pve1,2 and 3:
auto lo
iface lo inet loopback
auto ens224
iface ens224 inet static
address 172.168.29.220
netmask 255.255.255.0
iface ens192 inet manual
iface ens256 inet manual
auto vmbr0
iface vmbr0 inet static
address 172.168.28.220
netmask 255.255.255.0
gateway 172.168.28.1
bridge-ports ens256
bridge-stp off
bridge-fd 0
#VLAN 28
auto vmbr10
iface vmbr10 inet static
address 10.0.0.220
netmask 255.255.254.0
bridge-ports ens192
bridge-stp off
bridge-fd 0
bridge_vlan_aware yes
#VLAN 10