Hello, this is my first post in this great Forum. Please, excuse me for my English, it's not my language.
I've run a two node Proxmox HA Cluster with quorum disk and fencing devices. Two Fujitsu servers (Proxmox01 and Proxmox02) that have each one a bonding of 2 Intel ethernet interfaces each one connected to a LAG in a 2 units Netgear GST24TS stack switch.
I've created a KVM guest in Proxmox02 with two bridged virtual nic's each one in a different vlan tag over the same bonding interface:
eth0: Vlan tag 301 type Intel e1000 MAC Addres BA:FE:3A:4A:4C:6D bridge vmbr0
eth1: 1Vlan tag 206 type Intel e1000 MAC Addres 62:95:2E:C4:F9:3C bridge vmbr0
Then I've installed CentOS Linux in the guest VM and configured the network with:
eth0 10.140.131.170/27
eth1: 5.10.206.199/26
When I have the guest VM totally stopped I can ping 10.140.131.170 and I've got response :
If I do an arping to 10.140.131.170 I've got response from the MAC address of the guest virtual nick: BA:FE:3A:4A:4C:6D. So it's not another host answering to ping because I have two hosts with the same IP address in my network.
I've cleared the ARP cache of my computer and arp cache of the switch and the router. And still get response from ping and arping when the guest VM is stopped. Also I tried to reboot the Proxmox02 server and still get response from ping and arping, once rebooted.
Correct me if I'm wrong, but if the VM guest is stopped (totally shutdown) I don't have to get response from ping or arping.
The same thing is happening with eth1 but with a difference. If the guest VM is stopped ping to 5.10.206.199 is not responding, but arping gets response.
I've tried the same with another guest VM in Proxmox01 and it's working like it's supposed to. When the VM guest is stopped I can't get any ping or arping response from its ip addres (192.168.100.200). Then I've migrated this guest VM to Proxmox02 and stopped it. And I get the same problem. Now ping and arping get response with the guest VM shutdown and stopped. Even I can do an nmap and i'ts showing the same ports that have the guest VM whn is running:
I can try to connect to the Tomcat server running on the stopped guest VM and I get a 404 error:
Or try to connect to SSH in port 222 when the VM is stopped:
I'ts like a ghost VM!
What is happening? Maybe a Proxmox bug?
It's like if when I stop a guest VM, Proxmox, only unmount the filesystem and the system is still running in memory, instead of completely shutdown the system.
Is there any command tool to show the arp table of the VM guest?
I've tried with arp -n in Proxmox but is only showing the arp entries of the IP's assigned to the physical interfaces not the entries relative to the virtual interfaces of the guests.
This is a big problem for me for two reasons:
Proxmox details:
root@proxmox01:~# pveversion -v
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pve-manager: 3.3-1 (running version: 3.3-1/a06c9f73)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-5
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
root@proxmox02:~# pveversion -v
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pve-manager: 3.3-1 (running version: 3.3-1/a06c9f73)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-5
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
Thank you.
I've run a two node Proxmox HA Cluster with quorum disk and fencing devices. Two Fujitsu servers (Proxmox01 and Proxmox02) that have each one a bonding of 2 Intel ethernet interfaces each one connected to a LAG in a 2 units Netgear GST24TS stack switch.
I've created a KVM guest in Proxmox02 with two bridged virtual nic's each one in a different vlan tag over the same bonding interface:
eth0: Vlan tag 301 type Intel e1000 MAC Addres BA:FE:3A:4A:4C:6D bridge vmbr0
eth1: 1Vlan tag 206 type Intel e1000 MAC Addres 62:95:2E:C4:F9:3C bridge vmbr0
Then I've installed CentOS Linux in the guest VM and configured the network with:
eth0 10.140.131.170/27
eth1: 5.10.206.199/26
When I have the guest VM totally stopped I can ping 10.140.131.170 and I've got response :
ping 10.140.131.170
PING 10.140.131.170 (10.140.131.170) 56(84) bytes of data.
64 bytes from 10.140.131.170: icmp_seq=1 ttl=63 time=43.1 ms
64 bytes from 10.140.131.170: icmp_seq=2 ttl=63 time=3.94 ms
64 bytes from 10.140.131.170: icmp_seq=3 ttl=63 time=2.93 ms
64 bytes from 10.140.131.170: icmp_seq=4 ttl=63 time=2.57 ms
64 bytes from 10.140.131.170: icmp_seq=5 ttl=63 time=2.88 ms
64 bytes from 10.140.131.170: icmp_seq=6 ttl=63 time=2.54 ms
64 bytes from 10.140.131.170: icmp_seq=7 ttl=63 time=3.48 ms
64 bytes from 10.140.131.170: icmp_seq=8 ttl=63 time=5.22 ms
^C
--- 10.140.131.170 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7010ms
rtt min/avg/max/mdev = 2.540/8.340/43.136/13.177 ms
PING 10.140.131.170 (10.140.131.170) 56(84) bytes of data.
64 bytes from 10.140.131.170: icmp_seq=1 ttl=63 time=43.1 ms
64 bytes from 10.140.131.170: icmp_seq=2 ttl=63 time=3.94 ms
64 bytes from 10.140.131.170: icmp_seq=3 ttl=63 time=2.93 ms
64 bytes from 10.140.131.170: icmp_seq=4 ttl=63 time=2.57 ms
64 bytes from 10.140.131.170: icmp_seq=5 ttl=63 time=2.88 ms
64 bytes from 10.140.131.170: icmp_seq=6 ttl=63 time=2.54 ms
64 bytes from 10.140.131.170: icmp_seq=7 ttl=63 time=3.48 ms
64 bytes from 10.140.131.170: icmp_seq=8 ttl=63 time=5.22 ms
^C
--- 10.140.131.170 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7010ms
rtt min/avg/max/mdev = 2.540/8.340/43.136/13.177 ms
If I do an arping to 10.140.131.170 I've got response from the MAC address of the guest virtual nick: BA:FE:3A:4A:4C:6D. So it's not another host answering to ping because I have two hosts with the same IP address in my network.
I've cleared the ARP cache of my computer and arp cache of the switch and the router. And still get response from ping and arping when the guest VM is stopped. Also I tried to reboot the Proxmox02 server and still get response from ping and arping, once rebooted.
Correct me if I'm wrong, but if the VM guest is stopped (totally shutdown) I don't have to get response from ping or arping.
The same thing is happening with eth1 but with a difference. If the guest VM is stopped ping to 5.10.206.199 is not responding, but arping gets response.
I've tried the same with another guest VM in Proxmox01 and it's working like it's supposed to. When the VM guest is stopped I can't get any ping or arping response from its ip addres (192.168.100.200). Then I've migrated this guest VM to Proxmox02 and stopped it. And I get the same problem. Now ping and arping get response with the guest VM shutdown and stopped. Even I can do an nmap and i'ts showing the same ports that have the guest VM whn is running:
nmap 192.168.100.200
Starting Nmap 6.40 ( http://nmap.org ) at 2014-11-04 17:26 CET
Nmap scan report for 192.168.100.200
Host is up (0.0083s latency).
Not shown: 996 filtered ports
PORT STATE SERVICE
80/tcp open http
222/tcp open rsh-spx
443/tcp open https
9080/tcp open glrpc
Starting Nmap 6.40 ( http://nmap.org ) at 2014-11-04 17:26 CET
Nmap scan report for 192.168.100.200
Host is up (0.0083s latency).
Not shown: 996 filtered ports
PORT STATE SERVICE
80/tcp open http
222/tcp open rsh-spx
443/tcp open https
9080/tcp open glrpc
I can try to connect to the Tomcat server running on the stopped guest VM and I get a 404 error:
HTTP Status 404 - /debug.jsp;jsessionid=3C7377341682232DBEE7A5214104F503
type Status report
message /debug.jsp;jsessionid=3C7377341682232DBEE7A5214104F503
description The requested resource is not available.
type Status report
message /debug.jsp;jsessionid=3C7377341682232DBEE7A5214104F503
description The requested resource is not available.
Or try to connect to SSH in port 222 when the VM is stopped:
ssh 192.168.100.200 -p222
root@192.168.100.200's password:
Permission denied, please try again.
root@192.168.100.200's password:
Permission denied, please try again.
root@192.168.100.200's password:
Connection closed by 192.168.100.200
root@192.168.100.200's password:
Permission denied, please try again.
root@192.168.100.200's password:
Permission denied, please try again.
root@192.168.100.200's password:
Connection closed by 192.168.100.200
I'ts like a ghost VM!
What is happening? Maybe a Proxmox bug?
It's like if when I stop a guest VM, Proxmox, only unmount the filesystem and the system is still running in memory, instead of completely shutdown the system.
Is there any command tool to show the arp table of the VM guest?
I've tried with arp -n in Proxmox but is only showing the arp entries of the IP's assigned to the physical interfaces not the entries relative to the virtual interfaces of the guests.
This is a big problem for me for two reasons:
- If a server goes down my monitoring system still gets response from ping and don't send me the alert.
- If I want to assign one public IP address to a new VM guest and before the same address was used in another VM guest I don't have connectivity because the MAC address of the old VM is still responding and generating fake arp entries.
Proxmox details:
root@proxmox01:~# pveversion -v
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pve-manager: 3.3-1 (running version: 3.3-1/a06c9f73)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-5
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
root@proxmox02:~# pveversion -v
proxmox-ve-2.6.32: 3.2-136 (running kernel: 2.6.32-32-pve)
pve-manager: 3.3-1 (running version: 3.3-1/a06c9f73)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-34
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-5
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
Thank you.