Crazy behavior - Turning off Node 1 makes a VM in Node 3 inaccessible

[pjkenned]

The quick background on what I am trying to accomplish:
I have multiple-node per chassis rack units. I am trying to remove the top chassis and replace with newer hardware. As a result, I am trying to move all VMs to the second chassis.

Here is the basic setup
Chassis - Node - What is on node
Chassis0 - Prox1 - old home of vbulletin VM
Chassis1 - Prox2 - WordpressVM
Chassis1 - Prox3 - Prox3 - new home of vbulletinVM

The vbulletin VM is given a static route to the internal (10.0.x.x) IP address of the CentOS VM. Triple checked this.

We copied the drive from Prox1 to Prox3 and I can confirm that it is on the local directory for Prox3:

root@proxhv03:~# find / -name vm-102-disk-1.raw
/var/lib/vz/images/102/vm-102-disk-1.raw

I can also see that it is using CPU memory on Prox3 and I can see it running in the web interface on Prox3.

The issue
When I shutdown Prox1 the following happens:
Prox2 - wordpressVM works fine
Prox3 - vbulletinVM becomes inaccessible

Any ideas? I have read a bunch of documentation but cannot find a lead as to what this might be.

 

[udo]

Hi,
have you anything mounted inside the VM from Node 1 (perhaps you have an nfs-share configured?).

Udo

 

[pjkenned]

Hi,
have you anything mounted inside the VM from Node 1 (perhaps you have an nfs-share configured?).

Udo

Should not be:

[root@forums ~]# cat /proc/mountsrootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=6083960k,nr_inodes=1520990,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/vda1 / ext4 rw,relatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/vda3 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

 

[udo]

Should not be:

[root@forums ~]# cat /proc/mountsrootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=6083960k,nr_inodes=1520990,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/vda1 / ext4 rw,relatime,barrier=1,data=ordered 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/vda3 /home ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

Hi,
have you defined any iptables-rules to access VM3 on node1?
Do you see traffic on node1 when the VM run on node3 ( tcpdump -i vmbr0 host ip.of.vm.3 ).

Udo

 

[pjkenned]

Hi,
have you defined any iptables-rules to access VM3 on node1?
Do you see traffic on node1 when the VM run on node3 ( tcpdump -i vmbr0 host ip.of.vm.3 ).

Udo

Ended up just dumping the raw disk, converting it to a vhd and putting it on a Hyper-V cluster. Interestingly enough, did the same thing with a Wordpress installation and it is running faster (same dual L5520 base) on the Windows Hyper-V Server 2012 R2 machine (the "free" one). Good news is that the Proxmox->Hyper-V migration was very easy so I can re-do the cluster now.