Failed migration from Proxmox1 to Proxmox2 and related issues

[Erick]

I have been moving VM's from our Proxmox1.9 machine to our Proxmox2.0-59 machine. After transferring the 4th VM I started it's restore. It took longer that usual, so I waited another 45 mins before I (ctrl -c)'d to kill the process. I then tried again, same result. I then realized that I was calling the wrong logical volume, I corrected for it and tried again. Again no joy. Last I went back to my proxmox1 machine compressed the VM again, transferred it, changed the ID, and started the restore, this time I let it work for a few hours, still no good. During this another issues cropped up. My Proxmox 2 web interface, in the "server view" field shows a red icon for the server I was trying to restore the VM on. In addition all of the VM's listed under that server show offline icons, and there labels are not displayed in the search window. If you then click on the VM's there summary states that they are still running.

What is the red icon for in the server view?
Why are the VM's Not showing that they are running? "when they are"
How do I recover from this with minimal downtime?

Thank you for your time,
~Erick


More information on the process I was using to transfer the VM's:

ssh -a root@proxmox1a
vzdump --compress --dumpdir /tmp $ID
cd /tmp/
rsync -avP --bwlimit 20000 file.tgz root@(proxmox2a's Ip address):/tmp/

ssh root@proxmox2a
qmrestore --unique --storage VG2a /tmp/<file> 301

 

[dietmar]

Please can you post the output from the restore process?

 

[Erick]

Thank's Dietmar for your responce. I'm hunting for the output from qmrestore command unfortunatly it's been a few day's, and I <ctrl-c> out of it.

Update: We have found a hung process unfortunatly a kill -9 won't kill this process. Any ideas?
LVM commands on that volume group hang due to that lock.

root@proxmox2a:~# ps -ef | grep lv
root 932754 1 0 Jun05 ? 00:00:00 /sbin/lvcreate -aly --addtag pve-vm-307 --size 10485760.9990234k --name vm-307-disk-1 VG2b
root 983660 977632 0 10:50 pts/0 00:00:00 grep lv
root@proxmox2a:~#

Thank you for your time,
~Erick

 

[dietmar]

Do you use CLVM?

 

[jag3773]

A few more details about our configuration:

We are running a 2-node Proxmox 2.0 VE DRBD Cluster (2a and 2b). The main caveat to the wiki page describing this setup is that we run 2 DRBD volumes with a separate volume group on each. One is VG2a and the other is VG2b. We do this primarily in case we run into a split brain situation, we consider VG2b to be canonical on the 2b host and VG2a to be canonical on the 2a host because we ensure that VMs in the 2b volume group are running on the 2b host whenever possible (and the same for 2a). The goal is to lose less data in that circumstance

There is no reason I can think of for that lvcreate command to fail, except that its parent command was killed with Ctrl-C. It's not clear whether the stuck lvcreate command caused the parent command to hang (which prompted the Ctrl-C) or if the Ctrl-C on the parent command caused the lvcreate to hang. What is clear is that our normal restore takes 10-15 minutes and this VM restore was running for 40+ minutes (its no bigger than the others either).

LVM operations are successful on our 2b server. I don't think there is a problem with the volume group itself, I think it's just that the stuck process has a lock file in place on the 2a server, so any new LVM commands on 2a fail because they think there is an active lock on it.

We did try a `vgchange --refresh` on the 2b server, which completed just fine. The same command on the 2a server fails though, likely due to the lock.

Our current plan is to to stop rgmanager on 2a and thereby force all the VMs on that node to be restarted onto the other node. Then proceed with a full reboot of 2a which *should* clear out the problem.

I'm not convinced that this is a Proxmox code problem, but is there any debugging output that we can provide that would be helpful to you?

 

[dietmar]

but is there any debugging output that we can provide that would be helpful to you?

I don't really have an idea what cause such lock. If you cant kill the process, you need to reboot.