proxmox 3.4-6 cluster vzdump hanging

[sptnelson]

vzdump hanging is the observable issue, but I believe the root cause has something to do with acquiring locks.

I have a 3 node cluster, and the main node with the GUI seems to be working fine. The other two nodes are not reporting any status back to the gui.

I've restarted pvestatd, and looked at /etc/pve/.rrd, and no VMs from the two broken nodes are listed there.

I'd prefer to not reboot these nodes, as the VMs are running properly.

Any vzdump simply hangs forever. They are trying to write to an NFS mounted directory that I have verified is properly mounted, and has plenty of disk space available.

What else can I try to resolve this issue? What other information can I provide to help solve this? I've considered restarting the corosync service, but I've never done that before, and I'm not sure if it's a good idea.

Thanks in advance
Tony Nelson
Starpoint Solutions

 

[dietmar]

The other two nodes are not reporting any status back to the gui.

Please make sure the 'pvestad' service is running there. Also make sure all storages ar online and accessible - verify with

# pvesm status

Any vzdump simply hangs forever.

Maybe there is already a vzdump process running?

 

[sptnelson]

All of my storage looks good.

root@ny-vm02:/var/lock/qemu-server# pvesm status
local dir 1 304619824 252179656 52440168 83.29%
vmbackups nfs 1 4466830464 992064384 3474766080 22.71%

The only thing that changes on the other nodes is the %used in local.

This morning I had to kill vzdump on 2 of the 3 nodes. Unfortunately there isn't much information in the logs.

root@ny-vm03:/var/log/vzdump# ls -lart
total 64
drwxr-xr-x 2 root root 4096 Aug 21 2015 .
-rw-r--r-- 1 root root 12153 Apr 29 02:13 qemu-115.log
-rw-r--r-- 1 root root 10080 Apr 30 01:10 qemu-103.log
-rw-r--r-- 1 root root 853 Apr 30 01:10 qemu-105.log
-rw-r--r-- 1 root root 9852 Apr 30 01:15 qemu-107.log
-rw-r--r-- 1 root root 4444 Apr 30 01:23 qemu-109.log
-rw-r--r-- 1 root root 94 Apr 30 01:23 qemu-111.log
-rw-r--r-- 1 root root 94 May 4 01:00 qemu-101.log
drwxr-xr-x 18 root root 4096 May 4 06:25 ..
root@ny-vm03:/var/log/vzdump# cat qemu-101.log
May 04 01:00:01 INFO: Starting Backup of VM 101 (qemu)
May 04 01:00:01 INFO: status = running
root@ny-vm03:/var/log/vzdump#

Is there another log I can look in?

 

[sptnelson]

Also, I stopped pvestatd, verified that it was stopped, and restarted it.

Two of the nodes are still not updating in the UI.