unable to join via gui - login failed

liska_ · Mar 21, 2014

Hi all,
last two days I have a problem to login via gui. I created a cluster, no HA, and everything was working (well two linux KVM guest - at all I have around twenty kvm guests and five openvz containers - got problems, that they were not backed up and they were shutted down during that, but if I run just a backup of these machines then everything was ok).

Anyway, on wednesday I moved two containers from local to a nfs shared storage, where is located the rest of containers. Until this time there had been no problem. But this backup run via cron did not finished. it got stucked on backuping one container. For some time I was able to log in but it is not possible anymore. Login via ssh is still working. I tried manually unmounting backup nfs server as it was not responding, restarting pveproxy on both server, but nothing helps. I found on this forum that it could be happening because of different time, but unfortunatelly time on both machines is correct.
In pveproxy/access.log I can see
ip - - [21/Mar/2014:09:12:43 +0100] "POST /api2/json/access/ticket HTTP/1.1" 500 -
ip - - [21/Mar/2014:09:12:53 +0100] "POST /api2/json/access/ticket HTTP/1.1" 500 -
ip - - [21/Mar/2014:09:15:06 +0100] "POST /api2/json/access/ticket HTTP/1.1" 500 -
ip - - [21/Mar/2014:09:15:29 +0100] "POST /api2/json/access/ticket HTTP/1.1" 500 -

while in syslog
pveproxy[844530]: WARNING: proxy detected vanished client connection
pveproxy[844530]: WARNING: proxy detected vanished client connection

I use also monitoring account created through web interface, which stopped working at the same time. Can you please give me some advises how to solve this?

On both servers same versions:
pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-24 (running version: 3.1-24/060bd5a6)
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-9
libpve-access-control: 3.0-8
libpve-storage-perl: 3.0-18
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-6
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1

udo · Mar 21, 2014

liska_ said:
Hi all,
last two days I have a problem to login via gui. I created a cluster, no HA, and everything was working (well two linux KVM guest - at all I have around twenty kvm guests and five openvz containers - got problems, that they were not backed up and they were shutted down during that, but if I run just a backup of these machines then everything was ok).
...

Hi,
how is the status of your nodes?
Can you post the output of following commands:

Code:

clustat
pvecm nodes
df -h
cat /etc/pve/storage.cfg
cat /var/log/pve/tasks/active
cat /proc/loadavg

and what is the output of following command during an login on the gui?

Code:

tail -f /var/log/syslog

Udo

liska_ · Mar 24, 2014

Hi, thanks for answer. In syslog I can see only this repeating message cluster3 pveproxy[935179]: WARNING: proxy detected vanished client connection There was mounted another nfs storage for nfs, which I unmounted using mount -l and delete from storage.cfg. I added it via gui, no special mount options. Bellow are outputs of those commands: clustat Cluster Status for cluster @ Mon Mar 24 10:07:42 2014 Member Status: Quorate Member Name ID Status ------ ---- ---- ------ cluster3 1 Online, Local cluster 2 Online pvecm nodes Node Sts Inc Joined Name 1 M 4 2014-01-27 11:59:20 cluster3 2 M 356 2014-02-18 16:06:03 cluster df -H Filesystem Size Used Avail Use% Mounted on udev 11M 0 11M 0% /dev tmpfs 5.1G 467k 5.1G 1% /run /dev/mapper/pve-root 15G 1.9G 12G 14% / tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs 11G 49M 11G 1% /run/shm /dev/mapper/pve-data 30G 1.9G 28G 7% /var/lib/vz /dev/sda1 519M 66M 427M 14% /boot 192.168.130.14:/mnt/virtualy/virtualy 4.3T 4.2T 178G 96% /mnt/pve/skladka_virtualy /dev/fuse 32M 33k 32M 1% /etc/pve 192.168.130.18:/fast/virtualy 968G 310G 659G 33% /mnt/pve/indian_virtualy 192.168.130.18:/zalohy/virtualy 5.0T 4.4T 580G 89% /mnt/pve/indian_zalohy cat /etc/pve/storage.cfg nfs: skladka_virtualy path /mnt/pve/skladka_virtualy server 192.168.130.14 export /mnt/virtualy/virtualy options vers=3 content images,iso,vztmpl,rootdir,backup maxfiles 2 dir: local path /var/lib/vz content images,iso,vztmpl,rootdir maxfiles 0 nfs: indian_virtualy path /mnt/pve/indian_virtualy server 192.168.130.18 export /fast/virtualy options vers=3 content images,rootdir maxfiles 1 nfs: indian_zalohy path /mnt/pve/indian_zalohy server 192.168.130.18 export /zalohy/virtualy options vers=3 content iso maxfiles 1 cat /var/log/pve/tasks/active UPID:cluster3:000C8F5C:1D5B78C4:532AD907:vncproxy:122:root@pam: 1 532AD911 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1 UPID:cluster3:000C8DB4:1D59A52E:532AD45A:vncproxy:122:root@pam: 1 532AD464 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1 UPID:cluster3:000B6131:1CDDBE47:5329972A:vzdump::root@pam: 1 53299799 OK UPID:cluster3:000B28A1:1CCC58E6:53296AA1:vzdump::root@pam: 1 53296B0C OK UPID:cluster3:0009CFFE:1C617A0C:5328590A:vncproxy:131:root@pam: 1 532864AA OK UPID:cluster3:0009CFE4:1C617276:532858F7:vncproxy:131:root@pam: 1 53285901 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1 UPID:cluster3:0009CE01:1C60E072:53285781:vncproxy:131:root@pam: 1 532857B0 OK UPID:cluster3:0009CDE8:1C60D9CD:53285770:vncproxy:131:root@pam: 1 5328577E command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009CD7F:1C60B6B3:53285716:vncproxy:131:root@pam: 1 53285752 OK UPID:cluster3:0009C40D:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FD3 OK UPID:cluster3:0009C40C:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FA4 OK UPID:cluster3:0009C3C3:1C5DBA2D:53284F71:vncproxy:131:root@pam: 1 53284FA1 OK UPID:cluster3:0009C3B1:1C5DB3D0:53284F61:vncproxy:131:root@pam: 1 53284F6D command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009C2FE:1C5D8792:53284EF0:vncproxy:131:root@pam: 1 53284F3B OK UPID:cluster3:0009C2C0:1C5D7448:53284EBE:vncproxy:131:root@pam: 1 53284EED OK UPID:cluster3:0009C2B2:1C5D6E09:53284EAE:vncproxy:131:root@pam: 1 53284EBA command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009C266:1C5D582A:53284E76:vncproxy:131:root@pam: 1 53284E80 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1 UPID:cluster3:0009C1D2:1C5D28CB:53284DFD:vncproxy:131:root@pam: 1 53284E4D OK UPID:cluster3:0009C17D:1C5D0E32:53284DB9:vncproxy:131:root@pam: 1 53284DC4 OK UPID:cluster3:0009C105:1C5CEDB1:53284D66:vncproxy:131:root@pam: 1 53284DB1 OK UPID:cluster3:0009BFDE:1C5C944F:53284C81:vncproxy:131:root@pam: 1 53284CB2 OK UPID:cluster3:0009A6EE:1C54EF67:532838EF:vncproxy:131:root@pam: 1 532843EE OK UPID:cluster3:00099E2C:1C523B88:53283204:vncproxy:131:root@pam: 1 5328386A OK UPID:cluster3:00099A82:1C5119FF:53282F1F:vncproxy:131:root@pam: 1 53282F52 OK UPID:cluster3:000996C2:1C4FF936:53282C3C:vncproxy:131:root@pam: 1 53282C85 OK cat /proc/loadavg 8.29 8.31 8.32 4/681 940169

liska_ · Mar 24, 2014

I have a problem with text formatting. Last week it was working but now it keeps deleting all the EOL marks. How can I paste logs correctly?

liska_ said:
Hi, thanks for answer. In syslog I can see only this repeating message cluster3 pveproxy[935179]: WARNING: proxy detected vanished client connection There was mounted another nfs storage for nfs, which I unmounted using mount -l and delete from storage.cfg. I added it via gui, no special mount options. Bellow are outputs of those commands: clustat Cluster Status for cluster @ Mon Mar 24 10:07:42 2014 Member Status: Quorate Member Name ID Status ------ ---- ---- ------ cluster3 1 Online, Local cluster 2 Online pvecm nodes Node Sts Inc Joined Name 1 M 4 2014-01-27 11:59:20 cluster3 2 M 356 2014-02-18 16:06:03 cluster df -H Filesystem Size Used Avail Use% Mounted on udev 11M 0 11M 0% /dev tmpfs 5.1G 467k 5.1G 1% /run /dev/mapper/pve-root 15G 1.9G 12G 14% / tmpfs 5.3M 0 5.3M 0% /run/lock tmpfs 11G 49M 11G 1% /run/shm /dev/mapper/pve-data 30G 1.9G 28G 7% /var/lib/vz /dev/sda1 519M 66M 427M 14% /boot 192.168.130.14:/mnt/virtualy/virtualy 4.3T 4.2T 178G 96% /mnt/pve/skladka_virtualy /dev/fuse 32M 33k 32M 1% /etc/pve 192.168.130.18:/fast/virtualy 968G 310G 659G 33% /mnt/pve/indian_virtualy 192.168.130.18:/zalohy/virtualy 5.0T 4.4T 580G 89% /mnt/pve/indian_zalohy cat /etc/pve/storage.cfg nfs: skladka_virtualy path /mnt/pve/skladka_virtualy server 192.168.130.14 export /mnt/virtualy/virtualy options vers=3 content images,iso,vztmpl,rootdir,backup maxfiles 2 dir: local path /var/lib/vz content images,iso,vztmpl,rootdir maxfiles 0 nfs: indian_virtualy path /mnt/pve/indian_virtualy server 192.168.130.18 export /fast/virtualy options vers=3 content images,rootdir maxfiles 1 nfs: indian_zalohy path /mnt/pve/indian_zalohy server 192.168.130.18 export /zalohy/virtualy options vers=3 content iso maxfiles 1 cat /var/log/pve/tasks/active UPID:cluster3:000C8F5C:1D5B78C4:532AD907:vncproxy:122:root@pam: 1 532AD911 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1 UPID:cluster3:000C8DB4:1D59A52E:532AD45A:vncproxy:122:root@pam: 1 532AD464 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1 UPID:cluster3:000B6131:1CDDBE47:5329972A:vzdump::root@pam: 1 53299799 OK UPID:cluster3:000B28A1:1CCC58E6:53296AA1:vzdump::root@pam: 1 53296B0C OK UPID:cluster3:0009CFFE:1C617A0C:5328590A:vncproxy:131:root@pam: 1 532864AA OK UPID:cluster3:0009CFE4:1C617276:532858F7:vncproxy:131:root@pam: 1 53285901 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1 UPID:cluster3:0009CE01:1C60E072:53285781:vncproxy:131:root@pam: 1 532857B0 OK UPID:cluster3:0009CDE8:1C60D9CD:53285770:vncproxy:131:root@pam: 1 5328577E command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009CD7F:1C60B6B3:53285716:vncproxy:131:root@pam: 1 53285752 OK UPID:cluster3:0009C40D:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FD3 OK UPID:cluster3:0009C40C:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FA4 OK UPID:cluster3:0009C3C3:1C5DBA2D:53284F71:vncproxy:131:root@pam: 1 53284FA1 OK UPID:cluster3:0009C3B1:1C5DB3D0:53284F61:vncproxy:131:root@pam: 1 53284F6D command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009C2FE:1C5D8792:53284EF0:vncproxy:131:root@pam: 1 53284F3B OK UPID:cluster3:0009C2C0:1C5D7448:53284EBE:vncproxy:131:root@pam: 1 53284EED OK UPID:cluster3:0009C2B2:1C5D6E09:53284EAE:vncproxy:131:root@pam: 1 53284EBA command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009C266:1C5D582A:53284E76:vncproxy:131:root@pam: 1 53284E80 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1 UPID:cluster3:0009C1D2:1C5D28CB:53284DFD:vncproxy:131:root@pam: 1 53284E4D OK UPID:cluster3:0009C17D:1C5D0E32:53284DB9:vncproxy:131:root@pam: 1 53284DC4 OK UPID:cluster3:0009C105:1C5CEDB1:53284D66:vncproxy:131:root@pam: 1 53284DB1 OK UPID:cluster3:0009BFDE:1C5C944F:53284C81:vncproxy:131:root@pam: 1 53284CB2 OK UPID:cluster3:0009A6EE:1C54EF67:532838EF:vncproxy:131:root@pam: 1 532843EE OK UPID:cluster3:00099E2C:1C523B88:53283204:vncproxy:131:root@pam: 1 5328386A OK UPID:cluster3:00099A82:1C5119FF:53282F1F:vncproxy:131:root@pam: 1 53282F52 OK UPID:cluster3:000996C2:1C4FF936:53282C3C:vncproxy:131:root@pam: 1 53282C85 OK cat /proc/loadavg 8.29 8.31 8.32 4/681 940169

m.ardito · Mar 24, 2014

weird but happens for some users...
try embedding all in "CODE" markers (see edit in advanced mode)

Marco

dietmar · Mar 24, 2014

liska_ said:
I have a problem with text formatting. Last week it was working but now it keeps deleting all the EOL marks. How can I paste logs correctly?

use the CODE tag

liska_ · Mar 24, 2014

Hi, thanks for answer. In syslog I can see only this repeating message

Code:

 cluster3 pveproxy[935179]: WARNING: proxy detected vanished client connection

There was mounted another nfs storage for nfs, which I unmounted using mount -l and delete from storage.cfg. I added it via gui, no special mount options. Bellow are outputs of those commands:

Code:

 clustat Cluster Status for cluster @ Mon Mar 24 10:07:42 2014 Member Status: Quorate   Member Name                                                ID   Status  ------ ----                                                ---- ------  cluster3                                                       1 Online, Local  cluster                                                        2 Online

Code:

 pvecm nodes Node  Sts   Inc   Joined               Name    1   M      4   2014-01-27 11:59:20  cluster3    2   M    356   2014-02-18 16:06:03  cluster

Code:

 df -H Filesystem                             Size  Used Avail Use% Mounted on udev                                    11M     0   11M   0% /dev tmpfs                                  5.1G  467k  5.1G   1% /run /dev/mapper/pve-root                    15G  1.9G   12G  14% / tmpfs                                  5.3M     0  5.3M   0% /run/lock tmpfs                                   11G   49M   11G   1% /run/shm /dev/mapper/pve-data                    30G  1.9G   28G   7% /var/lib/vz /dev/sda1                              519M   66M  427M  14% /boot 192.168.130.14:/mnt/virtualy/virtualy  4.3T  4.2T  178G  96% /mnt/pve/skladka_virtualy /dev/fuse                               32M   33k   32M   1% /etc/pve 192.168.130.18:/fast/virtualy          968G  310G  659G  33% /mnt/pve/indian_virtualy 192.168.130.18:/zalohy/virtualy        5.0T  4.4T  580G  89% /mnt/pve/indian_zalohy

Code:

 cat /etc/pve/storage.cfg nfs: skladka_virtualy     path /mnt/pve/skladka_virtualy     server 192.168.130.14     export /mnt/virtualy/virtualy     options vers=3     content images,iso,vztmpl,rootdir,backup     maxfiles 2  dir: local     path /var/lib/vz     content images,iso,vztmpl,rootdir     maxfiles 0  nfs: indian_virtualy     path /mnt/pve/indian_virtualy     server 192.168.130.18     export /fast/virtualy     options vers=3     content images,rootdir     maxfiles 1  nfs: indian_zalohy     path /mnt/pve/indian_zalohy     server 192.168.130.18     export /zalohy/virtualy     options vers=3     content iso     maxfiles 1

Code:

 cat /var/log/pve/tasks/active UPID:cluster3:000C8F5C:1D5B78C4:532AD907:vncproxy:122:root@pam: 1 532AD911 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1 UPID:cluster3:000C8DB4:1D59A52E:532AD45A:vncproxy:122:root@pam: 1 532AD464 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1 UPID:cluster3:000B6131:1CDDBE47:5329972A:vzdump::root@pam: 1 53299799 OK UPID:cluster3:000B28A1:1CCC58E6:53296AA1:vzdump::root@pam: 1 53296B0C OK UPID:cluster3:0009CFFE:1C617A0C:5328590A:vncproxy:131:root@pam: 1 532864AA OK UPID:cluster3:0009CFE4:1C617276:532858F7:vncproxy:131:root@pam: 1 53285901 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1 UPID:cluster3:0009CE01:1C60E072:53285781:vncproxy:131:root@pam: 1 532857B0 OK UPID:cluster3:0009CDE8:1C60D9CD:53285770:vncproxy:131:root@pam: 1 5328577E command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009CD7F:1C60B6B3:53285716:vncproxy:131:root@pam: 1 53285752 OK UPID:cluster3:0009C40D:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FD3 OK UPID:cluster3:0009C40C:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FA4 OK UPID:cluster3:0009C3C3:1C5DBA2D:53284F71:vncproxy:131:root@pam: 1 53284FA1 OK UPID:cluster3:0009C3B1:1C5DB3D0:53284F61:vncproxy:131:root@pam: 1 53284F6D command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009C2FE:1C5D8792:53284EF0:vncproxy:131:root@pam: 1 53284F3B OK UPID:cluster3:0009C2C0:1C5D7448:53284EBE:vncproxy:131:root@pam: 1 53284EED OK UPID:cluster3:0009C2B2:1C5D6E09:53284EAE:vncproxy:131:root@pam: 1 53284EBA command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2 UPID:cluster3:0009C266:1C5D582A:53284E76:vncproxy:131:root@pam: 1 53284E80 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1 UPID:cluster3:0009C1D2:1C5D28CB:53284DFD:vncproxy:131:root@pam: 1 53284E4D OK UPID:cluster3:0009C17D:1C5D0E32:53284DB9:vncproxy:131:root@pam: 1 53284DC4 OK UPID:cluster3:0009C105:1C5CEDB1:53284D66:vncproxy:131:root@pam: 1 53284DB1 OK UPID:cluster3:0009BFDE:1C5C944F:53284C81:vncproxy:131:root@pam: 1 53284CB2 OK UPID:cluster3:0009A6EE:1C54EF67:532838EF:vncproxy:131:root@pam: 1 532843EE OK UPID:cluster3:00099E2C:1C523B88:53283204:vncproxy:131:root@pam: 1 5328386A OK UPID:cluster3:00099A82:1C5119FF:53282F1F:vncproxy:131:root@pam: 1 53282F52 OK UPID:cluster3:000996C2:1C4FF936:53282C3C:vncproxy:131:root@pam: 1 53282C85 OK

Code:

 cat /proc/loadavg 8.29 8.31 8.32 4/681 940169

liska_ · Mar 25, 2014

Hi,
I am now on different computer with same Linux Mint and Firefox and text pasting is working as expected, so I post outputs again.

clustat
Cluster Status for cluster @ Tue Mar 25 09:21:42 2014
Member Status: Quorate

Member Name ID Status
------ ---- ---- ------
cluster3 1 Online
cluster 2 Online, Local

pvecm nodes
Node Sts Inc Joined Name
1 M 356 2014-02-18 16:06:03 cluster3
2 M 320 2014-02-18 16:05:35 cluster

I had another nfs server mounted used only for backup, when it failed. I had to unmount it and delete from storage.cfg manually.

df -H
Filesystem Size Used Avail Use% Mounted on
udev 11M 0 11M 0% /dev
tmpfs 5.1G 467k 5.1G 1% /run
/dev/mapper/pve-root 15G 1.9G 12G 14% /
tmpfs 5.3M 0 5.3M 0% /run/lock
tmpfs 11G 49M 11G 1% /run/shm
/dev/mapper/pve-data 30G 1.9G 28G 7% /var/lib/vz
/dev/sda1 519M 66M 427M 14% /boot
192.168.130.14:/mnt/virtualy/virtualy 4.3T 4.2T 178G 96% /mnt/pve/skladka_virtualy
/dev/fuse 32M 33k 32M 1% /etc/pve
192.168.130.18:/fast/virtualy 968G 310G 659G 32% /mnt/pve/indian_virtualy
192.168.130.18:/zalohy/virtualy 5.0T 4.4T 580G 89% /mnt/pve/indian_zalohy

cat /var/log/pve/tasks/active
UPID:cluster3:000C8F5C:1D5B78C4:532AD907:vncproxy:122:root@pam: 1 532AD911 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1
UPID:cluster3:000C8DB4:1D59A52E:532AD45A:vncproxy:122:root@pam: 1 532AD464 command '/bin/nc -l -p 5900 -w 10 -c '/usr/sbin/qm vncproxy 122 2>/dev/null'' failed: exit code 1
UPID:cluster3:000B6131:1CDDBE47:5329972A:vzdump::root@pam: 1 53299799 OK
UPID:cluster3:000B28A1:1CCC58E6:53296AA1:vzdump::root@pam: 1 53296B0C OK
UPID:cluster3:0009CFFE:1C617A0C:5328590A:vncproxy:131:root@pam: 1 532864AA OK
UPID:cluster3:0009CFE4:1C617276:532858F7:vncproxy:131:root@pam: 1 53285901 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1
UPID:cluster3:0009CE01:1C60E072:53285781:vncproxy:131:root@pam: 1 532857B0 OK
UPID:cluster3:0009CDE8:1C60D9CD:53285770:vncproxy:131:root@pam: 1 5328577E command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2
UPID:cluster3:0009CD7F:1C60B6B3:53285716:vncproxy:131:root@pam: 1 53285752 OK
UPID:cluster3:0009C40D:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FD3 OK
UPID:cluster3:0009C40C:1C5DCCC6:53284FA1:vncproxy:131:root@pam: 1 53284FA4 OK
UPID:cluster3:0009C3C3:1C5DBA2D:53284F71:vncproxy:131:root@pam: 1 53284FA1 OK
UPID:cluster3:0009C3B1:1C5DB3D0:53284F61:vncproxy:131:root@pam: 1 53284F6D command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2
UPID:cluster3:0009C2FE:1C5D8792:53284EF0:vncproxy:131:root@pam: 1 53284F3B OK
UPID:cluster3:0009C2C0:1C5D7448:53284EBE:vncproxy:131:root@pam: 1 53284EED OK
UPID:cluster3:0009C2B2:1C5D6E09:53284EAE:vncproxy:131:root@pam: 1 53284EBA command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 2
UPID:cluster3:0009C266:1C5D582A:53284E76:vncproxy:131:root@pam: 1 53284E80 command '/bin/nc -l -p 5900 -w 10 -c '/usr/bin/ssh -T -o BatchMode=yes 147.231.80.190 /usr/sbin/qm vncproxy 131 2>/dev/null'' failed: exit code 1
UPID:cluster3:0009C1D2:1C5D28CB:53284DFD:vncproxy:131:root@pam: 1 53284E4D OK
UPID:cluster3:0009C17D:1C5D0E32:53284DB9:vncproxy:131:root@pam: 1 53284DC4 OK
UPID:cluster3:0009C105:1C5CEDB1:53284D66:vncproxy:131:root@pam: 1 53284DB1 OK
UPID:cluster3:0009BFDE:1C5C944F:53284C81:vncproxy:131:root@pam: 1 53284CB2 OK
UPID:cluster3:0009A6EE:1C54EF67:532838EF:vncproxy:131:root@pam: 1 532843EE OK
UPID:cluster3:00099E2C:1C523B88:53283204:vncproxy:131:root@pam: 1 5328386A OK
UPID:cluster3:00099A82:1C5119FF:53282F1F:vncproxy:131:root@pam: 1 53282F52 OK
UPID:cluster3:000996C2:1C4FF936:53282C3C:vncproxy:131:root@pam: 1 53282C85 OK

cat /proc/loadavg
8.11 8.23 8.44 2/677 970407

And in syslog there is just message
Mar 25 09:25:27 cluster3 pveproxy[966474]: WARNING: proxy detected vanished client connection
and this message keeps repeating every few minutes without any other attempt for login on both servers. There is no difference if I fill in correct credentials or not. It just wait for 30secs and then print message unable to login. I checked again time and it is synced correctly. It all started with long running backup of openvz container to nfs storage.

In pveproxy/access.log I can found strange logs during backup on one node

- - [25/Mar/2014:09:41:34 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=cpu&timeframe=hour&cf=AVERAGE&_dc=471 HTTP/1.1" 401 -
- - [25/Mar/2014:09:41:35 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=mem,maxmem&timeframe=hour&cf=AVERAGE&_dc=471 HTTP/1.1" 401 -
- - [25/Mar/2014:09:41:35 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=netin,netout&timeframe=hour&cf=AVERAGE&_dc=471 HTTP/1.1" 401 -
- - [25/Mar/2014:09:41:35 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=diskread,diskwrite&timeframe=hour&cf=AVERAGE&_dc=471 HTTP/1.1" 401 -
- - [25/Mar/2014:09:42:04 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=cpu&timeframe=hour&cf=AVERAGE&_dc=472 HTTP/1.1" 401 -
- - [25/Mar/2014:09:42:05 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=mem,maxmem&timeframe=hour&cf=AVERAGE&_dc=472 HTTP/1.1" 401 -
- - [25/Mar/2014:09:42:05 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=diskread,diskwrite&timeframe=hour&cf=AVERAGE&_dc=472 HTTP/1.1" 401 -
- - [25/Mar/2014:09:42:05 +0100] "GET /api2/png/nodes/cluster/qemu/131/rrd?ds=netin,netout&timeframe=hour&cf=AVERAGE&_dc=472 HTTP/1.1" 401 -

and when I try to login on second server I can see only
- - [25/Mar/2014:09:45:38 +0100] "POST /api2/extjs/access/ticket HTTP/1.1" 500 -

udo · Mar 25, 2014

liska_ said:
...
cat /proc/loadavg
8.11 8.23 8.44 2/677 970407
...
It all started with long running backup of openvz container to nfs storage.

Hi,
is the backup still running?
Why you have an load of 8? If there trouble with IO-Devices the load is normaly much higher, but 8 is also not few...

I guess, you can login via ssh?

If I login via gui I got such an output:

Code:

tail /var/log/syslog
...
Mar 25 10:14:21 pve1 pvedaemon[2616]: <root@pam> successful auth for user 'root@pam'

Udo

liska_ · Mar 25, 2014

No, backup process is not running anymore as I killed it manually. I hoped it would resolve my issue.
I have a load more than eight on both servers, top shows some kvm processes on the top of the list. But I am not sure if this has changed since that backup failure.

Yes, login via ssh is furtunatelly still working. But login of user created via gui stopped working as well.

liska_ · Mar 25, 2014

I restarted one node and I can now login to this one. But it can not get details of vms running on the second node or details about storages. It ends up with Communication failure. Restarting pveproxy did not help.
I tried to add again that failed nfs store but the servers did not mount it but it was present in storage.cfg. No interesting messages in logs. I remove it again. Once I got this problems when I had a failed nfs server but now I can no t find any clue of stalled storage connection.

liska_ · Mar 25, 2014

This problem was solved by hard reseting remote nfs server. So maybe the solution next time would be restarting pvestatd to get rid of the "broken" nfs server?

This is what I found in syslog:

Mar 25 13:46:19 cluster3 pmxcfs[181847]: [status] notice: received log
Mar 25 13:46:38 cluster3 pveproxy[981491]: WARNING: proxy detected vanished client connection
Mar 25 13:46:49 cluster3 kernel: ct0 nfs: server 192.168.130.45 OK
Mar 25 13:46:49 cluster3 kernel: ct0 nfs: server 192.168.130.45 OK
Mar 25 13:46:49 cluster3 kernel: ct0 nfs: server 192.168.130.45 OK
Mar 25 13:46:49 cluster3 kernel: ct0 nfs: server 192.168.130.45 OK
Mar 25 13:46:49 cluster3 kernel: ct0 nfs: server 192.168.130.45 OK
Mar 25 13:46:49 cluster3 pvedaemon[730143]: WARNING: unable to activate storage 'sun_data' - directory '/mnt/pve/sun_data' does no
t exist
Mar 25 13:46:49 cluster3 pvedaemon[820624]: WARNING: unable to activate storage 'sun_data' - directory '/mnt/pve/sun_data' does no
t exist
Mar 25 13:46:49 cluster3 pvedaemon[654492]: WARNING: unable to activate storage 'sun_data' - directory '/mnt/pve/sun_data' does no
t exist
Mar 25 13:46:49 cluster3 pvestatd[3586]: status update time (464268.966 seconds)

Search

Search

unable to join via gui - login failed

liska_

Member

udo

Distinguished Member

liska_

Member

liska_

Member

m.ardito

Famous Member

dietmar

Proxmox Staff Member

liska_

Member

liska_

Member

udo

Distinguished Member

liska_

Member

liska_

Member

liska_

Member

We value your privacy