"Login Failed"

[sigma]

Hi ! I need help. My Proxmox web interface stopped recognizing me with a "LOGIN FAILED" message.

I have a single computer, running proxmox latest version pre 4.0 with only two virtual machines inside.

It began with the following symptom: Some hours ago my two virtual machines started appearing "off" (monitors in black) on the web interface (but they were working just fine). All performance graphs had nothing being graphed on them.

Some minutes later, the whole web interface refreshed into the PROXMOX VE LOGIN window.

From that moment on, my password did not log me into Proxmox web interface anymore. But I can log as root with no problems at all on SSH shell.

I had no firewall ON, did not install anything prior to this behavior, nothing it has been working fine for years, and suddenly this problem. I am totally baffled about this.

Any suggestion or idea is very much welcome. This is a production server, and since the virtual machines are working just fine, I had no chance yet to perform a reset on the barebone server that runs proxmox.

pveversion -v returns the following:

proxmox-ve-2.6.32: 3.4-165 (running kernel: 2.6.32-42-pve)
pve-manager: 3.4-11 (running version: 3.4-11/6502936f)
pve-kernel-2.6.32-40-pve: 2.6.32-160
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-41-pve: 2.6.32-164
pve-kernel-2.6.32-42-pve: 2.6.32-165
pve-kernel-2.6.32-38-pve: 2.6.32-155
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-19
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

Regards,

Enrique

 

[wolfgang]

Hi,
I'm a little confused about your settings you wrote you have PVE4.0 but your pveversion -v tell us you run 3.4?

try to flush the browser cache.
restart pveproxy

 

[sigma]

Hi Wolfgang,

I am sorry I was not clear enough: As I stated, I have the latest proxmox version PRE 4.0 (before 4.0). The first thing I did when this started happening to me was getting into this forum and searching for similar cases (before posting anything). So I already did a flush of the browser cache, and even tried different browsers. Also tried to restart pveproxy.

The behavior is still the same: I am left out of the web interface! I am mildly panicking, since I am not that proficient on linux administration, as to manage (backup, check usage, etc.) my proxmox through command line.

Regards,

Enrique.

 

[wolfgang]

Do you have OpenVz container to?

What output you get from

cat /etc/hosts
cat /etc/network/interfaces ifconfig
service pveproxy status

 

[sigma]

Wolfgang,

I am not using OpenVz as VM engine.

root@proxmox:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
X.X.X.194 proxmox.avalle.net proxmox pvelocalhost


# The following lines are desirable for IPv6 capable hosts


ip6-localhost ip6-loopback
ip6-localnet
ip6-mcastprefix
ip6-allnodes
ip6-allrouters
ip6-allhosts




root@proxmox:~# cat /etc/network/interfaces ifconfig
auto lo
iface lo inet loopback


auto vmbr0
iface vmbr0 inet static
address X.X.X.194
netmask 255.255.255.224
gateway X.X.X.193
bridge_ports eth0
bridge_stp off
bridge_fd 0
cat: ifconfig: No such file or directory




root@proxmox:~# service pveproxy status
Usage: /etc/init.d/pveproxy {start|stop|restart|reload|force-reload}


Since that last command had no status parameter, but I could see the available parameters, I did just try the following:


root@proxmox:~# /etc/init.d/pveproxy force-reload
Restarting PVE API Proxy Server: pveproxy.

And tried again to log into the administrator web interface, but I keep getting the same: Login Failed. Please try again.

Thanks very much Wolfgang for your help. Hopefully this walk-through looking for a solution can be useful for other persons.

Enrique

 

[wolfgang]

Config looks ok.
Do you have as Login Realm "Linux PAM" or PVE?
PAM is the correct one where root works.

 

[sigma]

Wolfgang,

Yes, I supposed too it would look ok, as I never changed anything. This happened with no apparent reason. It worked fine for years, until now.

I am using, yes Linux PAM (although I did try PVE Authentication, just in case) as log in realm.

I just paid an anual subscription for Proxmox (I could afford only the community subscription). I know it is a minimal gesture, and it was long due for me to at least jump into it.

I understand that the community subscription does not entitle me for real ticket support, or anything like that.

Anyway, if there is any other suggestion or test for me to try, I am closely reading this thread, for the next hours.

Regards,

Enrique

 

[wolfgang]

try to reinstall the manager and change the password

passwd root

apt-get update && apt-get install pve-manager

 

[sigma]

Wolfgang,

I have a (great) partial success:

I had to issue the following commands:

apt-get update
apt-get remove pve-manager
apt-get install pve-manager

I regained access to web interface. But, virtual machines appear just as right before loosing access yesterday: "Powered off". Also all server graphs have nothing plotted on them (but VM are working fine).



... At this time I am kind of tempted of backing up my two VM and try to do Proxmox 4 upgrade.

Regards,

Enrique

 

[wolfgang]

restart the service pve-cluster and pvestatd.

 

[sigma]

I have everything working now. I stopped my two virtual machines and issued a reboot command on the proxmox ssh linux shell.

Thank you very much for your directions, Wolfgang!

Right now I am backing up both VM, just in case, and tomorrow I will try to upgrade proxmox into version 4.0

Regards,

Enrique.

 

[sigma]

Sorry to "reopoen" this thread: The problem depicted in here kept happening to me. Now I am up to date with Proxmox version 4. Always the same:

After rebooting proxmox server and everything working ok for several days, first I wake up into a web interface depicting my virtual machines powered off, no cpu / bandwitdth graphs active, but the "Status" panel (the upper panel showing the Uptime, load, cpu,io, ram, swap usage, etc. in text mode) is still being refreshed and showing what seems to be the correct values.

If at this time, I logoff from the web interface, then there is no way to log in again, as stated on my first post.

This is haunting me big time, and I would rather NOT reset my server only to regain proper web interface usage!!

Just in case:


pveversion -v

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "es_UY.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("es_UY.UTF-8").

proxmox-ve: not correctly installed (running kernel: 4.2.3-2-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-2.6.32-43-pve: 2.6.32-166
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-30
qemu-server: 4.0-46
pve-firmware: 1.1-7
libpve-common-perl: 4.0-43
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-21
pve-container: 1.0-37
pve-firewall: 2.0-15
pve-ha-manager: 1.0-18
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1

I don't know if it is related, but when inspecting the Syslog, around the time I suspect everything goes berzerk, I find a flood of lines like these:


Feb 06 10:31:02 proxmox sshd[26311]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:03 proxmox sshd[26313]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:04 proxmox sshd[26315]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:05 proxmox sshd[26317]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:06 proxmox sshd[26324]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:07 proxmox sshd[26326]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:08 proxmox sshd[26328]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:09 proxmox sshd[26330]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:10 proxmox sshd[26334]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:11 proxmox sshd[26336]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:12 proxmox sshd[26338]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:13 proxmox sshd[26341]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:14 proxmox sshd[26343]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]
Feb 06 10:31:15 proxmox sshd[26345]: fatal: no matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com [preauth]


Any help or hint is appreciated!

Regards,

Sigma!

 

[sigma]

I am trying to restart pvestatd which refuses to start:

More info:


systemctl status pvestatd.service

● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled)
Active: failed (Result: timeout) since Wed 2016-02-10 10:28:28 UYT; 25min ago
Process: 2234 ExecStop=/usr/bin/pvestatd stop (code=exited, status=0/SUCCESS)
Process: 3819 ExecStart=/usr/bin/pvestatd start (code=exited, status=255)
Main PID: 1096
Feb 10 10:25:27 proxmox pvestatd[3819]: start failed - can't aquire lock '/var/run/pvestatd.pid.lock' - Recurso no disponible temporalmente
Feb 10 10:25:27 proxmox pvestatd[3819]: start failed - can't aquire lock '/var/run/pvestatd.pid.lock' - Recurso no disponible temporalmente
Feb 10 10:25:27 proxmox systemd[1]: pvestatd.service: control process exited, code=exited status=255
Feb 10 10:26:58 proxmox systemd[1]: pvestatd.service stop-final-sigterm timed out. Killing.
Feb 10 10:28:28 proxmox systemd[1]: pvestatd.service still around after final SIGKILL. Entering failed mode.
Feb 10 10:28:28 proxmox systemd[1]: Failed to start PVE Status Daemon.
Feb 10 10:28:28 proxmox systemd[1]: Unit pvestatd.service entered failed state.

journalctl -xn

-- Logs begin at Sat 2016-01-30 09:24:03 UYT, end at Wed 2016-02-10 10:22:15 UYT. --
Feb 10 10:18:41 proxmox pveproxy[2089]: proxy detected vanished client connection
Feb 10 10:19:10 proxmox systemd[1]: Cannot add dependency job for unit watchdog-mux.socket, ignoring: Unit watchdog-mux.socket failed to load: No
Feb 10 10:19:15 proxmox pvestatd[3542]: start failed - can't aquire lock '/var/run/pvestatd.pid.lock' - Recurso no disponible temporalmente
Feb 10 10:19:15 proxmox pvestatd[3542]: start failed - can't aquire lock '/var/run/pvestatd.pid.lock' - Recurso no disponible temporalmente
Feb 10 10:19:15 proxmox systemd[1]: pvestatd.service: control process exited, code=exited status=255
Feb 10 10:20:45 proxmox systemd[1]: pvestatd.service stop-final-sigterm timed out. Killing.
Feb 10 10:21:57 proxmox systemd-timesyncd[10447]: interval/delta/delay/jitter/drift 2048s/+0.031s/0.084s/0.017s/-14ppm (ignored)
Feb 10 10:22:15 proxmox systemd[1]: pvestatd.service still around after final SIGKILL. Entering failed mode.
Feb 10 10:22:15 proxmox systemd[1]: Failed to start PVE Status Daemon.
-- Subject: Unit pvestatd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit pvestatd.service has failed.
--
-- The result is failed.

 

[Daniel S]

Hi.

Have you checked that the disk isn't on readonly mode?
It looks like that could be the problem ('/var/run/pvestatd.pid.lock' - Recurso no disponible)

Coud you write somthing on the disk? (touch /var/example.txt)
It could be that you have a disk problem (bad sectors) and the system disk was remounted in read only mode. With "dmesg" you should see it.

 

[sigma]

Hi Daniel, Thanks for the concern, nope it is in RW state:


root@proxmox:/# touch /var/example.txt

root@proxmox:/# ls -al /var/ex*

-rw-r--r-- 1 root root 0 Feb 10 11:48 /var/example.txt

 

[sigma]

I managed to restart pvestatd daemon, by deleting the pvestatd.pid.lock file. But now proxmox web interface got into it's "next" phase: I am left out of the web interface, my root password no longer recognized.

On the other hand, I can open an SSH shell with the same root / password combination just ok.

 

[sigma]

I managed to regain access to web interface and show all virtual machines in correct state, after issueing the following commands:

/etc/init.d/pve-cluster restart
/etc/init.d/pvestatd restart

At this point, the pvestatd restart gave an error about pvestatd.pid.lock being present.

I had to delete the file:
rm /var/run/pvestatd.pid.lock

And re-issue the:

/etc/init.d/pvestatd restart

And finally:

apt-get remove pve-manager
apt-get install pve-manager

 

[Sand]

I'm ocd asf an having the same issues

 

[Ivan Gonzalez]

Im now having this issue, running 4.4

 

[silvestrefigueroa]

same problem here!

pve-manager/5.1-41/0b958203 (running kernel: 4.13.13-2-pve)

at fresh install!!!

without any update....

suggestion?