[SOLVED] Unable to Login to Proxmox - Authentication Fails

[juspriss]

Hello,

I am unable to log in to my Proxmox VE 8.x server. Until yesterday, I could access it via the Web UI without any issues. However, today, authentication fails both via Web UI and directly through the server shell.

I checked the following log files using a live USB drive:
- /var/log/auth.log
- /var/log/pveproxy/access.log
- /var/log/syslog
- /var/log/dpkg.log

I only see my successful logins from previous days and today's failed attempts. There are no other login attempts recorded.

I also tried booting into single-user and recovery modes, but it still prompts for a password. Even with the correct password, login fails. Additionally, I checked via the live USB if the root password had been changed recently using chage, but it hasn't been. The VMs on the server seem to be running.

Any advice or guidance on how to resolve this issue would be greatly appreciated.

Thank you!

 

[bbgeek17]

- /var/log/syslog

you should be checking "journalctl" in PVE8, ie 'journalctl -f" and then try to login.

Any advice or guidance on how to resolve this issue would be greatly appreciated.

Do you have free space on your root disk?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[juspriss]

you should be checking "journalctl" in PVE8, ie 'journalctl -f" and then try to login.

I attempted to boot using a live USB and inspected the system.journal via journalctl, but my attempts were unsuccessful. Unfortunately, I didn't find anything else of interest in the logs.

I also managed to boot into a root shell by editing the Grub boot options and adding init=/bin/bash. However, when attempting to use systemctl -f, I encountered the error message: System has not been booted with systemd as init system (PID1). Can't operate.

Do you have free space on your root disk?

Yes, it was quite full, but I freed up some space using the live system. Now, the root partition is 11% free, which is approximately 1.5GB.

Now that it seems that I am in a root shell, I tried to change the password via passwd, but I received the error authentication token manipulation error, password unchanged

 

[juspriss]

Now that it seems that I am in a root shell, I tried to change the password via passwd, but I received the error authentication token manipulation error, password unchanged

Silly me, I was trying to change the password on a read-only mount. After remounting the partition as read-write, I was able to successfully update the password. Now, I can log in from both the shell and the Web UI.

However, I'm unsure if I should mark this issue as "resolved" since I'm still uncertain about the root cause. Do you have any suggestions?

 

[leesteken]

However, I'm unsure if I should mark this issue as "resolved" since I'm still uncertain about the root cause. Do you have any suggestions?

Maybe disk corruption? Or maybe you run some software that locks the account? Did you check the user line in /etc/shadow before changing the password?

 

[juspriss]

Maybe disk corruption? Or maybe you run some software that locks the account? Did you check the user line in /etc/shadow before changing the password?

I haven't used any different software than usual. During my last sessions, I was only checking the temperatures. Nevertheless, I will double-check for disk corruption. Thank you.

Yes, I did check the user line in /etc/shadow, which contained the hashes of the encrypted passwords, so nothing unusual there. However, I cannot guarantee the correctness of the hash.

 

[leesteken]

That does not sound like corruption (and how will you check?). Did you really need to change the password or was making some free space maybe enough?

 

[juspriss]

That does not sound like corruption (and how will you check?). Did you really need to change the password or was making some free space maybe enough?

I ran the long test with smartctl, and the results indicate that the disk is fine. I had to change the password because freeing up some space did not resolve the password issue. Do you have any particular ideas regarding what might have caused the problem?

 

[leesteken]

I ran the long test with smartctl, and the results indicate that the disk is fine. I had to change the password because freeing up some space did not resolve the password issue. Do you have any particular ideas regarding what might have caused the problem?

Only file corruption (for which you cannot really test) or someone else changed the password (and will have had the permissions to remove logging that would show this). Or you misremembering the password or forgetting that you changed it (check for passwd in .bash_history). Or you installed Proxmox twice on different drives and the system booted now from the other (due to BIOS settings wiped or changed). If you did not do it, then maybe someone else did it (over the network) and you should wipe and reinstall (or investigate login attempts). Without more information and because I know nothing about you or your setup or network, it's impossible to tell what is more likely (and what other possibilities there might be).
EDIT: Or maybe you simply locked the root user, like Ubuntu is setup by default, after adding a new user?
EDIT2: If you did do it yourself but don't remember then check for elevated levels of carbon-monoxide (CO) in the house.

 

[juspriss]

Only file corruption (for which you cannot really test) or someone else changed the password (and will have had the permissions to remove logging that would show this). Or you misremembering the password or forgetting that you changed it (check for passwd in .bash_history). Or you installed Proxmox twice on different drives and the system booted now from the other (due to BIOS settings wiped or changed). If you did not do it, then maybe someone else did it (over the network) and you should wipe and reinstall (or investigate login attempts). Without more information and because I know nothing about you or your setup or network, it's impossible to tell what is more likely (and what other possibilities there might be).
EDIT: Or maybe you simply locked the root user, like Ubuntu is setup by default, after adding a new user?
EDIT2: If you did do it yourself but don't remember then check for elevated levels of carbon-monoxide (CO) in the house.

Thank you for your detailed response. Here are my thoughts and actions based on your suggestions:

File Corruption: I ran smartctl, chkrootkit, and rkhunter scans, and all indicated no apparent disk issues or rootkits. While I know these aren't definitive, they should provide some level of assurance.

Password Change: I do not remember that I changed the password, but I verified my .bash_history for any passwd commands and didn't find anything unusual.

Proxmox Installations on Different Drives: This is a single mounted drive setup. I double-checked the BIOS settings, and there haven't been any changes or resets. The system is booting from the correct drive.

Root User Lock: The root account wasn't locked and I did not add a new user.

External Access: I checked the logs (auth.log, syslog, journalctl, etc.) and did not find any unusual login attempts or activities.

Carbon Monoxide: Thankfully, no elevated levels of CO in the house!

While the root password issue is resolved after resetting it, I still don't have a clear understanding of what caused it. I believe I will wipe the disk. I wanted to change the disk to migrate from SSD SATA to NVMe, and this event might be an excellent reason to speed up the process.

If you have any further suggestions or ideas on what else I could investigate, please let me know. Your help is greatly appreciated!

Thanks again for your assistance!

 

[bbgeek17]

Is there nothing in the log, from the time when it was broken, about failed login attempts?
Did you save /etc/shadow before changing it? Comparing the hash could have been interesting.

Next time, if you suspect any foul play and/or need to get to the bottom of things, duplicate the disk at the time of the failure, so you can go back for analyses or reproduction.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[juspriss]

Is there nothing in the log, from the time when it was broken, about failed login attempts?

Unfortunately, the only failed log in attempts are mines.

Did you save /etc/shadow before changing it? Comparing the hash could have been interesting.

Next time, if you suspect any foul play and/or need to get to the bottom of things, duplicate the disk at the time of the failure, so you can go back for analyses or reproduction.

Good luck

I did not save it, silly me!

Next time, I will make sure to back things up before any modifications and keep a copy for analysis and reproduction. This time, I did some analysis but didn't think about making a copy beforehand.

Thanks!

 

[Hannibal1987]

Today i am facing a simular problem (v1:0:18:4).
I set up a HA Cluster with 3 machines, everything is up and running well. I whatn to login into Proxmox WebUI or via SHH and my passwort (for sure the right one) is not letting me in. I had a simular problem a year ago, this was caused because of one node of the 3 node HA cluster was offline. This is a strange and known problem of proxmox HA Cluster, but today not the issue. After this i noted the password to be sure it is correct for the next unexpected login issue, and here we go. It makes me affraid again, that i am loosing the control of my machines without any reason. Today i am not onside and not able to do anything, tomorrow i will sit in front of the device and need to figure out the reason.

Any ideas in advance?

Thank you

 

[leesteken]

Today i am facing a simular problem (v1:0:18:4).

I don't understand that number, sorry.

I whatn to login into Proxmox WebUI or via SHH and my passwort (for sure the right one) is not letting me in.

Does it tell you the password or username is incorrect or does it just not log you in. The first could be cause by someone else (if the system is reachable via the internet). The latter can be cause by lack of free memory or some kinds of hardware or disk (read-only filesystem) or network failures.
Did you setup the system to email you before logging you in? Maybe it cannot send the email and won't log you in, for example.