I am having MAJOR problems logging in

[uscpsycho]

I just installed Proxmox and I'm having a hell of a time logging in.

For the first day or so, I was able to login, but it would take 5-10 attempts before it worked. This is true for both local login and remote login. Yes, of course I am 100% sure I am correctly entering my password every time. It just keeps rejecting my password for some reason. I tried a fresh install of Proxmox and I had the same problem.

At the moment I am logged in remotely but cannot login locally no many how many times I try. I have tried over 20 times at this point and I can't get in. The only reason I haven't tried to restart the box is because I do have a working remote connection right now and I'm afraid I won't be able to login at all if I lose that connection.

If there is no way to fix this then this is not a usable product for me. I can't do any real work in these VM's if I have to worry about not being able to access them. This is crazy.

Any idea what is going on and how to fix it?

 

[bbgeek17]

Is you Proxmox directly exposed to internet and therefore being credential attacked to a point where it starts rejecting authentications?
If you have an active sessions - examine your logs. Run "journalctl -f" and then login from another console, anything in the log?
Anything in the "last" output? Anything in "dmesg -w" ? "auth.log" ?
Do you, perhaps, have bad keys on your keyboard? Type your password in clear text ten times, does that look ok?

Keep in mind that PVE is a set of packages on top of a Debian Linux. The login functionality is the same as on millions of servers running around the world. Its extremely more likely that the problem is localized to your environment, than that it is something wide spread...

good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[wbk]

Any idea what is going on and how to fix it?

Without logs or description of _how_ you log in: not really.

Do you log in over SSH or via the web interface? Local, as in, with a keyboard connected to the machine?

• Via SSH: fail2ban can lock you out if some process on your machine tries (and fails) to connect to often
• Via the webinterface: pay attention to the realm that you chose for logging in, see the image

 

[uscpsycho]

Is you Proxmox directly exposed to internet and therefore being credential attacked to a point where it starts rejecting authentications?
If you have an active sessions - examine your logs. Run "journalctl -f" and then login from another console, anything in the log?
Anything in the "last" output? Anything in "dmesg -w" ? "auth.log" ?
Do you, perhaps, have bad keys on your keyboard? Type your password in clear text ten times, does that look ok?

Keep in mind that PVE is a set of packages on top of a Debian Linux. The login functionality is the same as on millions of servers running around the world. Its extremely more likely that the problem is localized to your environment, than that it is something wide spread...

good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I tried to get help on Reddit yesterday and did something like this, although the person suggesting this wasn't able to help me resolve it.

He asked me to run " journalctl --since=yesterday" and I got thousands and thousands of lines in the result. I stopped scrolling after 5,000 lines because it just seemed endless. He said that isn't normal but then offered no insights into what might be wrong. Perhaps this is my Proxmox being attacked, but I have no idea why it would be exposed. Promox is behind a Ubiquiti router that still has all of the default security settings. That should be enough keep it from this kind of attack. I was instantly having problems as soon as I installed it. I had trouble logging in from the first minute I installed it. If my Proxmox is exposed it was discovered instantly.

He also suggested running dmesg without any options. This was the output: https://pastebin.com/bSvt5qb4

One thing I did at installation is run a script that was recommended by this installation guide I followed. The reason for the script: "First, we need to update Proxmox with the latest packages. Please note you MUST run the post-configuration script before you do any Proxmox updates or deploy HAOS. If not, you will likely see 401 errors with the enterprise repositories since you (likely) don’t have a Proxmox paid license."

Proxmox VE Post Install​

This script provides options for managing Proxmox VE repositories, including disabling the Enterprise Repo, adding or correcting PVE sources, enabling the No-Subscription Repo, adding the test Repo, disabling the subscription nag, updating Proxmox VE, and rebooting the system.

Run the command below in the Proxmox VE Shell.

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/misc/post-pve-install.sh)"

It is recommended to answer “yes” (y) to all options presented during the process.

The script is here: https://raw.githubusercontent.com/tteck/Proxmox/main/misc/post-pve-install.sh

Could this script have done anything to expose me?

Does any of this help in identifying my login problem?

 

[uscpsycho]

Anything in the "last" output? Anything in "dmesg -w" ? "auth.log" ?
Do you, perhaps, have bad keys on your keyboard? Type your password in clear text ten times, does that look ok?

Forgot to reply to a couple of things.

No, the keyboard is not the issue. I am having the same problem with two different keyboards and the keyboards don't have any issues outside of Promox.

Another person yesterday asked me if I had anything weird in /var/log/auth.log.

I'm not super linux savvy but I didn't see an auth.log file in /var/log. Maybe I looked in the wrong place but the only log files I saw there are fontconfig.log, pve-firewall.log, dpkg.log. Am I looking in the wrong place?

 

[Dunuin]

Whats the CPU load, IO delay and is there any storage not available? Your PVE could just be busy and login requests will time out.
Even stuff like an unavailable PBS storage could cause login fails: https://forum.proxmox.com/threads/unavailable-storage-will-make-the-webui-unusable.119107/

 

[uscpsycho]

Without logs or description of _how_ you log in: not really.

Do you log in over SSH or via the web interface? Local, as in, with a keyboard connected to the machine?

• Via SSH: fail2ban can lock you out if some process on your machine tries (and fails) to connect to often
• Via the webinterface: pay attention to the realm that you chose for logging in, see the image

View attachment 61632

When I say local I do mean with a keyboard connected directly to the machine via USB. When logging in remotely, the problem I have is via the web interface. I have logged in a few times via SSH and it always seems to work without any issues. I can't say for sure that the problem doesn't exist when logging in over SSH but that's how it seems with a small sample size.

I tried both realms but I have the same problem with both. You didn't say which one is the correct one, from your screenshot am I to infer that the Linux PAM realm is the one I should be using?

 

[uscpsycho]

Whats the CPU load, IO delay and is there any storage not available? Your PVE could just be busy and login requests will time out.
Even stuff like an unavailable PBS storage could cause login fails: https://forum.proxmox.com/threads/unavailable-storage-will-make-the-webui-unusable.119107/

I'm not sure how to check this from a prompt. But I can tell you that this problem started from the get go. It was basically bare Proxmox and bare Home Assistant (I haven't even created a Home Assistant account for it to do anything). Since then I've managed to install Windows in another VM but it's not doing anything either.

I doubt these are causing the problem but if you tell me how to check CPU load and IO delay I will. Storage definitely shouldn't be a problem, I've got a 500GB drive with fresh installs of everything.

 

[Dunuin]

Try top and pvesm status.

 

[bbgeek17]

He asked me to run " journalctl --since=yesterday" and I got thousands and thousands of lines in the result. I stopped scrolling after 5,000 lines because it just seemed endless. He said that isn't normal but then offered no insights into what might be wrong.

This information is not really useful without presenting at least 200 out of those 5000 lines.

Promox is behind a Ubiquiti router that still has all of the default security settings. That should be enough keep it from this kind of attack

Take the cable out of the PVE, wait a few min (up to 20), then see if you can reliably login via physical console.

One thing I did at installation is run a script that was recommended by this installation guide I followed

The source seems to be somewhat trustworthy, probably many people used it without issues. That said, executing script from internet blindly is never a good idea. On a quick glance the script does nothing that you shouldnt be able to do yourself by following official PVE guides.

I'm not super linux savvy but I didn't see an auth.log file in /var/log. Maybe I looked in the wrong place

I am not sure what happened but this file is present in every one of our multiple test systems...
ls -al /var/log/auth*

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[uscpsycho]

Try top and pvesm status.

I'm not a Linux expert but I don't think there are any resource issues. Here are my results for both.

 

[uscpsycho]

This information is not really useful without presenting at least 200 out of those 5000 lines.

Here are the first 250. I'm not even saying there were 5000 lines. It was probably way way more, I just got tired of scrolling endlessly after 5000 lines.

The source seems to be somewhat trustworthy, probably many people used it without issues. That said, executing script from internet blindly is never a good idea. On a quick glance the script does nothing that you shouldnt be able to do yourself by following official PVE guides.

I know, I didn't exactly follow it blindly. The source seems to be very trusted among HA users across different platforms. I wouldn't have used it if I wasn't confident it isn't shady.

I am not sure what happened but this file is present in every one of our multiple test systems...
ls -al /var/log/auth*

No dice.

root@pve1:~# ls -al /var/log/auth*
ls: cannot access '/var/log/auth*': No such file or directory

 

[bbgeek17]

No dice.

root@pve1:~# ls -al /var/log/auth*
ls: cannot access '/var/log/auth*': No such file or directory

It moved to journal in pve8.

Run "journalctl -f" and then login from another console, anything in the log?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Dunuin]

I'm not a Linux expert but I don't think there are any resource issues. Here are my results for both.

View attachment 61667

Yes, looks fine.

I wouldn't have used it if I wasn't confident it isn't shady.

It's disabling the "nag" popup...which is probably at least a bit shady from the point of view of Proxmox. The correct way would be to pay for a license if you are so annoyed by it that you can't spend that 0.5 seconds to close it.
Also not recommended as a hack removing it already caused problems...

 

[uscpsycho]

This information is not really useful without presenting at least 200 out of those 5000 lines.

Oops, I forgot to post the link to my log in my last reply. These are the first ~250 lines: https://pastebin.com/Df7dW3QF

One thing I've noticed is that the volume of log entries is very high at times. For instance all the lines above have the exact same timestamp to the second. In the log samples below there are only a few lines at a time. Not sure if this has any significance.

Take the cable out of the PVE, wait a few min (up to 20), then see if you can reliably login via physical console.

I assume this is something I should try when I am unable to login, correct?

Run "journalctl -f" and then login from another console, anything in the log?

First, here is a block I pulled from the log when I had several login failures: https://pastebin.com/vEjDC5ti

And here is a sample from when I successfully login:
Jan 18 17:02:02 pve1 pveproxy[1076]: worker 123658 started
Jan 18 17:02:02 pve1 pveproxy[1076]: starting 1 worker(s)
Jan 18 17:02:02 pve1 pveproxy[1076]: worker 117012 finished
Jan 18 17:02:02 pve1 pveproxy[117012]: worker exit
Jan 18 16:57:44 pve1 pveproxy[1076]: worker 123054 started
Jan 18 16:57:44 pve1 pveproxy[1076]: starting 1 worker(s)
Jan 18 16:57:44 pve1 pveproxy[1076]: worker 115743 finished
Jan 18 16:57:44 pve1 pveproxy[115743]: worker exit
Jan 18 16:56:08 pve1 pvedaemon[118964]: <root@pam> successful auth for user 'root@pam'
Jan 18 16:53:37 pve1 pvedaemon[122481]: starting vnc proxy UPIDve1:0001DE71:0045E3F8:65A9C811:vncproxy:101:root@pam:
Jan 18 16:53:37 pve1 pvedaemon[118528]: <root@pam> starting task UPIDve1:0001DE71:0045E3F8:65A9C811:vncproxy:101:root@pam:
Jan 18 16:53:37 pve1 pvedaemon[118214]: <root@pam> end task UPIDve1:0001D637:004483F9:65A9C48C:vncproxy:101:root@pam: OK
Jan 18 16:51:31 pve1 pvedaemon[118528]: <root@pam> successful auth for user 'root@pam'
Jan 18 16:51:09 pve1 systemd-logind[739]: Removed session 22.
Jan 18 16:51:09 pve1 systemd-logind[739]: Session 22 logged out. Waiting for processes to exit.
Jan 18 16:51:09 pve1 systemd[1]: session-22.scope: Deactivated successfully.
Jan 18 16:51:09 pve1 sshd[121966]: pam_unix(sshd:session): session closed for user root
Jan 18 16:50:19 pve1 sshd[121966]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 18 16:50:19 pve1 systemd[1]: Started session-22.scope - Session 22 of User root.
Jan 18 16:50:19 pve1 systemd-logind[739]: New session 22 of user root.
Jan 18 16:50:19 pve1 sshd[121966]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 18 16:50:19 pve1 sshd[121966]: Accepted password for root from 192.168.5.150 port 38692 ssh2

 

[bbgeek17]

One thing I've noticed is that the volume of log entries is very high at times. For instance all the lines above have the exact same timestamp to the second. In the log samples below there are only a few lines at a time. Not sure if this has any significance.

those lines you selected are from boot time, essentially initializing the hardware and are not pertinent to your stable run operations or any login issues.

May be you have a bad motherboard connection that is sending garbage to console and elsewhere... Clearly bad authentication is being sent and you are being throttled:

Jan 18 05:42:12 pve1 login[21635]: TOO MANY LOGIN TRIES (5) on '/dev/tty1' FOR 'UNKNOWN'
Jan 18 05:42:12 pve1 login[21635]: FAILED LOGIN (5) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan 18 05:42:09 pve1 login[21635]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jan 18 05:42:09 pve1 login[21635]: pam_unix(login:auth): check pass; user unknown
Jan 18 05:42:05 pve1 login[21635]: FAILED LOGIN (4) on '/dev/tty1' FOR 'root', Authentication failure
Jan 18 05:41:54 pve1 login[21635]: FAILED LOGIN (3) on '/dev/tty1' FOR 'root', Authentication failure
Jan 18 05:41:45 pve1 login[21635]: FAILED LOGIN (2) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan 18 05:41:42 pve1 login[21635]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jan 18 05:41:42 pve1 login[21635]: pam_unix(login:auth): check pass; user unknown
Jan 18 05:41:39 pve1 login[21635]: FAILED LOGIN (1) on '/dev/tty1' FOR 'root', Authentication failure

even a cursory overview should have given you a pause:

Jan 18 05:44:28 pve1 login[22016]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jan 18 05:44:28 pve1 login[22016]: pam_unix(login:auth): check pass; user unknown
Jan 18 05:44:27 pve1 login[22016]: FAILED LOGIN (3) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan 18 05:44:24 pve1 login[22016]: pam_unix(login:auth): bad username [--poweroff]
Jan 18 05:44:20 pve1 login[22016]: FAILED LOGIN (2) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure


Jan 18 05:44:38 pve1 agetty[22101]: tty1: invalid character 0x18 in login name

I dont know what is going on there, but its not normal, nor does it have anything to do with "Proxmox VE: installation and configuration".
I'd recommend to start peeling back any complexity you can find, shut down VMs, disconnect unnecessary devices, disconnect from network.
Boot from Live Debian ISO and monitor system health. If you find a steady working state - start adding things back.

And start learning basic Linux administration if you want to find a solution.

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[uscpsycho]

those lines you selected are from boot time, essentially initializing the hardware and are not pertinent to your stable run operations or any login issues.

May be you have a bad motherboard connection that is sending garbage to console and elsewhere... Clearly bad authentication is being sent and you are being throttled:

Jan 18 05:42:12 pve1 login[21635]: TOO MANY LOGIN TRIES (5) on '/dev/tty1' FOR 'UNKNOWN'
Jan 18 05:42:12 pve1 login[21635]: FAILED LOGIN (5) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan 18 05:42:09 pve1 login[21635]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jan 18 05:42:09 pve1 login[21635]: pam_unix(login:auth): check pass; user unknown
Jan 18 05:42:05 pve1 login[21635]: FAILED LOGIN (4) on '/dev/tty1' FOR 'root', Authentication failure
Jan 18 05:41:54 pve1 login[21635]: FAILED LOGIN (3) on '/dev/tty1' FOR 'root', Authentication failure
Jan 18 05:41:45 pve1 login[21635]: FAILED LOGIN (2) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan 18 05:41:42 pve1 login[21635]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jan 18 05:41:42 pve1 login[21635]: pam_unix(login:auth): check pass; user unknown
Jan 18 05:41:39 pve1 login[21635]: FAILED LOGIN (1) on '/dev/tty1' FOR 'root', Authentication failure

even a cursory overview should have given you a pause:

Jan 18 05:44:28 pve1 login[22016]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=
Jan 18 05:44:28 pve1 login[22016]: pam_unix(login:auth): check pass; user unknown
Jan 18 05:44:27 pve1 login[22016]: FAILED LOGIN (3) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure
Jan 18 05:44:24 pve1 login[22016]: pam_unix(login:auth): bad username [--poweroff]
Jan 18 05:44:20 pve1 login[22016]: FAILED LOGIN (2) on '/dev/tty1' FOR 'UNKNOWN', Authentication failure


Jan 18 05:44:38 pve1 agetty[22101]: tty1: invalid character 0x18 in login name

I dont know what is going on there, but its not normal, nor does it have anything to do with "Proxmox VE: installation and configuration".
I'd recommend to start peeling back any complexity you can find, shut down VMs, disconnect unnecessary devices, disconnect from network.
Boot from Live Debian ISO and monitor system health. If you find a steady working state - start adding things back.

And start learning basic Linux administration if you want to find a solution.

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Those failed logins are from when I repeatedly tried to login and I couldn't. That was me, I wasn't being throttled by something dubious. I identified and included that part of the log to see if it might show what was causing my logins to fail.

The login problem started right away, when the only thing installed was Proxmox, so the VM's aren't to blame. To my untrained eye it looks like the problem could possibly be related to the "root" username because I see things like "bad username" and "invalid character 0x18 in login name" in the log. And I never have a problem when logging in with SSH where the username is preset.

I wonder if creating a second user might not have these username errors. is it possible to create a second admin account that can use Proxmox and all the VMs without restriction?

You mentioned disconnecting unnecessary devices which got me wondering if the wireless mouse connected to the console might not be playing well with it. I disconnected it and will see what happens. Also monitoring journalctl -f for anything that looks weird, most of it doesn't make a lot of sense to me at this point but I might be able to identify red flags.

 

[leesteken]

Maybe you already checked this, but what if some key presses on the keyboard don't always register properly? Since login does not show any feedback while typing the password, it could be that characters are dropped. Can you try a different keyboard locally (or test that keyboard)?

 

[uscpsycho]

Maybe you already checked this, but what if some key presses on the keyboard don't always register properly? Since login does not show any feedback while typing the password, it could be that characters are dropped. Can you try a different keyboard locally (or test that keyboard)?

Thanks, it's been suggested and responded to several times already. I also mentioned that I was having the same problem logging in locally and remotely so it's happening with two keyboards. I haven't had so many issues today but hopefully I can figure out what the problem is so I don't have to worry about it happening anymore.

 

[bbgeek17]

I also mentioned that I was having the same problem logging in locally and remotely so it's happening with two keyboards.

You've said several times that logging in via SSH always works. In that case, what do you mean by logging in "remotely"? To most people "remote" equals to SSH. Did you mean GUI? In that case - try to use on-screen keyboard app (built-into most OS) and type with your mouse.

Yes, you can create another user (man useradd) and provide that user with elevated privileges' https://linuxize.com/post/how-to-add-user-to-sudoers-in-ubuntu/ and https://pve.proxmox.com/wiki/User_Management.

This user will not have identical functionality as root. However, you should be able to manage most things in combination with "sudo". Dont disable root - its essential for PVE functions.

As previously mentioned - make a live CD/USB of another flavor of Linux, boot it and check functionality. Everything points to either bad hardware or hardware incompatibility. Perhaps the Locale is broken/messed up in some way as it relates to your Keyboard/Client Locale.

Its also possible that one of the "helper" scripts you ran modified the users/agetty or some other file, accidentally or on purpose.
I dont like advising to "reinstall Linux to fix an issue" but you could be trying to track down a needle in a haystack here without a magnet (necessary skills). Hence, a fresh install may be in order.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox