Root account forgot the password

[cicciociccio333]

Hi all,
since a couple of months i have a strange issue login to my proxmox server.
The root account password it's not recognized and when i try to login to web console I see " Login failed. Please try again ".
If i ssh with root account I see the error "Access denied".
The VMs are all online and reachable.

It already occurred 4 times and everytime i follow this guide to reset root account password: https://pve.proxmox.com/wiki/Root_Password_Reset
Last time i created a new account with administration privilege to use in case like this, but even the new account gives me the same error.
Clearly i'm sure the password is correct and to be sure i also saved it in my usual MobaXterm session.

As i use my server as test environment not every day i don't know how long pass before the error happens.

root@pve:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.60-1-pve)
pve-manager: 6.2-11 (running version: 6.2-11/22fb4983)
pve-kernel-5.4: 6.2-6
pve-kernel-helper: 6.2-6
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-12
pve-cluster: 6.1-8
pve-container: 3.2-1
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-2
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1

Googling i didn't find any relevant info.
Has it ever happened to someone else?

Thanks
Regards

 

[apoc]

No, but it makes me very very sceptical.
Is the PVE server by any chance accessible directly from the internet? This could be a hack...

 

[leesteken]

I had a similar thing happen to me a long time ago. It turned you that one of the shift-keys on the keyboard was failing intermittently, preventing me from typing the password correctly, without it being visible on screen (all stars/dot or invisible) When this happened when setting a new password, even copy-pasting the (believed to be) correct password would not work.

Apologies if I wasted your time with this story, but because it happened twice, you might want to try a different keyboard (to change the password).

 

[cicciociccio333]

Thanks for your time.
PVE is accessible from the internet but i doubt anyone will hack it 4 times. It may be. How could i verify?
About the correct password i'm sure it is correct because after the reset i login with the password in clear using copy and paste from notepad and it works.

 

[leesteken]

PVE is accessible from the internet but i doubt anyone will hack it 4 times. It may be. How could i verify?

I think password changes are visible in the syslog, but if someone knows the root password, they can easily change the log to hide it.
You could enable logging to a remote system, but if the system is compromised, you cannot trust it in any way.
Or maybe some "you need to change your password every month otherwise your account gets locked"-policy is active? I don't know how to check for that.
PS: Maybe your have enabled something that locks the account or login after so many wrong attempts. Because you are connected to the internet, you are not the only one trying to login.

 

[leesteken]

Can you perhaps save a copy of /etc/shadow, next time you change the password?
If the password of an account is changes you can see that the hashcode has changed and if the account is locked, it will contain a *.
Maybe you can compare the /etc/shadow against the copy every few minutes and have an email sent by the system automatically using cron?
That way, you might discover when the change happens and what whether the password is changed of the account is locked.

 

[apoc]

Its just not good practice to expose something like an virtualization host to the open world. It might be convenient, but it is a dangerous thing to do. Of course you can harden the box but in the end the experience shows noone really does that. So bottom line on my end is: don't do it.
A hack can be as easy as 1,2,3 if your system has a given vulnerability, no firewall up and running, etc. There are scripts just browsing the internet, issuing requests randomly to find a vulnerable system. If I check my firewall log each minute are dozens of requests trying to get in.

 

[devawpz]

Its just not good practice to expose something like an virtualization host to the open world. It might be convenient, but it is a dangerous thing to do. Of course you can harden the box but in the end the experience shows noone really does that. So bottom line on my end is: don't do it.

Not trying to hijack the thread, but this got me thinking, for sure. Any resources you'd recommend as a good starting point for reading some more about this?

 

[apoc]

Sorry, nothing I could refer to directly and out of my mind.
But it is similar driving a car. Everyone thinking about it very logical will put the seatbelt on. It just makes sense.

IT-Security is like securing a building. You just not would put the safe right into the place where you would put the front door.
You have your perimeter, which gets protected by a firewall. That should be as rigid as possible. In- and Outbound connections should be regulated.
Inbound do most of the people (by accident, because they use NAT). Outbound? hell you are going to be surprised what is leaving your network where you never ever had a clue about.
Next step is to segregate networks where it makes sense - similar to different rooms in a building. You dont want to have your kids in the basement on their own, or in your workshop where they can cut their legs of with the chainsaw or tablesaw.

And if you really take it serious you lock everything very critical away as much as possible. Depending on your needs you will place proxy, reverse-proxies and content scanner / mail filters into place. But all that is nonsense if you dont take care of your hypervisor.
Your hypervisor is your most critical asset. The attacker how got it under control is "king of the hill".

 

[mhakim]

after following reset root password, i can't access my /root directory and some of command line iptables, helppp