Help, I think I nuked Proxmox - Login incorrect even with root

VGE

Member
Apr 28, 2020
28
3
23
So first, I did this myself. I had both my production and test proxmox server open trough GUI and trough ssh. I did some testing to add a NFS share and unfortunately entered all commands in my production install instead of my test install. very stupid I know.

The production was still on Proxmox 6.7.X (I think)
Test is on 7.4.16

now my production version is not accessible even when directly accessing trough screen keyboard and mouse.

I get the login screen but when I enter the username root, I get "login Incorrect".

Any tips to trouble shoot? I have VM backups from this night so I could reinstall proxmox, but I rather try to fix this if possible, even just to look at some settings to prepare for a new install.


Edit: These are the lasts commands I did:

I changed /etc/apt/sources.list to:

Code:
deb http://ftp.debian.org/debian bookworm main contrib[/HEADING]
[HEADING=3]deb http://ftp.debian.org/debian bookworm-updates main contrib[/HEADING]
[HEADING=3][/HEADING]
[HEADING=3]# Proxmox VE pve-no-subscription repository provided by proxmox.com,[/HEADING]
[HEADING=3]# NOT recommended for production use[/HEADING]
[HEADING=3]deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription[/HEADING]
[HEADING=3][/HEADING]
[HEADING=3]# security updates[/HEADING]
[HEADING=3]deb http://security.debian.org/debian-security bookworm-security main contrib

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories

And then I run:

Code:
`update-initramfs`
`proxmox-boot-tool refresh`
`update-grub`
and rebooting

https://forum.proxmox.com/threads/could-not-mount-nfs-storage.102042/post-439864

Edit2:

I did do a 'apt update && upgrade -y' after sources.list edit.
 
Last edited:
I did some testing to add a NFS share and unfortunately entered all commands in my production install instead of my test install.
Happens to the best of us, even GitLab: https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/

So, don't beat yourself up about it :)

And then I run:

Code:
`update-initramfs`
`proxmox-boot-tool refresh`
`update-grub`
and rebooting
These commands aren't related to adding NFS shares... which filesystem are you using on your host? (e.g. ext4, ZFS, ...)

Also, that's probably unlikely, but... are you sure you don't have caps lock enabled? If your system is able to boot without any problems (as in, the host is reachable and the web GUI is online) then none of the commands above should affect your user's password. Also make sure you're trying to log into the correct host.
 
Happens to the best of us, even GitLab: https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/

So, don't beat yourself up about it :)
Good, now I feel a bit better thank you.

These commands aren't related to adding NFS shares... which filesystem are you using on your host? (e.g. ext4, ZFS, ...)

Also, that's probably unlikely, but... are you sure you don't have caps lock enabled? If your system is able to boot without any problems (as in, the host is reachable and the web GUI is online) then none of the commands above should affect your user's password. Also make sure you're trying to log into the correct host.
I am 99% sure that I am using ZFS. I was adding my NFS Share trough the gui of the test server when I got:
create storage failed: mount error: mount.nfs: access denied by server while mounting 192.168.5.160:/volume1/Proxmox-share-2 (500)

Thats when I started googling and landed on the post where those commands are suggested: https://forum.proxmox.com/threads/could-not-mount-nfs-storage.102042/

Just to clarify, my production-host is not reachable anymore trough CLI and SSH. I have connected a screen, mouse and keyboard directly to my server and there I see the standard: 'Welcome to the proxmox VE, please use your browser.... connect to: proxmox-url:8006
See the attached file
 

Attachments

  • proxmox login incorrect.jpg
    proxmox login incorrect.jpg
    94.9 KB · Views: 18
Okay, that seems strange. What happens when you try to log in via the web UI?

Do you remember anything else that you did? Did you maybe run an update after changing /etc/apt/sources.list?
 
@Max Carrara, No webUI connection possible! See my ping result:

Code:
C:\Users\Vin>ping 192.168.5.230

Pinging 192.168.5.230 with 32 bytes of data:
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.

Ping statistics for 192.168.5.230:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Yes sorry I did do a 'apt update && upgrade -y'
 
@Max Carrara, No webUI connection possible! See my ping result:

Code:
C:\Users\Vin>ping 192.168.5.230

Pinging 192.168.5.230 with 32 bytes of data:
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.

Ping statistics for 192.168.5.230:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Yes sorry I did do a 'apt update && upgrade -y'

Oh boy. Since you have backups, it's probably best if you just reinstall PVE in this case, anything else would just take way too long.

Theoretically, you could boot a live distribution via USB and then do some triaging, but that's just going to be painful.
 
  • Like
Reactions: VGE
when you boot, stop in grub with "e" edit
then edit the boot command with additional init= /bin/bash

then boot
you are landing in a bash with / mounted read only. remount / rw and you can change or deactivate root pasword.

after reboot you can login and investigate (probably interface renaming after new kernel etc...)

gaining root access to a physical accessable machine is quite easy....
 
  • Like
Reactions: VGE and Max Carrara
Oh boy. Since you have backups, it's probably best if you just reinstall PVE in this case, anything else would just take way too long.

Theoretically, you could boot a live distribution via USB and then do some triaging, but that's just going to be painful.
Im running a 2 ssd 120GB zfs mirror for just proxmox and a seperate SSD for the vms. Good opportunity to start with a clean proxmox install then. Even with the external backups, I can mount the VM SSD and run them correct?



when you boot, stop in grub with "e" edit
then edit the boot command with additional init= /bin/bash

then boot
you are landing in a bash with / mounted read only. remount / rw and you can change or deactivate root pasword.

after reboot you can login and investigate (probably interface renaming after new kernel etc...)

gaining root access to a physical accessable machine is quite easy....
Thats a good tip! Will try that first. Thanks. Will report back.
 
@pvps1 when I hold e, I get the following screen and line at the bottem which I can edit. Should I add init= /bin/bash to that line?
 

Attachments

  • IMG_5042222.jpg
    IMG_5042222.jpg
    307.8 KB · Views: 11
Ok I got in to bash and added init= /bin/bash . But no success.

Im getting ' authentication token manipulation error' after passwd.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!