Help, I think I nuked Proxmox - Login incorrect even with root

[VGE]

So first, I did this myself. I had both my production and test proxmox server open trough GUI and trough ssh. I did some testing to add a NFS share and unfortunately entered all commands in my production install instead of my test install. very stupid I know.

The production was still on Proxmox 6.7.X (I think)
Test is on 7.4.16

now my production version is not accessible even when directly accessing trough screen keyboard and mouse.

I get the login screen but when I enter the username root, I get "login Incorrect".

Any tips to trouble shoot? I have VM backups from this night so I could reinstall proxmox, but I rather try to fix this if possible, even just to look at some settings to prepare for a new install.


Edit: These are the lasts commands I did:

I changed /etc/apt/sources.list to:

deb http://ftp.debian.org/debian bookworm main contrib[/HEADING]
[HEADING=3]deb http://ftp.debian.org/debian bookworm-updates main contrib[/HEADING]
[HEADING=3][/HEADING]
[HEADING=3]# Proxmox VE pve-no-subscription repository provided by proxmox.com,[/HEADING]
[HEADING=3]# NOT recommended for production use[/HEADING]
[HEADING=3]deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription[/HEADING]
[HEADING=3][/HEADING]
[HEADING=3]# security updates[/HEADING]
[HEADING=3]deb http://security.debian.org/debian-security bookworm-security main contrib

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories

And then I run:

`update-initramfs`
`proxmox-boot-tool refresh`
`update-grub`
and rebooting

https://forum.proxmox.com/threads/could-not-mount-nfs-storage.102042/post-439864

Edit2:

I did do a 'apt update && upgrade -y' after sources.list edit.

 

[Max Carrara]

I did some testing to add a NFS share and unfortunately entered all commands in my production install instead of my test install.

Happens to the best of us, even GitLab: https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/

So, don't beat yourself up about it

And then I run:

`update-initramfs`
`proxmox-boot-tool refresh`
`update-grub`
and rebooting

These commands aren't related to adding NFS shares... which filesystem are you using on your host? (e.g. ext4, ZFS, ...)

Also, that's probably unlikely, but... are you sure you don't have caps lock enabled? If your system is able to boot without any problems (as in, the host is reachable and the web GUI is online) then none of the commands above should affect your user's password. Also make sure you're trying to log into the correct host.

 

[VGE]

Happens to the best of us, even GitLab: https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/

So, don't beat yourself up about it

Good, now I feel a bit better thank you.

These commands aren't related to adding NFS shares... which filesystem are you using on your host? (e.g. ext4, ZFS, ...)

Also, that's probably unlikely, but... are you sure you don't have caps lock enabled? If your system is able to boot without any problems (as in, the host is reachable and the web GUI is online) then none of the commands above should affect your user's password. Also make sure you're trying to log into the correct host.

I am 99% sure that I am using ZFS. I was adding my NFS Share trough the gui of the test server when I got:

create storage failed: mount error: mount.nfs: access denied by server while mounting 192.168.5.160:/volume1/Proxmox-share-2 (500)

Thats when I started googling and landed on the post where those commands are suggested: https://forum.proxmox.com/threads/could-not-mount-nfs-storage.102042/

Just to clarify, my production-host is not reachable anymore trough CLI and SSH. I have connected a screen, mouse and keyboard directly to my server and there I see the standard: 'Welcome to the proxmox VE, please use your browser.... connect to: proxmox-url:8006
See the attached file

 

[Max Carrara]

Okay, that seems strange. What happens when you try to log in via the web UI?

Do you remember anything else that you did? Did you maybe run an update after changing /etc/apt/sources.list?

 

[VGE]

@Max Carrara, No webUI connection possible! See my ping result:

C:\Users\Vin>ping 192.168.5.230

Pinging 192.168.5.230 with 32 bytes of data:
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.

Ping statistics for 192.168.5.230:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Yes sorry I did do a 'apt update && upgrade -y'

 

[Max Carrara]

@Max Carrara, No webUI connection possible! See my ping result:

C:\Users\Vin>ping 192.168.5.230

Pinging 192.168.5.230 with 32 bytes of data:
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.
Reply from 192.168.5.101: Destination host unreachable.

Ping statistics for 192.168.5.230:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Yes sorry I did do a 'apt update && upgrade -y'

Oh boy. Since you have backups, it's probably best if you just reinstall PVE in this case, anything else would just take way too long.

Theoretically, you could boot a live distribution via USB and then do some triaging, but that's just going to be painful.

 

[pvps1]

when you boot, stop in grub with "e" edit
then edit the boot command with additional init= /bin/bash

then boot
you are landing in a bash with / mounted read only. remount / rw and you can change or deactivate root pasword.

after reboot you can login and investigate (probably interface renaming after new kernel etc...)

gaining root access to a physical accessable machine is quite easy....

 

[VGE]

Oh boy. Since you have backups, it's probably best if you just reinstall PVE in this case, anything else would just take way too long.

Theoretically, you could boot a live distribution via USB and then do some triaging, but that's just going to be painful.

Im running a 2 ssd 120GB zfs mirror for just proxmox and a seperate SSD for the vms. Good opportunity to start with a clean proxmox install then. Even with the external backups, I can mount the VM SSD and run them correct?

when you boot, stop in grub with "e" edit
then edit the boot command with additional init= /bin/bash

then boot
you are landing in a bash with / mounted read only. remount / rw and you can change or deactivate root pasword.

after reboot you can login and investigate (probably interface renaming after new kernel etc...)

gaining root access to a physical accessable machine is quite easy....

Thats a good tip! Will try that first. Thanks. Will report back.

 

[VGE]

@pvps1 when I hold e, I get the following screen and line at the bottem which I can edit. Should I add init= /bin/bash to that line?

 

[VGE]

Ok I got in to bash and added init= /bin/bash . But no success.

Im getting ' authentication token manipulation error' after passwd.