Completely lost Proxmox after update

[thusband]

I updated Proxmox on my mini PC and then rebooted. I waited on the update due to the issues but after folks were saying it's fixed so I went ahead. I refreshed the update and noticed an increase in the total number of changes. The update completed successfully and I rebooted. I have a couple containers and a few VMs. The Ubuntu server VM wouldn't reboot. Now nothing comes up. I can't SSH. It's just gone.

I know I'm not giving a lot of information but I don't what my next step should be. Do I need to reinstall everything or is there something else I could try?

Any hints greatly appreciated

 

[UweV]

Is the status unknown for all VMs etc.? Then this might help: systemctl restart pvestatd.service (execute on shell/console)

 

[thusband]

OK this is weird. I was about ready to reinstall Proxmox so I attached a monitor and keyboard to my PC, inserted the flash card, rebooted the PC and, If I remember correctly, it then connected on my laptop via the IP address. All my containers and VMs seem to be back. I ran your suggestion to be sure but I don't know why or how it all corrected. What could I have done to drop the IP connection in the first place and then find it again after connecting a monitor to the PC?

I'm not sure if your suggestion fixed it but thanks, anyway, for the help. I'll need to figure out why and how this all happened.

 

[Spaneta]

Hello,

Can you check the server is booting correctly without keyboard ?
Or is it throwing an error telling it is stuck at the POST of the mainboard because of no keyboard (or other error like this).
If yes I would recommend you check the BIOS settings and also the mainboard battery coin cell.
If it is empty the BIOS may revert to default settings that prevent mainboard from completing POST in this type of situation.
Thus the fact that screen + keyboard could have "solved" the problem.

Have a nice day,

 

[thusband]

Hello,

Can you check the server is booting correctly without keyboard ?
Or is it throwing an error telling it is stuck at the POST of the mainboard because of no keyboard (or other error like this).
If yes I would recommend you check the BIOS settings and also the mainboard battery coin cell.
If it is empty the BIOS may revert to default settings that prevent mainboard from completing POST in this type of situation.
Thus the fact that screen + keyboard could have "solved" the problem.

Have a nice day,

Hmmm, you mean booting up without the USB keyboard attached to the mini PC? So I'd have to hook it all up and then do a reboot and quickly pull the keyboard USB connector out and see if it boots? Or could I just power it off and then on and see if it boots?

 

[gfngfn256]

Just reboot your server headless (either from GUI or SSH) & see what happens.

 

[Spaneta]

Not exactly headless but with only a screen to look at during the POST and the boot messages, which imply that yes you have to reboot it remotely as gfngfn256 told.
Once you see if the system stop booting it is always time to plug the USB keyboard after that.

 

[Kingneutron]

Start doing full bare-metal backups of the OS and your critical files. Always Have Something To Restore From.

https://www.youtube.com/watch?v=g9J-mmoCLTs

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, check my post history for details

 

[thusband]

Start doing full bare-metal backups of the OS and your critical files. Always Have Something To Restore From.

https://www.youtube.com/watch?v=g9J-mmoCLTs

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, check my post history for details

I will. Lesson learned. Thanks!

 

[esi_y]

Start doing full bare-metal backups of the OS and your critical files. Always Have Something To Restore From.

https://www.youtube.com/watch?v=g9J-mmoCLTs

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, check my post history for details

Isn't the whole point of "hypervisor" not to have to resort to this, ever?

The other thing is, what is the point of using proprietary tools for a Linux install when:

1. a simple dd [1] would suffice;

2. when applicable for a zfs deployment , a rollback to a snaphost or even send|receive [2] (you might want to ssh tunnel it for a truly remote backup) would be easy to automate and forget.

[1] https://www.cyberciti.biz/faq/unix-linux-dd-create-make-disk-image-commands/
[2] https://blog.shalman.org/zfs-send-receive/

* no affiliation to the blog posts/authors whatsover, they appear top of the regular web search and address one of the many ways to approach this

 

[Kingneutron]

Snapshots are NOT a Backup. I recently helped someone who had BOTH zfs boot/root mirror disks fail at the same time.

Feel free to use dd if you want to waste disk space on the destination. Veeam only backs up in-use sectors, keeps several backup versions, and restores lvm structure.

As Proxmox is a flexible hypervisor, you can customize it. Being able to restore your entire primary environment back to a workable state, hopefully with a minimum of time and effort, is priceless.

Don't knock it if you haven't had to use it (or something similar) in a DR situation - I have quite a bit of sysadmin + DR experience and have tested the stuff that I recommend.

 

[Spaneta]

Although a working backup and restore process is of recognized importance, may we ask @thusband to clarify the current situation ?
Was it a POST problem or something else simple that was solved ?

Have a nice day,

 

[esi_y]

Snapshots are NOT a Backup.

Yes?

I recently helped someone who had BOTH zfs boot/root mirror disks fail at the same time.

So having it continuously replicated on another machine would not have assisted you?

Feel free to use dd if you want to waste disk space on the destination. Veeam only backs up in-use sectors, keeps several backup versions, and restores lvm structure.

I wonder how it does that on a LUKS volume. Several backup versions are the snapshots? LVM structure is part of the partition I suppose. Of course if you want to back up any Linux from a live boot, rsync is your best choice.

As Proxmox is a flexible hypervisor, you can customize it. Being able to restore your entire primary environment back to a workable state, hopefully with a minimum of time and effort, is priceless.

I am afraid even the official take of Proxmox on this whole topic is in disgreement, why wouldn't they have provided some sort of facility within PBS for exactly this?

Don't knock it if you haven't had to use it (or something similar) in a DR situation - I have quite a bit of sysadmin + DR experience and have tested the stuff that I recommend.

I do not recommend that line of thinking and I would not brag about it.

 

[thusband]

Although a working backup and restore process is of recognized importance, may we ask @thusband to clarify the current situation ?
Was it a POST problem or something else simple that was solved ?

Have a nice day,

Thanks for asking. Everything is back to normal but I still can't explain what happened. When I hooked up a monitor and keyboard then rebooted and logged in everything returned as normal. I was then able to log in via the IP address. I've looked at the logs but don't see anything although I'd probably miss it anyway not being very experienced.

It seems, though, that a good backup outside of Proxmox is very good idea. I didn't need it this time but at some point I probably will.

 

[Spaneta]

Maybe it is working currently but also it could fail to boot for the same reason the next time you reboot, or have a power outage, or what elese.

If not already done I would recommend you remotely reboot the PVE node _without_ keyboard attached to it to see if it boots correclty this time.
If booting is stuck at some stage please let us know what is told on screen.
If it tells there is a "No keyboard present" error, please follow THIS post and see if that allow for normal booting each time after that

With kind regards,

 

[gfngfn256]

I am afraid even the official take of Proxmox on this whole topic is in disgreement, why wouldn't they have provided some sort of facility within PBS for exactly this?

Your logic doesn't make sense - If Proxmox don't provide a facility for full host restorability then don't bother with one? Read these forums how many reinstalls/reconfigs take place.

Proxmox is being very prudent not to provide a full host backup/restore system - since a full host OS backup/restore working architecture is very dicey in any OS. (Maybe read Windows forums on the subject). Add to that, that a HV is not expected to have downtime, so creating a working restorable backup from a running system is no small feat. Add to that automation & you've got your work set out.
Also boot/system/OS configs are so different from one PVE instance to another, that until a reliable, one-solution-fits-all exist, I imagine they won't
release any real sort of host backup.

I feel (hope - as per their own roadmap) Proxmox will add in the future host backup support, but one thing I'd be very concerned about - if some botched, knocked together system is introduced - this is the worst type of backup possible; giving the user the feeling of being "covered" until calamity strikes, & his/her expectations are shattered. This goes also for anyone else providing their own backup solutions / scripts to others - users must be warned - to actually attempt to rebuild their system OS/configs from scratch using nothing but a backup source. Once that works, maybe they will actually be successful again on D-day in a real-world situation.

Feel free to use dd if you want to waste disk space on the destination

Compression takes care of some of that, albeit without deduplication. But in my experience I'd rather write a complete image each time - and just prune the old ones.

 

[esi_y]

Your logic doesn't make sense - If Proxmox don't provide a facility for full host restorability then don't bother with one? Read these forums how many reinstalls/reconfigs take place.

I was part of some of the discussions, then I figured out that the "backup" of the PVE node is basically about keeping the config.db and if I want to reproduce exactly same node ... the original ISO (even I do not install from ISO, but on top of Debian, easier to automate too). I believe this is why PVE is not providing any "full host restorability" - that and that updates break everything. There's nothing difficult about rsync and grub install either, you can script it (but that's why snapshots, if your filesystem supports it, are the way to go if you do not want to be taking the node down for the occassion).

Proxmox is being very prudent not to provide a full host backup/restore system - since a full host OS backup/restore working architecture is very dicey in any OS.

See above.

Add to that, that a HV is not expected to have downtime

So you have a cluster ...

, so creating a working restorable backup

Just no. Also if you have to take your single node down to back up, you just introduced a downtime, how regularly do you do this? To have no downtime?

Also boot/system/OS configs are so different from one PVE instance to another

Can you explain this?

giving the user the feeling of being "covered" until calamity strikes, & his/her expectations are shattered

If you have a cluster and calamity strikes your single node, it does not matter, the issue is when e.g. datacentre gets flushed out all at once. But if you need 24/7 then you do not run it in one availability zone, hence not use PVE at all.

Compression takes care of some of that, albeit without deduplication.

If you run a serious - quote "enterprise" setup (with encryption at rest) you can't compress anything, but what is the problem of having a dd image of 10GB partition? This is for general Linux, you do not need that for PVE anyhow, it could be counterproductive for a cluster (you will bring up old node with old config.db into existing cluster.... meh).

But in my experience I'd rather write a complete image each time - and just prune the old ones.

I suspect all these comments are from people who do not run a cluster, i.e. do not even need PVE.

 

[esi_y]

I've looked at the logs but don't see anything although I'd probably miss it anyway not being very experienced.

You could have posted it.

 

[thusband]

Maybe it is working currently but also it could fail to boot for the same reason the next time you reboot, or have a power outage, or what elese.

If not already done I would recommend you remotely reboot the PVE node _without_ keyboard attached to it to see if it boots correclty this time.
If booting is stuck at some stage please let us know what is told on screen.
If it tells there is a "No keyboard present" error, please follow THIS post and see if that allow for normal booting each time after that

With kind regards,

You could have posted it.

 

[esi_y]

@thusband I think you are missing the attachment.