Messed up PVE - need to backup manually

j.io

Active Member
Mar 16, 2021
104
7
58
40
Hi all,

an update and kernel clean messed up my grub and I end up with

Code:
KERNEL PANIC!

Please reboot your computer.

VFS: Unable to mount root fs on unknown-block(0,0)

I am using dropbear and LUKS. I tried all the info with mounting the device via a live Linux and updating grub etc. - but all did not work. Don't ask me ...

Maybe it is dropbear that breaks everything ... I have no idea what to do.


The good news is I have access to the root filesystem of the PVE host via the live Linux - so I decided to pull the latest backups of all VMs / LXCs as well as the whole configuration and set up a new Proxmox host to manually move it over there.

(Yes, I have backups, but I want exactly the state of the VM / LXCs at the point where I rebooted the system (and never got it back up again).

What files do I need to copy to a USB drive in order to bring the VMs / LXCs up on a fresh PVE install?

Thank for the help, I am desperate now and wasted already multiple hours on this ... running this PVE host for at least 4-5 years without issue, but now is the time for a fresh start I guess.
 
Since the live USB can read the old root, the first file I'd save is /var/lib/pve-cluster/config.db from the mounted old system. If a matching config.db-wal is beside it, save that too so the copy retains recent committed configuration changes. /etc/pve is normally mounted by pmxcfs, so an empty directory in the live session is expected. The database contains the VM and container definitions and storage.cfg. Keep it private, since it can also contain credentials. It doesn't contain the guest disks. Proxmox pmxcfs recovery · Proxmox database source

For the newest data still on disk, use the guest definitions and storage.cfg to find the actual volumes. Save every attached VM disk, including any EFI and TPM state volumes, plus each container's rootfs and extra mpN mounts. For a directory backend, look under the configured storage path and preserve any base images a linked clone depends on. LVM-thin guest disks live in logical volumes. ZFS guest storage can be zvols or datasets. A container bind mount points at a separate host path, and a device mount points at a separate device. Preserve those sources separately. /var/lib/vz/dump can hold older backup archives, but copying it won't capture changes made after those backups. Proxmox storage configuration · Container mount points

I'd keep the original disks out of the reinstall, set up the new host on a separate disk, and test recovery on copies before starting the guests on the production network. That gives you a chance to check that each disk and mount is present and that guest files and databases are usable. If you can share the storage types in use (for example, lvmthin, zfspool, or dir) and whether the guest storage shares the install disk, that would narrow down what to copy. Please leave out credentials and addresses.

OpenAI Codex assisted with drafting this reply.
 
OpenAI Codex assisted with drafting this reply.
Op is perfectly capable of asking codex/grok/gemini/mysuperllm himself ;)

Thank for the help, I am desperate now and wasted already multiple hours on this ... running this PVE host for at least 4-5 years without issue, but now is the time for a fresh start I guess.
I read your post but am unsure what you'd like help with.

You already mounted the filesystem and recovered your payload. reinstalling pve and reloading your payload should be trivial. as to why it happened- you didnt provide any logs to troubleshoot...
 
  • Like
Reactions: IsThisThingOn
Thanks so far.

So I failed on repairing my kernel GRUB. Because I am obviously too dumb I thought I go with a reinstall of a new Proxmox host and copy all the required files over. I just wanted to know which there are.
 
... so am I right that I have to only save all contents of /etc/pve and merge it with the ones on a new system and that's it?
 
No. Even if you boot from the Live CD and access the PVE disk, `config.db` is not mounted in `/etc/pve`, so `/etc/pve` is empty.

Since `config.db` is a pmxcfs file system mounted in `/etc/pve` as part of PVE’s service operations, it does not exist even when booting from the Live CD.

To restore the data, you'll need the `config.db` file, the network settings, and the hostname.

https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html#_recovery

Code:
Recovery
If you have major problems with your Proxmox VE host, for example hardware issues, it could be helpful to copy the pmxcfs database file /var/lib/pve-cluster/config.db, and move it to a new Proxmox VE host. On the new host (with nothing running), you need to stop the pve-cluster service and replace the config.db file (required permissions 0600). Following this, adapt /etc/hostname and /etc/hosts according to the lost Proxmox VE host, then reboot and check (and don’t forget your VM/CT data).

So, all you need is the following.

Code:
/etc/hostname
/etc/hosts
/etc/network/interfaces
/var/lib/pve-cluster/config.db
*Of course, if the virtual disk was located on the boot disk, it goes without saying that you would need to remove it, so I’ll skip that part.

If you still don't understand this (and this is a basic concept), your skill level isn't high enough to get help on this forum.

Given the nature of the question here, you likely won’t be able to resolve this without purchasing a subscription, opening a support ticket, and contacting support.

If you have the technical know-how, it should be a simple matter of following the instructions in the documentation.

After recovery, you should perform regular backups.
Of course, you’ll also need to back up the virtual disk separately.

Code:
/var/lib/pve-cluster/config.db
/etc/network/interfaces
/etc/pve/virtual-guest/cpu-models.conf
/etc/pve/storage.cfg
/etc/pve/qemu-server/*
/etc/pve/lxc/*
/var/lib/vz/snippets/*
/etc/modprobe.d/*
/etc/kernel/cmdline
/etc/default/grub
/usr/share/kvm/*
/etc/modules
/etc/sensors.d
/etc/passwd
/etc/resolv.conf
/etc/apt/sources.list.d/
/usr/local/lib/systemd/network/*
 
Last edited:
First - thank you for your answer.

No. Even if you boot from the Live CD and access the PVE disk, `config.db` is not mounted in `/etc/pve`, so `/etc/pve` is empty.
There is a way to mount /etc/pve even from a live CD. I achieved this last weekend and could see all the content of /etc/pve.

If you still don't understand this (and this is a basic concept), your skill level isn't high enough to get help on this forum.
I understand this, but my Proxmox host runs flawless since 3 years or so, so I forgot a lot about the background mechanics of Proxmox in the meantime and need to freshen it up.

After recovery, you should perform regular backups.
Of course, you’ll also need to back up the virtual disk separately.
I am doing this both on local as well on the network. My problem was that I removed some Kernels due to a full harddrive and did not take the backup of the CURRENT system state before rebooting. I only have to backups from a couple of hours earlier, so I am missing some data.

I should have triggered a backup job rights before restarting (that I did not do for the last 2 years to be honest and it always worked out).

Anyway, thank you for your time and help. :-)
 
Thanks so far.

So I failed on repairing my kernel GRUB. Because I am obviously too dumb I thought I go with a reinstall of a new Proxmox host and copy all the required files over. I just wanted to know which there are.
All you need to re-run your payload can be found in
/etc/pve/lxc/ctid.conf
/etc/pve/qemu-server/vmid.conf

and, of course, the virtual disks. PVE is designed to be an appliance for the most part- you dont really need to back it up in that sense; just reinstall and redeploy the payload.

So, all you need is the following.
These files serve no purpose. on a single host they're not relevent; in a cluster you wouldnt need to restore the node in the first place.

I understand this, but my Proxmox host runs flawless since 3 years or so, so I forgot a lot about the background mechanics of Proxmox in the meantime and need to freshen it up.
If this was relevant, you wouldnt be here. you would also not need to buy car insurance.
My problem was that I removed some Kernels due to a full harddrive
You kinda buried the lede here. YOU caused the problem to begin with. incidentally, full recovery of this situation is achievable with chroot.
 
  • Like
Reactions: j.io
If this was relevant, you wouldnt be here. you would also not need to buy car insurance.
If my grandma had wheels, she would be a bike. :p

You kinda buried the lede here. YOU caused the problem to begin with. incidentally, full recovery of this situation is achievable with chroot.
Yes, may be. I followed all the tutorials in the internet to recover from this issue - of course also including the one from the Proxmox docs - but it did not work. So unless someone's coming over to help me in person, I don't know what to do except setting um a fresh node and move the required disks etc. over.

Hence the question, which files I'd need ...

I don't want to be rude, but this is the only thing I am asking for. :-) I know that I caused the issue, and I told so in my original post, so no need to defend Proxmox. ;)