[REINSTALLED FRESH] I broke my partitions AND grub with sgdisk -e /dev/sda* - any way to fix this?

[AndrzejL]

TRY WATCHING AND UNDERSTANDING AND FOLLOWING THIS VIDEO.

https://youtu.be/VR5TP9E9wio

MAYBE YOU WILL HAVE MORE LUCK THAN I DID. I DISCOVERED THIS VIDEO AFTER I REALLY MESSED UP MY SSD WITH TESTDISK.

Hello beautiful people.

I messed up. I noticed that my proxmox installation was throwing errors in fdisk -l

The backup GPT table is corrupt, but the primary appears OK, so that will be used.

I did the worst possible thing - I found an advice where someone was recommending sgdisk -e /dev/sda but I misread it.

I ran that command on all 3 partitions... sda1 sda2 and sda3

sgdisk -e /dev/sda1
sgdisk -e /dev/sda2
sgdisk -e /dev/sda3

It was 3 am... I was tired. My bad...



Those are the messages I got.

After a reboot all I get is GRUB LOADING.

I was trying to rescue the installation and was reading "recovering from grub failure" from proxmox here https://pve.proxmox.com/wiki/Recover_From_Grub_Failure but when I boot to arch linux live and I run sudo vgscan and sudo vgchange -ay there is no output and the sudo mount /dev/pve/root /media/RESCUE/ says that the special device does not exist.

Rescue boot from Proxmox ISO shows:



/dev/sdb2 can be mounted it contains EFI folder. Other two won't mount or fsck.

I am booted into arch linux live trying to chroot into the installation but the most basic steps are failing.



I don't want to mess with this on my own - I am afraid I will do more damage...

Yes I know... its a bit late for being cautious

Thanks in advance for any help provided.

Kindest regards.

AndrzejL

 

[leesteken]

Sounds like you corrupted /dev/sda1 but that can easily be fixed using GRUB rescue, chroot into your existing installation and a grub-install /dev/sda.
Looks like /dev/sda2 was not touched (not saving changes. Use -g to override.) and the FAT filesystem on that ESP-partition is probably fine.
You might have corrupted a little part your LVM storage and I'm not sure if and which logical volume (with Proxmox or a VM) is corrupted (if it was not free space). ZFS would detect such a thing but LVM probably not. Run a filesystem check of each VM (after fixing GRUB) to look for corrupted but if it is file corruption, it might be silent and a restore from backup might be best.

Either reinstall (and re-setup your Proxmox) and restore everything from backups or try if just doing a GRUB rescue and grub-install fixes it: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure#General_advice

Then again, I might be wrong and make everything worse, so a byte-for-byte copy to another drive might be the safest thing to do first...

 

[AndrzejL]

Sounds like you corrupted /dev/sda1 but that can easily be fixed using GRUB rescue, chroot into your existing installation and a grub-install /dev/sda.
Looks like /dev/sda2 was not touched (not saving changes. Use -g to override.) and the FAT filesystem on that ESP-partition is probably fine.
You might have corrupted a little part your LVM storage and I'm not sure if and which logical volume (with Proxmox or a VM) is corrupted (if it was not free space). ZFS would detect such a thing but LVM probably not. Run a filesystem check of each VM (after fixing GRUB) to look for corrupted but if it is file corruption, it might be silent and a restore from backup might be best.

Either reinstall (and re-setup your Proxmox) and restore everything from backups or try if just doing a GRUB rescue and grub-install fixes it: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure#General_advice

Then again, I might be wrong and make everything worse, so a byte-for-byte copy to another drive might be the safest thing to do first...

I wish it was that simple but unfortunately I cannot even chroot into it because of the lvm corruption.

I was trying to rescue the installation and was reading "recovering from grub failure" from proxmox here https://pve.proxmox.com/wiki/Recover_From_Grub_Failure but when I boot to arch linux live and I run sudo vgscan and sudo vgchange -ay there is no output and the sudo mount /dev/pve/root /media/RESCUE/ says that the special device does not exist.

Rescue boot from Proxmox ISO shows:

View attachment 68279

Thank you for your suggestion. I hope it becomes part of the solution once I recover the lvm. Hopefully... Anyone has a lvm uncorruptor or a time machine?

Kindest regards.

Andrzej

 

[AndrzejL]

BAD IDEA, LVM AND TESTDISK DON'T MIX WELL...

Cheers.

Andrzej

I am reading that a partition table can be recovered with a use of a tool like testdisk.

Tried installing it in ubuntu live but I don't see it in the repo.

https://www.cgsecurity.org/wiki/TestDisk_Livecd

GParted Live is supposed to contain it. Gonna try using that and update with progress.

Cheers.

Andrzej

 

[leesteken]

I am reading that a partition table can be recovered with a use of a tool like testdisk.

I don't think your partition table is broken. I think you accidentally overwrite a disk block (or two) at the end of two of the partitions, which apparently broke GRUB and LVM (and the first can easily be fixed if the last one can be fixed).

GParted Live is supposed to contain it. Gonna try using that and update with progress.

Nonetheless, maybe testdisk can inded repair your drive and/or LVM, after which you can GRUB rescue and chroot into it and fix GRUB on /dev/sda1. Best of luck to you!

 

[Kingneutron]

Bro... You're not going to want to grasp this right now, so maybe come back in a day or so if you get back to a stable state.

You need to start. making. backups. Especially before you mess around with the system.

https://github.com/kneutron/ansitest/tree/master/proxmox

Take a look at least at the bkpcrit script - you want to get to a point where you can just reinstall proxmox from ISO, redefine your backup storage (either separate disk or NAS), and do a plain restore on all your VMs and containers. Backing up the critical files should even allow you to recover the VM and CTR config files if all you have left are the virtual storage disks.

This is your wakeup call to do proper Disaster Recovery prep and have a plan for the future. Ideally you want to get your primary environment back up and running in less than 12-24 hours.

Not trying to bag on you at a bad time, but this was avoidable with standard DR. Unfortunately, most ppl don't go that far until they've experienced data loss personally - or learned from others.

Yes it sucks and I've been there, so at least know that I'm sorry for your loss. But make a plan so you can recover from something like this, even if you enter the wrong command.

 

[AndrzejL]

I don't think your partition table is broken. I think you accidentally overwrite a disk block (or two) at the end of two of the partitions, which apparently broke GRUB and LVM (and the first can easily be fixed if the last one can be fixed).

Nonetheless, maybe testdisk can inded repair your drive and/or LVM, after which you can GRUB rescue and chroot into it and fix GRUB on /dev/sda1. Best of luck to you!

It didn't and if the partition table was not broken then... it probably is now.

As soon as I did what I did (wrote a weird looking recovery from testdisk I found this video

https://youtu.be/VR5TP9E9wio

It probably would help me recover but now my drive looks like this:



Yeah I am a freaking genius.

Cheers.

Andrzej

 

[AndrzejL]

Bro... You're not going to want to grasp this right now, so maybe come back in a day or so if you get back to a stable state.

You need to start. making. backups. Especially before you mess around with the system.

https://github.com/kneutron/ansitest/tree/master/proxmox

Take a look at least at the bkpcrit script - you want to get to a point where you can just reinstall proxmox from ISO, redefine your backup storage (either separate disk or NAS), and do a plain restore on all your VMs and containers. Backing up the critical files should even allow you to recover the VM and CTR config files if all you have left are the virtual storage disks.

This is your wakeup call to do proper Disaster Recovery prep and have a plan for the future. Ideally you want to get your primary environment back up and running in less than 12-24 hours.

Not trying to bag on you at a bad time, but this was avoidable with standard DR. Unfortunately, most ppl don't go that far until they've experienced data loss personally - or learned from others.

Yes it sucks and I've been there, so at least know that I'm sorry for your loss. But make a plan so you can recover from something like this, even if you enter the wrong command.

Words of wisdom. And yes... I was going to backup after I fixed that error mentioned in my first post... Oh the irony...

Thank you.

Andrzej

 

[leesteken]

It didn't and if the partition table was not broken then... it probably is now.

As soon as I did what I did (wrote a weird looking recovery from testdisk I found this video

https://youtu.be/VR5TP9E9wio

It probably would help me recover but now my drive looks like this:

View attachment 68334

Yeah. that does not look good. I guess you did you not make a byte-for-byte copy before starting "repairs"? I don't have any experience with testdisk myself to help you, sorry.

I wiped my partitions once by resetting the system while they were still being moved by gparted. On the bright side, my last made backup was instantly up to date again. ;-)

 

[AndrzejL]

Yeah. that does not look good. I guess you did you not make a byte-for-byte copy before starting "repairs"? I don't have any experience with testdisk myself to help you, sorry.

I wiped my partitions once by resetting the system while they were still being moved by gparted. On the bright side, my last made backup was instantly up to date again. ;-)

No stress... Its not like its a company server. Just a private server.

One positive thing - I got the machine replaced. From Dell Optiplex SFF 990 with i7 2600 to Dell Optiplex SFF 7010 with i7 3770.



Fresh install.

Regards.

Andrzej

 

[Kingneutron]

I noticed you have sdb on the system, if you have stray VM and CTR disks still on there but no configs for the VMs, read thru this script:

https://github.com/kneutron/ansitest/blob/master/proxmox/proxmox-recover-vm-disks-without-backup.sh

Basically tried to document a whole HOWTO in there. Might help you get back up and running quicker.

BTW I would strongly recommend that you document your recovery process, and maybe share with others.

 

[AndrzejL]

I noticed you have sdb on the system, if you have stray VM and CTR disks still on there but no configs for the VMs, read thru this script:

https://github.com/kneutron/ansitest/blob/master/proxmox/proxmox-recover-vm-disks-without-backup.sh

Basically tried to document a whole HOWTO in there. Might help you get back up and running quicker.

BTW I would strongly recommend that you document your recovery process, and maybe share with others.

What I did is I grabbed a smaller but very usable 120G WD Green SSD, and put it into the Dell machine with my 2 x 2TB Crucial drives and 1 USB 3.0 6TB WD MyBook, I diskpart cleaned the WD Green, installed proxmox 8.2 on it, mounted my 2 x 2TB Crucial drives containing vm drives and isos and am 75% done moving the content to the 6TB MyBook.

Once this is done I will diskpart clean the 2 x 2TB Crucial drives using Windows installer LiveUSB, then I will properly initialize the drives with GPT using the Proxmox GUI and then I will mount the 6TB drive and I will move the content back to the 2 x 2TB Crucial SSDs.

Unfortunately .conf files for the vms are not kept with the vm .qcow2 files and I lost them - so once this is done I will create a VMs that will correspond with the VM drive numbers and assign them accordingly.

I have quite the task ahead of me. I will be at it for a while. Putting all my little fixes and stuff in place... That's the price I pay for not having a backup. I had this installation for a good few years. You know how after a while you get it "just right"...

There are two types of people in this world. The ones who backup and the ones that will backup...

Cheers.

Andrzej

 

[Kingneutron]

PROTIP - take a look at that bkpcrit script I posted earlier when you have a chance. Use it often and make sure it's backing up to separate storage (not the OS/root disk - use NAS or at least a separate disk) Then you should have the .conf files for VMs and CTRs as well available for restore ;-)

If the backing storage is ZFS, you can even have a snapshot history of critical-file backups.

/ I've got mine in crontab and also run it manually before any system changes - including package upgrades

 

[AndrzejL]

PROTIP - take a look at that bkpcrit script I posted earlier when you have a chance. Use it often and make sure it's backing up to separate storage (not the OS/root disk - use NAS or at least a separate disk) Then you should have the .conf files for VMs and CTRs as well available for restore ;-)

If the backing storage is ZFS, you can even have a snapshot history of critical-file backups.

/ I've got mine in crontab and also run it manually before any system changes - including package upgrades

Will do. I am pretty upset about this whole ordeal but weirdly not even remotely as much as I thought I would be... Maybe because I know that I can set it all back up and I do not have to recreate all the VMs from scratch because I have the vm disks...

Cheers.

Andrzej

 

[AndrzejL]

I noticed you have sdb on the system, if you have stray VM and CTR disks still on there but no configs for the VMs, read thru this script:

https://github.com/kneutron/ansitest/blob/master/proxmox/proxmox-recover-vm-disks-without-backup.sh

Basically tried to document a whole HOWTO in there. Might help you get back up and running quicker.

BTW I would strongly recommend that you document your recovery process, and maybe share with others.

Slowly but surely...



Thanks for the qm rescan tip.

Cheers.

Andrzej

 

[AndrzejL]

Ok I am back online.

I wanted to thank everyone that replied in this thread. The fact that someone cared enough to reply was a help on its own.

Kindest regards.

Have a fantastic day.

AndrzejL

 

[Kingneutron]

Cool Now don't forget to setup bkpcrit and do a Full Backup ;-)

 

[AndrzejL]

Cool Now don't forget to setup bkpcrit and do a Full Backup ;-)

Heck yeah.

So... Did you wrote that script?

https://github.com/kneutron/ansitest/blob/master/proxmox/bkpcrit-proxmox.sh

Run it as root right?

What needs to be edited? It seems like I found those places but... I don't see anywhere to set the "Backup drive or folder"?

# xxx TODO EDITME
primaryuser=dave

# xxx TODO EDITME - set 1 if dest is ZFS compressed (lz4,zstd)
comprdest=0
if [ "$comprdest" = "1" ]; then
  taropts="-cpf "; tarsfx="tar"
else
#  taropts="--use-compress-program lzop -cpf "
  taropts="--lzop -cpf "; tarsfx="tar.lzop"
fi

# xxx TODO EDITME
echo $dest # = PK

# xxx TODO editme
distro="debian"
testd=$(grep ID_LIKE /etc/os-release |awk -F\" '{print $2}') # rhel fedora
testd=${testd// /-}    # bash inline sed, replace space with dash
[ "$testd" = "" ] || distro="$testd"

Cheers.

AndrzejL

 

[Kingneutron]

Yep, I wrote it a loooong time ago and improved it over the years. Has saved my bacon a few times.

Yes, run it as root and you can set it up to fire off automatically in crontab, that's why I put the PATH in.

You can comment out the "primary user" stuff if you only have root account on proxmox, otherwise set it to your non-root user login.

The next EDITME is if you're saving the tar files to a ZFS compressed target - the script assumes not, and enables lzop compression. Change it to 1 if you're saving to a ZFS/shared drive; or you could change it to use zstd or gzip if you like.

You can ignore the PK, I should probably comment that out...

> I don't see anywhere to set the "Backup drive or folder"?

That is the " BKPDEST.mrg " file, which by default on my system is located in /root/bin/boojum . You can put it elsewhere and soft-symlink to it if you want, that is where the backup destination (partition / shared drive) is defined. It's in a separate file since it is sourced / used by several scripts.

The OS is debian, so you don't have to worry about the rest. I adapted my existing script for Proxmox.

 

[AndrzejL]

Yep, I wrote it a loooong time ago and improved it over the years. Has saved my bacon a few times.

Yes, run it as root and you can set it up to fire off automatically in crontab, that's why I put the PATH in.

You can comment out the "primary user" stuff if you only have root account on proxmox, otherwise set it to your non-root user login.

The next EDITME is if you're saving the tar files to a ZFS compressed target - the script assumes not, and enables lzop compression. Change it to 1 if you're saving to a ZFS/shared drive; or you could change it to use zstd or gzip if you like.

You can ignore the PK, I should probably comment that out...

> I don't see anywhere to set the "Backup drive or folder"?

That is the " BKPDEST.mrg " file, which by default on my system is located in /root/bin/boojum . You can put it elsewhere and soft-symlink to it if you want, that is where the backup destination (partition / shared drive) is defined. It's in a separate file since it is sourced / used by several scripts.

The OS is debian, so you don't have to worry about the rest. I adapted my existing script for Proxmox.

That is very cool. Thank you again! Your script-foo be strong...

I will analyse that script to learn stuff.

Cheers.

Andrzej

NOTE TO SELF: https://github.com/yjajkiew/dynhost-ovh/blob/master/README.md <<< You used this to set your OVH dyn host.

#!/usr/bin/env sh


# Account configuration
HOST=DOMAINE_NAME
LOGIN=LOGIN
PASSWORD=PASSWORD
DNSSERVER=@dns102.ovh.net


PATH_LOG=/var/log/dynhostovh.log


# Get current IPv4 and corresponding configured
HOST_IP=$(dig $DNSSERVER +short $HOST A)
CURRENT_IP=$(curl -m 5 -4 ifconfig.co 2>/dev/null)
if [ -z $CURRENT_IP ]
then
  CURRENT_IP=$(dig +short myip.opendns.com @resolver1.opendns.com)
fi
CURRENT_DATETIME=$(date -R)


# Update dynamic IPv4, if needed
if [ -z $CURRENT_IP ] || [ -z $HOST_IP ]
then
  echo "[$CURRENT_DATETIME]: No IP retrieved" >> $PATH_LOG
else
  if [ "$HOST_IP" != "$CURRENT_IP" ]
  then
    RES=$(curl -m 5 -L --location-trusted --user "$LOGIN:$PASSWORD" "https://www.ovh.com/nic/update?system=dyndns&hostname=$HOST&myip=$CURRENT_IP")
    echo "[$CURRENT_DATETIME]: IPv4 has changed - request to OVH DynHost: $RES" >> $PATH_LOG
  fi
fi