Upgrade Gone Horribly Wrong - Help

IceProx

New Member
Dec 20, 2015
11
0
1
55
Hi All,

I have been running Proxmox for years and it has always been very stable. For the most part I run Linux KVMs with only 1 Windows KVM.

The Windows KVM was Windows 7 Pro and had been running since the summer very stable. I decided to upgrade to Windows 10 Pro and since that time I would get a daily crash with the BSOD relating to PFL or memory issues. It wasn't a big issue since it was my Windows sandbox VM.

Thursday I decided to try an fix the issue I was having with it. I started doing to updates to Proxmox to make sure it was up to date and when it was updated I added the source list for Proxmox 4.1 and started the dist-upgrade.

My fault I did not read the migration instructions (until after) and this is when everything began to blow apart. By the end, it would not even boot. I pulled the drive and installed Proxmox 4.1 fresh from a CD on a new drive.

All of my VM storage is on iSCSI. However, my network/storage/VM config files were not backed up. I shutdown and added the original drive back in. When I booted up I realized I had a problem with the LVM names being the same so I ended up changing the names on the original drive from pve to pve_old. This way there was not any conflicts.

The bad part is this: when I mounted the original drive and went the the /etc/pve directory it is now empty. I have searched high and low for answers trying to figure out how to retrieve this data to no avail. The one bit of hope I have is this.

When I run:
Code:
root@moxy002:~# fsck -f -v /dev/pve_old/root
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

       52486 inodes used (0.83%, out of 6291456)
         837 non-contiguous files (1.6%)
          94 non-contiguous directories (0.2%)
             # of inodes with ind/dind/tind blocks: 2949/17/0
      747464 blocks used (2.97%, out of 25165824)
           0 bad blocks
           1 large file

       38191 regular files
        4164 directories
         855 character device files
        4530 block device files
           3 fifos
           8 links
        4707 symbolic links (4530 fast symbolic links)
          27 sockets
------------
       52485 files
root@moxy002:~#
You can see there are not any errors.

End of Part 1.

Thanks,
ProxMox
 
However, when I run fsck using a different superblock I get the following:
Code:
root@moxy002:~# fsck -f -v -b 32768 /dev/pve_old/root
fsck from util-linux 2.25.2
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #20 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #21 (32254, counted=22162).
Fix<y>? yes
Free blocks count wrong for group #22 (32254, counted=8839).
Fix<y>? yes
Free blocks count wrong for group #23 (32254, counted=23439).
Fix<y>? yes
Free blocks count wrong for group #24 (32254, counted=23491).
Fix<y>? yes
Free blocks count wrong for group #25 (31229, counted=23404).
Fix<y>? yes
Free blocks count wrong for group #26 (32254, counted=30932).
Fix<y>? yes
Free blocks count wrong for group #35 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #46 (32254, counted=29308).
Fix<y>? yes
Free blocks count wrong for group #178 (32254, counted=7052).
Fix<y>? yes
Free blocks count wrong for group #179 (32254, counted=24401).
Fix<y>? yes
Free blocks count wrong for group #180 (32254, counted=29677).
Fix<y>? yes
Free blocks count wrong for group #181 (32254, counted=31079).
Fix<y>? yes
Free blocks count wrong for group #182 (32254, counted=30900).
Fix<y>? yes
Free blocks count wrong for group #224 (32254, counted=30866).
Fix<y>? yes
Free blocks count wrong for group #238 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #257 (32254, counted=32238).
Fix<y>? yes
Free blocks count wrong for group #269 (32254, counted=4229).
Fix<y>? yes
Free blocks count wrong for group #270 (32254, counted=16184).
Fix<y>? yes
Free blocks count wrong for group #271 (32254, counted=29641).
Fix<y>? yes
Free blocks count wrong for group #272 (32254, counted=25568).
Fix<y>? yes
Free blocks count wrong for group #273 (32254, counted=24208).
Fix<y>? yes
Free blocks count wrong for group #274 (32254, counted=9673).
Fix<y>? yes
Free blocks count wrong for group #275 (32254, counted=5677).
Fix<y>? yes
Free blocks count wrong for group #276 (32254, counted=23627).
Fix<y>? yes
Free blocks count wrong for group #277 (32254, counted=23522).
Fix<y>? yes
Free blocks count wrong for group #278 (32254, counted=23590).
Fix<y>? yes
Free blocks count wrong for group #279 (32254, counted=29248).
Fix<y>? yes
Free blocks count wrong for group #280 (32254, counted=24709).
Fix<y>? yes
Free blocks count wrong for group #281 (32254, counted=23484).
Fix<y>? yes
Free blocks count wrong for group #282 (32254, counted=23139).
Fix<y>? yes
Free blocks count wrong for group #283 (32254, counted=20135).
Fix<y>? yes
Free blocks count wrong for group #284 (32254, counted=26468).
Fix<y>? yes
Free blocks count wrong for group #285 (32254, counted=26260).
Fix<y>? yes
Free blocks count wrong for group #286 (32254, counted=32228).
Fix<y>? yes
Free blocks count wrong for group #339 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #358 (32254, counted=32251).
Fix<y>? yes
Free blocks count wrong for group #369 (32254, counted=22483).
Fix<y>? yes
Free blocks count wrong for group #393 (32254, counted=32235).
Fix<y>? yes
Free blocks count wrong for group #394 (32254, counted=32237).
Fix<y>? yes
Free blocks count wrong for group #445 (32254, counted=32236).
Fix<y>? yes
Free blocks count wrong for group #476 (32254, counted=32247).
Fix<y>? yes
Free blocks count wrong for group #644 (32254, counted=29365).
Fix<y>? yes
Free blocks count wrong for group #657 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #676 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #721 (32254, counted=32253).
Fix<y>? yes
Free blocks count wrong for group #728 (32254, counted=32180).
Fix<y>? yes
Free blocks count wrong (24722890, counted=24418360).
Fix<y>? yes
Free inodes count wrong for group #20 (8192, counted=8191).
Fix<y>? yes
Directories count wrong for group #20 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #21 (8192, counted=7157).
Fix<y>? yes
Directories count wrong for group #21 (0, counted=40).
Fix<y>? yes
Free inodes count wrong for group #22 (8192, counted=7089).
Fix<y>? yes
Directories count wrong for group #22 (0, counted=129).
Fix<y>? yes
Free inodes count wrong for group #23 (8192, counted=7245).
Fix<y>? yes
Directories count wrong for group #23 (0, counted=127).
Fix<y>? yes
Free inodes count wrong for group #24 (8192, counted=7619).
Fix<y>? yes
Directories count wrong for group #24 (0, counted=131).
Fix<y>? yes
Free inodes count wrong for group #25 (8192, counted=7852).
Fix<y>? yes
Directories count wrong for group #25 (0, counted=96).
Fix<y>? yes
Free inodes count wrong for group #26 (8192, counted=8035).
Fix<y>? yes
Directories count wrong for group #26 (0, counted=39).
Fix<y>? yes
Free inodes count wrong for group #35 (8192, counted=8190).
Fix<y>? yes
Directories count wrong for group #35 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #46 (8192, counted=7976).
Fix<y>? yes
Directories count wrong for group #46 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #178 (8192, counted=6860).
Fix<y>? yes
Directories count wrong for group #178 (0, counted=151).
Fix<y>? yes
Free inodes count wrong for group #179 (8192, counted=7712).
Fix<y>? yes
Directories count wrong for group #179 (0, counted=75).
Fix<y>? yes
Free inodes count wrong for group #180 (8192, counted=6352).
Fix<y>? yes
Directories count wrong for group #180 (0, counted=30).
Fix<y>? yes
Free inodes count wrong for group #181 (8192, counted=7594).
Fix<y>? yes
Directories count wrong for group #181 (0, counted=4).
Fix<y>? yes
Free inodes count wrong for group #182 (8192, counted=7553).
Fix<y>? yes
Directories count wrong for group #182 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #224 (8192, counted=6264).
Fix<y>? yes
Directories count wrong for group #224 (0, counted=196).
Fix<y>? yes
Free inodes count wrong for group #238 (8192, counted=8191).
Fix<y>? yes
Directories count wrong for group #238 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #257 (8192, counted=8176).
Fix<y>? yes
Directories count wrong for group #257 (0, counted=11).
Fix<y>? yes
Free inodes count wrong for group #269 (8192, counted=6506).
Fix<y>? yes
Directories count wrong for group #269 (0, counted=32).
Fix<y>? yes
Free inodes count wrong for group #271 (8192, counted=5948).
Fix<y>? yes
Directories count wrong for group #271 (0, counted=87).
Fix<y>? yes
Free inodes count wrong for group #272 (8192, counted=6007).
Fix<y>? yes
Directories count wrong for group #272 (0, counted=299).
Fix<y>? yes
Free inodes count wrong for group #273 (8192, counted=5619).
Fix<y>? yes
Directories count wrong for group #273 (0, counted=209).
Fix<y>? yes
Free inodes count wrong for group #274 (8192, counted=4830).
Fix<y>? yes
Directories count wrong for group #274 (0, counted=231).
Fix<y>? yes
Free inodes count wrong for group #275 (8192, counted=4567).
Fix<y>? yes

End of Part 2.

Thanks,
IceProx
 
Code:
Directories count wrong for group #275 (0, counted=80).
Fix<y>? yes
Free inodes count wrong for group #276 (8192, counted=6250).
Fix<y>? yes
Directories count wrong for group #276 (0, counted=76).
Fix<y>? yes
Free inodes count wrong for group #277 (8192, counted=6645).
Fix<y>? yes
Directories count wrong for group #277 (0, counted=185).
Fix<y>? yes
Free inodes count wrong for group #278 (8192, counted=6067).
Fix<y>? yes
Directories count wrong for group #278 (0, counted=291).
Fix<y>? yes
Free inodes count wrong for group #279 (8192, counted=5271).
Fix<y>? yes
Directories count wrong for group #279 (0, counted=65).
Fix<y>? yes
Free inodes count wrong for group #280 (8192, counted=5867).
Fix<y>? yes
Directories count wrong for group #280 (0, counted=131).
Fix<y>? yes
Free inodes count wrong for group #281 (8192, counted=6345).
Fix<y>? yes
Directories count wrong for group #281 (0, counted=226).
Fix<y>? yes
Free inodes count wrong for group #282 (8192, counted=6754).
Fix<y>? yes
Directories count wrong for group #282 (0, counted=267).
Fix<y>? yes
Free inodes count wrong for group #283 (8192, counted=6691).
Fix<y>? yes
Directories count wrong for group #283 (0, counted=238).
Fix<y>? yes
Free inodes count wrong for group #284 (8192, counted=6621).
Fix<y>? yes
Directories count wrong for group #284 (0, counted=517).
Fix<y>? yes
Free inodes count wrong for group #285 (8192, counted=5979).
Fix<y>? yes
Directories count wrong for group #285 (0, counted=158).
Fix<y>? yes
Free inodes count wrong for group #286 (8192, counted=8169).
Fix<y>? yes
Directories count wrong for group #286 (0, counted=5).
Fix<y>? yes
Free inodes count wrong for group #339 (8192, counted=8191).
Fix<y>? yes
Directories count wrong for group #339 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #358 (8192, counted=8189).
Fix<y>? yes
Directories count wrong for group #358 (0, counted=3).
Fix<y>? yes
Free inodes count wrong for group #369 (8192, counted=7633).
Fix<y>? yes
Directories count wrong for group #369 (0, counted=7).
Fix<y>? yes
Free inodes count wrong for group #393 (8192, counted=5604).
Fix<y>? yes
Directories count wrong for group #393 (0, counted=7).
Fix<y>? yes
Free inodes count wrong for group #394 (8192, counted=5374).
Fix<y>? yes
Directories count wrong for group #394 (0, counted=2).
Fix<y>? yes
Free inodes count wrong for group #445 (8192, counted=8174).
Fix<y>? yes
Directories count wrong for group #445 (0, counted=3).
Fix<y>? yes
Free inodes count wrong for group #476 (8192, counted=8186).
Fix<y>? yes
Directories count wrong for group #476 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #644 (8192, counted=8056).
Fix<y>? yes
Directories count wrong for group #644 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #657 (8192, counted=8191).
Fix<y>? yes
Directories count wrong for group #657 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #676 (8192, counted=8191).
Fix<y>? yes
Directories count wrong for group #676 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #721 (8192, counted=8191).
Fix<y>? yes
Directories count wrong for group #721 (0, counted=1).
Fix<y>? yes
Free inodes count wrong for group #728 (8192, counted=8185).
Fix<y>? yes
Directories count wrong for group #728 (0, counted=3).
Fix<y>? yes
Free inodes count wrong (6291445, counted=6238970).
Fix<y>? yes
/dev/mapper/pve_old-root: ***** FILE SYSTEM WAS MODIFIED *****

       52486 inodes used (0.83%, out of 6291456)
         837 non-contiguous files (1.6%)
          94 non-contiguous directories (0.2%)
             # of inodes with ind/dind/tind blocks: 2949/17/0
      747464 blocks used (2.97%, out of 25165824)
           0 bad blocks
           1 large file

       38191 regular files
        4164 directories
         855 character device files
        4530 block device files
           3 fifos
           8 links
        4707 symbolic links (4530 fast symbolic links)
          27 sockets
------------
       52485 files
root@moxy002:~#
As you can see from above, it shows it is reporting (for example) 0 directories but 3 counted. But, even though I selected yes to fix each issue and it shows the filesystem was modified, if I run the same command again I get the same errors reported.

I have a feeling (hope) that if fsck was repairing these errors then my empty /etc/pve would be populated again. I also tried extundelete and get an error leading back to the fsck issue referring to inodes errors.

I hope someone has a magic bullet to help me out on this. Otherwise, I will have lots of trial and error to get things going again.

End of Part 3.

Thanks,
IceProx
 
Hello IceProx,

It is not a problem with your filesystem gone wrong, /etc/pve is actually a mountpoint of Proxmox's own cluster file system, so it is empty on your root file system.

I think the data is all in /var/lib/pve-cluster/config.db in the form of a sqlite database. There is a sqlite3 tool on disk (/usr/bin/sqlite3). If you open the .db file with that tool, you can use .dump and .output to get at the data in textual form.

Or:
Code:
select * from tree;

Also see here for recovery: https://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)
 
Last edited:
Hey, any chance you did not fail on the update instructions, but got caught by windows 10 update ?
check
https://forum.proxmox.com/threads/do-not-use-suse-virtio-drivers-from-windows-update.25094/

if that vm uses either scsi or virtio and the windows update happens it replaces the working drivers for your drive with horribly broken ones.

On a fresh install of windows 10 it happens during the install process as well.

If so, there are instructions in the thread above on how to temp-fix it.
 
Hello IceProx,

It is not a problem with your filesystem gone wrong, /etc/pve is actually a mountpoint of Proxmox's own cluster file system, so it is empty on your root file system.

I think the data is all in /var/lib/pve-cluster/config.db in the form of a sqlite database. There is a sqlite3 tool on disk (/usr/bin/sqlite3). If you open the .db file with that tool, you can use .dump and .output to get at the data in textual form.

Or:
Code:
select * from tree;

Also see here for recovery: https://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)

windinternet - you are my hero! I saw your response this morning and have been working on it for most of the day but your instructions got all of my configs back. Thank you, thank you, thank you!

I ran into another issue along the way that help me up. Even though it was the same hardware, the eth ports were not mapped the same. When I setup my bridge my server connection would crash as well and I could not get my iSCSI connect to come online. Anyway, all is fixed and everything is back up and running.

I would like to give you a donation to your coffee, beer, wine, etc fund. Please send me a link.

Thanks again,
IceProx