[SOLVED] All vz directories lost after upgrade. Server not restarting

[jctual]

I have just updated proxmox 4 with latest upgrades.
After reboot the server did not restart.
After connecting in rescue mode, I see that all the vz directories (/var/lib/vz) have disappeared or at least are not visible.

/etc/fstab is empty.
/dev/pve/data is still there.

Should I remount the partition /var/lib/vz ? How ?


Thanks for your help.
My server is down and it's critical for me.

 

[oguz]

Most likely /var/lib/vz (and probably /etc/pve) is not mounted because you're having problems with systemd starting. Check here for possible similar problem:
https://forum.proxmox.com/threads/upgrade-to-kernel-4-15-18-12-pve-goes-boom.52564/

How did you perform the upgrade from PVE 4 to 5? Because recently the upgrade instructions have been changed to avoid a problem between zfs and systemd. See here:
https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0

edit: correction

 

[Stoiko Ivanov]

* Please check the journal after the boot for errors (there most probably is one causing the whole boot-process to not run smoothly)
* Also take a look at the console/screen output of the node
* if your local storage is on LVM also check the output of `pvs`, `vgs`, `lvs`

* your `/etc/fstab` should not be empty (unless you're only in initramfs - that could explain the empty /etc/fstab)

If this does not resolve the problem please post the log and output of the commands

hope this helps!


EDIT: just checked that fstab is empty in initramfs

 

[jctual]

Thanks for your replies.
My server is a remote server on OVH provider.
Since my server was not booting, OVH gave me a connection to a rescue OS on my server, which is not my proxmox OS.
This is why, I do not see the proxmox directories.
I have to find a way to mount the proxmox partitions on this OS.
I need at least to save my VMs.

 

[Stoiko Ivanov]

hm - check the output of `pvs`, `vgs`, `lvs`
it might be that you need to activate the LVM before `vgchange -ay` usually gets this done
* afterwards you should be able to see and mount your volumes (/, /var/lib/vz) or dd your disk-images

 

[jctual]

I got access to proxmox data with vgchange and mount /dev/pve/ data /temp.
I can copy my VMs;

Now I need to explain why proxmox is not booting after the upgrade.

The update button of the GUI did probably an upgrade from 4.x to 5.x.
Since it is a remote server I can't see what going on during reboot.

I can try to recover grub following hese instructions
https://pve.proxmox.com/wiki/Recover_From_Grub_Failure

Any better idea ?

 

[jctual]

I tried to recover grub with the above procedure.
It does not work.
/dev/pve/root does not exist. I have only /dev/pve/data.

here the result of fdisk
Device Start End Sectors Size Type
/dev/sda1 40 2048 2009 1004.5K BIOS boot
/dev/sda2 4096 40962047 40957952 19.5G Linux RAID
/dev/sda3 40962048 43057151 2095104 1023M Linux swap
/dev/sda4 43057152 3907018751 3863961600 1.8T Linux RAID

Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BC8A8549-EA8A-4239-AC1B-E47829F323B4

Device Start End Sectors Size Type
/dev/sdb1 40 2048 2009 1004.5K BIOS boot
/dev/sdb2 4096 40962047 40957952 19.5G Linux RAID
/dev/sdb3 40962048 43057151 2095104 1023M Linux swap
/dev/sdb4 43057152 3907018751 3863961600 1.8T Linux RAID

Disk /dev/md4: 1.8 TiB, 1978348273664 bytes, 3863961472 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/md2: 19.5 GiB, 20970405888 bytes, 40957824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/pve-data: 1.8 TiB, 1974049177600 bytes, 3855564800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

and the result of lvdisplay
--- Logical volume ---
LV Path /dev/pve/data
LV Name data
VG Name pve
LV UUID sBKqKE-0mY1-6dwK-ZcHx-nHh5-ucQo-7M1AFS
LV Write Access read/write
LV Creation host, time rescue.ovh.net, 2016-06-24 18:40:33 +0200
LV Status available
# open 1
LV Size 1.80 TiB
Current LE 470650
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0

How can I reinstall grub from the rescue OS ?

 

[Stoiko Ivanov]

please post the output of:
* `lsblk`
* `cat /proc/mdadm` (on a side-note - proxmox does not support software-raid)
* `pvs`
* `vgs`
* `lvs`

what is on /dev/md2 and /dev/md4? (maybe one of those contains your root-fs directly?)

 

[jctual]

root@rescue:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 1.8T 0 disk
├─sdb4 8:20 0 1.8T 0 part
│ └─md4 9:4 0 1.8T 0 raid1
│ └─pve-data 252:0 0 1.8T 0 lvm /root/proxmox
├─sdb2 8:18 0 19.5G 0 part
│ └─md2 9:2 0 19.5G 0 raid1
├─sdb3 8:19 0 1023M 0 part
└─sdb1 8:17 0 1004.5K 0 part
sda 8:0 0 1.8T 0 disk
├─sda4 8:4 0 1.8T 0 part
│ └─md4 9:4 0 1.8T 0 raid1
│ └─pve-data 252:0 0 1.8T 0 lvm /root/proxmox
├─sda2 8:2 0 19.5G 0 part
│ └─md2 9:2 0 19.5G 0 raid1
├─sda3 8:3 0 1023M 0 part
└─sda1 8:1 0 1004.5K 0 part


root@rescue:~# cat /proc/mdadm
cat: /proc/mdadm: No such file or directory

root@rescue:~# pvs
PV VG Fmt Attr PSize PFree
/dev/md4 pve lvm2 a-- 1.80t 4.00g

root@rescue:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 1 0 wz--n- 1.80t 4.00g


root@rescue:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve -wi-ao---- 1.80t


I don't know what is on /dev/md2 and /dev/md4.

When I try mount /dev§md4 temp
it fails
root@rescue:~# mount /dev/md4 temp
mount: unknown filesystem type 'LVM2_member'

 

[Stoiko Ivanov]

root@rescue:~# cat /proc/mdadm
cat: /proc/mdadm: No such file or directory

sorry typo - `cat /proc/mdstat`

md4 seems like the physical volume (PV) for the volume group (VG) 'pve' containing your data lv - what is saved there? (output of `ls -la`)

the wiki-article assumes a standard PVE-disk layout from our installer (which you don't have)

can you mount /dev/md2?

 

[jctual]

yes I can mount md2

root@rescue:~# mount /dev/md2 temp
root@rescue:~# cd temp
root@rescue:~/temp# ls
bin dev home lib64 media proc run srv tmp var
boot etc lib lost+found opt root sbin sys usr vz
root@rescue:~/temp#

can I rescue grub there ?

 

[jctual]

here the result of mdstat

root@rescue:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md2 : active raid1 sda2[0] sdb2[1]
20478912 blocks [2/2] [UU]

md4 : active raid1 sda4[0] sdb4[1]
1931980736 blocks [2/2] [UU]
bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

 

[jctual]

Could you confirm that the following commands are correct to re-install grub on my configuration.

I am not sure of the command
mount /dev/md2/root /media/RESCUE
should it be : mount /dev/md2 /media/RESCUE ???

mkdir /media/RESCUE
mount /dev/md2/root /media/RESCUE/
mount /dev/sda1 /media/RESCUE/boot
mount -t proc proc /media/RESCUE/proc
mount -t sysfs sys /media/RESCUE/sys
mount -o bind /dev /media/RESCUE/dev

Chroot into your proxmox install.

chroot /media/RESCUE

Then upgrade grub and install it.

grub-upgrade
grub-install /dev/sda

 

[jctual]

I have just reinstalled grub with the previous procedure.
Now it works again.

Thanks for your help.

 

[Stoiko Ivanov]

You're welcome - Glad it worked out! - please mark the thread as SOLVED so that others know what to expect. Thanks!