Hello experts,
I am experiencing a serious issue which seems to be related to a power failure in my village several days ago.
After starting the PVE server again I got the following errors and the boot process was stuck at the (initramfs) prompt. I have searched and tried many suggestions from different articles but I am still stuck and don't know what to do anymore.
I am a Proxmox newbie and have been using it for several weeks without major issues.
I will try to provide some information below:
I am using the following hardware for the PVE host:
- ProLiant ML350 Gen10 server with 256Gb of RAM and 2x Xeon E5-2680 v3 @ 2.50Ghz
- HPE Smart Array P440ar controller
- 2x 300Gb RAID 1 -> used for boot and system
- 4x 900Gb RAID 5 -> used for storage of VMs
I have resized the root filesystem in the past to 60Gib to enable storage of iso images. Default PVE installation had 10Gib. I extend the LV at that time using:
lvresize --size 60G --resizefs /dev/mapper/pve-root
and that worked all fine. To my recollection I did reboot the PVE host several times after the increase of the root LV & filesystem and this work all fine. But, I am not 100% sure the current issue is caused by the power failure or the increase of root filesystem
I am not a Linux LVM expert so Google is my best friend in this.
During boot of PVE host after the power failure:
/sbin/fsck.xfs: XFS file system.
mount: mounting /dev/mapper/pve-root on /root failed: Input/output error.
Failed to mount /dev/mapper/pve-root as root file system.
Have create a GParted USB keydrive which I used to boot the server. I enable root login and ssh access to easily copy/paste command and output
Trying to mount manually:
root@debian:/# mount /dev/mapper/pve-root /mnt/pve-root
mount: /mnt/pve-root: can't read superblock on /dev/mapper/pve-root.
Tried to perform xfs_repair:
root@debian:/# xfs_repair /dev/mapper/pve-root
Phase 1 - find and verify superblock...
superblock read failed, offset 64424509440, size 131072, ag 96, rval 0
fatal error -- Success
Viewed last 10 entries of dmesg
root@debian:/# dmesg | tail -n 10
[ 7742.206610] XFS (dm-1): last sector read failed
[ 8194.083402] attempt to access beyond end of device
[ 8194.083407] dm-1: rw=4096, want=126042112, limit=125829120
[ 8194.083411] XFS (dm-1): last sector read failed
[ 8213.161167] attempt to access beyond end of device
[ 8213.161171] dm-1: rw=4096, want=126042112, limit=125829120
[ 8213.161175] XFS (dm-1): last sector read failed
[ 9542.671614] attempt to access beyond end of device
[ 9542.671616] dm-1: rw=4096, want=126042112, limit=125829120
[ 9542.671619] XFS (dm-1): last sector read failed
Based on different articles I have gathered the following information:
root@debian:/# blkid (output filtered due to character limit of post)
/dev/sda2: UUID="5E89-E59E" TYPE="vfat" PARTUUID="ea84b74e-3bb8-4504-a066-3d3ca77fc7a5"
/dev/sda3: UUID="ApYL4D-OeeH-mIvy-aHGQ-pSUC-20gI-i39bAD" TYPE="LVM2_member" PARTUUID="a9983011-cbb2-4dc7-85ae-18b9c53f96f5"
/dev/sdb: UUID="rbuLqF-z0gL-3lzl-Xuzm-U5bD-R1NE-ye3800" TYPE="LVM2_member"
/dev/sdc1: LABEL_FATBOOT="UNTITLED" LABEL="UNTITLED" UUID="4125-1F0A" TYPE="vfat"
/dev/loop0: TYPE="squashfs"
/dev/mapper/pve-swap: UUID="b3562cd7-3759-4990-a521-84fe34e2140d" TYPE="swap"
/dev/mapper/pve-root: UUID="b05cc3f5-ac2b-4df5-8f0b-4bcf931d3696" TYPE="xfs"
/dev/sda1: PARTUUID="3eb98dfb-90ac-4fe7-a212-17c828ab3659"
root@debian:/# pvscan
PV /dev/sdb VG datastore2 lvm2 [<2.46 TiB / 512.00 MiB free]
PV /dev/sda3 VG pve lvm2 [278.86 GiB / 0 free]
Total: 2 [<2.73 TiB] / in use: 2 [<2.73 TiB] / in no VG: 0 [0 ]
root@debian:/# vgscan
Found volume group "datastore2" using metadata type lvm2
Found volume group "pve" using metadata type lvm2
root@debian:/# lvscan
ACTIVE '/dev/datastore2/datastore2' [2.42 TiB] inherit
ACTIVE '/dev/datastore2/vm-100-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-100-disk-1' [150.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9000-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9999-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9100-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9100-disk-1' [100.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-8000-disk-0' [120.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-1' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-4' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-1' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-1' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-4' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-1' [30.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-4' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-1' [30.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-1' [30.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9102-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9102-disk-1' [100.00 GiB] inherit
ACTIVE '/dev/pve/swap' [8.00 GiB] inherit
ACTIVE '/dev/pve/root' [60.00 GiB] inherit
ACTIVE '/dev/pve/datastore1' [210.65 GiB] inherit
ACTIVE '/dev/pve/vm-8001-disk-0' [10.00 GiB] inherit
root@debian:/# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 278.86g 0
/dev/sdb datastore2 lvm2 a-- <2.46t 512.00m
root@debian:/# vgs
VG #PV #LV #SN Attr VSize VFree
datastore2 1 37 0 wz--n- <2.46t 512.00m
pve 1 4 0 wz--n- 278.86g 0
root@debian:/# lvs pve
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
datastore1 pve twi-aotz-- 210.65g 0.00 10.22
root pve -wi-a----- 60.00g
swap pve -wi-a----- 8.00g
vm-8001-disk-0 pve Vwi-a-tz-- 10.00g datastore1 0.00
root@debian:/# lvdisplay /dev/mapper/pve-root
--- Logical volume ---
LV Path /dev/pve/root
LV Name root
VG Name pve
LV UUID iH2fdK-EZTz-0raa-GdPn-D8Ji-CQEk-JVmS2J
LV Write Access read/write
LV Creation host, time proxmox, 2020-04-04 23:19:04 +0000
LV Status available
# open 0
LV Size 60.00 GiB
Current LE 15360
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 254:1
Output of fdisk
root@debian:/# fdisk -l /dev/sda
Disk /dev/sda: 279.37 GiB, 299966445568 bytes, 585871964 sectors
Disk model: LOGICAL VOLUME
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disklabel type: gpt
Disk identifier: 060C87EF-3DE1-4E9C-8755-E2CB47C0ECD3
Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 1050623 1048576 512M EFI System
/dev/sda3 1050624 585871930 584821307 278.9G Linux LVM
Partition 1 does not start on physical sector boundary.
Output of dumpe2fs
root@debian:/# dumpe2fs /dev/mapper/pve-root | grep superblock
dumpe2fs 1.45.5 (07-Jan-2020)
dumpe2fs: Bad magic number in super-block while trying to open /dev/mapper/pve-root
Couldn't find valid filesystem superblock.
I apologize for the overwhelming information but I understand this information is helpful to determine the possible cause.
Hope someone can provide some suggestions on how to fix this.
Many thanks in advance,
Rob
I am experiencing a serious issue which seems to be related to a power failure in my village several days ago.
After starting the PVE server again I got the following errors and the boot process was stuck at the (initramfs) prompt. I have searched and tried many suggestions from different articles but I am still stuck and don't know what to do anymore.
I am a Proxmox newbie and have been using it for several weeks without major issues.
I will try to provide some information below:
I am using the following hardware for the PVE host:
- ProLiant ML350 Gen10 server with 256Gb of RAM and 2x Xeon E5-2680 v3 @ 2.50Ghz
- HPE Smart Array P440ar controller
- 2x 300Gb RAID 1 -> used for boot and system
- 4x 900Gb RAID 5 -> used for storage of VMs
I have resized the root filesystem in the past to 60Gib to enable storage of iso images. Default PVE installation had 10Gib. I extend the LV at that time using:
lvresize --size 60G --resizefs /dev/mapper/pve-root
and that worked all fine. To my recollection I did reboot the PVE host several times after the increase of the root LV & filesystem and this work all fine. But, I am not 100% sure the current issue is caused by the power failure or the increase of root filesystem
I am not a Linux LVM expert so Google is my best friend in this.
During boot of PVE host after the power failure:
/sbin/fsck.xfs: XFS file system.
mount: mounting /dev/mapper/pve-root on /root failed: Input/output error.
Failed to mount /dev/mapper/pve-root as root file system.
Have create a GParted USB keydrive which I used to boot the server. I enable root login and ssh access to easily copy/paste command and output
Trying to mount manually:
root@debian:/# mount /dev/mapper/pve-root /mnt/pve-root
mount: /mnt/pve-root: can't read superblock on /dev/mapper/pve-root.
Tried to perform xfs_repair:
root@debian:/# xfs_repair /dev/mapper/pve-root
Phase 1 - find and verify superblock...
superblock read failed, offset 64424509440, size 131072, ag 96, rval 0
fatal error -- Success
Viewed last 10 entries of dmesg
root@debian:/# dmesg | tail -n 10
[ 7742.206610] XFS (dm-1): last sector read failed
[ 8194.083402] attempt to access beyond end of device
[ 8194.083407] dm-1: rw=4096, want=126042112, limit=125829120
[ 8194.083411] XFS (dm-1): last sector read failed
[ 8213.161167] attempt to access beyond end of device
[ 8213.161171] dm-1: rw=4096, want=126042112, limit=125829120
[ 8213.161175] XFS (dm-1): last sector read failed
[ 9542.671614] attempt to access beyond end of device
[ 9542.671616] dm-1: rw=4096, want=126042112, limit=125829120
[ 9542.671619] XFS (dm-1): last sector read failed
Based on different articles I have gathered the following information:
root@debian:/# blkid (output filtered due to character limit of post)
/dev/sda2: UUID="5E89-E59E" TYPE="vfat" PARTUUID="ea84b74e-3bb8-4504-a066-3d3ca77fc7a5"
/dev/sda3: UUID="ApYL4D-OeeH-mIvy-aHGQ-pSUC-20gI-i39bAD" TYPE="LVM2_member" PARTUUID="a9983011-cbb2-4dc7-85ae-18b9c53f96f5"
/dev/sdb: UUID="rbuLqF-z0gL-3lzl-Xuzm-U5bD-R1NE-ye3800" TYPE="LVM2_member"
/dev/sdc1: LABEL_FATBOOT="UNTITLED" LABEL="UNTITLED" UUID="4125-1F0A" TYPE="vfat"
/dev/loop0: TYPE="squashfs"
/dev/mapper/pve-swap: UUID="b3562cd7-3759-4990-a521-84fe34e2140d" TYPE="swap"
/dev/mapper/pve-root: UUID="b05cc3f5-ac2b-4df5-8f0b-4bcf931d3696" TYPE="xfs"
/dev/sda1: PARTUUID="3eb98dfb-90ac-4fe7-a212-17c828ab3659"
root@debian:/# pvscan
PV /dev/sdb VG datastore2 lvm2 [<2.46 TiB / 512.00 MiB free]
PV /dev/sda3 VG pve lvm2 [278.86 GiB / 0 free]
Total: 2 [<2.73 TiB] / in use: 2 [<2.73 TiB] / in no VG: 0 [0 ]
root@debian:/# vgscan
Found volume group "datastore2" using metadata type lvm2
Found volume group "pve" using metadata type lvm2
root@debian:/# lvscan
ACTIVE '/dev/datastore2/datastore2' [2.42 TiB] inherit
ACTIVE '/dev/datastore2/vm-100-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-100-disk-1' [150.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9000-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9999-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9100-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9100-disk-1' [100.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-8000-disk-0' [120.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-1' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9001-disk-4' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-1' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9005-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-1' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9006-disk-4' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-1' [30.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9002-disk-4' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-1' [30.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9003-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-1' [30.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-2' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9004-disk-3' [20.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9102-disk-0' [50.00 GiB] inherit
ACTIVE '/dev/datastore2/vm-9102-disk-1' [100.00 GiB] inherit
ACTIVE '/dev/pve/swap' [8.00 GiB] inherit
ACTIVE '/dev/pve/root' [60.00 GiB] inherit
ACTIVE '/dev/pve/datastore1' [210.65 GiB] inherit
ACTIVE '/dev/pve/vm-8001-disk-0' [10.00 GiB] inherit
root@debian:/# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 278.86g 0
/dev/sdb datastore2 lvm2 a-- <2.46t 512.00m
root@debian:/# vgs
VG #PV #LV #SN Attr VSize VFree
datastore2 1 37 0 wz--n- <2.46t 512.00m
pve 1 4 0 wz--n- 278.86g 0
root@debian:/# lvs pve
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
datastore1 pve twi-aotz-- 210.65g 0.00 10.22
root pve -wi-a----- 60.00g
swap pve -wi-a----- 8.00g
vm-8001-disk-0 pve Vwi-a-tz-- 10.00g datastore1 0.00
root@debian:/# lvdisplay /dev/mapper/pve-root
--- Logical volume ---
LV Path /dev/pve/root
LV Name root
VG Name pve
LV UUID iH2fdK-EZTz-0raa-GdPn-D8Ji-CQEk-JVmS2J
LV Write Access read/write
LV Creation host, time proxmox, 2020-04-04 23:19:04 +0000
LV Status available
# open 0
LV Size 60.00 GiB
Current LE 15360
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 254:1
Output of fdisk
root@debian:/# fdisk -l /dev/sda
Disk /dev/sda: 279.37 GiB, 299966445568 bytes, 585871964 sectors
Disk model: LOGICAL VOLUME
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disklabel type: gpt
Disk identifier: 060C87EF-3DE1-4E9C-8755-E2CB47C0ECD3
Device Start End Sectors Size Type
/dev/sda1 34 2047 2014 1007K BIOS boot
/dev/sda2 2048 1050623 1048576 512M EFI System
/dev/sda3 1050624 585871930 584821307 278.9G Linux LVM
Partition 1 does not start on physical sector boundary.
Output of dumpe2fs
root@debian:/# dumpe2fs /dev/mapper/pve-root | grep superblock
dumpe2fs 1.45.5 (07-Jan-2020)
dumpe2fs: Bad magic number in super-block while trying to open /dev/mapper/pve-root
Couldn't find valid filesystem superblock.
I apologize for the overwhelming information but I understand this information is helpful to determine the possible cause.
Hope someone can provide some suggestions on how to fix this.
Many thanks in advance,
Rob
