Backups started failing after resizing LVM

D

deranjer

Active Member
Dec 28, 2010
50
1
28
I resized my LVM a little bit ago... and now my backups are failing..it appears as if the LVM expansion ran into issues, although everything appears to work. Anyone have any idea how to solve?



vzdump 105 103 --quiet 1 --mailto xxxxx@xxxx.com --mode snapshot --compress gzip --storage Backups

103: Mar 02 00:00:02 INFO: Starting Backup of VM 103 (qemu)
103: Mar 02 00:00:02 INFO: status = running
103: Mar 02 00:00:03 INFO: backup mode: snapshot103: Mar 02 00:00:03 INFO: ionice priority: 7103: Mar 02 00:00:03 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:03 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
103: Mar 02 00:00:04 INFO: FATAL: Could not load /lib/modules/2.6.32-14-pve/modules.dep: No such file or directory
103: Mar 02 00:00:04 INFO: /sbin/modprobe failed: 1
103: Mar 02 00:00:04 INFO: Can't process LV vzsnap-proxmox-0: snapshot target support missing from kernel?
103: Mar 02 00:00:04 INFO: Failed to suspend origin data
103: Mar 02 00:00:06 ERROR: Backup of VM 103 failed - command 'lvcreate --size 1024M --snapshot --name 'vzsnap-proxmox-0' '/dev/pve/data'' failed: exit code 5
 
Please post the output of pvscan and lvscan.

Did you extend or reduce the LV? Which command did you use? If you reduced the LV withouth passing the -r option to lvresize, you may have corrupted the filesystem.
 
malakatronis said:
Please post the output of pvscan and lvscan.

Did you extend or reduce the LV? Which command did you use? If you reduced the LV withouth passing the -r option to lvresize, you may have corrupted the filesystem.
Click to expand...


I extended the LV /dev/pve/data, if I remember it was a simple pvcreate, then vgextend, then lvextend.

Code: 
root@proxmox:~# lvscan  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  ACTIVE            '/dev/pve/swap' [5.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [18.50 GiB] inherit
  ACTIVE            '/dev/pve/data' [1.11 TiB] inherit

Code: 
root@proxmox:~# pvscan
  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  WARNING: Volume Group pve is not consistent
  PV /dev/sda2   VG pve   lvm2 [74.03 GiB / 50.53 GiB free]
  PV /dev/sdb1   VG pve   lvm2 [931.51 GiB / 31.51 GiB free]
  PV /dev/sdd2   VG pve   lvm2 [931.87 GiB / 698.90 GiB free]
  Total: 3 [1.89 TiB] / in use: 3 [1.89 TiB] / in no VG: 0 [0   ]
 
Did you forget to resize the file system inside the volume? If this is the case then the restore might get confused and try to write outside the volume.
 
OK, how about fdisk -l, pvdisplay, vgdisplay and lvdisplay?

Maybe you can find the command you entered inside /root/.bash_history.
 
malakatronis said:
OK, how about fdisk -l, pvdisplay, vgdisplay and lvdisplay?

Maybe you can find the command you entered inside /root/.bash_history.
Click to expand...


Too far back for bash history unfortunately... and I honestly can't remember if I resized the partition... which usually means I didn't :(

Code: 
root@proxmox:~# fdisk -l

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00061d9b


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          66      523264   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              66        9730    77626368   8e  Linux LVM


Disk /dev/sdb: 1000.2 GB, 1000203804160 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x610d31db


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      121601   976760001   83  Linux


Disk /dev/sdc: 500.1 GB, 500106780160 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x44fdfe06


   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       60801   488384001   83  Linux


Disk /dev/dm-0: 19.9 GB, 19864223744 bytes
255 heads, 63 sectors/track, 2415 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/dm-0 doesn't contain a valid partition table


WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


Note: sector size is 4096 (not 512)


Disk /dev/sdd: 3000.6 GB, 3000592977920 bytes
255 heads, 63 sectors/track, 45600 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1       45601  2930266576   ee  GPT
Partition 1 does not start on physical sector boundary.


Disk /dev/dm-1: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/dm-1 doesn't contain a valid partition table
Note: sector size is 4096 (not 512)


Disk /dev/dm-2: 1216.5 GB, 1216515932160 bytes
255 heads, 63 sectors/track, 18487 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/dm-2 doesn't contain a valid partition table

Code: 
root@proxmox:~# pvdisplay  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  WARNING: Volume Group pve is not consistent
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               pve
  PV Size               74.03 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              18951
  Free PE               12935
  Allocated PE          6016
  PV UUID               cyWdzo-HFei-HHwF-5446-j6A9-J3jG-DMFHmI


  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               pve
  PV Size               931.51 GiB / not usable 3.19 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              238466
  Free PE               8066
  Allocated PE          230400
  PV UUID               Pbyrpl-oYOq-ZzBq-8wW6-k31d-qckK-SU5LF3


  --- Physical volume ---
  PV Name               /dev/sdd2
  VG Name               pve
  PV Size               931.87 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              238559
  Free PE               178919
  Allocated PE          59640
  PV UUID               Fg7BYq-Zpvl-fCHf-ttAo-XzeI-PfaZ-OANCZf

Code: 
 Incorrect metadata area header checksum on /dev/sdb1 at offset 4096  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  --- Volume group ---
  VG Name               pve
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  113
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               1.89 TiB
  PE Size               4.00 MiB
  Total PE              495976
  Alloc PE / Size       296056 / 1.13 TiB
  Free  PE / Size       199920 / 780.94 GiB
  VG UUID               ntKi9a-8ujb-YaW9-MBMq-bSHA-IZce-iRyaJ0

Code: 
 root@proxmox:~# lvdisplay
Incorrect metadata area header checksum on /dev/sdb1 at offset 4096  Incorrect metadata area header checksum on /dev/sdb1 at offset 4096
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                fGB0zs-7WIQ-3BN8-eL9a-7YiE-5dDY-d70viT
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1


  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                f0mVM9-2A1I-zV0W-Mepm-ibF7-W4G5-PWaDoT
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                18.50 GiB
  Current LE             4736
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0


  --- Logical volume ---
  LV Path                /dev/pve/data
  LV Name                data
  VG Name                pve
  LV UUID                6q19dx-4rRa-f7wJ-nPf7-HmeQ-91OB-dohVK6
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                1.11 TiB
  Current LE             290040
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
 
Note that partition table ID for /dev/sdb1 should be 8e (LVM) instead of 83 (Linux). Although one can have a perfectly healthy Physical Volume on top of a partition with ID=83, that's not very orthodox, so I tend to think that the partition /dev/sdb1 has been overwritten.

If it's just the partition table on /dev/sdb what has been overwritten, maybe you could try to restore a backup file of the volume group "pve" from /etc/lvm/archive. What's the output of pvck /dev/sdb1 ?

Are you able to boot from sysrescd and run fsck /dev/pve/root and /dev/pve/data?
 
Wait a moment...

Perhaps you issued a pvcreate command on a partition that was already a PV member of VG /dev/pve, i.e. you mistyped 'pvcreate /dev/sdb1' instead of 'pvcreate /dev/sdd2'. (If a partition is a PV member of a VG, you mustn't issue a pvcreate again in order to extend any LV that sits on top of it.)

If that's the case, you just have to restore the metadata. Boot from sysrescd and follow this guide: https://access.redhat.com/knowledge...lume_Manager_Administration/mdatarecover.html
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back