[SOLVED] all vms down and lvm-thin not backupable

M

Mr.Gosh

Member
Jan 5, 2016
36
0
6
34
after this night all vms on one of our proxmox srevers are down and it seems that the lvm-thin has problems.
lvs prints a lot of warnings:
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 68719411200: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 68719468544: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4831830016: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4831830016: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 214748299264: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 214748356608: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 599549476864: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 599549534208: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 107374116864: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 107374174208: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 34359672832: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 34359730176: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1999844081664: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1999844139008: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 1999843033088: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 1999843090432: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 4096: Input/output error
Found duplicate PV YNYObiWn0kcRWriXVvryPvC15WSMRrie: using /dev/sdf1 not /dev/sdd1
Found duplicate PV WaNwRKhHcTVjSaXUwapF8xBiy9GmYcB6: using /dev/sdg1 not /dev/sde1
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin pool params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
Failed to parse thin params: Fail.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-a-tz-- 154.85g 0.00 0.42
root pve -wi-ao---- 59.50g
swap pve -wi-ao---- 8.00g
snap_vm-140-disk-1_AfterImport vg_2T Vri---tz-k 64.00g thin2tb vm-140-disk-1
snap_vm-140-disk-1_mysqlImport vg_2T Vri---tz-k 64.00g thin2tb vm-140-disk-1
thin2tb vg_2T twi---tzX- 1.82t
vm-140-disk-1 vg_2T Vwi-a-tz-- 64.00g thin2tb
vm-140-state-AfterImport vg_2T Vwi-a-tz-- 4.50g thin2tb
vm-140-state-mysqlImport vg_2T Vwi-a-tz-- 4.50g thin2tb
vm-148-disk-1 vg_2T Vwi-a-tz-- 200.00g thin2tb
vm-150-disk-1 vg_2T Vwi-a-tz-- 32.00g thin2tb
thin600gb vg_600G twi---tzX- 558.16g
vm-149-disk-1 vg_600G Vwi-a-tz-- 100.00g thin600gb

in the syslog there seems one harddisk out of these mirror raids to be defect:
Jul 1 01:01:46 pve01 kernel: [306820.300853] sd 4:1:0:0: attempting task abort! scmd(ffff88065b9f6b80)
Jul 1 01:01:46 pve01 kernel: [306820.300860] sd 4:1:0:0: [sdb] tag#3 CDB: Write(10) 2a 00 00 04 26 a0 00 00 48 00
Jul 1 01:01:46 pve01 kernel: [306820.300864] scsi target4:1:0: volume handle(0x004f), volume wwid(0x0451ba54dbc95d1a)
Jul 1 01:01:46 pve01 kernel: [306820.300866] sd 4:1:0:0: task abort: FAILED scmd(ffff88065b9f6b80)
Jul 1 01:01:46 pve01 kernel: [306820.300869] sd 4:1:0:0: attempting task abort! scmd(ffff880815871080)
Jul 1 01:01:46 pve01 kernel: [306820.300871] sd 4:1:0:0: [sdb] tag#2 CDB: Write(10) 2a 00 45 c8 10 00 00 00 08 00
Jul 1 01:01:46 pve01 kernel: [306820.300873] scsi target4:1:0: volume handle(0x004f), volume wwid(0x0451ba54dbc95d1a)
Jul 1 01:01:46 pve01 kernel: [306820.300876] sd 4:1:0:0: task abort: FAILED scmd(ffff880815871080)
Jul 1 01:01:46 pve01 kernel: [306820.300878] sd 4:1:0:0: attempting task abort! scmd(ffff88065b9f4600)
Jul 1 01:01:46 pve01 kernel: [306820.300881] sd 4:1:0:0: [sdb] tag#1 CDB: Write(10) 2a 00 00 04 88 d8 00 00 10 00
Jul 1 01:01:46 pve01 kernel: [306820.300883] scsi target4:1:0: volume handle(0x004f), volume wwid(0x0451ba54dbc95d1a)



how can i get a backup out of these machines?
 
Some more Info:
# pvs
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 68719411200: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 68719468544: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4831830016: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4831830016: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 214748299264: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 214748356608: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 599549476864: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 599549534208: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 107374116864: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 107374174208: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 34359672832: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 34359730176: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1999844081664: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1999844139008: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 1999843033088: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 1999843090432: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 4096: Input/output error
Found duplicate PV YNYObiWn0kcRWriXVvryPvC15WSMRrie: using /dev/sdf1 not /dev/sdd1
Found duplicate PV WaNwRKhHcTVjSaXUwapF8xBiy9GmYcB6: using /dev/sdg1 not /dev/sde1
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 238.35g 15.84g
/dev/sdf1 vg_2T lvm2 a-- 1.82t 0
/dev/sdg1 vg_600G lvm2 a-- 558.34g 0
 
# vgs
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 68719411200: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 214748299264: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 599549476864: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 107374116864: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 34359672832: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1999844081664: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 1999843033088: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 4096: Input/output error
Found duplicate PV YNYObiWn0kcRWriXVvryPvC15WSMRrie: using /dev/sdf1 not /dev/sdd1
Found duplicate PV WaNwRKhHcTVjSaXUwapF8xBiy9GmYcB6: using /dev/sdg1 not /dev/sde1
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 238.35g 15.84g
vg_2T 1 8 0 wz--n- 1.82t 0
vg_600G 1 2 0 wz--n- 558.34g 0
 
# lvs
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 68719411200: Input/output error
/dev/vg_2T/vm-140-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-mysqlImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4831772672: Input/output error
/dev/vg_2T/vm-140-state-AfterImport: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 214748299264: Input/output error
/dev/vg_2T/vm-148-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 599549476864: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 107374116864: Input/output error
/dev/vg_600G/vm-149-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 0: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 34359672832: Input/output error
/dev/vg_2T/vm-150-disk-1: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 1999844081664: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 1999843033088: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc1: read failed after 0 of 4096 at 4096: Input/output error
Found duplicate PV YNYObiWn0kcRWriXVvryPvC15WSMRrie: using /dev/sdf1 not /dev/sdd1
Found duplicate PV WaNwRKhHcTVjSaXUwapF8xBiy9GmYcB6: using /dev/sdg1 not /dev/sde1
Failed to parse thin pool params: Fail.
...
Failed to parse thin pool params: Fail.
Failed to parse thin params: Fail.
...
Failed to parse thin params: Fail.
Failed to parse thin pool params: Fail.
...
Failed to parse thin pool params: Fail.
Failed to parse thin params: Fail.
...
Failed to parse thin params: Fail.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-a-tz-- 154.85g 0.00 0.42
root pve -wi-ao---- 59.50g
swap pve -wi-ao---- 8.00g
snap_vm-140-disk-1_AfterImport vg_2T Vri---tz-k 64.00g thin2tb vm-140-disk-1
snap_vm-140-disk-1_mysqlImport vg_2T Vri---tz-k 64.00g thin2tb vm-140-disk-1
thin2tb vg_2T twi---tzX- 1.82t
vm-140-disk-1 vg_2T Vwi-a-tz-- 64.00g thin2tb
vm-140-state-AfterImport vg_2T Vwi-a-tz-- 4.50g thin2tb
vm-140-state-mysqlImport vg_2T Vwi-a-tz-- 4.50g thin2tb
vm-148-disk-1 vg_2T Vwi-a-tz-- 200.00g thin2tb
vm-150-disk-1 vg_2T Vwi-a-tz-- 32.00g thin2tb
thin600gb vg_600G twi---tzX- 558.16g
vm-149-disk-1 vg_600G Vwi-a-tz-- 100.00g thin600gb
 
Hey,

/proc/mdstat doesn't work here cause it's a hardware raid.
but there are two hardware raids and i poked around a bit to realize that the "not so important" 600gb mirror raid had the problems - so i removed that one and *tataaaa* the system is up and running again: memo to myself: "get rid of the hardware raids... ;-)

so for now this problem is "solved" but there stays this problem that a degraded hardware raid killed the availability of other raids... this is a problem on our side...
 
By using thin provisioning, a storage administrator can over-commit the physical storage, often avoiding the need to purchase additional storage. For example, if ten users each request a 100GB file system for their application, the storage administrator can create what appears to be a 100GB file system for each user but which is backed by less actual storage that is used only when needed. When using thin provisioning, it is important that the storage administrator monitor the storage pool and add more capacity if it starts to become full.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom