whats wrong with my discsystem?

S

seechiller

Renowned Member
Jan 5, 2014
18
1
68
Hi,

I've just run the update to upgrade to 3.4 and noticed some strange output:

Code: 
Installing for i386-pc platform.
Installation finished. No error reported.
Generating grub configuration file ...
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found linux image: /boot/vmlinuz-2.6.32-39-pve
Found linux image: /boot/vmlinuz-2.6.32-34-pve
Found initrd image: /boot/initrd.img-2.6.32-34-pve
Found linux image: /boot/vmlinuz-2.6.32-33-pve
Found initrd image: /boot/initrd.img-2.6.32-33-pve
Found linux image: /boot/vmlinuz-2.6.32-32-pve
Found initrd image: /boot/initrd.img-2.6.32-32-pve
Found linux image: /boot/vmlinuz-2.6.32-30-pve
Found initrd image: /boot/initrd.img-2.6.32-30-pve
Found linux image: /boot/vmlinuz-2.6.32-26-pve
Found initrd image: /boot/initrd.img-2.6.32-26-pve
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
Setting up libnuma1 (2.0.8~rc4-1) ...
Setting up libpve-common-perl (3.0-24) ...
Setting up openssl (1.0.1e-2+deb7u16) ...
Setting up pve-cluster (3.0-17) ...
Restarting pve cluster filesystem: pve-cluster.
Setting up libpve-access-control (3.0-16) ...
Setting up libpve-storage-perl (3.0-33) ...
Setting up libxml2-utils (2.8.0+dfsg1-7+wheezy4) ...
Setting up numactl (2.0.8~rc4-1) ...
Setting up parted (3.2-6~bpo70+1) ...
Setting up pve-kernel-2.6.32-39-pve (2.6.32-156) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-39-pve /boot/vmlinuz-2.6.32-39-pve
update-initramfs: Generating /boot/initrd.img-2.6.32-39-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-39-pve /boot/vmlinuz-2.6.32-39-pve
Generating grub configuration file ...
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found linux image: /boot/vmlinuz-2.6.32-39-pve
Found initrd image: /boot/initrd.img-2.6.32-39-pve
Found linux image: /boot/vmlinuz-2.6.32-34-pve
Found initrd image: /boot/initrd.img-2.6.32-34-pve
Found linux image: /boot/vmlinuz-2.6.32-33-pve
Found initrd image: /boot/initrd.img-2.6.32-33-pve
Found linux image: /boot/vmlinuz-2.6.32-32-pve
Found initrd image: /boot/initrd.img-2.6.32-32-pve
Found linux image: /boot/vmlinuz-2.6.32-30-pve
Found initrd image: /boot/initrd.img-2.6.32-30-pve
Found linux image: /boot/vmlinuz-2.6.32-26-pve
Found initrd image: /boot/initrd.img-2.6.32-26-pve
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
Setting up pve-firmware (1.1-4) ...
Setting up pve-qemu-kvm (2.2-10) ...

I've checked the raid status, looks okay, I also checked the discs within the controller, looks also okay. The system runns on SSD raid 1.

I've checked messages and found this:
Code: 
May 21 02:18:45 pm3 kernel: EXT3-fs: barriers disabled
May 21 02:18:45 pm3 kernel: kjournald starting.  Commit interval 5 seconds
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): using internal journal
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): 95 orphan inodes deleted
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): recovery complete
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): mounted filesystem with ordered data mode
May 21 02:19:19 pm3 kernel: EXT3-fs: barriers disabled
May 21 02:19:19 pm3 kernel: kjournald starting.  Commit interval 5 seconds
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): using internal journal
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): 95 orphan inodes deleted
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): recovery complete
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): mounted filesystem with ordered data mode
May 21 02:20:16 pm3 kernel: __ratelimit: 2819 callbacks suppressed
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:32 pm3 kernel: EXT3-fs: barriers disabled
May 21 02:21:32 pm3 kernel: kjournald starting.  Commit interval 5 seconds
May 21 02:21:32 pm3 kernel: EXT3-fs (dm-3): using internal journal
May 21 02:21:45 pm3 kernel: __ratelimit: 26 callbacks suppressed
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: <35893589805
May 21 02:21:45 pm3 kernel: <3589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <3589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: 3589805
May 21 02:21:45 pm3 kernel: 3589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <3589805
May 21 02:21:45 pm3 kernel: <3589805
this filled messages log until the / partition was full

on rebooting I got this on console:
Code: 
Buffer I/O error on device dm-5

I run pvscan:
Code: 
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]
  Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]

dmsetup info:
Code: 
Name:              pve-data-real
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        2
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnyYL6vSp40fai6CmO3CTYBNFLcJ0gaSogi-real

Name:              pve-swap
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnyRQeMl6rgi9dIeT4T1P7qN4DlDQSueY0R

Name:              pve-root
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywEL4YeJLlVI0imVh2UDU3YBANLOqJuCC

Name:              pve-vzsnap--pm3--0-cow
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK-cow

Name:              pve-data
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnyYL6vSp40fai6CmO3CTYBNFLcJ0gaSogi

Name:              pve-vzsnap--pm3--0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK

Drive Infos (from controller)
Code: 
Enclosure Device ID: 24
Slot Number: 0
Drive's position: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 1
Device Id: 37
WWN: 50026b723a09ee02
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA

Raw Size: 111.790 GB [0xdf94bb0 Sectors]
Non Coerced Size: 111.290 GB [0xde94bb0 Sectors]
Coerced Size: 110.827 GB [0xdda7800 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Commissioned Spare : No
Emergency Spare : No
Device Firmware Level: C4
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5005076028d068f9
Connected Port Number: 0(path0)
Inquiry Data: 50026B723A09EE02    KINGSTON SH103S3120G                    507KC4
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Solid State Device
Drive:  Not Certified
Drive Temperature : N/A
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's NCQ setting : N/A
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Enclosure Device ID: 24
Slot Number: 1
Drive's position: DiskGroup: 0, Span: 0, Arm: 1
Enclosure position: 1
Device Id: 38
WWN: 50026b723a09ef7f
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA

Raw Size: 111.790 GB [0xdf94bb0 Sectors]
Non Coerced Size: 111.290 GB [0xde94bb0 Sectors]
Coerced Size: 110.827 GB [0xdda7800 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Commissioned Spare : No
Emergency Spare : No
Device Firmware Level: C4
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5005076028d068fa
Connected Port Number: 0(path0)
Inquiry Data: 50026B723A09EF7F    KINGSTON SH103S3120G                    507KC4
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Solid State Device
Drive:  Not Certified
Drive Temperature : N/A
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's NCQ setting : N/A
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Array infos:
Code: 
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 110.827 GB
Sector Size         : 512
Mirror Data         : 110.827 GB
State               : Optimal
Strip Size          : 128 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Is VD Cached: No

I'm not sure whats wrong, usually if a disc fails the disc is kicked out and I replace it with a new one and rebuild the array but none of the ssd's was kicked out until now. Any idea whats wrong here?

Thanks!
 
The logical volume dev/pve/vzsnap-pm3-0 loock like an old vzdump snapshot, and maybe you run out of free space inside the VG?
I would just delete that snapshot (I assume vzdump is not running?).
 
Thanks Dietmar!

I deleted this files:
Code: 
/dev/pve/vzsnap-pm3-0
/dev/mapper/pve-vzsnap--pm3--0
/dev/disk/by-id/dm-name-pve-vzsnap--pm3--0
/dev/disk/by-id/dm-uuid-LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK
/dev/dm-5

Now pvscan looks much better
Code: 
root@pm3:/dev/disk/by-id# pvscan
  PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]
  Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]

I'll reboot at the night befor the backup starts with the console and watch if all errors are gone

The server has from time to time full / & backup partitions, I'll try to clean out some unused files.
 
I’ve now rebooted my server, on startup I found this on the console:

Code: 
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 1
Buffer I/O error on device dm-5, logical block 14503935
Buffer I/O error on device dm-5, logical block 14503935

If I run again pvscan I get again:

Code: 
root@pm3:~# pvscan  
 /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error   /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error   /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error   /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error   PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]   Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]


I deleted again this files:

Code: 
/dev/pve/vzsnap-pm3-0
/dev/mapper/pve-vzsnap--pm3--0
/dev/disk/by-id/dm-name-pve-vzsnap--pm3--0
/dev/disk/by-id/dm-uuid-LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK
 /dev/dm-5
pvscan looks again better
Code: 
  PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]
  Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]

but after I rebooted again it starts over with the same i/o errors, I did this 3 times, always with the same result.

I've absolutly no idea whats wrong here and if I can trust the server that nothing strange will happens
 
Last edited:
dmesg:
Code: 
megasas: 06.703.11.00 Mar. 27, 17:00:00 PDT 2014
megasas: 0x1000:0x0079:0x1014:0x03b2: bus 36:slot 0:func 0
megaraid_sas 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
megaraid_sas 0000:24:00.0: setting latency timer to 64
megasas: FW now in Ready state
  alloc irq_desc for 50 on node -1
  alloc kstat_irqs on node -1
megaraid_sas 0000:24:00.0: irq 50 for MSI/MSI-X
[scsi0]: FW supports <0> MSIX vector, OnlineCPUs: <24>, Current MSIX <1>
ata_piix 0000:00:1f.2: version 2.13
ata_piix 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
ata_piix 0000:00:1f.2: setting latency timer to 64
scsi1 : ata_piix
megasas: cpx is not supported.
scsi2 : ata_piix
ata1: SATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0x20f0 irq 14
ata2: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x20f8 irq 15
  alloc irq_desc for 21 on node -1
  alloc kstat_irqs on node -1
ata_piix 0000:00:1f.5: PCI INT C -> GSI 21 (level, low) -> IRQ 21
ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
ata_piix 0000:00:1f.5: setting latency timer to 64
scsi3 : ata_piix
scsi4 : ata_piix
ata3: SATA max UDMA/133 cmd 0x2108 ctl 0x2124 bmdma 0x20d0 irq 21
ata4: SATA max UDMA/133 cmd 0x2100 ctl 0x2120 bmdma 0x20d8 irq 21
megasas: INIT adapter done
megaraid_sas: Controller type: MR, Memory size is: 512MB
LSI Debug instance->disableOnlineCtrlReset 0
scsi0 : LSI SAS based MegaRAID driver
scsi 0:0:24:0: Enclosure         IBM-ESXS SAS EXP BP       632A PQ: 0 ANSI: 4
scsi 0:2:0:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:1:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:2:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:3:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:4:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
sd 0:2:0:0: [sda] 232421376 512-byte logical blocks: (118 GB/110 GiB)
sd 0:2:1:0: [sdb] 5267576832 512-byte logical blocks: (2.69 TB/2.45 TiB)
sd 0:2:0:0: [sda] Write Protect is off
sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
sd 0:2:0:0: [sda] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:1:0: [sdb] Write Protect is off
sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
sd 0:2:2:0: [sdc] 1951170560 512-byte logical blocks: (998 GB/930 GiB)
sd 0:2:1:0: [sdb] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:2:0: [sdc] Write Protect is off
sd 0:2:2:0: [sdc] Mode Sense: 1f 00 00 08
sd 0:2:3:0: [sdd] 5853511680 512-byte logical blocks: (2.99 TB/2.72 TiB)
sd 0:2:2:0: [sdc] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:3:0: [sdd] Write Protect is off
sd 0:2:3:0: [sdd] Mode Sense: 1f 00 00 08
sd 0:2:4:0: [sde] 7808589824 512-byte logical blocks: (3.99 TB/3.63 TiB)
 sdb:
sd 0:2:3:0: [sdd] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:4:0: [sde] Write Protect is off
sd 0:2:4:0: [sde] Mode Sense: 1f 00 00 08
 sdc:
sd 0:2:4:0: [sde] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
 sdd:
 sda:
 sde: sdd1
 sda1 sda2
sd 0:2:3:0: [sdd] Attached SCSI disk
sd 0:2:0:0: [sda] Attached SCSI disk
 sde1
usb 2-1: New USB device found, idVendor=04b3, idProduct=4012
usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-1: Product: IBM Composite Device-0
usb 2-1: Manufacturer: IBM
usb 2-1: SerialNumber: 20070221-7
usb 2-1: configuration #2 chosen from 1 choice
sd 0:2:4:0: [sde] Attached SCSI disk
 sdc1
sd 0:2:2:0: [sdc] Attached SCSI disk
input: IBM IBM Composite Device-0 as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:2.0/input/input2
generic-usb 0003:04B3:4012.0001: input,hidraw0: USB HID v1.00 Keyboard [IBM IBM Composite Device-0] on usb-0000:00:1d.7-1/input0
input: IBM IBM Composite Device-0 as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:2.1/input/input3
generic-usb 0003:04B3:4012.0002: input,hidraw1: USB HID v1.00 Mouse [IBM IBM Composite Device-0] on usb-0000:00:1d.7-1/input1
input: IBM IBM Composite Device-0 as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:2.2/input/input4
generic-usb 0003:04B3:4012.0003: input,hidraw2: USB HID v1.00 Mouse [IBM IBM Composite Device-0] on usb-0000:00:1d.7-1/input2
 sdb1
sd 0:2:1:0: [sdb] Attached SCSI disk
Refined TSC clocksource calibration: 2666.760 MHz.
Switching to clocksource tsc
ata4: SATA link down (SStatus 0 SControl 300)
ata3: SATA link down (SStatus 0 SControl 300)
usb 5-2: new full speed USB device number 2 using uhci_hcd
usb 5-2: New USB device found, idVendor=04b3, idProduct=4010
usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-2: Product: RNDIS/CDC ETHER
usb 5-2: Manufacturer: IBM
usb 5-2: configuration #1 chosen from 2 choices
ata2.00: SATA link down (SStatus 0 SControl 300)
ata2.01: SATA link down (SStatus 0 SControl 300)
ata1.00: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.01: SATA link down (SStatus 0 SControl 300)
ata1.01: link offline, clearing class 3 to NONE
ata1.00: ATAPI: MATSHITADVD-RAM UJ890, SAA5, max UDMA/100
ata1.00: configured for UDMA/100
scsi 1:0:0:0: CD-ROM            MATSHITA DVD-RAM UJ890    SAA5 PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
scsi 0:0:24:0: Attached scsi generic sg0 type 13
sd 0:2:0:0: Attached scsi generic sg1 type 0
sd 0:2:1:0: Attached scsi generic sg2 type 0
sd 0:2:2:0: Attached scsi generic sg3 type 0
sd 0:2:3:0: Attached scsi generic sg4 type 0
sd 0:2:4:0: Attached scsi generic sg5 type 0
sr 1:0:0:0: Attached scsi generic sg6 type 5
ses 0:0:24:0: Attached Enclosure device
waiting module removal not supported: please upgrade
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (dm-0): mounted filesystem with ordered data mode
<30>udevd[796]: starting version 175
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ipmi message handler version 39.2
EDAC MC: Ver: 2.1.0 Jun 24 2015
dca service started, version 1.12.1
ioatdma: Intel(R) QuickData Technology Driver 4.00
ioatdma 0000:00:16.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ioatdma 0000:00:16.0: setting latency timer to 64
  alloc irq_desc for 51 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.0: irq 51 for MSI/MSI-X
ioatdma 0000:00:16.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
ioatdma 0000:00:16.1: setting latency timer to 64
  alloc irq_desc for 52 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.1: irq 52 for MSI/MSI-X
ioatdma 0000:00:16.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ioatdma 0000:00:16.2: setting latency timer to 64
  alloc irq_desc for 53 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.2: irq 53 for MSI/MSI-X
ioatdma 0000:00:16.3: PCI INT D -> GSI 19 (level, low) -> IRQ 19
ioatdma 0000:00:16.3: setting latency timer to 64
  alloc irq_desc for 54 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.3: irq 54 for MSI/MSI-X
ioatdma 0000:00:16.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ioatdma 0000:00:16.4: setting latency timer to 64
  alloc irq_desc for 55 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.4: irq 55 for MSI/MSI-X
ioatdma 0000:00:16.5: PCI INT B -> GSI 17 (level, low) -> IRQ 17
ioatdma 0000:00:16.5: setting latency timer to 64
  alloc irq_desc for 56 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.5: irq 56 for MSI/MSI-X
ioatdma 0000:00:16.6: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ioatdma 0000:00:16.6: setting latency timer to 64
  alloc irq_desc for 57 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.6: irq 57 for MSI/MSI-X
ioatdma 0000:00:16.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
ioatdma 0000:00:16.7: setting latency timer to 64
  alloc irq_desc for 58 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.7: irq 58 for MSI/MSI-X
IPMI System Interface driver.
ipmi_si: probing via SMBIOS
ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
ACPI Warning: SystemIO range 0x00000000000005a8-0x00000000000005af conflicts with OpRegion 0x00000000000005a8-0x00000000000005af (\_SB_.PCI0.LPC0.GPE0) (20090903/utaddress-254)
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000043f conflicts with OpRegion 0x0000000000000439-0x0000000000000439 (\_SB_.PCI0.RIL_) (20090903/utaddress-254)
ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000043f conflicts with OpRegion 0x000000000000040e-0x000000000000040e (\_SB_.PCI0.RIT_) (20090903/utaddress-254)
ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000043f conflicts with OpRegion 0x000000000000040c-0x000000000000040c (\_SB_.PCI0.RTY_) (20090903/utaddress-254)
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
lpc_ich: Resource conflict(s) found affecting gpio_ich
  alloc irq_desc for 22 on node -1
  alloc kstat_irqs on node -1
i801_smbus 0000:00:1f.3: PCI INT B -> GSI 22 (level, low) -> IRQ 22
tpm_tis 00:09: 1.2 TPM (device-id 0xFE, rev-id 70)
input: PC Speaker as /devices/platform/pcspkr/input/input5
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: Found a ICH10 TCO device (Version=2, TCOBASE=0x05e0)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
cdc_ether 5-2:1.0: usb0: register 'cdc_ether' at usb-0000:00:1d.0-2, CDC Ethernet Device, e6:1f:13:a6:2c:df
usbcore: registered new interface driver cdc_ether
ipmi_si ipmi_si.0: Found new BMC (man_id: 0x000002, prod_id: 0x00dc, dev_id: 0x20)
ipmi_si ipmi_si.0: IPMI kcs interface initialized
EXT3-fs (dm-0): using internal journal
fuse init (API version 7.14)
device-mapper: snapshots: Snapshot is marked invalid.
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 1
Buffer I/O error on device dm-5, logical block 14503935
Buffer I/O error on device dm-5, logical block 14503935
Adding 14417916k swap on /dev/mapper/pve-swap.  Priority:-1 extents:1 across:14417916k
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (dm-3): using internal journal
EXT3-fs (dm-3): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sda1): using internal journal
EXT3-fs (sda1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sdb1): using internal journal
EXT3-fs (sdb1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sde1): using internal journal
EXT3-fs (sde1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sdc1): using internal journal
EXT3-fs (sdc1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sdd1): using internal journal
EXT3-fs (sdd1): mounted filesystem with ordered data mode
  alloc irq_desc for 59 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 59 for MSI/MSI-X
  alloc irq_desc for 60 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 60 for MSI/MSI-X
  alloc irq_desc for 61 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 61 for MSI/MSI-X
  alloc irq_desc for 62 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 62 for MSI/MSI-X
  alloc irq_desc for 63 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 63 for MSI/MSI-X
  alloc irq_desc for 64 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 64 for MSI/MSI-X
  alloc irq_desc for 65 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 65 for MSI/MSI-X
  alloc irq_desc for 66 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 66 for MSI/MSI-X
  alloc irq_desc for 67 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 67 for MSI/MSI-X
bnx2 0000:0b:00.1: eth1: using MSIX
device eth0 entered promiscuous mode
  alloc irq_desc for 68 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 68 for MSI/MSI-X
  alloc irq_desc for 69 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 69 for MSI/MSI-X
  alloc irq_desc for 70 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 70 for MSI/MSI-X
  alloc irq_desc for 71 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 71 for MSI/MSI-X
  alloc irq_desc for 72 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 72 for MSI/MSI-X
  alloc irq_desc for 73 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 73 for MSI/MSI-X
  alloc irq_desc for 74 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 74 for MSI/MSI-X
  alloc irq_desc for 75 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 75 for MSI/MSI-X
  alloc irq_desc for 76 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 76 for MSI/MSI-X
bnx2 0000:0b:00.0: eth0: using MSIX
NET: Registered protocol family 10
ADDRCONF(NETDEV_UP): eth0: link is not ready
ADDRCONF(NETDEV_UP): eth1: link is not ready
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Slow work thread pool: Starting up
Slow work thread pool: Ready
FS-Cache: Loaded
 
seechiller said:
Thanks Dietmar!

I deleted this files:
Code: 
/dev/pve/vzsnap-pm3-0
/dev/mapper/pve-vzsnap--pm3--0
/dev/disk/by-id/dm-name-pve-vzsnap--pm3--0
/dev/disk/by-id/dm-uuid-LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK
/dev/dm-5
Click to expand...

Hi,
delete the snapshot, like Dietmar suggested,doesn't mean to delete the files!!
It's mean to use
Code: 
lvremove /dev/pve/vzsnap-pm3-0

What is the output of following commands
Code: 
pvs
vgs
lvs
lvdisplay /dev/pve/vzsnap-pm3-0
Udo
 
Hi Udo

udo said:
delete the snapshot, like Dietmar suggested,doesn't mean to delete the files!!
It's mean to use
Code: 
lvremove /dev/pve/vzsnap-pm3-0
Click to expand...

Thanks, I didn't know that the snampshot has to be removed with lvremove, just did it.



udo said:
What is the output of following commands
Code: 
pvs
vgs
lvs
lvdisplay /dev/pve/vzsnap-pm3-0
Click to expand...

Here are the outputs:
Code: 
root@pm3:/mnt/backup-1# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  pve  lvm2 a--  110.32g 13.75g

root@pm3:/mnt/backup-1# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 110.32g 13.75g

root@pm3:/mnt/backup-1# lvs
  LV   VG   Attr      LSize  Pool Origin Data%  Move Log Copy%  Convert
  data pve  -wi-ao--- 55.33g
  root pve  -wi-ao--- 27.50g
  swap pve  -wi-ao--- 13.75g

root@pm3:/mnt/backup-1# lvdisplay /dev/pve/vzsnap-pm3-0
  One or more specified logical volume(s) not found.

I'll reboot again at midnight and check again the logs
 
You must log in or register to reply here.
Top Bottom