whats wrong with my discsystem?

S

seechiller

Renowned Member
Jan 5, 2014
18
1
68
Hi,

I've just run the update to upgrade to 3.4 and noticed some strange output:

Code: 
Installing for i386-pc platform.
Installation finished. No error reported.
Generating grub configuration file ...
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found linux image: /boot/vmlinuz-2.6.32-39-pve
Found linux image: /boot/vmlinuz-2.6.32-34-pve
Found initrd image: /boot/initrd.img-2.6.32-34-pve
Found linux image: /boot/vmlinuz-2.6.32-33-pve
Found initrd image: /boot/initrd.img-2.6.32-33-pve
Found linux image: /boot/vmlinuz-2.6.32-32-pve
Found initrd image: /boot/initrd.img-2.6.32-32-pve
Found linux image: /boot/vmlinuz-2.6.32-30-pve
Found initrd image: /boot/initrd.img-2.6.32-30-pve
Found linux image: /boot/vmlinuz-2.6.32-26-pve
Found initrd image: /boot/initrd.img-2.6.32-26-pve
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
Setting up libnuma1 (2.0.8~rc4-1) ...
Setting up libpve-common-perl (3.0-24) ...
Setting up openssl (1.0.1e-2+deb7u16) ...
Setting up pve-cluster (3.0-17) ...
Restarting pve cluster filesystem: pve-cluster.
Setting up libpve-access-control (3.0-16) ...
Setting up libpve-storage-perl (3.0-33) ...
Setting up libxml2-utils (2.8.0+dfsg1-7+wheezy4) ...
Setting up numactl (2.0.8~rc4-1) ...
Setting up parted (3.2-6~bpo70+1) ...
Setting up pve-kernel-2.6.32-39-pve (2.6.32-156) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 2.6.32-39-pve /boot/vmlinuz-2.6.32-39-pve
update-initramfs: Generating /boot/initrd.img-2.6.32-39-pve
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 2.6.32-39-pve /boot/vmlinuz-2.6.32-39-pve
Generating grub configuration file ...
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found linux image: /boot/vmlinuz-2.6.32-39-pve
Found initrd image: /boot/initrd.img-2.6.32-39-pve
Found linux image: /boot/vmlinuz-2.6.32-34-pve
Found initrd image: /boot/initrd.img-2.6.32-34-pve
Found linux image: /boot/vmlinuz-2.6.32-33-pve
Found initrd image: /boot/initrd.img-2.6.32-33-pve
Found linux image: /boot/vmlinuz-2.6.32-32-pve
Found initrd image: /boot/initrd.img-2.6.32-32-pve
Found linux image: /boot/vmlinuz-2.6.32-30-pve
Found initrd image: /boot/initrd.img-2.6.32-30-pve
Found linux image: /boot/vmlinuz-2.6.32-26-pve
Found initrd image: /boot/initrd.img-2.6.32-26-pve
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
Setting up pve-firmware (1.1-4) ...
Setting up pve-qemu-kvm (2.2-10) ...

I've checked the raid status, looks okay, I also checked the discs within the controller, looks also okay. The system runns on SSD raid 1.

I've checked messages and found this:
Code: 
May 21 02:18:45 pm3 kernel: EXT3-fs: barriers disabled
May 21 02:18:45 pm3 kernel: kjournald starting.  Commit interval 5 seconds
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): using internal journal
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): 95 orphan inodes deleted
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): recovery complete
May 21 02:18:45 pm3 kernel: EXT3-fs (dm-3): mounted filesystem with ordered data mode
May 21 02:19:19 pm3 kernel: EXT3-fs: barriers disabled
May 21 02:19:19 pm3 kernel: kjournald starting.  Commit interval 5 seconds
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): using internal journal
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): 95 orphan inodes deleted
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): recovery complete
May 21 02:19:19 pm3 kernel: EXT3-fs (dm-3): mounted filesystem with ordered data mode
May 21 02:20:16 pm3 kernel: __ratelimit: 2819 callbacks suppressed
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:20:16 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:32 pm3 kernel: EXT3-fs: barriers disabled
May 21 02:21:32 pm3 kernel: kjournald starting.  Commit interval 5 seconds
May 21 02:21:32 pm3 kernel: EXT3-fs (dm-3): using internal journal
May 21 02:21:45 pm3 kernel: __ratelimit: 26 callbacks suppressed
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: lost page write due to I/O error on dm-3
May 21 02:21:45 pm3 kernel: <35893589805
May 21 02:21:45 pm3 kernel: <3589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <3589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: 3589805
May 21 02:21:45 pm3 kernel: 3589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <73589805
May 21 02:21:45 pm3 kernel: <3589805
May 21 02:21:45 pm3 kernel: <3589805
this filled messages log until the / partition was full

on rebooting I got this on console:
Code: 
Buffer I/O error on device dm-5

I run pvscan:
Code: 
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error
  PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]
  Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]

dmsetup info:
Code: 
Name:              pve-data-real
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        2
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnyYL6vSp40fai6CmO3CTYBNFLcJ0gaSogi-real

Name:              pve-swap
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnyRQeMl6rgi9dIeT4T1P7qN4DlDQSueY0R

Name:              pve-root
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywEL4YeJLlVI0imVh2UDU3YBANLOqJuCC

Name:              pve-vzsnap--pm3--0-cow
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK-cow

Name:              pve-data
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 3
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnyYL6vSp40fai6CmO3CTYBNFLcJ0gaSogi

Name:              pve-vzsnap--pm3--0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK

Drive Infos (from controller)
Code: 
Enclosure Device ID: 24
Slot Number: 0
Drive's position: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 1
Device Id: 37
WWN: 50026b723a09ee02
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA

Raw Size: 111.790 GB [0xdf94bb0 Sectors]
Non Coerced Size: 111.290 GB [0xde94bb0 Sectors]
Coerced Size: 110.827 GB [0xdda7800 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Commissioned Spare : No
Emergency Spare : No
Device Firmware Level: C4
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5005076028d068f9
Connected Port Number: 0(path0)
Inquiry Data: 50026B723A09EE02    KINGSTON SH103S3120G                    507KC4
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Solid State Device
Drive:  Not Certified
Drive Temperature : N/A
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's NCQ setting : N/A
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Enclosure Device ID: 24
Slot Number: 1
Drive's position: DiskGroup: 0, Span: 0, Arm: 1
Enclosure position: 1
Device Id: 38
WWN: 50026b723a09ef7f
Sequence Number: 2
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA

Raw Size: 111.790 GB [0xdf94bb0 Sectors]
Non Coerced Size: 111.290 GB [0xde94bb0 Sectors]
Coerced Size: 110.827 GB [0xdda7800 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Commissioned Spare : No
Emergency Spare : No
Device Firmware Level: C4
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5005076028d068fa
Connected Port Number: 0(path0)
Inquiry Data: 50026B723A09EF7F    KINGSTON SH103S3120G                    507KC4
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Solid State Device
Drive:  Not Certified
Drive Temperature : N/A
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's NCQ setting : N/A
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Array infos:
Code: 
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 110.827 GB
Sector Size         : 512
Mirror Data         : 110.827 GB
State               : Optimal
Strip Size          : 128 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Is VD Cached: No

I'm not sure whats wrong, usually if a disc fails the disc is kicked out and I replace it with a new one and rebuild the array but none of the ssd's was kicked out until now. Any idea whats wrong here?

Thanks!
 
The logical volume dev/pve/vzsnap-pm3-0 loock like an old vzdump snapshot, and maybe you run out of free space inside the VG?
I would just delete that snapshot (I assume vzdump is not running?).
 
Thanks Dietmar!

I deleted this files:
Code: 
/dev/pve/vzsnap-pm3-0
/dev/mapper/pve-vzsnap--pm3--0
/dev/disk/by-id/dm-name-pve-vzsnap--pm3--0
/dev/disk/by-id/dm-uuid-LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK
/dev/dm-5

Now pvscan looks much better
Code: 
root@pm3:/dev/disk/by-id# pvscan
  PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]
  Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]

I'll reboot at the night befor the backup starts with the console and watch if all errors are gone

The server has from time to time full / & backup partitions, I'll try to clean out some unused files.
 
I’ve now rebooted my server, on startup I found this on the console:

Code: 
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 1
Buffer I/O error on device dm-5, logical block 14503935
Buffer I/O error on device dm-5, logical block 14503935

If I run again pvscan I get again:

Code: 
root@pm3:~# pvscan  
 /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408056320: Input/output error   /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 59408113664: Input/output error   /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 0: Input/output error   /dev/pve/vzsnap-pm3-0: read failed after 0 of 4096 at 4096: Input/output error   PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]   Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]


I deleted again this files:

Code: 
/dev/pve/vzsnap-pm3-0
/dev/mapper/pve-vzsnap--pm3--0
/dev/disk/by-id/dm-name-pve-vzsnap--pm3--0
/dev/disk/by-id/dm-uuid-LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK
 /dev/dm-5
pvscan looks again better
Code: 
  PV /dev/sda2   VG pve   lvm2 [110.32 GiB / 12.75 GiB free]
  Total: 1 [110.32 GiB] / in use: 1 [110.32 GiB] / in no VG: 0 [0   ]

but after I rebooted again it starts over with the same i/o errors, I did this 3 times, always with the same result.

I've absolutly no idea whats wrong here and if I can trust the server that nothing strange will happens
 
Last edited:
dmesg:
Code: 
megasas: 06.703.11.00 Mar. 27, 17:00:00 PDT 2014
megasas: 0x1000:0x0079:0x1014:0x03b2: bus 36:slot 0:func 0
megaraid_sas 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
megaraid_sas 0000:24:00.0: setting latency timer to 64
megasas: FW now in Ready state
  alloc irq_desc for 50 on node -1
  alloc kstat_irqs on node -1
megaraid_sas 0000:24:00.0: irq 50 for MSI/MSI-X
[scsi0]: FW supports <0> MSIX vector, OnlineCPUs: <24>, Current MSIX <1>
ata_piix 0000:00:1f.2: version 2.13
ata_piix 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
ata_piix 0000:00:1f.2: setting latency timer to 64
scsi1 : ata_piix
megasas: cpx is not supported.
scsi2 : ata_piix
ata1: SATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0x20f0 irq 14
ata2: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x20f8 irq 15
  alloc irq_desc for 21 on node -1
  alloc kstat_irqs on node -1
ata_piix 0000:00:1f.5: PCI INT C -> GSI 21 (level, low) -> IRQ 21
ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
ata_piix 0000:00:1f.5: setting latency timer to 64
scsi3 : ata_piix
scsi4 : ata_piix
ata3: SATA max UDMA/133 cmd 0x2108 ctl 0x2124 bmdma 0x20d0 irq 21
ata4: SATA max UDMA/133 cmd 0x2100 ctl 0x2120 bmdma 0x20d8 irq 21
megasas: INIT adapter done
megaraid_sas: Controller type: MR, Memory size is: 512MB
LSI Debug instance->disableOnlineCtrlReset 0
scsi0 : LSI SAS based MegaRAID driver
scsi 0:0:24:0: Enclosure         IBM-ESXS SAS EXP BP       632A PQ: 0 ANSI: 4
scsi 0:2:0:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:1:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:2:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:3:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
scsi 0:2:4:0: Direct-Access     IBM      ServeRAID M5015  2.13 PQ: 0 ANSI: 5
sd 0:2:0:0: [sda] 232421376 512-byte logical blocks: (118 GB/110 GiB)
sd 0:2:1:0: [sdb] 5267576832 512-byte logical blocks: (2.69 TB/2.45 TiB)
sd 0:2:0:0: [sda] Write Protect is off
sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
sd 0:2:0:0: [sda] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:1:0: [sdb] Write Protect is off
sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
sd 0:2:2:0: [sdc] 1951170560 512-byte logical blocks: (998 GB/930 GiB)
sd 0:2:1:0: [sdb] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:2:0: [sdc] Write Protect is off
sd 0:2:2:0: [sdc] Mode Sense: 1f 00 00 08
sd 0:2:3:0: [sdd] 5853511680 512-byte logical blocks: (2.99 TB/2.72 TiB)
sd 0:2:2:0: [sdc] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:3:0: [sdd] Write Protect is off
sd 0:2:3:0: [sdd] Mode Sense: 1f 00 00 08
sd 0:2:4:0: [sde] 7808589824 512-byte logical blocks: (3.99 TB/3.63 TiB)
 sdb:
sd 0:2:3:0: [sdd] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:2:4:0: [sde] Write Protect is off
sd 0:2:4:0: [sde] Mode Sense: 1f 00 00 08
 sdc:
sd 0:2:4:0: [sde] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
 sdd:
 sda:
 sde: sdd1
 sda1 sda2
sd 0:2:3:0: [sdd] Attached SCSI disk
sd 0:2:0:0: [sda] Attached SCSI disk
 sde1
usb 2-1: New USB device found, idVendor=04b3, idProduct=4012
usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-1: Product: IBM Composite Device-0
usb 2-1: Manufacturer: IBM
usb 2-1: SerialNumber: 20070221-7
usb 2-1: configuration #2 chosen from 1 choice
sd 0:2:4:0: [sde] Attached SCSI disk
 sdc1
sd 0:2:2:0: [sdc] Attached SCSI disk
input: IBM IBM Composite Device-0 as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:2.0/input/input2
generic-usb 0003:04B3:4012.0001: input,hidraw0: USB HID v1.00 Keyboard [IBM IBM Composite Device-0] on usb-0000:00:1d.7-1/input0
input: IBM IBM Composite Device-0 as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:2.1/input/input3
generic-usb 0003:04B3:4012.0002: input,hidraw1: USB HID v1.00 Mouse [IBM IBM Composite Device-0] on usb-0000:00:1d.7-1/input1
input: IBM IBM Composite Device-0 as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:2.2/input/input4
generic-usb 0003:04B3:4012.0003: input,hidraw2: USB HID v1.00 Mouse [IBM IBM Composite Device-0] on usb-0000:00:1d.7-1/input2
 sdb1
sd 0:2:1:0: [sdb] Attached SCSI disk
Refined TSC clocksource calibration: 2666.760 MHz.
Switching to clocksource tsc
ata4: SATA link down (SStatus 0 SControl 300)
ata3: SATA link down (SStatus 0 SControl 300)
usb 5-2: new full speed USB device number 2 using uhci_hcd
usb 5-2: New USB device found, idVendor=04b3, idProduct=4010
usb 5-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 5-2: Product: RNDIS/CDC ETHER
usb 5-2: Manufacturer: IBM
usb 5-2: configuration #1 chosen from 2 choices
ata2.00: SATA link down (SStatus 0 SControl 300)
ata2.01: SATA link down (SStatus 0 SControl 300)
ata1.00: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.01: SATA link down (SStatus 0 SControl 300)
ata1.01: link offline, clearing class 3 to NONE
ata1.00: ATAPI: MATSHITADVD-RAM UJ890, SAA5, max UDMA/100
ata1.00: configured for UDMA/100
scsi 1:0:0:0: CD-ROM            MATSHITA DVD-RAM UJ890    SAA5 PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
scsi 0:0:24:0: Attached scsi generic sg0 type 13
sd 0:2:0:0: Attached scsi generic sg1 type 0
sd 0:2:1:0: Attached scsi generic sg2 type 0
sd 0:2:2:0: Attached scsi generic sg3 type 0
sd 0:2:3:0: Attached scsi generic sg4 type 0
sd 0:2:4:0: Attached scsi generic sg5 type 0
sr 1:0:0:0: Attached scsi generic sg6 type 5
ses 0:0:24:0: Attached Enclosure device
waiting module removal not supported: please upgrade
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (dm-0): mounted filesystem with ordered data mode
<30>udevd[796]: starting version 175
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ipmi message handler version 39.2
EDAC MC: Ver: 2.1.0 Jun 24 2015
dca service started, version 1.12.1
ioatdma: Intel(R) QuickData Technology Driver 4.00
ioatdma 0000:00:16.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ioatdma 0000:00:16.0: setting latency timer to 64
  alloc irq_desc for 51 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.0: irq 51 for MSI/MSI-X
ioatdma 0000:00:16.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
ioatdma 0000:00:16.1: setting latency timer to 64
  alloc irq_desc for 52 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.1: irq 52 for MSI/MSI-X
ioatdma 0000:00:16.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ioatdma 0000:00:16.2: setting latency timer to 64
  alloc irq_desc for 53 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.2: irq 53 for MSI/MSI-X
ioatdma 0000:00:16.3: PCI INT D -> GSI 19 (level, low) -> IRQ 19
ioatdma 0000:00:16.3: setting latency timer to 64
  alloc irq_desc for 54 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.3: irq 54 for MSI/MSI-X
ioatdma 0000:00:16.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ioatdma 0000:00:16.4: setting latency timer to 64
  alloc irq_desc for 55 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.4: irq 55 for MSI/MSI-X
ioatdma 0000:00:16.5: PCI INT B -> GSI 17 (level, low) -> IRQ 17
ioatdma 0000:00:16.5: setting latency timer to 64
  alloc irq_desc for 56 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.5: irq 56 for MSI/MSI-X
ioatdma 0000:00:16.6: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ioatdma 0000:00:16.6: setting latency timer to 64
  alloc irq_desc for 57 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.6: irq 57 for MSI/MSI-X
ioatdma 0000:00:16.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19
ioatdma 0000:00:16.7: setting latency timer to 64
  alloc irq_desc for 58 on node -1
  alloc kstat_irqs on node -1
ioatdma 0000:00:16.7: irq 58 for MSI/MSI-X
IPMI System Interface driver.
ipmi_si: probing via SMBIOS
ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0
ACPI Warning: SystemIO range 0x00000000000005a8-0x00000000000005af conflicts with OpRegion 0x00000000000005a8-0x00000000000005af (\_SB_.PCI0.LPC0.GPE0) (20090903/utaddress-254)
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000043f conflicts with OpRegion 0x0000000000000439-0x0000000000000439 (\_SB_.PCI0.RIL_) (20090903/utaddress-254)
ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000043f conflicts with OpRegion 0x000000000000040e-0x000000000000040e (\_SB_.PCI0.RIT_) (20090903/utaddress-254)
ACPI Warning: SystemIO range 0x0000000000000400-0x000000000000043f conflicts with OpRegion 0x000000000000040c-0x000000000000040c (\_SB_.PCI0.RTY_) (20090903/utaddress-254)
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
lpc_ich: Resource conflict(s) found affecting gpio_ich
  alloc irq_desc for 22 on node -1
  alloc kstat_irqs on node -1
i801_smbus 0000:00:1f.3: PCI INT B -> GSI 22 (level, low) -> IRQ 22
tpm_tis 00:09: 1.2 TPM (device-id 0xFE, rev-id 70)
input: PC Speaker as /devices/platform/pcspkr/input/input5
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: Found a ICH10 TCO device (Version=2, TCOBASE=0x05e0)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
cdc_ether 5-2:1.0: usb0: register 'cdc_ether' at usb-0000:00:1d.0-2, CDC Ethernet Device, e6:1f:13:a6:2c:df
usbcore: registered new interface driver cdc_ether
ipmi_si ipmi_si.0: Found new BMC (man_id: 0x000002, prod_id: 0x00dc, dev_id: 0x20)
ipmi_si ipmi_si.0: IPMI kcs interface initialized
EXT3-fs (dm-0): using internal journal
fuse init (API version 7.14)
device-mapper: snapshots: Snapshot is marked invalid.
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503920
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 14503934
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 0
Buffer I/O error on device dm-5, logical block 1
Buffer I/O error on device dm-5, logical block 14503935
Buffer I/O error on device dm-5, logical block 14503935
Adding 14417916k swap on /dev/mapper/pve-swap.  Priority:-1 extents:1 across:14417916k
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (dm-3): using internal journal
EXT3-fs (dm-3): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sda1): using internal journal
EXT3-fs (sda1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sdb1): using internal journal
EXT3-fs (sdb1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sde1): using internal journal
EXT3-fs (sde1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sdc1): using internal journal
EXT3-fs (sdc1): mounted filesystem with ordered data mode
EXT3-fs: barriers disabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (sdd1): using internal journal
EXT3-fs (sdd1): mounted filesystem with ordered data mode
  alloc irq_desc for 59 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 59 for MSI/MSI-X
  alloc irq_desc for 60 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 60 for MSI/MSI-X
  alloc irq_desc for 61 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 61 for MSI/MSI-X
  alloc irq_desc for 62 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 62 for MSI/MSI-X
  alloc irq_desc for 63 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 63 for MSI/MSI-X
  alloc irq_desc for 64 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 64 for MSI/MSI-X
  alloc irq_desc for 65 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 65 for MSI/MSI-X
  alloc irq_desc for 66 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 66 for MSI/MSI-X
  alloc irq_desc for 67 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.1: irq 67 for MSI/MSI-X
bnx2 0000:0b:00.1: eth1: using MSIX
device eth0 entered promiscuous mode
  alloc irq_desc for 68 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 68 for MSI/MSI-X
  alloc irq_desc for 69 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 69 for MSI/MSI-X
  alloc irq_desc for 70 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 70 for MSI/MSI-X
  alloc irq_desc for 71 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 71 for MSI/MSI-X
  alloc irq_desc for 72 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 72 for MSI/MSI-X
  alloc irq_desc for 73 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 73 for MSI/MSI-X
  alloc irq_desc for 74 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 74 for MSI/MSI-X
  alloc irq_desc for 75 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 75 for MSI/MSI-X
  alloc irq_desc for 76 on node -1
  alloc kstat_irqs on node -1
bnx2 0000:0b:00.0: irq 76 for MSI/MSI-X
bnx2 0000:0b:00.0: eth0: using MSIX
NET: Registered protocol family 10
ADDRCONF(NETDEV_UP): eth0: link is not ready
ADDRCONF(NETDEV_UP): eth1: link is not ready
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Slow work thread pool: Starting up
Slow work thread pool: Ready
FS-Cache: Loaded
 
seechiller said:
Thanks Dietmar!

I deleted this files:
Code: 
/dev/pve/vzsnap-pm3-0
/dev/mapper/pve-vzsnap--pm3--0
/dev/disk/by-id/dm-name-pve-vzsnap--pm3--0
/dev/disk/by-id/dm-uuid-LVM-pmLkYaq8onYG7zkPUhzHxZXc1mwyJvnywaO21v568C4VFdfYGv0TYMOe8B8fIEyK
/dev/dm-5
Click to expand...

Hi,
delete the snapshot, like Dietmar suggested,doesn't mean to delete the files!!
It's mean to use
Code: 
lvremove /dev/pve/vzsnap-pm3-0

What is the output of following commands
Code: 
pvs
vgs
lvs
lvdisplay /dev/pve/vzsnap-pm3-0
Udo
 
Hi Udo

udo said:
delete the snapshot, like Dietmar suggested,doesn't mean to delete the files!!
It's mean to use
Code: 
lvremove /dev/pve/vzsnap-pm3-0
Click to expand...

Thanks, I didn't know that the snampshot has to be removed with lvremove, just did it.



udo said:
What is the output of following commands
Code: 
pvs
vgs
lvs
lvdisplay /dev/pve/vzsnap-pm3-0
Click to expand...

Here are the outputs:
Code: 
root@pm3:/mnt/backup-1# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  pve  lvm2 a--  110.32g 13.75g

root@pm3:/mnt/backup-1# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 110.32g 13.75g

root@pm3:/mnt/backup-1# lvs
  LV   VG   Attr      LSize  Pool Origin Data%  Move Log Copy%  Convert
  data pve  -wi-ao--- 55.33g
  root pve  -wi-ao--- 27.50g
  swap pve  -wi-ao--- 13.75g

root@pm3:/mnt/backup-1# lvdisplay /dev/pve/vzsnap-pm3-0
  One or more specified logical volume(s) not found.

I'll reboot again at midnight and check again the logs
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom