Old Proxmox installation and problem with backup

David Calvache Casas

Well-Known Member
Jun 14, 2013
35
2
48
Almería, Spain
Hello to all.

Yes i know i must upgrade the host, but this machine it a very critical machine running 24/7/365 and isolated from the corp NET.
Now i have a problem, last day during the backup of the second visrtual machine something goes very bad.

Here is the log:

Jun 12 03:44:09 INFO: Starting Backup of VM 104 (qemu)
Jun 12 03:44:09 INFO: running
Jun 12 03:44:09 INFO: status = running
Jun 12 03:44:09 INFO: backup mode: snapshot
Jun 12 03:44:09 INFO: ionice priority: 7
Jun 12 03:44:09 INFO: suspend vm to make snapshot
Jun 12 03:44:10 INFO: Logical volume "vzsnap-proxmox92-0" created
Jun 12 03:44:10 INFO: resume vm
Jun 12 03:44:10 INFO: vm is online again after 1 seconds
Jun 12 03:44:10 INFO: creating archive '/mnt/backup/proxmox92/vzdump-qemu-104-2013_06_12-03_44_09.tgz'




And from kern.log

Jun 12 03:44:10 proxmox92 kernel: EXT3-fs: barriers disabled
Jun 12 03:44:10 proxmox92 kernel: kjournald starting. Commit interval 5 seconds
Jun 12 04:00:23 proxmox92 kernel: r8169 0000:02:00.0: eth0: link up
Jun 12 04:19:20 proxmox92 kernel: r8169 0000:02:00.0: eth0: link up
Jun 12 04:37:19 proxmox92 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Jun 12 04:37:19 proxmox92 kernel: __ratelimit: 61187 callbacks suppressed
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888699
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888700
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888701
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888702
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888703
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888704
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888705
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888706
Jun 12 04:37:19 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888707
Jun 12 04:37:24 proxmox92 kernel: __ratelimit: 4380356 callbacks suppressed
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:24 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: __ratelimit: 4366968 callbacks suppressed
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:29 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: __ratelimit: 4357097 callbacks suppressed
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:34 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: __ratelimit: 4393335 callbacks suppressed
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:39 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: __ratelimit: 4379377 callbacks suppressed
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:44 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: __ratelimit: 4275234 callbacks suppressed
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 04:37:49 proxmox92 kernel: Buffer I/O error on device dm-4, logical block 7888698
Jun 12 06:25:01 proxmox92 kernel: Kernel logging (proc) stopped.
Jun 12 06:25:01 proxmox92 kernel: imklog 3.18.6, log source = /proc/kmsg started.
Jun 12 23:03:48 proxmox92 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.


i am stoped (SIGKILL) then all backup process (vzdump, vmtar,etc) and unlocked the vm. Then i umount the vzsnapshots (one of then shows me a error notice). and when i go to remove the snapshots i found this:

$ lvs
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
data pve -wi-ao 311.41G
hdsiam pve owi-ao 60.00G
root pve -wi-ao 96.00G
swap pve -wi-ao 7.00G
vzsnap-proxmox92-0 pve Swi-I- 1.00G hdsiam 100.00

$pvscan
/dev/dm-4: read failed after 0 of 4096 at 64424443904: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 64424501248: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 4096: Input/output error
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
PV /dev/sda2 VG pve lvm2 [558.41 GB / 82.99 GB free]
Total: 1 [558.41 GB] / in use: 1 [558.41 GB] / in no VG: 0 [0 ]

$lvscan
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
ACTIVE '/dev/pve/swap' [7.00 GB] inherit
ACTIVE '/dev/pve/root' [96.00 GB] inherit
ACTIVE '/dev/pve/data' [311.41 GB] inherit
inactive Original '/dev/pve/hdsiam' [60.00 GB] inherit
inactive Snapshot '/dev/pve/vzsnap-proxmox92-0' [1.00 GB] inherit


And now, here is the problem, I think the snapshot goes full during backup (i dont know how) but th VM is running in production without stop, will i encounter any problem removing the snapshot?, where is storing the changes sinde the date of the crash. The hd of the vm in producction is the /dev/pve/hdsiam wich is "inactive". Must i activate manually(vgchange -ay) the volume /dev/pve/hdsiam?

What will be the best practiques to recover from this state?



PD. other info:
pveversion -v
pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 1.9-55+ovzfix-2
pve-kernel-2.6.32-6-pve: 2.6.32-55+ovzfix-1
pve-kernel-2.6.32-7-pve: 2.6.32-55+ovzfix-2
qemu-server: 1.1-32
pve-firmware: 1.0-15
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-2
ksm-control-daemon: 1.0-6





Lots of thanks.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!