This is very unusual so I am posting this to the forum to see if others know why. The system that ran into this reporting issue is running:
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-8
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-17
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-4
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1
The system become non responsive which I had to cold reboot. Per my inspection of syslog I found the following errors around the time of incidence:
Feb 28 04:15:13 vpshost36 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs error (device dm-3): ext3_get_inode_loc: unable to read inode block - inode=17235982, block=
68943874
Feb 28 04:15:14 vpshost36 kernel: __ratelimit: 108 callbacks suppressed
Feb 28 04:15:14 vpshost36 kernel: Buffer I/O error on device dm-3, logical block 0
Feb 28 04:15:14 vpshost36 kernel: lost page write due to I/O error on dm-3
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs (dm-3): I/O error while writing superblock
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs (dm-3): error in ext3_reserve_inode_write: IO failure
Feb 28 04:15:14 vpshost36 kernel: Buffer I/O error on device dm-3, logical block 0
Feb 28 04:15:14 vpshost36 kernel: lost page write due to I/O error on dm-3
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs (dm-3): I/O error while writing superblock
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs error (device dm-3): ext3_get_inode_loc: unable to read inode block - inode=17235981, block=
68943874
I also noticed that per each back up there were file system fixes similar to:
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054124
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054091
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054088
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054074
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054071
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062696
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062695
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062694
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062693
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062690
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187023
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187022
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187021
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187020
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187013
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196439
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196435
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196434
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196433
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196430
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 1869926
This happened when the system was doing a backs to nfs mount file system. My guess is this is related to NFS but not certain. Any ideas why this happened?
Thanks
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-8
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-17
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-4
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1
The system become non responsive which I had to cold reboot. Per my inspection of syslog I found the following errors around the time of incidence:
Feb 28 04:15:13 vpshost36 kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs error (device dm-3): ext3_get_inode_loc: unable to read inode block - inode=17235982, block=
68943874
Feb 28 04:15:14 vpshost36 kernel: __ratelimit: 108 callbacks suppressed
Feb 28 04:15:14 vpshost36 kernel: Buffer I/O error on device dm-3, logical block 0
Feb 28 04:15:14 vpshost36 kernel: lost page write due to I/O error on dm-3
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs (dm-3): I/O error while writing superblock
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs (dm-3): error in ext3_reserve_inode_write: IO failure
Feb 28 04:15:14 vpshost36 kernel: Buffer I/O error on device dm-3, logical block 0
Feb 28 04:15:14 vpshost36 kernel: lost page write due to I/O error on dm-3
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs (dm-3): I/O error while writing superblock
Feb 28 04:15:14 vpshost36 kernel: EXT3-fs error (device dm-3): ext3_get_inode_loc: unable to read inode block - inode=17235981, block=
68943874
I also noticed that per each back up there were file system fixes similar to:
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054124
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054091
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054088
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054074
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 9054071
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062696
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062695
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062694
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062693
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 6062690
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187023
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187022
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187021
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187020
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 5187013
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196439
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196435
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196434
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196433
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 4196430
Feb 28 03:23:37 vpshost36 kernel: ext3_orphan_cleanup: deleting unreferenced inode 1869926
This happened when the system was doing a backs to nfs mount file system. My guess is this is related to NFS but not certain. Any ideas why this happened?
Thanks