Hello,
we have a 10 node setup with a shared GFS2 Storage (we have used the gfs2-utils in the enterprise repo) and kernel 2.6.32-39-pve
Since our latest upgrate to Proxmox 3.4, we're actually experiencing a serious bug already recognized by RedHat: https://bugzilla.redhat.com/show_bug.cgi?id=712139
When we try to delete a single file, whole cluster hang with filesystem consistency error. We have already checked the whole filesystem with fsck.gfs2 but the issue happen again.
The bug it's already patched by RH in kernel-2.6.32-166.el6.
Could you provide an update version of the pve kernel with this patches?
Thank you in advance for your reply.
Following the panic log:
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: fatal: filesystem consistency error
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: inode = 10 2363351
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1421
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: about to withdraw this file system
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: gfs2_delete_inode: -5
Jul 3 16:13:53 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: telling LM to unmount
Jul 3 16:13:53 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: withdrawn
Jul 3 16:13:53 VMFO07 kernel: Pid: 541404, comm: vi veid: 0 Not tainted 2.6.32-39-pve #1
Jul 3 16:13:53 VMFO07 kernel: Call Trace:
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa05795f8>] ? gfs2_lm_withdraw+0x128/0x160 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa0555880>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa057982d>] ? gfs2_consist_inode_i+0x5d/0x60 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa0575210>] ? gfs2_dinode_dealloc+0x60/0x1e0 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa055c819>] ? gfs2_glock_nq+0x269/0x400 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa0575861>] ? gfs2_delete_inode+0x281/0x530 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff810703e5>] ? __cond_resched+0x65/0x70
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa057567a>] ? gfs2_delete_inode+0x9a/0x530 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa05755e0>] ? gfs2_delete_inode+0x0/0x530 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811cd8c6>] ? generic_delete_inode+0xa6/0x1c0
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811cda35>] ? generic_drop_inode+0x55/0x70
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa05753c7>] ? gfs2_drop_inode+0x37/0x40 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811cbdf6>] ? iput+0xc6/0x100
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811c0466>] ? do_unlinkat+0x1d6/0x240
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811afa95>] ? fput+0x25/0x30
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811aa280>] ? filp_close+0x60/0x90
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811c04e6>] ? sys_unlink+0x16/0x20
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 3 16:13:53 VMFO07 kernel: no_formal_ino = 10
Jul 3 16:13:53 VMFO07 kernel: no_addr = 2363351
Jul 3 16:13:53 VMFO07 kernel: i_size = 0
Jul 3 16:13:53 VMFO07 kernel: blocks = 2
Jul 3 16:13:53 VMFO07 kernel: i_goal = 2363351
Jul 3 16:13:53 VMFO07 kernel: i_diskflags = 0x00000000
Jul 3 16:13:53 VMFO07 kernel: i_height = 0
Jul 3 16:13:53 VMFO07 kernel: i_depth = 0
Jul 3 16:13:53 VMFO07 kernel: i_entries = 0
Jul 3 16:13:53 VMFO07 kernel: i_eattr = 0
Jul 3 16:13:53 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: gfs2_delete_inode: -5
Jul 3 16:13:53 VMFO07 kernel: gdlm_unlock 5,240fd7 err=-22
we have a 10 node setup with a shared GFS2 Storage (we have used the gfs2-utils in the enterprise repo) and kernel 2.6.32-39-pve
Since our latest upgrate to Proxmox 3.4, we're actually experiencing a serious bug already recognized by RedHat: https://bugzilla.redhat.com/show_bug.cgi?id=712139
When we try to delete a single file, whole cluster hang with filesystem consistency error. We have already checked the whole filesystem with fsck.gfs2 but the issue happen again.
The bug it's already patched by RH in kernel-2.6.32-166.el6.
Could you provide an update version of the pve kernel with this patches?
Thank you in advance for your reply.
Following the panic log:
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: fatal: filesystem consistency error
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: inode = 10 2363351
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1421
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: about to withdraw this file system
Jul 3 16:13:52 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: gfs2_delete_inode: -5
Jul 3 16:13:53 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: telling LM to unmount
Jul 3 16:13:53 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: withdrawn
Jul 3 16:13:53 VMFO07 kernel: Pid: 541404, comm: vi veid: 0 Not tainted 2.6.32-39-pve #1
Jul 3 16:13:53 VMFO07 kernel: Call Trace:
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa05795f8>] ? gfs2_lm_withdraw+0x128/0x160 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa0555880>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa057982d>] ? gfs2_consist_inode_i+0x5d/0x60 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa0575210>] ? gfs2_dinode_dealloc+0x60/0x1e0 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa055c819>] ? gfs2_glock_nq+0x269/0x400 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa0575861>] ? gfs2_delete_inode+0x281/0x530 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff810703e5>] ? __cond_resched+0x65/0x70
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa057567a>] ? gfs2_delete_inode+0x9a/0x530 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa05755e0>] ? gfs2_delete_inode+0x0/0x530 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811cd8c6>] ? generic_delete_inode+0xa6/0x1c0
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811cda35>] ? generic_drop_inode+0x55/0x70
Jul 3 16:13:53 VMFO07 kernel: [<ffffffffa05753c7>] ? gfs2_drop_inode+0x37/0x40 [gfs2]
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811cbdf6>] ? iput+0xc6/0x100
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811c0466>] ? do_unlinkat+0x1d6/0x240
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811afa95>] ? fput+0x25/0x30
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811aa280>] ? filp_close+0x60/0x90
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff811c04e6>] ? sys_unlink+0x16/0x20
Jul 3 16:13:53 VMFO07 kernel: [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
Jul 3 16:13:53 VMFO07 kernel: no_formal_ino = 10
Jul 3 16:13:53 VMFO07 kernel: no_addr = 2363351
Jul 3 16:13:53 VMFO07 kernel: i_size = 0
Jul 3 16:13:53 VMFO07 kernel: blocks = 2
Jul 3 16:13:53 VMFO07 kernel: i_goal = 2363351
Jul 3 16:13:53 VMFO07 kernel: i_diskflags = 0x00000000
Jul 3 16:13:53 VMFO07 kernel: i_height = 0
Jul 3 16:13:53 VMFO07 kernel: i_depth = 0
Jul 3 16:13:53 VMFO07 kernel: i_entries = 0
Jul 3 16:13:53 VMFO07 kernel: i_eattr = 0
Jul 3 16:13:53 VMFO07 kernel: GFS2: fsid=ClusterFO:SANStorage.6: gfs2_delete_inode: -5
Jul 3 16:13:53 VMFO07 kernel: gdlm_unlock 5,240fd7 err=-22
Last edited: