BUG: GFS2 Issue Kernel Panic when delete on a new fresh filesystem.

[robynhub]

Hello everyone.

We are using the enterprise repo on a 10-node cluster running Proxmox 3.4 (latest version).
We're actually experiencing a strange behaviour with GFS2. We're trying to use GFS2 but with a new and clean filesystem we receive a kernel panic when trying to delete some file.
Step by step to reproduce the issue follow (VMFO07 it's the only node logged into the SAN):


root@VMFO07:~# dpkg -l gfs2*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================-================-================-====================================================
un gfs2-tools <none> (no description available)
ii gfs2-utils 3.1.3-1 amd64 Global file system 2 tools

root@VMFO07:~# uname -a
Linux VMFO07 2.6.32-39-pve #1 SMP Fri May 8 11:27:35 CEST 2015 x86_64 GNU/Linux

root@VMFO07:~# /etc/init.d/open-iscsi start
Starting iSCSI initiator service: iscsid.
Setting up iSCSI targets:
Logging in to [iface: default, target: [snip], portal: 192.168.193.170,3260] (multiple)
Logging in to [iface: default, target: [snip], portal: 192.168.193.171,3260] (multiple)
Logging in to [iface: default, target: [snip], portal: 192.168.194.171,3260] (multiple)
Logging in to [iface: default, target: i[snip], portal: 192.168.194.170,3260] (multiple)
Login to [iface: default, target: [snip]: 192.168.193.170,3260] successful.
Login to [iface: default, target: [snip], portal: 192.168.193.171,3260] successful.
Login to [iface: default, target: [snip], portal: 192.168.194.171,3260] successful.
Login to [iface: default, target: [snip], portal: 192.168.194.170,3260] successful.

root@VMFO07:~# multipath -ll
[snip] dm-0 HP,MSA 1040 SAN
size=9.1T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 7:0:0:0 sdb 8:16 active ready running
|- 9:0:0:0 sdc 8:32 active ready running
|- 10:0:0:0 sdd 8:48 active ready running
`- 8:0:0:0 sde 8:64 active ready running

root@VMFO07:~# mkfs.gfs2 -V
gfs2_mkfs master (built Mar 15 2013 08:54:14)
Copyright (C) Red Hat, Inc. 2004-2010 All rights reserved.

root@VMFO07:~# mkfs.gfs2 -p lock_dlm -t ClusterFO:SANStorage -j 16 /dev/dm-0
Are you sure you want to proceed? [y/n]y
Device: /dev/dm-0
Blocksize: 4096
Device Size 9305.13 GB (2439283712 blocks)
Filesystem Size: 9305.13 GB (2439283710 blocks)
Journals: 16
Resource Groups: 9306
Locking Protocol: "lock_dlm"
Lock Table: "ClusterFO:SANStorage"
UUID: 1b7a065f-d126-a302-0ee9-c9682e5326f0

root@VMFO07:~# fsck.gfs2 -V
GFS2 fsck master (built Mar 15 2013 08:54:10)
Copyright (C) Red Hat, Inc. 2004-2010 All rights reserved.

root@VMFO07:~# fsck.gfs2 -f -y /dev/dm-0
Initializing fsck
Validating Resource Group index.
Level 1 rgrp check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Starting pass1
Pass1 complete
Starting pass1b
Pass1b complete
Starting pass1c
Pass1c complete
Starting pass2
Pass2 complete
Starting pass3
Pass3 complete
Starting pass4
Pass4 complete
Starting pass5
Pass5 complete
gfs2_fsck complete

root@VMFO07:~# mount -t gfs2 /dev/dm-0 /mnt/SANStorage
root@VMFO07:~#
root@VMFO07:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=8238674,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6593064k,mode=755)
[snip]
/dev/dm-0 on /mnt/SANStorage type gfs2 (rw,relatime,hostdata=jid=0)

root@VMFO07:~# cd /mnt/SANStorage
root@VMFO07:/mnt/SANStorage# touch testfile
root@VMFO07:/mnt/SANStorage# rm testfile
root@VMFO07:/mnt/SANStorage# ls
ls: cannot open directory .: Input/output error
root@VMFO07:/mnt/SANStorage#

At this state if I try to unmount the system hang and i need to manually fence the node to get it running.

Here the logs:

Loading iSCSI transport class v2.0-870.
iscsi: registered transport (tcp)
iscsi: registered transport (iser)
scsi7 : iSCSI Initiator over TCP/IP
scsi8 : iSCSI Initiator over TCP/IP
scsi9 : iSCSI Initiator over TCP/IP
scsi10 : iSCSI Initiator over TCP/IP
scsi 7:0:0:0: Direct-Access HP MSA 1040 SAN G105 PQ: 0 ANSI: 5
scsi 9:0:0:0: Direct-Access HP MSA 1040 SAN G105 PQ: 0 ANSI: 5
sd 7:0:0:0: Attached scsi generic sg3 type 0
sd 9:0:0:0: Attached scsi generic sg4 type 0
sd 7:0:0:0: [sdb] 19514269696 512-byte logical blocks: (9.99 TB/9.08 TiB)
scsi 10:0:0:0: Direct-Access HP MSA 1040 SAN G105 PQ: 0 ANSI: 5
sd 9:0:0:0: [sdc] 19514269696 512-byte logical blocks: (9.99 TB/9.08 TiB)
sd 10:0:0:0: Attached scsi generic sg5 type 0
scsi 8:0:0:0: Direct-Access HP MSA 1040 SAN G105 PQ: 0 ANSI: 5
sd 10:0:0:0: [sdd] 19514269696 512-byte logical blocks: (9.99 TB/9.08 TiB)
sd 8:0:0:0: Attached scsi generic sg6 type 0
sd 8:0:0:0: [sde] 19514269696 512-byte logical blocks: (9.99 TB/9.08 TiB)
sd 7:0:0:0: [sdb] Write Protect is off
sd 7:0:0:0: [sdb] Mode Sense: fb 00 00 08
sd 9:0:0:0: [sdc] Write Protect is off
sd 9:0:0:0: [sdc] Mode Sense: fb 00 00 08
sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 10:0:0:0: [sdd] Write Protect is off
sd 10:0:0:0: [sdd] Mode Sense: fb 00 00 08
sd 9:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 8:0:0:0: [sde] Write Protect is off
sd 8:0:0:0: [sde] Mode Sense: fb 00 00 08
sd 10:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 8:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdb:
sdc:
sdd: unknown partition table
sde: unknown partition table
unknown partition table
unknown partition table
sd 7:0:0:0: [sdb] Attached SCSI disk
sd 9:0:0:0: [sdc] Attached SCSI disk
sd 10:0:0:0: [sdd] Attached SCSI disk
sd 8:0:0:0: [sde] Attached SCSI disk
GFS2 (built Jun 24 2015 06:26:56) installed
GFS2: fsid=: Trying to join cluster "lock_dlm", "ClusterFO:SANStorage"
GFS2: fsid=ClusterFO:SANStorage.0: Joined cluster. Now mounting FS...
GFS2: fsid=ClusterFO:SANStorage.0: jid=0, already locked for use
GFS2: fsid=ClusterFO:SANStorage.0: jid=0: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=0: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=1: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=1: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=1: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=2: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=2: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=2: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=3: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=3: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=3: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=4: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=4: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=4: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=5: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=5: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=5: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=6: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=6: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=6: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=7: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=7: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=7: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=8: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=8: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=8: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=9: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=9: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=9: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=10: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=10: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=10: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=11: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=11: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=11: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=12: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=12: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=12: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=13: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=13: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=13: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=14: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=14: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=14: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=15: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=15: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=15: Done
GFS2: fsid=ClusterFO:SANStorage.0: fatal: filesystem consistency error
GFS2: fsid=ClusterFO:SANStorage.0: inode = 1 529788
GFS2: fsid=ClusterFO:SANStorage.0: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1421
GFS2: fsid=ClusterFO:SANStorage.0: about to withdraw this file system
GFS2: fsid=ClusterFO:SANStorage.0: telling LM to unmount
GFS2: fsid=ClusterFO:SANStorage.0: withdrawn
Pid: 7059, comm: rm veid: 0 Not tainted 2.6.32-39-pve #1
Call Trace:
[<ffffffffa07cb5f8>] ? gfs2_lm_withdraw+0x128/0x160 [gfs2]
[<ffffffffa07a7880>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]
[<ffffffffa07cb82d>] ? gfs2_consist_inode_i+0x5d/0x60 [gfs2]
[<ffffffffa07c7210>] ? gfs2_dinode_dealloc+0x60/0x1e0 [gfs2]
[<ffffffffa07ae819>] ? gfs2_glock_nq+0x269/0x400 [gfs2]
[<ffffffffa07c7861>] ? gfs2_delete_inode+0x281/0x530 [gfs2]
[<ffffffffa07c767a>] ? gfs2_delete_inode+0x9a/0x530 [gfs2]
[<ffffffffa07c75e0>] ? gfs2_delete_inode+0x0/0x530 [gfs2]
[<ffffffff811cd9a6>] ? generic_delete_inode+0xa6/0x1c0
[<ffffffff811cdb15>] ? generic_drop_inode+0x55/0x70
[<ffffffffa07c73c7>] ? gfs2_drop_inode+0x37/0x40 [gfs2]
[<ffffffff811cbed6>] ? iput+0xc6/0x100
[<ffffffff811c0546>] ? do_unlinkat+0x1d6/0x240
[<ffffffff811b3f8a>] ? sys_newfstatat+0x2a/0x40
[<ffffffff811c1c4b>] ? sys_unlinkat+0x1b/0x50
[<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b
no_formal_ino = 1
no_addr = 529788
i_size = 0
blocks = 2
i_goal = 529788
i_diskflags = 0x00000000
i_height = 0
i_depth = 0
i_entries = 0
i_eattr = 0
GFS2: fsid=ClusterFO:SANStorage.0: gfs2_delete_inode: -5
gdlm_unlock 5,8157c err=-22

Can this be caused by an older version of gfs2 utils? With the previous kernel we don't have any issue.
I understand that GFS2 isn't officially supported by proxmox but we really need a working clustered filesystem usable in a Proxmox Environment.


Thank you in advance.

 

[robynhub]

UPDATE:

The bug it's kernel related. If i boot with 2.6.32-26-pve the issue isn't there:

root@VMFO07:~# uname -a
Linux VMFO07 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 GNU/Linux
root@VMFO07:~# multipath -ll
[snip] dm-0 HP,MSA 1040 SAN
size=9.1T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 4:0:0:0 sdc 8:32 active ready running
|- 1:0:0:0 sdb 8:16 active ready running
|- 3:0:0:0 sde 8:64 active ready running
`- 2:0:0:0 sdd 8:48 active ready running
root@VMFO07:~# mkfs.gfs2 -p lock_dlm -t ClusterFO:SANStorage -j 16 /dev/dm-0
This will destroy any data on /dev/dm-0.
Are you sure you want to proceed? [y/n]y
Device: /dev/dm-0
Blocksize: 4096
Device Size 9305.13 GB (2439283712 blocks)
Filesystem Size: 9305.13 GB (2439283710 blocks)
Journals: 16
Resource Groups: 9306
Locking Protocol: "lock_dlm"
Lock Table: "ClusterFO:SANStorage"
UUID: 1762b00f-8e4d-2194-727f-29f4b8319183

root@VMFO07:~# mount -t gfs2 /dev/dm-0 /mnt/SANStorage
root@VMFO07:~# cd /mnt/SANStorage/
root@VMFO07:/mnt/SANStorage# ls
root@VMFO07:/mnt/SANStorage# touch pippo
root@VMFO07:/mnt/SANStorage# rm pippo
root@VMFO07:/mnt/SANStorage# ls
root@VMFO07:/mnt/SANStorage# rm testfile
root@VMFO07:/mnt/SANStorage# ls
root@VMFO07:/mnt/SANStorage# dd if=/dev/urandom of=testfile bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 10.3469 s, 10.1 MB/s
root@VMFO07:/mnt/SANStorage# ls
testfile
root@VMFO07:/mnt/SANStorage# rm testfile
root@VMFO07:/mnt/SANStorage#

The logs show nothing wrong:

GFS2 (built Oct 14 2013 08:11:53) installed
GFS2: fsid=: Trying to join cluster "lock_dlm", "ClusterFO:SANStorage"
GFS2: fsid=ClusterFO:SANStorage.0: Joined cluster. Now mounting FS...
GFS2: fsid=ClusterFO:SANStorage.0: jid=0, already locked for use
GFS2: fsid=ClusterFO:SANStorage.0: jid=0: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=0: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=1: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=1: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=1: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=2: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=2: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=2: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=3: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=3: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=3: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=4: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=4: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=4: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=5: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=5: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=5: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=6: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=6: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=6: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=7: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=7: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=7: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=8: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=8: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=8: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=9: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=9: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=9: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=10: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=10: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=10: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=11: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=11: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=11: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=12: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=12: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=12: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=13: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=13: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=13: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=14: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=14: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=14: Done
GFS2: fsid=ClusterFO:SANStorage.0: jid=15: Trying to acquire journal lock...
GFS2: fsid=ClusterFO:SANStorage.0: jid=15: Looking at journal...
GFS2: fsid=ClusterFO:SANStorage.0: jid=15: Done



PLS Fix this huge bug asap. The GFS2 support is broken!

Seriusly guys, please include GFS2 in the official support for Proxmox VE.
It need only a one-block-device working clustered filesystem to become a Virtualization Enviroment worthy of its name.

Thank you in advance for your support.

 

[robynhub]

I can't belive that i'm the only one with this issue...

 

[robynhub]

2 K euros/year in community licenses and no answers in 4 days.
Not even to confirm or deny the bug. Great service.

 

[dietmar]

What is the output of

# pveversion -v

 

[Thomas Lamprecht]

Hey I can reproduce your problem, even with lock_nolock (so without fencing), testet with 2.6.32-39-pve kernel...

The problem already showed up on the OpenVZ bug tracker, and also some other trackers, and was less or more marked as wont fix (at least in the OpenVZ one) and seems to persist only in some kernel, older and newer work fine. Debugging that seems tricky.
I guess migrating to an something other than GFS2 isn't an option for you? Would seem easier...

 

[robynhub]

What is the output of

# pveversion -v

root@VMFO07:~# pveversion -v
proxmox-ve-2.6.32: 3.4-157 (running kernel: 2.6.32-26-pve)
pve-manager: 3.4-6 (running version: 3.4-6/102d4547)
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-18
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

 

[mir]

Looks like you have forgotten to reboot your server after upgrade!!!

running kernel: 2.6.32-26-pve

available kernels:
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-26-pve: 2.6.32-114

 

[robynhub]

No sir. I've downgraded to the previous kernel to have an usble filesystem.

 

[mir]

pve-kernel-2.6.32-26-pve is not the previous kernel. It might be the previous kernel on your system but as said not the previous kernel from proxmox.

 

[robynhub]

It'is a recent installation and it was the only other kernel installed. As described in the previous posts, as far i know the issue it's present in the latest enterprise kernel. I don't know if other versions are affected.

 

[robynhub]

I can't choose another filesystem. It would be great but there aren't other usable options. Ocfs2 it's deprecated and not supported by proxmox kernels. Gfs2 it's the only option for a one-block-device clustered filesystem. Multipath iscsi it's a non sacrificable option for us.

 

[robynhub]

Hey I can reproduce your problem, even with lock_nolock (so without fencing), testet with 2.6.32-39-pve kernel...

The problem already showed up on the OpenVZ bug tracker, and also some other trackers, and was less or more marked as wont fix (at least in the OpenVZ one) and seems to persist only in some kernel, older and newer work fine. Debugging that seems tricky.
I guess migrating to an something other than GFS2 isn't an option for you? Would seem easier...

glad to hear it. Two weeks ago I opened a thread for the same problem without a solution: http://forum.proxmox.com/threads/22725-Bug-in-GFS2-Support-in-2-6-32-39-pve

 

[Thomas Lamprecht]

I got some news, I successfully (at least some simple test are working), ported GFS2 to the Proxmox VE 4beta, as I wanted to see how the status with PVE and GFS2 is now on a current kernel and it is alot easier to do that than to dig into 2.6.32 changes.
So if you're planning to update your PVE to version 4 as soon it's stable you MAY be good with GFS2. I have to say that besides the intern GFS2 testsuite and some simple tests (mount, touch, rm, umount, mount again, dd some data, ls, ...), I didn't test much. So no promises that everything works perfectly out of the box.
I used the newest stable GFS2 (3.1.8) and replaced cman with PVE's clustermanager (pvecm).

I'll try to look into the 2.6.32 kernel tomorrow, but - as usual ^^ - no promises...
Easiest thing for now, migrate to something like Ceph or wait for PVE4 (beta is out) and hope.

 

[robynhub]

I got some news, I successfully (at least some simple test are working), ported GFS2 to the Proxmox VE 4beta, as I wanted to see how the status with PVE and GFS2 is now on a current kernel and it is alot easier to do that than to dig into 2.6.32 changes.
So if you're planning to update your PVE to version 4 as soon it's stable you MAY be good with GFS2. I have to say that besides the intern GFS2 testsuite and some simple tests (mount, touch, rm, umount, mount again, dd some data, ls, ...), I didn't test much. So no promises that everything works perfectly out of the box.
I used the newest stable GFS2 (3.1.8) and replaced cman with PVE's clustermanager (pvecm).

I'll try to look into the 2.6.32 kernel tomorrow, but - as usual ^^ - no promises...
Easiest thing for now, migrate to something like Ceph or wait for PVE4 (beta is out) and hope.

Could be fine. But it's a production server and we won't upgrade to an unstable solution. We will wait for an updated and patched solution with stable 2.6 kernel. Until then we will use the 2.6.32-29 kernel.

I've almost lost 10 Tbyte of customers data cause of it. Thankfully there is the live storage migration that allow us to temporally migrate to a slower NAS, downgrade the kernel and put all the data back into the SAN (whe had to do that since the slowiness of the NAS).

In the next days, or better when a kernel update will be available, I will setup a testing cluster (more licenses to buy!) to verify if the bug it's still there before use it on the production servers.
It's a bit sad that I should do that since we have bought the enterprise licenses exactly to avoid this kind of works. Kernel testing isn't exactly our core business. We will not migrate to ceph anyway since our cluster it's not designed to support it (we don't have any dedicated 10Gigabit interfaces to use with Ceph).
99% of clusters and blade servers i saw have all a iSCSI or Fiber Channel SAN and many blades. This is the standard architecture for VMWare and HyperV clusters. Who want to avoid the huge license fee that those softwares needs, may choose proxmox but it should support a one-block-device clustered filesystem comparable with Microsoft's Cluster Shared Volumes or VMFS by VMWare. I think that is the first reason to not use Proxmox as Virtualization Enviroment in the real business world. And it's a shame 'couse the front-end and the others tools are very, very good products! Our company it's grown with our open source software and with our contributions to the community and we would like so much to adopt Proxmox VE in the next years. Please keep our suggestions and reports as a constructive critics to build a better VE.

Thank you for your support anyway. I will, hopefully, look forward for any solutions in the stable kernel branch or upgrade to 4.0 when stable in the enterprise repo.

Best Regards.

ps. I suggest to warn the others users to avoid the usage of GFS2 until this bug will be fixed, it's easy to loss data with it.

 

[dietmar]

99% of clusters and blade servers i saw have all a iSCSI or Fiber Channel SAN and many blades. This is the standard architecture for VMWare and HyperV clusters. Who want to avoid the huge license fee that those softwares needs, may choose proxmox but it should support a one-block-device clustered filesystem comparable with Microsoft's Cluster Shared Volumes or VMFS by VMWare.

So why don't you simply use LVM on top of those iSCSI devices?

 

[Thomas Lamprecht]

I understand that the situation is, at least, uncomfortable to you. I'll take a look into the changes from 2.6.32-26-pve and when there is a relative quick fix I'll patch it.

I wonder a bit why nobody else got trouble from GFS2, as packages which get into the enterprise repository are deployed and tested by a lot (every non subscriber and that's really a lot) of people and we didn't get a bug report before yours, AFAIK.

I think your best way is to wait until 4.0 is released, for now. I'll update this post if I got some news, though.

 

[robynhub]

So why don't you simply use LVM on top of those iSCSI devices?

For Two main reasons:

1. LVM on iSCSI doesn't support multipath. This is very important for us even for load balancing or fallback purposes.

2. Most important, LVM doesn't allow to overcommit the storage. Let me explain this. In one of our clusters we have almost 200 VMs where many of those have configured an hd of 500 Gb used only at 0,5,1%. If those storages are on LVM we should buy a storage of 100 Tb and almost fully unused! With a filesystem that support sparse files, i can overcommit and add storage whenever is needed with another SAN's storage enclosure and doing an online grow of the filesystem. It allow our company to grow more gradually even with our systems. Actually the 200 VMs use only 10 Tb out of 15 Tb of SAN storage!

 

[robynhub]

I understand that the situation is, at least, uncomfortable to you. I'll take a look into the changes from 2.6.32-26-pve and when there is a relative quick fix I'll patch it.

I wonder a bit why nobody else got trouble from GFS2, as packages which get into the enterprise repository are deployed and tested by a lot (every non subscriber and that's really a lot) of people and we didn't get a bug report before yours, AFAIK.

I think your best way is to wait until 4.0 is released, for now. I'll update this post if I got some news, though.

I think that many users have this bug but they haven't experienced yet becouse they don't have already deleted a file on the filesystem! I've ran my cluster almost for a month with this bug without issues just becouse I haven't deleted any file in this time.
It's a very dangerous situation that many users could are without knowing anything of it.



Un/Fortunately I have a natural predisposition to find bugs.

Ty for your time.

 

[mir]

1. LVM on iSCSI doesn't support multipath. This is very important for us even for load balancing or fallback purposes.

If you use multipath to your iSCSI luns then LVM per se will also support multipath.
http://forum.proxmox.com/threads/11938-Multipath-iSCSI-LVM-and-Cluster

2. Most important, LVM doesn't allow to overcommit the storage. Let me explain this. In one of our clusters we have almost 200 VMs where many of those have configured an hd of 500 Gb used only at 0,5,1%. If those storages are on LVM we should buy a storage of 100 Tb and almost fully unused! With a filesystem that support sparse files, i can overcommit and add storage whenever is needed with another SAN's storage enclosure and doing an online grow of the filesystem. It allow our company to grow more gradually even with our systems. Actually the 200 VMs use only 10 Tb out of 15 Tb of SAN storage!

As of RHEL 6.4 (means supported in Proxmox)
https://access.redhat.com/documenta...r_Administration/thinprovisioned_volumes.html