BUG: GFS2 Issue Kernel Panic when delete on a new fresh filesystem.

[robynhub]

As of RHEL 6.4 (means supported in Proxmox)
https://access.redhat.com/documenta...r_Administration/thinprovisioned_volumes.html

Proxmox use the -T option by default when create a new virtual disk or new pool?
In short words, proxmox uses thin volumes by default?

 

[dietmar]

In short words, proxmox uses thin volumes by default?

No, you cannot use dm-thin with shared storage (that is dangerous and does not work).

 

[mir]

Since thin provisioned lvm volumes cannot be shared across a cluster I guess not??

A work around could be to create the VM's using Qcow2 since Qcow2 by default, and also in Proxmox, are thin provisioned. Qcow2 has another positive benefit that you will gain the ability to do snapshots and clones.

Eg. create the pv over iSCSI multipath. On this pv create a normal vg and normal lv(s). The create a normal ext4 filesystem on these lv's and create your vm's using Qcow2.

Pros:
- Multipath backed lv's
- Thin provisioned virtual disks for vm's

Cons:
- You loose approximately 5-10% disk performance compared to raw but compared to gfs you will see 25-30% increase of performance (my experience).

 

[robynhub]

I don't think that my customers SQL servers would very happy. The performances are almost acceptables with 4 path iscsi multipath yet.
And live migration will continue to work? It won't lock the whole ext4 filesystem on a single cluster node? Or worst it won't corrupt the ext4 filesystem with multiple mounts and many journals opened simultaneusly?

With GFS2 I'm having a pretty good performances. I can't complain...

 

[mir]

In my example live migration will also work!!

As to your performance claim I do not think you have facts to proof your assumption. I have done extensive tests which have proofed my figures.

And since you are experiencing data loss while my suggestion is rock solid I would not think twice;-) Performance and data integrity are contradictory of which I favor data integrity and this is especially true for databases. But again YMMV.

 

[robynhub]

In my example live migration will also work!!

As to your performance claim I do not think you have facts to proof your assumption. I have done extensive tests which have proofed my figures.

And since you are experiencing data loss while my suggestion is rock solid I would not think twice;-) Performance and data integrity are contradictory of which I favor data integrity and this is especially true for databases. But again YMMV.

For the performances maybe you're right. But how it's possible the mounting the same ext4 filesystem on 10 nodes at the same time without corruption? It does not have any logic for me...

 

[mir]

You missed my point. You create a filesystem on the logical volume and your vm's is created as a qcow2 file in this filesystem. All you need to be sure of is that a vm cannot be running on more than one node at the time, this means proper fencing.

But do extensive tests before you deploy in production.

BTW. have you tried with the 3.10 kernel? (means you loose ability to create openvz containers)

 

[dietmar]

Eg. create the pv over iSCSI multipath. On this pv create a normal vg and normal lv(s). The create a normal ext4 filesystem on these lv's and create your vm's using Qcow2.

Note sure if I understand you correctly, but that sounds quite dangerous to me. You cannot mount a shared device with ext4 on more than one host.

 

[robynhub]

You missed my point. You create a filesystem on the logical volume and your vm's is created as a qcow2 file in this filesystem. All you need to be sure of is that a vm cannot be running on more than one node at the time, this means proper fencing.

But do extensive tests before you deploy in production.

BTW. have you tried with the 3.10 kernel? (means you loose ability to create openvz containers)

Maybe i could immagine what you want to do but seems too risky and complicated to use it in a production environment.
By the kernel plz read the other posts.

Ty anyway.

 

[Thomas Lamprecht]

Tried it today on PVE3.4 with the

$ uname -a
Linux pve 3.10.0-11-pve #1 SMP Tue Jul 21 08:59:46 CEST 2015 x86_64 GNU/Linux

Kernel and it works, at least I can write and delete as much as I want without running into problems yet.

So it's sure worth a try, but keep in mind that you will loose the OpenVZ container abillity. As long as you're only running VMs, you're good to go.

 

[robynhub]

Tried it today on PVE3.4 with the

$ uname -a
Linux pve 3.10.0-11-pve #1 SMP Tue Jul 21 08:59:46 CEST 2015 x86_64 GNU/Linux

Kernel and it works, at least I can write and delete as much as I want without running into problems yet.

So it's sure worth a try, but keep in mind that you will loose the OpenVZ container abillity. As long as you're only running VMs you're good to go.

Actually on the repo I can find only pve-kernel-3.10.0-10-pve. But i can give it a try on my testing cluster.
We don't use openvz containers at all.

I'll make some tests and I will let you know how it's going...

Ty.

 

[robynhub]

Tried it today on PVE3.4 with the

 $ uname -a Linux pve 3.10.0-11-pve #1 SMP Tue Jul 21 08:59:46 CEST 2015 x86_64 GNU/Linux

Kernel and it works, at least I can write and delete as much as I want without running into problems yet. So it's sure worth a try, but keep in mind that you will loose the OpenVZ container abillity. As long as you're only running VMs, you're good to go.

Bug confirmed and still present also with latest release 2.6.32-40-pve.

 

[t.lamprecht]

GFS2 for the newer OpenVZ 2.6.32 kernel probably won't work again, as OpenVZ, as far as I've seen, declared the bug as "won't fix" (see here), and between the GFS2 working and GFS2 not working releases of the OVZ Kernel they changed and refactored a lot, so no easy bug fix available, sorry.

It is working with the 3.10 kernel which is available in the latest stable PVE and also with the Kernel from PVE4beta, though.

So the options are:
- If you need the OpenVZ Kernel because you use OVZ Containers and GFS2 stay at the kernel where GFS2 is still working (2.6.32-26-pve)
- If you don't use OVZ Containers the 3.10 kernel is an good option, GFS2 is working there as far as I've tested it.
- In PVE4 (note that only a beta is available, not for production) you can use containers (LXC) and GFS2 is also working.

Note: I couldn't test everything from GFS2 on the 3.10 kernel or PVE4, so it would be great if someone could confirm the working status there.

 

[robynhub]

Hello,

We don't need an OpenVZ Kernel but it's a production installation e we would like to avoid unstable solutions. The 3.10 Kernel it's considered as part of the stable distribution?
We have bought almost 15 community licenses to gain some stabilty...

Thank you for your answer.

 

[t.lamprecht]

Yes 3.10 is considered as stable, it's the latest redhat kernel. You really should be good to update to 3.10.
Al tough as normally and understandable kernel updates always are a bit risky, it sure wouldn't hurt to try it on a test system first.

 

[robynhub]

Yes 3.10 is considered as stable, it's the latest redhat kernel. You really should be good to update to 3.10.
Al tough as normally and understandable kernel updates always are a bit risky, it sure wouldn't hurt to try it on a test system first.

There is any official procedure to upgrade? How i can install the 3.10 kernel without scramble everything (it's a running production cluster of 10 nodes).

 

[t.lamprecht]

A "normal" kernel update should do it. Execute that on every note

apt-get update
apt-get install pve-kernel-3.10.0-11-pve

and reboot.
If you haven't a test system, do it on one node only first and reboot that one and look if everything works as expected.
But as I said even with the stablest Kernels, these updates always have a little rest risk for production when you don't test them before, but I don't want to scare you out i made the upgrade and hadn't any problem at all.

 

[robynhub]

Hello again,

i've tryed to upgrade the kernel and, without mounting the production fs, i've tryed to create a local gfs2 filesystem to test the issue:

root@VMFO07:~# uname -a
Linux VMFO07 3.10.0-11-pve #1 SMP Tue Jul 21 08:59:46 CEST 2015 x86_64 GNU/Linux
root@VMFO07:~# dd if=/dev/zero of=testfs bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.165824 s, 1.3 GB/s
root@VMFO07:~# mkfs.gfs2 -p lock_nolock -j 1 -t test:testfs testfs
This will destroy any data on testfs.
It appears to contain: data
Are you sure you want to proceed? [y/n]y
Device: testfs
Blocksize: 4096
Device Size 0.20 GB (51200 blocks)
Filesystem Size: 0.20 GB (51197 blocks)
Journals: 1
Resource Groups: 1
Locking Protocol: "lock_nolock"
Lock Table: "test:testfs"
UUID: 8fba1ba7-63de-f08b-924d-9638a7f6065d
root@VMFO07:~# mkdir tempdir
root@VMFO07:~# mount -t gfs2 testfs tempdir

Message from syslogd@VMFO07 at Aug 4 11:30:19 ...
kernel:[ 216.420809] Kernel panic - not syncing: Fatal exception
Write failed: Broken pipe

I've tryed to get some informations of the panic but nothing it's shown in the logs. I've made a "real" screenshot with my phone:



I don't know if it can give some hints... In the mean time I will fallback to the 2.6.32-26-pve.

Ty.


-- Small Update:

After downgrade I haven't any issue:

root@VMFO07:~/test# uname -a
Linux VMFO07 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 GNU/Linux
root@VMFO07:~/test# cd tempdir/
root@VMFO07:~/test/tempdir# ls
root@VMFO07:~/test/tempdir# cd ..
root@VMFO07:~/test# ls
tempdir testfs
root@VMFO07:~/test# mount -t gfs2 testfs tempdir
root@VMFO07:~/test# cd tempdir/
root@VMFO07:~/test/tempdir# ls
root@VMFO07:~/test/tempdir# touch pippo
root@VMFO07:~/test/tempdir# rm pippo
root@VMFO07:~/test/tempdir# ls
root@VMFO07:~/test/tempdir# cd ..
root@VMFO07:~/test# umount tempdir
root@VMFO07:~/test#

Regards.

 

[t.lamprecht]

Did you rebbot the system after the update? And is testfs a valid device?


I upgraded from a 2.6.32 to 3.10 and tried it its working here. Something seems fishy here...

root@34compiler:/# mkfs.gfs2 -p lock_nolock -t test:testfs -j 1 /dev/vdb1 
It appears to contain an existing filesystem (gfs2)
This will destroy any data on /dev/vdb1
Are you sure you want to proceed? [y/n]y
Device:                    /dev/vdb1
Block size:                4096
Device size:               32.00 GB (8388347 blocks)
Filesystem size:           32.00 GB (8388346 blocks)
Journals:                  1
Resource groups:           129
Locking protocol:          "lock_nolock"
Lock table:                "test:testfs"
UUID:                      83111d7c-efb1-a7d4-6533-54cf6ba3bbc5
root@34compiler:/# mount -t gfs2 /dev/vdb1 /gfs/
root@34compiler:/# cd /gfs/
root@34compiler:/gfs# ls
root@34compiler:/gfs# touch test
root@34compiler:/gfs# dd if=/dev/zero of=test.file count=1 bs=100M
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 0.87393 s, 120 MB/s
root@34compiler:/gfs# rm test
root@34compiler:/gfs# ls
test.file
root@34compiler:/gfs# mkdir temp
root@34compiler:/gfs# cd temp/
root@34compiler:/gfs/temp# touch etr
root@34compiler:/gfs/temp# ls
etr
root@34compiler:/gfs/temp# rm etr 
root@34compiler:/gfs/temp# ls
root@34compiler:/gfs/temp# cd ..
root@34compiler:/gfs# uname -a
Linux 34compiler 3.10.0-9-pve #1 SMP Thu Apr 30 09:31:13 CEST 2015 x86_64 GNU/Linux

 

[robynhub]

Did you rebbot the system after the update? And is testfs a valid device?

Of course I've rebooted. But the kernel you tested is'nt the same:

3.10.0-9-pve != 3.10.0-11-pve

The device isn't a real block device. It's just a zero filled file as you done in your tests.