barriers deprecation and pve

R

resoli

Renowned Member
Mar 9, 2010
147
4
83
Hello,

I've recently read this article (found the link on this forum):
"The end of block barriers"
http://lwn.net/Articles/400541/

It explains fairly clearly why enabling barriers is so bad for fsync performance,
(try pveperf on a filesystem with and without barriers enabled),
and why kernel developers decided to deprecate that mechanism
(the article was posted in 2010).

The barriers mechanism was abandoned in favor to a more flexible one.
The new mechanism (WRITE_FLUSH_FUA) was introduced by Tejun Heo:
http://lwn.net/Articles/399715/

The article reports also the message on lkmlabout a set of patches
"replace barriers with explicit flush / FUA usage"
http://lwn.net/Articles/400777/
that: "converts over all filesystems to the new WRITE_FLUSH_FUA
primitive that Tejun added. XFS, btrfs, gfs2, reiserfs, ext3 and ext4
have passed extensive xfstests coverage with this"

What is the state of current pve kernel about this? Are these modifications already backported,
so we should consider safe to disable barriers on that filesystems (XFS, btrfs, gfs2, reiserfs,
ext3 and ext4) ?

bye,
rob
 
spirit said:
Hi,
AFAIK nobarrier option is deprecated, so It should change nothing to use it.
http://patchwork.ozlabs.org/patch/80457/
Click to expand...

nobarrier (or the equivalent barrier=0) makes a lot of difference, in terms of fsyncs/sec:

with barrier=0, from /proc/mounts:
/dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered 0 0

# pveperf /var/lib/vz
CPU BOGOMIPS: 38397.44
REGEX/SECOND: 372877
HD SIZE: 29.53 GB (/dev/mapper/pve-data)
BUFFERED READS: 176.09 MB/sec
AVERAGE SEEK TIME: 8.52 ms
FSYNCS/SECOND: 714.08
DNS EXT: 95.30 ms
DNS INT: 50.18 ms

with barrier=1:
/dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0

# pveperf /var/lib/vz
CPU BOGOMIPS: 38397.44
REGEX/SECOND: 374760
HD SIZE: 29.53 GB (/dev/mapper/pve-data)
BUFFERED READS: 183.27 MB/sec
AVERAGE SEEK TIME: 8.40 ms
FSYNCS/SECOND: 34.85
DNS EXT: 93.51 ms
DNS INT: 47.75 ms

spirit said:
I think that this is already backported in redhat 2.6.32 kernel that proxmox use.
Click to expand...

The tests above were done under pve 3.1,
# pveversion
pve-manager/3.1-24/060bd5a6 (running kernel: 2.6.32-26-pve)

# uname -a
Linux pve 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 GNU/Linux

bye,
rob
 
resoli said:
nobarrier (or the equivalent barrier=0) makes a lot of difference, in terms of fsyncs/sec:

with barrier=0, from /proc/mounts:
/dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered 0 0

# pveperf /var/lib/vz
CPU BOGOMIPS: 38397.44
REGEX/SECOND: 372877
HD SIZE: 29.53 GB (/dev/mapper/pve-data)
BUFFERED READS: 176.09 MB/sec
AVERAGE SEEK TIME: 8.52 ms
FSYNCS/SECOND: 714.08
DNS EXT: 95.30 ms
DNS INT: 50.18 ms

with barrier=1:
/dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0

# pveperf /var/lib/vz
CPU BOGOMIPS: 38397.44
REGEX/SECOND: 374760
HD SIZE: 29.53 GB (/dev/mapper/pve-data)
BUFFERED READS: 183.27 MB/sec
AVERAGE SEEK TIME: 8.40 ms
FSYNCS/SECOND: 34.85
DNS EXT: 93.51 ms
DNS INT: 47.75 ms



The tests above were done under pve 3.1,
# pveversion
pve-manager/3.1-24/060bd5a6 (running kernel: 2.6.32-26-pve)

# uname -a
Linux pve 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 GNU/Linux

bye,
rob
Click to expand...


do you have tried with ext4 ?

also maybe to be sure, can you test with 3.2 kernel from debian wheezy ?

(barrier has been removed in 2.6.37 kernel)
 
/dev/mapper/img1-vm--100--disk--1p1 /tmp/100 ext4 rw,relatime,barrier=0,data=ordered 0 0

# pveperf /tmp/100
CPU BOGOMIPS: 38397.44
REGEX/SECOND: 369248
HD SIZE: 9.39 GB (/dev/mapper/img1-vm--100--disk--1p1)
BUFFERED READS: 179.24 MB/sec
AVERAGE SEEK TIME: 7.58 ms
FSYNCS/SECOND: 868.87
DNS EXT: 127.17 ms
DNS INT: 48.32 ms

/dev/mapper/img1-vm--100--disk--1p1 /tmp/100 ext4 rw,relatime,barrier=1,data=ordered 0 0

# pveperf /tmp/100
CPU BOGOMIPS: 38397.44
REGEX/SECOND: 367612
HD SIZE: 9.39 GB (/dev/mapper/img1-vm--100--disk--1p1)
BUFFERED READS: 190.23 MB/sec
AVERAGE SEEK TIME: 7.48 ms
FSYNCS/SECOND: 44.43
DNS EXT: 81.46 ms
DNS INT: 48.79 ms

Will try on a vanilla wheezy ...

rob
 
You must log in or register to reply here.
Top Bottom