How safe is nobarrier and ext4? Huge fsync performance difference

[mmenaz]

Hi, I've a proxmox, with pvetest repo, kernel 3.10, for a school where I've installed 2 Samsung EVO840 250GB SSD, both formatted as EXT4.
I've seen in some post that the bad fsync performance with ext4 and ssd was due to TRIM command, and the suggestion to put in a crontab, that is what I did but with no luck.
The real issue seems the default "barrier" option, that is disabled in ext3 by proxmox, and enabled for ext4.

Here the mount options:

/dev/mapper/pve-root on / type ext4 (rw,noatime,errors=remount-ro,data=ordered)
/dev/sdb1 on /mnt/hd02 type ext4 (rw,noatime,nobarrier,data=ordered)

pveperf /mnt/hd02, with NO barrier, gives me:

CPU BOGOMIPS:      24744.56
REGEX/SECOND:      1788679
HD SIZE:           229.10 GB (/dev/sdb1)
BUFFERED READS:    524.02 MB/sec
AVERAGE SEEK TIME: 0.05 ms
FSYNCS/SECOND:     6652.47

while pveperf on standard mount option (barrier) gives me:

CPU BOGOMIPS:      24744.56
REGEX/SECOND:      1834370
HD SIZE:           39.25 GB (/dev/mapper/pve-root)
BUFFERED READS:    504.89 MB/sec
AVERAGE SEEK TIME: 0.05 ms
FSYNCS/SECOND:     209.79

The question is: since for ext3 is disabled, and there is a HUGE improvement disablling for ext4 as well, is as (un)safe as with ext3, so I have the same risk I'm currently having with ext3?
What do you usually do (I've no raid controller here, too expensive since needs to be high end with BBU)?
Thanks a lot

 

[tom]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

compare the mount options from both SSDs with:

cat /proc/mounts

any difference?

 

[mmenaz]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

Yes Tom, just noticed 30 seconds after the post, as often happens. Not being able to delete the post, I've changed the question, since the difference in performance is so huge that can't be not considered deeply. Hope you find the time to reply to the new question as well ;P

 

[mir]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

using nobarrier on ext4 is as safe or unsafe as using nobarrier on ext3. So if you are satisfied using nobarrier on ext3 you can also use it on ext4.

 

[tom]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

Just to note, using desktop SSDs like samsung 840 evo is risky. these SSDs does not have a power-loss data protection - so in the case of powerloss, you loose all data in the SSD cache.

if you use enterprise class SSDs, e.g. Intel DS S3500/3700 you have a power-loss data protection.

here is a configuration of one of your test lab servers with an Intel DC S3500, 80 GB - default setup with ext4, no special mount options set, kernel 3.10:

root@prox-ceph-2:~# cat /etc/fstab

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext4 defaults 0 1
UUID=3375400d-6048-429b-b0ec-e231ba33865d /boot ext4 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

cat /proc/mounts
...
/dev/mapper/pve-root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
...

root@prox-ceph-2:/# pveperf
CPU BOGOMIPS:      100624.08
REGEX/SECOND:      821393
HD SIZE:           18.08 GB (/dev/mapper/pve-root)
BUFFERED READS:    301.83 MB/sec
AVERAGE SEEK TIME: 0.06 ms
FSYNCS/SECOND:     1575.37
DNS EXT:           49.89 ms
DNS INT:           1.04 ms (proxmox.com)

 

[mir]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

if you mount a ext4 partition without special mount options on a recent kernel the default behavior is to mount with barrier=0, aka nobarrier.

 

[mmenaz]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

How can I tell? I've googled around and seems that ext4 default behaviour is with barrier ON (http://kernelnewbies.org/Ext4 "2.13. Barriers on by default"), and that "defaults" option in fstab does not act on barrier.
Is there a way to know that are the current mount options used, default included?

 

[mir]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

As of kernel 2.6.37 write_barrier is marked deprecated and has been replaced by WRITE_FLUSH, WRITE_FUA and WRITE_FLUSH_FUA

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4fed947cb311e5aa51781d316cefca836352f6ce

 

[mmenaz]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

Did not know about power-loss data protection! Just noted that you mention "Samsung SSD 840 PRO" as "Single, fast and reliable enterprise class SSD" in the Ceph wiki documentation, maybe you'd better modify and clarify why a "normal" ssd is good enough in that role.
http://pve.proxmox.com/wiki/Ceph_Server
I'm googling about power-loss data protection and reading horror story in tests where this is absent, and also about other disk brand that should have but is not as effective as Intel one.
I've choose SSD for safety and speed, and seems I can't have none of them, or I'm in high risk

 

[tom]

Re: same ssd, same ext4 nobarrier, abyssal pveperf fsync difference, why?

Did not know about power-loss data protection! Just noted that you mention "Samsung SSD 840 PRO" as "Single, fast and reliable enterprise class SSD" in the Ceph wiki documentation, maybe you'd better modify and clarify why a "normal" ssd is good enough in that role.

The wiki recommends enterprise class SSDs (e.g. intel DC S3500/3700), we just use a Samsung SSD 840 PRO in our TEST cluster as we got a bunch of them here. we do not recommend desktop class SSDs for production.

http://pve.proxmox.com/wiki/Ceph_Server
I'm googling about power-loss data protection and reading horror story in tests where this is absent, and also about other disk brand that should have but is not as effective as Intel one.
I've choose SSD for safety and speed, and seems I can't have none of them, or I'm in high risk