Where is the Bottleneck?

trendco · Feb 27, 2015

For a Fileserver Virtualization i passthrough a SATA Controller to a Nas4Free VM.
There i have et al an mirrored ZFS SSD Pool (2 x Samsung 840 Pro).
The Read-Performance from this Pool inside the Nas4Free VM is very fast: ~1GB/sec.
The Dataset of this Pool I've shared via NFS and added in Proxmox as NFS-Storage.
With VIRTIO Network in these VM i get this iperf-Values:

Code:

nas4free: ~ # iperf -c 192.168.0.3
------------------------------------------------------------
Client connecting to 192.168.0.3, TCP port 5001
TCP window size:  257 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.240 port 13603 connected with 192.168.0.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  45.5 GBytes  [COLOR=#0000ff]39.0 Gbits/sec[/COLOR]


nas4free: ~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.240 port 5001 connected with 192.168.0.3 port 55841
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  37.0 GBytes  [COLOR=#0000ff]31.7 Gbits/sec[/COLOR]

Code:

root@proxmox:/alex# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.3 port 5001 connected with 192.168.0.240 port 13603
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  45.5 GBytes  [COLOR=#0000ff]38.9 Gbits/sec[/COLOR]
root@proxmox:/alexiperf -c 192.168.0.240
------------------------------------------------------------
Client connecting to 192.168.0.240, TCP port 5001
TCP window size: 22.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.3 port 55841 connected with 192.168.0.240 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  37.0 GBytes  [COLOR=#0000ff]31.8 Gbits/sec[/COLOR]

But the Read-Performance from the Host is very bad:

Code:

root@proxmox:/# dd if=/mnt/pve/nas4free-ssd/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=2000
2000+0 records in
2000+0 records out
4194304000 bytes (4.2 GB) copied, 76.0466 s, [COLOR=#ff0000]55.2 MB/s[/COLOR]

When I use the "e1000" Network Card in Nas4Free VM, iperf shows ~"1.3 Gbits/sec" an the Result in dd increased to ~"130 MB/s"

I think the Speed with VIRTIO would still be nearly as fast as in the VM itself?
Where is the Bottleneck?

wolfgang · Mar 2, 2015

Hi,
I think Nas4Free has a virtio driver problem.

trendco · Mar 2, 2015

But why is then the iperf-Test so good?

Nemesiz · Mar 2, 2015

trendco said:
But why is then the iperf-Test so good?

iperf is a network tool and it does not have anything to do with hdd.

trendco · Mar 2, 2015

I know, but the Problem seems to be the Network.

wolfgang · Mar 2, 2015

I'm don't know mutch about free4NAS but may be the iperf output is not valid!
this problem occurs when you use iperf on Windows.

trendco · Mar 2, 2015

Today i tested the newest available Version of Nas4Free "9.3.0.2 - Nayla (Revision 1349)" and there, the dd Speed-Test increased to >750MB/s

Code:

root@proxmox:~# dd if=/mnt/pve/nas4free-ssd/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=2000
2000+0 records in
2000+0 records out
4194304000 bytes (4.2 GB) copied, 5.47898 s, 766 MB/s

root@proxmox:~# dd if=/mnt/pve/nas4free-ssd/images/104/vm-104-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes (6.3 GB) copied, 7.67795 s, 819 MB/s

But pveperf shows ~ the same as before for "FSYNCS/SECOND":

Code:

root@proxmox:~# pveperf /mnt/pve/nas4free-ssd/
CPU BOGOMIPS:      55997.52
REGEX/SECOND:      2283167
HD SIZE:           234.26 GB (192.168.0.240:/mnt/raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#ff0000]363.04[/COLOR]
DNS EXT:           88.40 ms
DNS INT:           58.28 ms (local)

Is this normaly (max.) for NFS Storage (FSYNCS/SECOND)?

trendco · Mar 2, 2015

I still have made some more Speed Tests, between the Tests I always rebooted, so that nothing is cached:

dd inside the Nas4Free VM (SATA Controller passed through):

Code:

nas4free: ~ # dd if=/mnt/raid1-ssd/pve/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes transferred in 5.985437 secs (1051127276 bytes/sec) -> [COLOR=#0000ff]1002 MB/s[/COLOR]

nas4free: ~ # dd if=/mnt/raid1-ssd/pve/images/104/vm-104-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes transferred in 5.829766 secs (1079195326 bytes/sec) -> [COLOR=#0000ff]1029 MB/s[/COLOR]

Here the Speed is great!

----------------------------------------------------------------------------------------------------------------------------

Exported the Pool of Nas4Free and imported directly to Proxmox:

Code:

root@proxmox:~# dd if=/mnt/raid1-ssd/pve/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes (6.3 GB) copied, 9.47247 s, [COLOR=#0000ff]664 MB/s[/COLOR]

root@proxmox:~# dd if=/mnt/raid1-ssd/pve/images/104/vm-104-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes (6.3 GB) copied, 8.42852 s, [COLOR=#0000ff]746 MB/s[/COLOR]


root@proxmox:~# pveperf /mnt/raid1-ssd/pve/
CPU BOGOMIPS:      56000.88
REGEX/SECOND:      2210844
HD SIZE:           234.26 GB (raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#0000ff]479.41[/COLOR]
DNS EXT:           94.83 ms
DNS INT:           62.33 ms (local)

The Speed has decreased, although the Pool now depends directly on the Proxmox System (it is even lower as the detour via VIRTIO/NFS)!?
"FSYNCS/SECOND" is only slightly better than over NFS!?

How could that be? Is FreeBSD so much better in interaction with ZFS as ZoL?
And why are my "FSYNCS/SECOND" Values so bad with a SSD? I do not understand how some People have Values of a few thousand?

dietmar · Mar 2, 2015

Without diging deeper, using dd this way is not a suitable benchmark (zfs compression?).

trendco · Mar 2, 2015

zfs compression?

no, not activated.

Nemesiz · Mar 3, 2015

If you are talking about ZFS check your ARC size and usage. Compare it with your Nas4Free

trendco · Mar 3, 2015

Yes, i talk about ZFS. And yes, i tested with the same Values of ARC min/max.
I do not understand why there is such a large Difference between Linux & FreeBSD.

Nemesiz · Mar 3, 2015

Try to test with dd and monitoring with arcstats. Look for missing %

trendco · Mar 4, 2015

I got this Info from the ZFS Mailing-List:

For the difference in performance:
Most likely ZoL isn't that optimized compared to the OpenZFS FreeBSD branch, in handling mirrors and non-rotating drives there are some patches which havn't made it into the Linux branch jet - plus the memory management in Linux isn't optimal at the moment.

Then so be it.

But again for the Question regarding "FSYNCS/SECOND":
How can it be, that I have only a Value of ~500 with SSD and others have Values of a few thousand with HD's?
-> http://forum.proxmox.com/threads/20...with-Performance-Data?highlight=FSYNCS/SECOND

Nemesiz · Mar 4, 2015

Try to set sync=disabled and test again just for curiosity.

trendco · Mar 4, 2015

I have included the SSD-Pool as local Storage (Directory).

Nemesiz · Mar 4, 2015

It ok but make one test with "zfs set sync=disabled volume_test_name"

trendco · Mar 4, 2015

OMG !

Code:

sync=standard

root@proxmox:~# pveperf /mnt/raid1-ssd/pve/
CPU BOGOMIPS:      55997.92
REGEX/SECOND:      2153260
HD SIZE:           189.98 GB (raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#0000ff]469.59[/COLOR]
DNS EXT:           93.08 ms
DNS INT:           84.53 ms (local)

Code:

sync=disabled

root@proxmox:~# pveperf /mnt/raid1-ssd/pve/
CPU BOGOMIPS:      55997.92
REGEX/SECOND:      2138047
HD SIZE:           189.98 GB (raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#0000ff]31697.61[/COLOR]
DNS EXT:           143.22 ms
DNS INT:           87.76 ms (local)

But "sync=disabled" is no Alternative.

Nemesiz · Mar 4, 2015

By default if your pool have no external ZIL and sync is set to standard or always you get double write. First as ZIL later as normal write.

p.s. sync=disabled ZFS keeps all data (sync and async requests) in RAM and flush it every 5 seconds to drives.

trendco · Mar 4, 2015

Ok, understand, thank you.
But should the Performance with SSD's not still be better than "FSYNCS/SECOND: 469.59" ?

Where is the Bottleneck?

Member

Proxmox Retired Staff

Member

Renowned Member

Member

Proxmox Retired Staff

Member

Member

Proxmox Staff Member

Member

Renowned Member

Member

Renowned Member

Member

Renowned Member

Member

Renowned Member

Member

Renowned Member

Member