Where is the Bottleneck?

T

trendco

Member
Proxmox Subscriber
Jan 9, 2012
282
2
18
For a Fileserver Virtualization i passthrough a SATA Controller to a Nas4Free VM.
There i have et al an mirrored ZFS SSD Pool (2 x Samsung 840 Pro).
The Read-Performance from this Pool inside the Nas4Free VM is very fast: ~1GB/sec.
The Dataset of this Pool I've shared via NFS and added in Proxmox as NFS-Storage.
With VIRTIO Network in these VM i get this iperf-Values:

Code: 
nas4free: ~ # iperf -c 192.168.0.3
------------------------------------------------------------
Client connecting to 192.168.0.3, TCP port 5001
TCP window size:  257 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.240 port 13603 connected with 192.168.0.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  45.5 GBytes  [COLOR=#0000ff]39.0 Gbits/sec[/COLOR]


nas4free: ~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.240 port 5001 connected with 192.168.0.3 port 55841
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  37.0 GBytes  [COLOR=#0000ff]31.7 Gbits/sec[/COLOR]

Code: 
root@proxmox:/alex# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.3 port 5001 connected with 192.168.0.240 port 13603
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  45.5 GBytes  [COLOR=#0000ff]38.9 Gbits/sec[/COLOR]
root@proxmox:/alexiperf -c 192.168.0.240
------------------------------------------------------------
Client connecting to 192.168.0.240, TCP port 5001
TCP window size: 22.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.3 port 55841 connected with 192.168.0.240 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  37.0 GBytes  [COLOR=#0000ff]31.8 Gbits/sec[/COLOR]

But the Read-Performance from the Host is very bad:

Code: 
root@proxmox:/# dd if=/mnt/pve/nas4free-ssd/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=2000
2000+0 records in
2000+0 records out
4194304000 bytes (4.2 GB) copied, 76.0466 s, [COLOR=#ff0000]55.2 MB/s[/COLOR]

When I use the "e1000" Network Card in Nas4Free VM, iperf shows ~"1.3 Gbits/sec" an the Result in dd increased to ~"130 MB/s"

I think the Speed with VIRTIO would still be nearly as fast as in the VM itself?
Where is the Bottleneck?
 
Hi,
I think Nas4Free has a virtio driver problem.
 
I'm don't know mutch about free4NAS but may be the iperf output is not valid!
this problem occurs when you use iperf on Windows.
 
Today i tested the newest available Version of Nas4Free "9.3.0.2 - Nayla (Revision 1349)" and there, the dd Speed-Test increased to >750MB/s :D

Code: 
root@proxmox:~# dd if=/mnt/pve/nas4free-ssd/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=2000
2000+0 records in
2000+0 records out
4194304000 bytes (4.2 GB) copied, 5.47898 s, 766 MB/s

root@proxmox:~# dd if=/mnt/pve/nas4free-ssd/images/104/vm-104-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes (6.3 GB) copied, 7.67795 s, 819 MB/s

But pveperf shows ~ the same as before for "FSYNCS/SECOND":

Code: 
root@proxmox:~# pveperf /mnt/pve/nas4free-ssd/
CPU BOGOMIPS:      55997.52
REGEX/SECOND:      2283167
HD SIZE:           234.26 GB (192.168.0.240:/mnt/raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#ff0000]363.04[/COLOR]
DNS EXT:           88.40 ms
DNS INT:           58.28 ms (local)

Is this normaly (max.) for NFS Storage (FSYNCS/SECOND)?
 
Last edited:
I still have made some more Speed Tests, between the Tests I always rebooted, so that nothing is cached:


dd inside the Nas4Free VM (SATA Controller passed through):

Code: 
nas4free: ~ # dd if=/mnt/raid1-ssd/pve/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes transferred in 5.985437 secs (1051127276 bytes/sec) -> [COLOR=#0000ff]1002 MB/s[/COLOR]

nas4free: ~ # dd if=/mnt/raid1-ssd/pve/images/104/vm-104-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes transferred in 5.829766 secs (1079195326 bytes/sec) -> [COLOR=#0000ff]1029 MB/s[/COLOR]

Here the Speed is great!


----------------------------------------------------------------------------------------------------------------------------


Exported the Pool of Nas4Free and imported directly to Proxmox:

Code: 
root@proxmox:~# dd if=/mnt/raid1-ssd/pve/images/110/vm-110-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes (6.3 GB) copied, 9.47247 s, [COLOR=#0000ff]664 MB/s[/COLOR]

root@proxmox:~# dd if=/mnt/raid1-ssd/pve/images/104/vm-104-disk-1.qcow2 of=/dev/null bs=2M count=3000
3000+0 records in
3000+0 records out
6291456000 bytes (6.3 GB) copied, 8.42852 s, [COLOR=#0000ff]746 MB/s[/COLOR]


root@proxmox:~# pveperf /mnt/raid1-ssd/pve/
CPU BOGOMIPS:      56000.88
REGEX/SECOND:      2210844
HD SIZE:           234.26 GB (raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#0000ff]479.41[/COLOR]
DNS EXT:           94.83 ms
DNS INT:           62.33 ms (local)

The Speed has decreased, although the Pool now depends directly on the Proxmox System (it is even lower as the detour via VIRTIO/NFS)!?
"FSYNCS/SECOND" is only slightly better than over NFS!?


How could that be? Is FreeBSD so much better in interaction with ZFS as ZoL?
And why are my "FSYNCS/SECOND" Values so bad with a SSD? I do not understand how some People have Values of a few thousand?
 
Without diging deeper, using dd this way is not a suitable benchmark (zfs compression?).
 
Last edited:
I got this Info from the ZFS Mailing-List:
For the difference in performance:
Most likely ZoL isn't that optimized compared to the OpenZFS FreeBSD branch, in handling mirrors and non-rotating drives there are some patches which havn't made it into the Linux branch jet - plus the memory management in Linux isn't optimal at the moment.
Click to expand...
Then so be it.


But again for the Question regarding "FSYNCS/SECOND":
How can it be, that I have only a Value of ~500 with SSD and others have Values of a few thousand with HD's?
-> http://forum.proxmox.com/threads/20...with-Performance-Data?highlight=FSYNCS/SECOND
 
OMG !

Code: 
sync=standard

root@proxmox:~# pveperf /mnt/raid1-ssd/pve/
CPU BOGOMIPS:      55997.92
REGEX/SECOND:      2153260
HD SIZE:           189.98 GB (raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#0000ff]469.59[/COLOR]
DNS EXT:           93.08 ms
DNS INT:           84.53 ms (local)

Code: 
sync=disabled

root@proxmox:~# pveperf /mnt/raid1-ssd/pve/
CPU BOGOMIPS:      55997.92
REGEX/SECOND:      2138047
HD SIZE:           189.98 GB (raid1-ssd/pve)
FSYNCS/SECOND:     [COLOR=#0000ff]31697.61[/COLOR]
DNS EXT:           143.22 ms
DNS INT:           87.76 ms (local)

But "sync=disabled" is no Alternative.
 
By default if your pool have no external ZIL and sync is set to standard or always you get double write. First as ZIL later as normal write.

p.s. sync=disabled ZFS keeps all data (sync and async requests) in RAM and flush it every 5 seconds to drives.
 
You must log in or register to reply here.
Top Bottom