Very low performance with Netapp NFS

[xpanisello]

Hello,

I'm installing a new virtualization core:

- 4 x HP Proliant Server (2 Intel Processors x8 cores and 96GB RAM each one), and another smaller one. With 4 10Gbase-T ports (two for LAN and two for SAN) each one.
- 2 x Netapp (release 8.3, FAS2552 and FAS2520) with 4 10GbaseSX for node
- 2 x Qnap with 4 10GbaseT ports (two for SAN and two for LAN)
- 2 x Switch Extreme with 48 10Gb ports.

As you can see all of them has two 10Gb ports for SAN, 1 on one switch and the other on the other switch, using LACP.

All it's working fine. I've created some vm over the qnaps, using iSCSI and NFS v3, and they are working fine.

But when I made some performance test with Netapp, in order to decide to use NFS v4 or iSCSI for vm's, I get a problem.

I copied, with rsync, a 10GB file to:

a Qnap iSCSI, it Works at 143MB/s
a Netapp iSCSI it Works at 143,33 MB/s
a Qnap NFS v3, it Works at 82 MB/s
a Netapp NFS v4, it Works at 152MB/s

Then I copied 3,3Gb in about 20.000 files to:

a Qnap iSCSI, it Works at 146MB/s
a Netapp iSCSI it Works at 141,33 MB/s
a Qnap NFS v3, it Works at 55 MB/s
a Netapp NFS v3, it Works at 6,8MB/s
a Netapp NFS v4, it Works at 6,4MB/s

Oppsss, what's happening with Netapp NFS with a lot of little files????

What do you think about it?

Sure, there is a problem with Netapp NFS and Proxmox!!!

Is anyone using Netapp NFS with Proxmox? Have you this problem? How you solved it? What performance has it?

I hope is a configuration problem.

Xavier.

 

[mir]

what wsize and rsize to you use for NFS to netapps?

 

[xpanisello]

Hello,

First I try with default parameters, then Netapp send me some modifications, last parameters were rsize=wsize=65536, and timeo=6000

 

[Whatever]

Same story but with FreeNas 9.3 (neither of tuning sets found on this forum has helped by the moment)

 

[manu]

NFS is slow when working with you have a lot of small files, due to disk seeks and network latency. I am not surprised from these results.

 

[xpanisello]

Hello,

I'm sorry, Manu, but i'm really surprised, as you can see in my statics the Qnap (TS-EC879U-RP, 8x3TB sata HD, it costs about 7.000€) NFS v3 reaches 55MB/s, the same proxmox server, same switchs, same files... but diferent NAS, Netapp (two controllers, 10Gb fiber connections, 24 x 1TB sas HD... about 30.000€ !!)

How is possible? Qnap reaches 55MB/s, that means is possible, and Netapp stays at 6MB/s? It's no possible. I made the test for three times, with nfs v3 and nfs v4.

Time ago I used FreeNas for Proxmox testing, before decide to migrate from VMWare, but It was just in laboratory in a 1Gb environment. Whatever has then same problem, may be a general poor performance with this kind of files? But then, why Qnap reaches 55MB/s?

 

[Whatever]

NFS is slow when working with you have a lot of small files, due to disk seeks and network latency. I am not surprised from these results.

in my scenario NFS is used as backup storage (not the small files amount is an issue)

 

[xpanisello]

Yes, I agree with you, how important is this issue? If we plan to use nfs for big files there is no problem. But I were surprised when testing. I was planning to use iSCSI for images and NFS for backups, iso... but I read several articles about the major performance of NFS v4 vs iSCSI, that's why I made these tests, if NFS v4 is faster and, by the other side, I can do vm snapshots (it is not possible in iSCSI) it will be better NFS v4 than iSCSI for images.

 

[hec]

You need working reverse lookup for NFS. For every file you write ther is a dns reverse lookup.

Did you read the NF Document from NFS?

Did you enable NFS 4.1?

Your idmap.conf should have the same domain name as configured in netapp

Please post the output of:

set diag
vserver show -vserver VSERVERNAME
nfs show -vserver VSERVERNAME
nfs show -vserver VSERVERNAME -fields v4-numeric-ids

 

[xpanisello]

Hi Hec,

I'm sorry, but I didn't read the NF Document from NFS, can you post a link, please?

No, during test there were only v3 or v4 enabled, never both neither 4.1.

No, Hec, I didn't set the Domain in idmap.conf. I'll set it.

Here are the outputs:

vserver show -vserver Maquines_virtuals_nfs1


Vserver: Maquines_virtuals_nfs1
Vserver Type: data
Vserver Subtype: default
Vserver UUID: 886c31b4-f237-11e4-8063-00a0987d8d1 4
Root Volume: Maquines_virtuals_nfs1_root
Aggregate: sas_data_1
NIS Domain: ajuntamentsalou.local
Root Volume Security Style: unix
LDAP Client: -
Default Volume Language Code: C.UTF-8
Snapshot Policy: default
Comment:
Quota Policy: default
List of Aggregates Assigned: -
Limit on Maximum Number of Volumes allowed: unlimited
Vserver Admin State: running
Vserver Operational State: running
Vserver Operational State Stopped Reason: -
Allowed Protocols: nfs
Disallowed Protocols: cifs, fcp, iscsi, ndmp
Is Vserver with Infinite Volume: false
QoS Policy Group: -
Config Lock: false
Volume Delete Retention Period: 12
IPspace Name: Default


******************************************************************************


nfs show -vserver Maquines_virtuals_nfs1


Vserver: Maquines_virtuals_nfs1
General NFS Access: true
RPC GSS Context Cache High Water Mark: 0
RPC GSS Context Idle: 0
NFS v3: enabled
NFS v4.0: enabled
UDP Protocol: enabled
TCP Protocol: enabled
Default Windows User: -
Enable NFSv3 EJUKEBOX error: true
Require All NFSv3 Reads to Return Read Attributes: false
Show Change in FSID as NFSv3 Clients Traverse Filesystems: enabled
Enable the Dropping of a Connection When an NFSv3 Request is Dropped: enabled
Vserver NTFS Unix Security Options: use_export_policy
Vserver Change Ownership Mode: use_export_policy
Force Usage of SpinNp Readdir Requests: false
NFS Response Trace Enabled: false
NFS Response Trigger (in secs): 60
UDP Maximum Transfer Size: 32768
TCP Maximum Transfer Size: 65536
NFSv3 TCP Maximum Read Size: 65536
NFSv3 TCP Maximum Write Size: 65536
NFSv4.0 ACL Support: enabled
NFSv4.0 Read Delegation Support: enabled
NFSv4.0 Write Delegation Support: enabled
Show Change in FSID as NFSv4 Clients Traverse Filesystems: enabled
NFSv4.0 Referral Support: disabled
NFSv4 ID Mapping Domain: defaultv4iddomain.com
NFSv4 Validate UTF-8 Encoding of Symbolic Link Data: disabled
NFSv4 Lease Timeout Value (in secs): 30
NFSv4 Grace Timeout Value (in secs): 45
Preserves and Modifies NFSv4 ACL (and NTFS File Permissions in Unified Security Style): enabled
NFSv4.1 Minor Version Support: enabled
Rquota Enable: disabled
NFSv4.1 Implementation ID Domain: netapp.com
NFSv4.1 Implementation ID Name: NetApp Release 8.3
NFSv4.1 Implementation ID Date: Tue Mar 10 00:01:28 2015
NFSv4.1 Parallel NFS Support: enabled
NFSv4.0 Migration Support: disabled
NFSv4.1 Referral Support: disabled
NFSv4.1 Migration Support: disabled
NFSv4.1 ACL Support: disabled
NFS vStorage Support: disabled
NFSv4 Support for Numeric Owner IDs: enabled
Default Windows Group: -
NFSv4.1 Read Delegation Support: disabled
NFSv4.1 Write Delegation Support: disabled
Number of Slots in the NFSv4.x Session slot tables: 180
Size of the Reply that will be Cached in Each NFSv4.x Session Slot (in bytes): 640
Maximum Number of ACEs per ACL: 400
NFS Mount Root Only: enabled
NFS Root Only: disabled
Qtree Exports Enabled: disabled
AUTH_SYS Extended Groups Enabled: disabled
AUTH_SYS and RPCSEC_GSS Auxillary Groups Limit: 32
Validation of Qtree IDs for Qtree File Operations: enabled
NFS Mount Daemon Port: 635
Network Lock Manager Port: 4045
Network Status Monitor Port: 4046
NFS Quota Daemon Port: 4049
Permitted Kerberos Encryption Types: des, des3, aes-128, aes-256
Showmount Enabled: disabled
Set the Protocol Used for Name Services Lookups for Exports: udp
Map Unknown UID to Default Windows User: enable
DNS Domain Search Enabled During Netgroup Lookup: enabled


********************************************************************************


nfs show -vserver Maquines_virtuals_nfs1 -fields v4-numeric-ids


vserver v4-numeric-ids
---------------------- --------------
Maquines_virtuals_nfs1 enabled

 

[spirit]

it could be network latency related.

As rsync write a file, wait for ack, write a file, wait for ack,... (+also nfs ops)

between each operation, you need to add some ms.

Can you a a write test with more parralel write ?


Also, do you have tuned MTU ? if yes, do you have setup the same of netapp and proxmox ?

 

[hec]

here is the link:

http://www.netapp.com/us/system/pdf-reader.aspx?m=tr-4067.pdf

If this does not help we could test the performance with sio_ntap. With this tool we can test nearly everything.

Please post the mount options of the nfs mount.

iSCSI will always be faster because it is block based. With NFS you have a filesystem on the netapp.

Maybe you could try some changes to sysctl

sunrpc.tcp_slot_table_entries = 128
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_mem = 362208 482944 724416
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.udp_mem = 363168 484224 726336

Lets see if this help to get up performance. Even with small files the FAS2552 should do more throughput and IOPs

 

[spirit]

you can also try to disable nis, which is enabled by default on ontap 8.3


#vserver services name-service nis-domain delete -vserver yourvserver