nfs performance

[ChappIO]

I have been testing different ways to set up my NAS storage for proxmox but I am finding odd results. Maybe you can help me clarify them.

Testing on local ssd I am finding ~500MB/s writes
Testing through iSCSI target ~100MB/s writes
Testing through NFS storage ~1.8GB/s writes

I find it weird to see the high speeds on NFS storage since I just have a bonded gigabit network. Is there a lot of caching happening or someting?

Also, do you have some tips? I have many compute heavy vms which don't need long-lasting storage and a few vms which require a lot of storage (some for slow storage, some for databases).

 

[mailinglists]

Probably a lot of caching.
Probably bad testing methods.

How did you test? You provide too little detail about testing and the environment for anyone to meaningfully comment on your results.

 

[ChappIO]

Probably a lot of caching.
Probably bad testing methods.

How did you test? You provide too little detail about testing and the environment for anyone to meaningfully comment on your results.

I just dd'd /dev/zero to disk in a vm. Probably a bad way to test this. Could you point me in a better direction for testing?

I am just trying to figure out what the pros and cons are of the various ways of storing disk images for my vms.

My setup:

2 compute nodes with limited local storage
1 NAS
1 network node which does ssl termination

I run mostly stateless compute services in the vms but also a handful of database/storage vms. I want to move (or at least backup) the storage of those vms to the NAS so that I have a central place which contains all important data.

I have never used iscsi before but I see some people recommending this so I figured I'd try it out. But would there be any benefits?

 

[r.jochum]

Bonnie++ is nice to do real tests on NFS.

 

[ChappIO]

Bonnie++ is nice to do real tests on NFS.

Thanks for the tip!
I ran bonnie++ on every configuration (not at the same time of course).
As far as I can see, nfs provides higher performance all round.

Did I draw the wrong conclusion? Is there a benefit of using iSCSI instead of NFS?

 

[David Pilibosian]

What kind of NAS are you testing to? If it's a FreeNAS and I recall correctly iSCSI does sync writes by default and NFS doesn't. Without a ZIL SSD you're going to see a pretty massive performance difference.

 

[ChappIO]

What kind of NAS are you testing to? If it's a FreeNAS and I recall correctly iSCSI does sync writes by default and NFS doesn't. Without a ZIL SSD you're going to see a pretty massive performance difference.

This is a Synology Rackstation RS818+ with mostly default settings. It contains 3 WD Red (4TB) drives using their SHR configuration.

 

[mailinglists]

I would go with NFS, due to easier debugging and maintenance.
However if your data is important, make sure NFS is not mounted / exported with async.