Thanks for the input. What about throughput? Is it as good as iSCSI? Also there are a few things that Proxmox can use with ZFS over iSCSI such as snapshot management, etc.
In my own testing I have found that with all things equal, NFS provides throughput as well as, or slightly better than iSCSI, but pay attention to the "all things being equal" part.
iSCSI uses async writes, which improves write performance by "lying" to the OS that has mounted it and telling it that data has been committed to disk, when in fact it has actually only been received in memory. It then writes it to disk as soon as it is able. This provides better write speeds, but it means that there is a potential for a few seconds worth of writes to get lost in case of sudden power loss, or a kernel panic/crash. (A few seconds of data may not sound like much, until it corrupts a drive image or database...)
NFS can be configured to use either async writes or sync writes with ZFS. With async writes it behaves much like iSCSI and there is a potential loss of a couple of seconds of write data. In async mode, it performs similarly too, or maybe slightly better than iSCSI in my experience. With sync writes on, it actually commits the written data to a stable disk before informing the client OS that the data is written. This eliminates the risk of that second or two of data being lost in case of an unclean shutdown, but it comes at the expense of a performance penalty, as now you essentially don't have a write buffer.
With ZFS you can compensate for this performance loss experienced from sync writes by using a - so called - SLOG device. ZFS always maintains a log of writes it intends to perform, with enough data to recreate those writes in case of an unclean shutdown. Every single write is noted in the ZIL (ZFS Intent Log) and maintained there until that write is committed from memory to disk, at which point it is discarded. The ZIL is only read from on boot after an unclean shutdown in the case where writes need to be recreated.
Normally this ZIL is written to the hard drives in the pool, which can be slow. A SLOG is a "Separate LOG" device, essentially a dedicated SSD (well, a mirrored pair is usually recommended for production systems) which contains this ZIL. Since it only needs to hold a second or two of data, it can be very small, but it is recommended that it has high write endurance and is low latency to speed things up, and also that it is of an enterprise variety that has capacitors that keep it powered on long enough to commit writes from its internal buffers to the flash in case of sudden power loss.
With a SLOG in place for the ZIL, sync writes can be much much faster than without one, but are still typically slightly slower than async writes. (nothing is faster than RAM, right?
)
So, to reiterate, in my testing NFS performed equal to or slightly better than iSCSI when NFS was in async mode. In sync mode it can range from equivalent to (with very good and expensive SLOG devices) to slightly slower (with typical SLOG devices) to much slower if the ZIL is in the pool with the hard drives. Keep in mind this is only for writes though. The reads should be the same either way.
Also, keep in mind, this is my testing in my workloads. I'm not sure what your workloads look like, and would recommend you do your own performance tests.
