iscsi-zfs or iscsi proxmox storage

frankz

Member
Nov 16, 2020
354
23
23
Hello everyone, I would like your opinion on iscsi. I use a nas xigmanas (for years), I have a disk in zfs to use as storage. I would like to try the iscsi share or the zfs-iscsi on proxmox. Mki have reported that in terms of performance iscsi is more powerful, moreover it is a block system therefore volume and not folder like nfs. What do you recommend?
 
iSCSI use blocks, it runs nearly like your computer talks to a local hard disk.
iSCSI is like a local disk but connected thru the network but the network is only another cable like SATA or SAS connectors

NFS is a network protocol used for sharing files with rights etc ... It runs on a server, with authentication, etc ... All that "network" fonctions slows ...

iSCSI is faster than NFS because it nearly only transports data packets, no network support packets and function.
iSCSI is far less secure, that is why it's used on dedicated networks not connected with anything.

On small install with small VMs or containers and good network, the difference is not very important because IOPS are small.
For VMs with lots IOPS, the iSCSI will be quicker.

The only interest in NFS shares is that you can very easily access the hard drive files of each VM in case of problem on the shared device.
In iSCSI, you cannot access to the iSCSI as easily.

If performance matters in iSCSI versus NFS, no doubt, iSCSI wins !

This link shown benchmark with Hyper-V (but it's the same thing in this case) but you will have several dashbords with IOPS comparison en network bandiwth. No doubt ... iSCSI wins in all case ...
https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/

regards
 
  • Like
Reactions: frankz
iSCSI use blocks, it runs nearly like your computer talks to a local hard disk.
iSCSI is like a local disk but connected thru the network but the network is only another cable like SATA or SAS connectors

NFS is a network protocol used for sharing files with rights etc ... It runs on a server, with authentication, etc ... All that "network" fonctions slows ...

iSCSI is faster than NFS because it nearly only transports data packets, no network support packets and function.
iSCSI is far less secure, that is why it's used on dedicated networks not connected with anything.

On small install with small VMs or containers and good network, the difference is not very important because IOPS are small.
For VMs with lots IOPS, the iSCSI will be quicker.

The only interest in NFS shares is that you can very easily access the hard drive files of each VM in case of problem on the shared device.
In iSCSI, you cannot access to the iSCSI as easily.

If performance matters in iSCSI versus NFS, no doubt, iSCSI wins !

This link shown benchmark with Hyper-V (but it's the same thing in this case) but you will have several dashbords with IOPS comparison en network bandiwth. No doubt ... iSCSI wins in all case ...
https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/

regards
Thanks for the indications, you know the differences but you don't know the benefits you can have on the cluster.
 
Thanks for the indications, you know the differences but you don't know the benefits you can have on the cluster.
I was thinking I was clear ... Sorry.
The benefits is that iSCSI is quicker than NFS for the reason I tried to explain.
In the normal life of the cluster, I don't see any difference between iSCSI and NFS.

Maybe your question is about iSCSI versus ZFS/iSCSI and not iSCSI versus NFS ?

Regards
 
iSCSI will not transfer the data in sync, thats the reason why "its faster", it will copy over the Data in den Background. If you mount an iSCSI Target twice, you can say good bye to your Data.

IMHO use NFS or SMB, avoid using iSCSI.
 
I was thinking I was clear ... Sorry.
The benefits is that iSCSI is quicker than NFS for the reason I tried to explain.
In the normal life of the cluster, I don't see any difference between iSCSI and NFS.

Maybe your question is about iSCSI versus ZFS/iSCSI and not iSCSI versus NFS ?

Regards
I was thinking I was clear ... Sorry.
The benefits is that iSCSI is quicker than NFS for the reason I tried to explain.
In the normal life of the cluster, I don't see any difference between iSCSI and NFS.

Maybe your question is about iSCSI versus ZFS/iSCSI and not iSCSI versus NFS ?

Regards
Yes as written I meant the difference between zfs-isci and iscsi.
 
Yes as written I meant the difference between zfs-isci and iscsi.
Ok Pierre, so mounting an iscsi or zf-iscsi target on a cluster doesn't make a difference with nfs at the performance level, so if I have an iscsi I better mount it eg. from windows server or other host that supports iscsi for higher performance?
 
iSCSI will not transfer the data in sync, thats the reason why "its faster", it will copy over the Data in den Background. If you mount an iSCSI Target twice, you can say good bye to your Data.

IMHO use NFS or SMB, avoid using iSCSI.
It is not as easy as that.
Iscsi enforces no sync write at tthestorage level. The reason for that is that typically you have iscsi in the datacenter where initiator and target are on server grade, datacenter hardware which is ups backed.
NFS and SMB was build for a different purpose initially and moving into the datacenter. NFS and SMB were not used for VMs and databases initially.

Since iscsi is a block level storage you need to have a coordination layer if multille hosts are accessing tbe same volume. A cluster filesystem takes care of that.
So the general statement is wrong.

NFS and SMB both have their challenges as well. Especially older implementations.
I'd recommend NFS and SMB only If version 4.1 or 3 onwards are used.

But in general you can't recommend one over the other!
Why? Simple: its only the protocol. It is only one part of your storage infrastructure.
There are system which work better with one or the other. You have to consider the whole chain!!!
 
It is not as easy as that.
Iscsi enforces no sync write at tthestorage level. The reason for that is that typically you have iscsi in the datacenter where initiator and target are on server grade, datacenter hardware which is ups backed.
NFS and SMB was build for a different purpose initially and moving into the datacenter. NFS and SMB were not used for VMs and databases initially.

Since iscsi is a block level storage you need to have a coordination layer if multille hosts are accessing tbe same volume. A cluster filesystem takes care of that.
So the general statement is wrong.

NFS and SMB both have their challenges as well. Especially older implementations.
I'd recommend NFS and SMB only If version 4.1 or 3 onwards are used.

But in general you can't recommend one over the other!
Why? Simple: its only the protocol. It is only one part of your storage infrastructure.
There are system which work better with one or the other. You have to consider the whole chain!!!
 
OK thanks . I use on nfs clusters, I must say that the performance is good. Considering that iscsi is a block network storage, so if I mount an iscsi as a disk on a cluster, I initialize it as zfs to be used as storage for virtual machine creations, in your opinion should I notice any differences compared to nfs?
 
Ok Pierre, so mounting an iscsi or zf-iscsi target on a cluster doesn't make a difference with nfs at the performance level, so if I have an iscsi I better mount it eg. from windows server or other host that supports iscsi for higher performance?
ZFS/iSCSI allows snapshot ... Not simple iSCSI ... Just take care that ZFS use 1 GB of RAM by TB of disk ...
Don't see any diffrence from the VM point of vue as they don't "use" iSCSI but the storage you setup in your VMs.

But for Windows Server, you will have to setup VirtIO virtual hardware and install specific drivers to speed up :
https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers

To install easily Windows server, I used to create 2 DVD device, one loaded with Windows Server OS, the other with the VirtIO drivers and I loaded the good drivers from the install of Windows ...

Regards
 
  • Like
Reactions: frankz
If performance matters in iSCSI versus NFS, no doubt, iSCSI wins !
Hi,

It depends a lot. For example NFS have a cache system so for some specific loads can be very effective. In the PMX context, if you use NFS(so the vHDD is a file) you will also benefit from the kernel caching sistem. As a general speaking, if you have mostly read IO I think that NFS will be better compared with iSCSI(because of caching).

Good luck / Bafta !
 
  • Like
Reactions: frankz
iSCSI use blocks, it runs nearly like your computer talks to a local hard disk.
iSCSI is like a local disk but connected thru the network but the network is only another cable like SATA or SAS connectors

NFS is a network protocol used for sharing files with rights etc ... It runs on a server, with authentication, etc ... All that "network" fonctions slows ...

iSCSI is faster than NFS because it nearly only transports data packets, no network support packets and function.
iSCSI is far less secure, that is why it's used on dedicated networks not connected with anything.

On small install with small VMs or containers and good network, the difference is not very important because IOPS are small.
For VMs with lots IOPS, the iSCSI will be quicker.

The only interest in NFS shares is that you can very easily access the hard drive files of each VM in case of problem on the shared device.
In iSCSI, you cannot access to the iSCSI as easily.

If performance matters in iSCSI versus NFS, no doubt, iSCSI wins !

This link shown benchmark with Hyper-V (but it's the same thing in this case) but you will have several dashbords with IOPS comparison en network bandiwth. No doubt ... iSCSI wins in all case ...
https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/

regards

Problem with Proxmox backup is that LXC containers on a NFS share are suspended during backup.

Are snapshots possible with LXC containers on a ISCSI drive?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!