Hello everyone, I would like your opinion on iscsi. I use a nas xigmanas (for years), I have a disk in zfs to use as storage. I would like to try the iscsi share or the zfs-iscsi on proxmox. Mki have reported that in terms of performance iscsi is more powerful, moreover it is a block system therefore volume and not folder like nfs. What do you recommend?
iscsi-zfs or iscsi proxmox storage
- Thread starter frankz
- Start date
iSCSI use blocks, it runs nearly like your computer talks to a local hard disk.
iSCSI is like a local disk but connected thru the network but the network is only another cable like SATA or SAS connectors
NFS is a network protocol used for sharing files with rights etc ... It runs on a server, with authentication, etc ... All that "network" fonctions slows ...
iSCSI is faster than NFS because it nearly only transports data packets, no network support packets and function.
iSCSI is far less secure, that is why it's used on dedicated networks not connected with anything.
On small install with small VMs or containers and good network, the difference is not very important because IOPS are small.
For VMs with lots IOPS, the iSCSI will be quicker.
The only interest in NFS shares is that you can very easily access the hard drive files of each VM in case of problem on the shared device.
In iSCSI, you cannot access to the iSCSI as easily.
If performance matters in iSCSI versus NFS, no doubt, iSCSI wins !
This link shown benchmark with Hyper-V (but it's the same thing in this case) but you will have several dashbords with IOPS comparison en network bandiwth. No doubt ... iSCSI wins in all case ...
https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/
regards
iSCSI is like a local disk but connected thru the network but the network is only another cable like SATA or SAS connectors
NFS is a network protocol used for sharing files with rights etc ... It runs on a server, with authentication, etc ... All that "network" fonctions slows ...
iSCSI is faster than NFS because it nearly only transports data packets, no network support packets and function.
iSCSI is far less secure, that is why it's used on dedicated networks not connected with anything.
On small install with small VMs or containers and good network, the difference is not very important because IOPS are small.
For VMs with lots IOPS, the iSCSI will be quicker.
The only interest in NFS shares is that you can very easily access the hard drive files of each VM in case of problem on the shared device.
In iSCSI, you cannot access to the iSCSI as easily.
If performance matters in iSCSI versus NFS, no doubt, iSCSI wins !
This link shown benchmark with Hyper-V (but it's the same thing in this case) but you will have several dashbords with IOPS comparison en network bandiwth. No doubt ... iSCSI wins in all case ...
https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/
regards
Thanks for the indications, you know the differences but you don't know the benefits you can have on the cluster.
I was thinking I was clear ... Sorry.
The benefits is that iSCSI is quicker than NFS for the reason I tried to explain.
In the normal life of the cluster, I don't see any difference between iSCSI and NFS.
Maybe your question is about iSCSI versus ZFS/iSCSI and not iSCSI versus NFS ?
Regards
Yes as written I meant the difference between zfs-isci and iscsi.
Ok Pierre, so mounting an iscsi or zf-iscsi target on a cluster doesn't make a difference with nfs at the performance level, so if I have an iscsi I better mount it eg. from windows server or other host that supports iscsi for higher performance?
It is not as easy as that.
Iscsi enforces no sync write at tthestorage level. The reason for that is that typically you have iscsi in the datacenter where initiator and target are on server grade, datacenter hardware which is ups backed.
NFS and SMB was build for a different purpose initially and moving into the datacenter. NFS and SMB were not used for VMs and databases initially.
Since iscsi is a block level storage you need to have a coordination layer if multille hosts are accessing tbe same volume. A cluster filesystem takes care of that.
So the general statement is wrong.
NFS and SMB both have their challenges as well. Especially older implementations.
I'd recommend NFS and SMB only If version 4.1 or 3 onwards are used.
But in general you can't recommend one over the other!
Why? Simple: its only the protocol. It is only one part of your storage infrastructure.
There are system which work better with one or the other. You have to consider the whole chain!!!
OK thanks . I use on nfs clusters, I must say that the performance is good. Considering that iscsi is a block network storage, so if I mount an iscsi as a disk on a cluster, I initialize it as zfs to be used as storage for virtual machine creations, in your opinion should I notice any differences compared to nfs?
ZFS/iSCSI allows snapshot ... Not simple iSCSI ... Just take care that ZFS use 1 GB of RAM by TB of disk ...
Don't see any diffrence from the VM point of vue as they don't "use" iSCSI but the storage you setup in your VMs.
But for Windows Server, you will have to setup VirtIO virtual hardware and install specific drivers to speed up :
https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers
To install easily Windows server, I used to create 2 DVD device, one loaded with Windows Server OS, the other with the VirtIO drivers and I loaded the good drivers from the install of Windows ...
Regards
Hi,
It depends a lot. For example NFS have a cache system so for some specific loads can be very effective. In the PMX context, if you use NFS(so the vHDD is a file) you will also benefit from the kernel caching sistem. As a general speaking, if you have mostly read IO I think that NFS will be better compared with iSCSI(because of caching).
Good luck / Bafta !
Problem with Proxmox backup is that LXC containers on a NFS share are suspended during backup.
Are snapshots possible with LXC containers on a ISCSI drive?
Last edited: