We'll soon outgrow our existing shared storage device (off the shelf NAS) and are looking at migrating to a Nexenta system. We're currently using NFS, but I'm wondering if anyone has done comparative testing on the same hardware between iSCSI and NFS for Proxmox Ve 2.0+? I haven't found much about why people choose one over the other or any hard performance metrics. Any suggestions would be appreciated.
NFS vs ISCSI
- Thread starter cyclones
- Start date
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Nexenta is a great system (actually ZFS is the great part) and I have used it for a couple of years now. Can be tricky to get tuned right for your workload but when you get i right it really good. But if you are going to use it for production make sure NOT to use the community edition.
Regarding NFS vs ISCSI on Nexenta when using it as a storage backend for virtualization you can go either way. There are pros and cons with them both. From a performance point of view it does not matter at all. They both perform equally well, but you have to tune your system for whatever protocol you chose to use. The question is rather if you want to be able to use the LVM group functions within proxmox then you need to go with ISCSI (to have proxmox do online backup of your VM's). But then again ZFS has the snapshot functionality in the file system so you could simply go with NFS and let Nexenta take care of the snapshots/backup of VM's. Using NFS you can access all the files directly from the file system within Nexenta so you can actually see and perform tasks om each VM's files. Using ISCSI you wont be able to do that.
Regarding NFS vs ISCSI on Nexenta when using it as a storage backend for virtualization you can go either way. There are pros and cons with them both. From a performance point of view it does not matter at all. They both perform equally well, but you have to tune your system for whatever protocol you chose to use. The question is rather if you want to be able to use the LVM group functions within proxmox then you need to go with ISCSI (to have proxmox do online backup of your VM's). But then again ZFS has the snapshot functionality in the file system so you could simply go with NFS and let Nexenta take care of the snapshots/backup of VM's. Using NFS you can access all the files directly from the file system within Nexenta so you can actually see and perform tasks om each VM's files. Using ISCSI you wont be able to do that.
with this you do not backup your entire vm, but only the hdd of your vm. the configuration file is not backed up. proxmox lvm backup gets you just that: one nice file with everything included, ready to get restored with one click. not incremental tho.But then again ZFS has the snapshot functionality in the file system so you could simply go with NFS and let Nexenta take care of the snapshots/backup of VM's.
That is true. But depending on you environment there are pros and cons with them both. Doing snapshots in ZFS is instant. Within a second you have your backup of all VM's image files witch could be hundreds or even thousands if you want. It does not impact the proxmox hosts them selfs. Restoring a single VM image file is done in seconds if needed. Saves a lot of time, resources, and disk space since each snapshot in ZFS is incremental as well.
I am new to iSCSI / NFS. Am I think in a correct way with following statement? "We use Nexenta and present the storage to proxmox by iSCSI. Making LVM with this iSCSI storage and those KVM use this LVM to store their harddisk image. 1) Once we do backup by proxmox, proxmox will perform a snapshoot backup? 2) who is doing the snapshoot? LVM or ZFS?That is true. But depending on you environment there are pros and cons with them both. Doing snapshots in ZFS is instant. Within a second you have your backup of all VM's image files witch could be hundreds or even thousands if you want. It does not impact the proxmox hosts them selfs. Restoring a single VM image file is done in seconds if needed. Saves a lot of time, resources, and disk space since each snapshot in ZFS is incremental as well.
Sorry for my english and Thank you
Thanks xlnt - yes, I'm aware of the licensing restrictions on the "community" version of Nexenta. I may also explore OpenIndiana if we decide that costs are an issue, but that probably won't be our primary concern in regards to our OS costs for storage. Our existing off the shelf system will be rotated to a secondary off site backup solution as a sunk cost.
Tim - If I understand your question correctly, then what you're saying is true and in that case the snapshot feature within Proxmox is using LVM features. xlnt was talking about using the ZFS options in Nexenta (or similar) to do backups independent of proxmox. GT1 is correct that this is then only the harddrives, but restoring from harddrive only is pretty quick and something I do for some of my systems already (archived ones).
Tim - If I understand your question correctly, then what you're saying is true and in that case the snapshot feature within Proxmox is using LVM features. xlnt was talking about using the ZFS options in Nexenta (or similar) to do backups independent of proxmox. GT1 is correct that this is then only the harddrives, but restoring from harddrive only is pretty quick and something I do for some of my systems already (archived ones).
I do use NexentaStor as well as OI (and illumian.org, the Debian-packaged illumos variant) but did not get the chance to use it in combination with Proxmox.
My overall experience with iSCSI vs. NFS was that albeit iSCSI might give you a tick more of performance, it's not always worth the time you have to invest into learning multipathing, tuning jumbo frames, and foremost configure initiators and MPIO. iSCSI and FC are about block devices while NFS and SMB are closer to each other and can accept temp. loss of IO while block devices via FC oder iSCSI can crash a system more easily when conenctivity is lost.
If you want to use the snapshots on ZFS I had to realize, that - although you can snapshot and clone easily zVols (this is what gets mapped as iSCSI LUN) you cannot expose a snapshot as LUN (i.e. for restore), you have - at least from what I currently know - to revert the zVol completely. With NFS you are more flexible from a illumos/Nexenta perspective.
In general about ZFS for virtualization, you want to have many small vdevs (disk groups) in a zPool to get decent performance.
You can do Raidz but best performance an easiest expanding is possible with Pools of mirrored vdevs.
My overall experience with iSCSI vs. NFS was that albeit iSCSI might give you a tick more of performance, it's not always worth the time you have to invest into learning multipathing, tuning jumbo frames, and foremost configure initiators and MPIO. iSCSI and FC are about block devices while NFS and SMB are closer to each other and can accept temp. loss of IO while block devices via FC oder iSCSI can crash a system more easily when conenctivity is lost.
If you want to use the snapshots on ZFS I had to realize, that - although you can snapshot and clone easily zVols (this is what gets mapped as iSCSI LUN) you cannot expose a snapshot as LUN (i.e. for restore), you have - at least from what I currently know - to revert the zVol completely. With NFS you are more flexible from a illumos/Nexenta perspective.
In general about ZFS for virtualization, you want to have many small vdevs (disk groups) in a zPool to get decent performance.
You can do Raidz but best performance an easiest expanding is possible with Pools of mirrored vdevs.
Yes, as many vdevs as possible (mirrored) but also put in at least 3 ssd drives as well. One large for read cache and two others (mirrored) for write cache. Then, if configured right, you will have a great perfoming disk system to host VM's.
The only drawback with Nexenta is that you really need to have two servers acting as clustered storage controllers if want to have some sort of redundancy or possibillty to maintenance of Nexenta it self without a complete shutdown of you virtual environment. And at this point the cost of Nexenta is in par or even higher than with a more traditional storage solution.
If you arre used to work with traditional storage systems (with dual controllers) where you often can perform online upgrades then you really dont like to have to shutdown everythig just to upgrade or even rebbot the Nexenta machine.
One last thing, be careful with the deduplication. Make sure you have LOTS of memory if you decide to do that in Nexenta.
The only drawback with Nexenta is that you really need to have two servers acting as clustered storage controllers if want to have some sort of redundancy or possibillty to maintenance of Nexenta it self without a complete shutdown of you virtual environment. And at this point the cost of Nexenta is in par or even higher than with a more traditional storage solution.
If you arre used to work with traditional storage systems (with dual controllers) where you often can perform online upgrades then you really dont like to have to shutdown everythig just to upgrade or even rebbot the Nexenta machine.
One last thing, be careful with the deduplication. Make sure you have LOTS of memory if you decide to do that in Nexenta.
I can only confirm xlnt's warning about ZFS deduplication. It's not always the way to go depending on your data, it's good with lots of duplicated small data (think VDI), much less for large pools with only some duplication. When the DDT (Deduplication Table) doesn't fit into RAM your pool performance will really go south. ZFS' default compression gives already good savings with far less impact on memory or CPU requirements.
It is unfortunately true about Nexenta's pricing for HA cluster vs. entry level dual controller appliances.
I just imagine Proxmox on top of an illumos distribution - that could be somewhat awesome. (Think Zones, ZFS and KVM which has been ported to the illumos) ;-)
It is unfortunately true about Nexenta's pricing for HA cluster vs. entry level dual controller appliances.
I just imagine Proxmox on top of an illumos distribution - that could be somewhat awesome. (Think Zones, ZFS and KVM which has been ported to the illumos) ;-)