What open source solutions are available to use "ZFS over iSCSI with Proxmox"?

al.semenenko88

New Member
Dec 25, 2017
6
0
1
31
What open source solutions are available to use "ZFS over iSCSI with Proxmox"?
Will FreeNas 9 or 10 work for "ZFS over iSCSI"?
What solutions are on Linux?
 

raku

Member
Apr 16, 2016
35
3
8
38
These patches added support for native Linux ZFS over ISCSI support. And what about FreeNAS support?
I've got problems with running OmniOS on my hardware. What would be a better choice for NAS/SAN storage in near future? Linux server with LIO/targetcli or FreeNAS?
 

raku

Member
Apr 16, 2016
35
3
8
38
Bump

I've managed to apply ILO patches on my testing virtual environment. Looks like it works - I can create, delete and migrate offline KVM VMs. Great work of Udo Rader !!!
Can't test it on running VMs yet, as Virtualbox doesn't allow to run nested virtualization. Right now I'm preparing to move my tests to bare metal hardware.
Is there any ETA to include patches to official Proxmox release?
 

raku

Member
Apr 16, 2016
35
3
8
38
Cool. Now I need to choose between ZFS On Linux and FreeBSD :)
I think, FreeBSD would be a better choice.

Are these patches going to be included in Proxmox in near future?
 

Knuuut

Member
Jun 7, 2018
91
8
8
54
I like the way how the connection from proxmox to freenas is set up. It's done over the api, not over ssh with config file manipulation.
I'd appreciate too, when this would be included into proxmox mainstream.
 

raku

Member
Apr 16, 2016
35
3
8
38
Raku - What kind of problems did you experience with OmniOS? This was just recommended to for an iSCSI shared storage solution for a PVE cluster. Is FreeNAS a better choice in your opinion?
Typical hardware compatibility problems:
1) OmniOS doesn't recognize my SATA drives connected to LSI based RAID controller. Only SAS drives are visible
2) OmniOS doesn't work with my Mellanox 10Gbit network adapter

Is FreeNAS better? I don't know. I've patched my cluster with freenas-proxmox from https://github.com/TheGrandWazoo/freenas-proxmox. I was able to connect the cluster to my FreeNAS storage (16 x 4 TB SAS). I can create, run, migrate, destroy KVM machines. I've tested sequential writes with dd and got transfers about 1GB/s on empty cluster. Everything works good so far.
What I miss is https access from proxmox to FreeNAS API. I'm working to fix that and when it's ready, I'll create pull request to include my patches to freenas-proxmox project.
 

mir

Well-Known Member
Apr 14, 2012
3,489
97
48
Copenhagen, Denmark
2) OmniOS doesn't work with my Mellanox 10Gbit network adapter
This is strange since the drivers for Mellanox 10Gbit in FreeNAS/FreeBSD is the ported drivers from OmniosCE. I have no problems with either Mellanox 10 or 20 Gbit in both Omnios and FreeNAS.
 
Thanks for the clarification, Raku - please keep me posted on your progress within this topic?

We are trying to establish a standard / preferred shared cluster storage to recommend to our Proxmox customers and prospects, and I would like to have assurances that people have had consistent success with specific configuration(s).

Thanks very much.
 

raku

Member
Apr 16, 2016
35
3
8
38
Progress report:
I've configured FreeNAS storage and found it faulting while cloning VMs and booting from virtio SCSI hard drive. So I moved to Ubuntu 18.04 and iSCSI via LIO. First look after patching Proxmox and configuring ZFS pool and iSCSI target - all works smoothly. But closer look revealed huge immature of ZoL and iSCSI soluton.

I've tested storage performance inside KVM VM:
FreeNAS: about 450-700 MB/s, 3500 IOPS
Ubuntu: about 80-120 MB/s, 700 IOPS

So I switched back to FreeNAS and just a couple minutes ago solved my issues:
https://github.com/TheGrandWazoo/freenas-proxmox/issues/9
https://github.com/TheGrandWazoo/freenas-proxmox/issues/10

I think, I'm going to stay with the FreeNAS solution. Zvol over iSCSI on linux sucks.
 
Last edited:

raku

Member
Apr 16, 2016
35
3
8
38
All features I need work OK. But so far, I run only 1-2 VMs simultaneously. I need to do some more tests running about 50-100 VMs. If that works, I'll say it is production ready on my site and start to migrate from the old XenServer cluster.
 
Apr 27, 2013
25
0
1
It would be simply amazing to use zvol over ISCSI with FreeNAS. I'm a bit nervous to put it into production without an official release though. Is there any thoughts on this from Proxmox devs? I am going to give it a shot on some test hardware.
 

LnxBil

Well-Known Member
Feb 21, 2015
4,217
407
83
Germany
@udo send another batch of patches in week - at least I saw the mails in pve-devel about this. It's still in development.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!