New HA setup suggestions (MP ISCSI with ZFS or ..?)

[mailinglists]

Hi guys,

we have been using CloudStack with ISCSI exported HA storage (pairs of DRBD nodes).
We want to try proxmox now and also move away from DRBD, GlusterFS or CEPH. KISS style.
Stating that, I am wondering what would selection of technology be for:
- production system,
- true HA,
- live snapshots, migration and backups,
- LXC and KVM supported.

I was thinking along the lines of:
At least 3+ CPU nodes with Proxmox VE 4.* in a cluster setup.
At least 1+ external dual controller storage appliance from the likes of EMC.
At least 2+ switches for SCSI multipath between dual controller storage and CPU nodes.
ZFS on top of ISCSI multipath target exported from the storage.
(because (c)LVM would not allow us snapshost or thin provisioning?)

What do you think about such setup? Is there a reason why ZFS would not work on top of a multipath ISCSI target?
We welcome all suggestions, especially on the storage side. Even better if you refer to a specific model or solution.

Here is a personal "out of the box" thought for storage. ZFS mirror over two VDEVs which are both different ISCSI devices on different non HA cheap storage solaris boxes?

 

[mailinglists]

Because ZFS is not a clustered file system, I guess the above setup will not work for live migration or true HA.
I guess the only option is then LVM over ISCSI with locking mechanism from proxmox (instead of CLVM) or NFS.
And in order to have snapshots, NFS with qcow2 images is the only option then?
Are there any other options?

 

[mir]

You could replace EMC with Nexenta to gain ZFS on the storage appliance. If you have a lot of cash you could even get a HA storage appliance since Nexenta supports HA given the right license.

I just noticed this product also: http://www.zeta.systems/

 

[mailinglists]

Thank you for your suggestion. It is appreciated.

If i understand correctly, ZFS on the storage appliance is not visible to Proxmox and therefor backups, snapshots, etc could not be manipulated or used from within Proxmox. In this case, if I export one ISCSI target and all nodes use it via (c)LVM, there is no simple way to revert a single VM disk, just exported volume as a whole.

 

[mir]

If i understand correctly, ZFS on the storage appliance is not visible to Proxmox and therefor backups, snapshots, etc could not be manipulated or used from within Proxmox

It is the opposite way. The plugin makes the storage visible to proxmox so snapshots, backups etc. can be used and manipulated within the proxmox gui.

 

[mailinglists]

That's awesome. I guess I just have to try it out to understand completely.
Does it require any specific storage solution, or it would work with basically any fully implemented ISCSI protocol where the backing device has ZFS?
I guess that for testing I could use openattic, but then buy a enterprise grade dual controller storage with ZFS if such even exist.

 

[mir]

Does it require any specific storage solution, or it would work with basically any fully implemented ISCSI protocol where the backing device has ZFS?

I would suggest that you only use a Solaris based solution. Personally I use Omnios (https://omnios.omniti.com/). Commercial support is available. With napp-it as gui (https://napp-it.org/index_en.html). For best hardware support buy a SuperMicro MB either with on-board LSI HBA or added LSI HBA. For Nics only use Intel - Any 1 Gb and 10 Gb only up to X520 (X540 and X540 is in beta). Infiniband is also supported up to ConnectX3. Remember to use ECC RAM for ZFS. If you want to add Log and L2ARC device to pool only use Datacenter certified SSD.

but then buy a enterprise grade dual controller storage with ZFS if such even exist.

I have listed some options in this post: https://forum.proxmox.com/threads/iscsi-any-way-to-get-snapshots.26824/#post-142400
I know for a fact that some large storage providers in Germany and Switzerland has made HA ZFS storage using Omnios with RSF-1 on-top (http://www.high-availability.com/high-availability-rsf-1/)

Forgot to mention that if you have a very large wallet Oracle has a ZFS HA storage appliance in-a-box (Oracle ZFS Storage ZS3-2)

 

[dignus]

Would definitely recommend http://zeta.systems. Been using it for about 2 years, very happy about it.
But.. proxmox's native ZFS implementation doesn't work since it uses a different iscsi daemon than Zeta.

 

[mir]

proxmox's native ZFS implementation doesn't work since it uses a different iscsi daemon than Zeta.

What iscsi daemon does Zeta use?

 

[dignus]

SCST

 

[mir]

Does Zeta have an official API?
Can a Zeta storage application be managed through cli via ssh?

 

[mir]

Seems their newest release comes with a REST based API. Sadly you cannot get a community version and since I only want to have a Zeta storage appliance to be able to write a storage plugin for Proxmox I see no such plugin coming :-(

 

[dignus]

Does Zeta have an official API?
Can a Zeta storage application be managed through cli via ssh?

Yes, it does have an official API.
SSH/CLI: yes, basically all zeta does is write configs to disk/system and monitor. I have to say their monitoring & management GUI is quite excellent.