Shared Storage for Proxmox Cluster

[Redhat]

Hi All,

We have a running cluster with 3 nodes based on Proxmox 5 latest. Our goal is to migrate all our stand-alone XenServer into that new environment.

We have a FreeNAS server with 24 400GB SSDs and 24 600GB-10k HDDs configured with two different ZFS pools to provide storage for the cluster via 10Gbps separate network.

We tried to connect the Proxmox cluster to the FreeNAS via ZFS over iSCSI but the current proxmox driver is istgt which is considered end-of-development, see here:
hXXps://forums.freenas.org/index.php?threads/istgt-or-ctld.48771/#post-334095

Moreover, we tried to follow the ZFS-Over-iSCSI proxmox guide and indeed, creating a VM failed due to the fact that istgt was not present.

So we need to find out a better way to provide shared storage to our Proxmox cluster.

We can install any NAS solution needed, given that it is open-source or has a free version.
We need to be able to live migrate VMs, take snapshots and configure HA for the cluster.

What are our options with FreeNAS? What are the other options?

Thank you all for viewing and commenting!

 

[tschanness]

Why not use NFS?

Regards, Jonas

 

[MewBie]

If you can put some additional drives in the Proxmox nodes Ceph could be an option to store the VM's.

The rest of the data stays on the NAS and made available via NFS?

 

[tschanness]

...or put the VM disks on an NFS store...

Regards,
Jonas

 

[Redhat]

Why not use NFS?

Regards, Jonas

Thank you Jonas for the reply.
AFAIK, NFS does not allow Snapshots of VMs, see the table here here:
https://pve.proxmox.com/wiki/Storage

Going with the NFS option is our last choice, we would like to first know if there is a better solution that will allow all operations described in the above mentioned table.

Seems like NexentaStor is not an option due to the 10TB limit, the NAS4Free also lacks the option for ZFS-Over-iSCSI and the native Linux with ZFS4Linux lacks the GUI needed for ease of operation.

Ceph is also not an option given the current hardware we have - Single x86 Server with 24 SAS Bays.

I will run now some tests with NFS export to see how it looks but I believe that the correct solution must not be limited to certain disk image format.

Thank you, Appreciate your contribution to this thread. You are welcome to comment on the above, we still check the options and need some advice from people with good knowledge of Proxmox.

 

[MewBie]

I'm no expert but keeping your VM's on a single NAS will never get you HA.
If you store the VM's inside the Proxmox cluster you get all the features you want. Easiest way to do that afaik is to run Ceph on the Proxmox nodes or have an external Ceph cluster.

 

[Redhat]

@tschanness

Here is why not to use NFS:

I created an NFS share on the FreeNAS box and got ~20MBps (MegaByte Per second) when moving data inside a VM.
Then i created an iSCSI target and on top of it an LVM storage (shared between all nodes) and now I get 200MBps when i copy data within a VM, on the exact same storage device.

NFS is well known to be slow, especially it v4.

With LVM over iSCSI i get very good speed but there are limitations.

When migrating a VM from a node to another node, the data gets copied inside the iSCSI share while with other methods (even NFS) it is just being assigned to the other node instead of being copied.
We can live with that since most of our VMs are less than 100GB disk size so migrating an average VM takes few minutes.

Snapshots does not work at all, I still need to figure out how to solve this.

Backups probably works but i need to check it out.

 

[tasya]

@tschanness

Here is why not to use NFS:

I created an NFS share on the FreeNAS box and got ~20MBps (MegaByte Per second) when moving data inside a VM.
Then i created an iSCSI target and on top of it an LVM storage (shared between all nodes) and now I get 200MBps when i copy data within a VM, on the exact same storage device.

NFS is well known to be slow, especially it v4.

With LVM over iSCSI i get very good speed but there are limitations.

When migrating a VM from a node to another node, the data gets copied inside the iSCSI share while with other methods (even NFS) it is just being assigned to the other node instead of being copied.
We can live with that since most of our VMs are less than 100GB disk size so migrating an average VM takes few minutes.

Snapshots does not work at all, I still need to figure out how to solve this.

Backups probably works but i need to check it out.

@Redhat

May I know if you are still using the freenas now as your share storage? and did you installed on decidate server or inside the proxmox ?

Thanks