Shared Storage Comparison

[ejmerkel]

We are looking at recommendations for shared storage for a new Proxmox cluster. We are considering the following 2 options:

1) Using Ceph via Proxmox nodes
2) Using a NAS such as TrueNAS https://www.ixsystems.com/truenas/ using either ISCSI or NFS

Can anyone share their recommendations and pluses and minuses of either approach?

Best regards,
Eric

 

[jeronimo89]

i have about 20 sites with proxmox (single or cluster) wth Freenas without any problems using NFS

 

[DANILO MONTAGNA]

The best option depends on the hardware/infrastructure environment you have...

If you have enough hardware with local disks and at minimum of 1 SSD disk per node, and is comfortable with the lost area when you choose the replication size of a ceph cluster (usually 3/2)... go to ceph..

If you prefer to use as much as available area possible, and have a good 10Gbit network for storage back-end connection, go to NFS.

Using iSCSI is also a good option, but it depends on what type of filesystem you want to use on top of if.. because you will have some limitations, like VM-Snapshots if you choose to use LVM and share it between nodes.

 

[ejmerkel]

The best option depends on the hardware/infrastructure environment you have...

If you have enough hardware with local disks and at minimum of 1 SSD disk per node, and is comfortable with the lost area when you choose the replication size of a ceph cluster (usually 3/2)... go to ceph..

If you prefer to use as much as available area possible, and have a good 10Gbit network for storage back-end connection, go to NFS.

Using iSCSI is also a good option, but it depends on what type of filesystem you want to use on top of if.. because you will have some limitations, like VM-Snapshots if you choose to use LVM and share it between nodes.

Well, I have 2 existing clusters using local storage so probably the easiest would be to add a NAS. All of our switches our 10G switches so that is not a problem.

In regards to NFS vs ISCSI, I have seen some benchmarks of FreeNAS/TrueNAS and it sounds like ISCSI out performs NFS but you are telling you cannot take snapshots on an ISCSI LUN?

On a different note, TrueNAS is using a ZFS file system. Would it make sense to do the backups on the NAS via ZFS rather than taking vzdumps of the images? Is there any integration between Proxmox and ZFS on a separate NAS?

Best regards,
Eric

 

[DANILO MONTAGNA]

In regards to NFS vs ISCSI, I have seen some benchmarks of FreeNAS/TrueNAS and it sounds like ISCSI out performs NFS but you are telling you cannot take snapshots on an ISCSI LUN?

On a different note, TrueNAS is using a ZFS file system. Would it make sense to do the backups on the NAS via ZFS rather than taking vzdumps of the images? Is there any integration between Proxmox and ZFS on a separate NAS?

Regarding the Snapshots, I was talking about taking snapshots from guests using proxmox GUI, not on LUN side, because the only filesystem on top of iSCSI that supports VM Snapshots is ZFS over iSCSI, LVM-thick does not support VM snapshots.

About backups, if your NAS is using ZFS for filysystem you can do that for sure, but for a better integration between backups and proxmox, the best option is to use vzdump.

 

[jayg30]

You'll find a bunch of benchmarks for freenas comparing nfs and iscsi. Just know that lots of them are poorly done unfortunately. They both perform equally well when doing the same thing. The big differences usually comes from how the ZFS storage looks to the VM and if it is lying about sync writes. Correctly setup and tuned so that are both "safe" for your data the difference will be small.

Now, if you were using esxi as your hypervisor (or hyperv since it doesn't do NFS) then this wouldn't be a question. You would want to pick iSCSI. FreeNAS only supports VAAI over iSCSI. Meaning you must use it to access the VAAI features like write same zero, xcopy, UNMAP, etc. Freenas never built this out for NFS.

Outside of that case, I prefer NFS for it's flexibility and ease of use. Make sure you have a fast slog device (with power loss protection) to handle the sync writes. A good UPS as well. Don't disable sync unless you really know the dangers.

good read https://www.ixsystems.com/community...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

 

[ejmerkel]

You'll find a bunch of benchmarks for freenas comparing nfs and iscsi. Just know that lots of them are poorly done unfortunately. They both perform equally well when doing the same thing. The big differences usually comes from how the ZFS storage looks to the VM and if it is lying about sync writes. Correctly setup and tuned so that are both "safe" for your data the difference will be small.

Now, if you were using esxi as your hypervisor (or hyperv since it doesn't do NFS) then this wouldn't be a question. You would want to pick iSCSI. FreeNAS only supports VAAI over iSCSI. Meaning you must use it to access the VAAI features like write same zero, xcopy, UNMAP, etc. Freenas never built this out for NFS.

Outside of that case, I prefer NFS for it's flexibility and ease of use. Make sure you have a fast slog device (with power loss protection) to handle the sync writes. A good UPS as well. Don't disable sync unless you really know the dangers.

good read https://www.ixsystems.com/community...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

Thanks for that information. So I am curious, what your opinion would be on backups of VM's if I were to use NFS on FreeNAS/TrueNAS. Would you use ZFS snapshots on the NAS device and then copy those off to another backup NAS? Or would you a VZDUMP from the Proxmox Hypervisor to another NFS share on another NAS?

It seems to me the NAS -> NAS backup would be faster than NAS -> PROXMOX -> NAS but I am guessing you would lose the ability to restore a VM from VZDUMP backups. That would have to all be handled on the NAS. I am just trying to think of the benefits and down sides of both approaches.

Eric

 

[LnxBil]

You also have to consider chances of failure. Every "cheap" NAS has only one controller, so a failure in the controller is a total failure in your cluster.

If you want true HA (everything can and will fail and has to be redundant), you need a HA storage aswell. This means either buying a enterprise grade SAN (dual-controllers, dual PSU, dual outbound connection) with at least two switches (is does not matter if it's 10GBE for iSCSI or FC) or you use CEPH (two switches apply here too). Everything else also works fine, but has a single point of failure (SPOF).

 

[ejmerkel]

You also have to consider chances of failure. Every "cheap" NAS has only one controller, so a failure in the controller is a total failure in your cluster.

If you want true HA (everything can and will fail and has to be redundant), you need a HA storage aswell. This means either buying a enterprise grade SAN (dual-controllers, dual PSU, dual outbound connection) with at least two switches (is does not matter if it's 10GBE for iSCSI or FC) or you use CEPH (two switches apply here too). Everything else also works fine, but has a single point of failure (SPOF).

Understood. We have redundant switches and and each hypervisor has 2 x 10G interfaces using LAG into different switches. We are also considering using the TrueNAS M40 (https://www.ixsystems.com/truenas/) NAS which has redundant controllers so this should be as HA as possible.

 

[LnxBil]

Understood. We have redundant switches and and each hypervisor has 2 x 10G interfaces using LAG into different switches. We are also considering using the TrueNAS M40 (https://www.ixsystems.com/truenas/) NAS which has redundant controllers so this should be as HA as possible.

Perfect!