Best way to use storage from TrueNAS for performance?

[j_s]

Hello. Relatively new to proxmox, but not TrueNAS. Been using TrueNAS for years.

Been reading Proxmox docs, and not coming to any conclusions on anything, so I'm resorting to asking and hoping to clarify a few things.

Currently I have a cluster of 3 nodes all connecting to a single NFS share on TrueNAS. I'm using qcow2 so I can do VM snapshots. I use them as quick backups when I want to do something like a risky OS upgrade and I want to roll back if things go badly. Usually the snapshots are around for just a few hours unless I forget to delete them (whoops).

In this configuration performance is "acceptable", but I can tell this will not scale well as I add more VMs. What is the best way to handle storage of VMs from the TrueNAS?

I keep reading about ZFS-on-iSCSI. My guess is that each VM gets it's own iscsi disk. That could turn into a lot of iSCSI disks! Is that really the best choice?

Is ZFS-on-iSCSI really the end-all for maximizing performance? What about some kind of Ceph implementation? I have no experience with Ceph (yet).

Also, is there any way to convert my current qcow2 VMs to work with ZFS-on-iSCSI if I get to that?

I'm hoping to have some kind of snapshotting functionality because it comes in handy during risky processes (like upgrading linux VM versions).

Thanks!

 

[floh8]

What is the best way to handle storage of VMs from the TrueNAS?

i think nfs is best

That could turn into a lot of iSCSI disks! Is that really the best choice?

its not that stable as nfs.

What about some kind of Ceph implementation?

much slower than nfs for writing.

Also, is there any way to convert my current qcow2 VMs to work with ZFS-on-iSCSI if I get to that?

yes, u can migrate the disk from a to b.

What is the best way to handle storage of VMs from the TrueNAS?

u should use mirrors with slog and l2arc with nvme or fast ssd.

 

[abufrejoval]

Short version: nothing can make a single TrueNAS go faster, if it's limited on storage or disk I/O.

Long version:

It's been a long time since network storage was faster than local storage, especially in these days of cheap NVMe. But if your TrueNAS is good enough, NFS will be good enough, no need complicating your life with something fancy.

Proxmox is essentially Linux and KVM plus orchestration around it.

When it comes to storage I/O performance, you don't need to ask around here but look at what others have measured and reported for anything that uses KVM.

Then again KVM tries to minimize the VM overhead via pass-through drivers like VirtIO, so the performance you can achieve from any Linux host to your storage should not be too much worse than what a VM inside can do.

These days it's mostly the network speed that limits your network file performance, if it's 1Gbit Ethernet, iSCSI vs. NFS shouldn't be much of a difference, especially if you use NFS 4.x and not v3. At 100, 400 or 800Gbit/s Ethernet things might be different again, but in these days of NVMe storage arrays the bottleneck tends to be in the network not the protocol, especially if your TrueNAS only has a single NIC.

Ceph is designed for scale-out storage, not a single host. It's slower for writing only if the write amplification for updating the redudant hosts is limiting the throughput. Again, that is more likely to happen if the network is undersized because the typical minimum 3 node Ceph cluster will want three copies of the data written from the client. The advantage is resilience and in some cases read performance can be better because they can be scattered across the Ceph nodes.

NFS has no scale-out and no out-of-the box redundancy so only a single copy of any written data will go over the network. No resilience outside the box, either.

If you have thousands of Ceph nodes and thousands of clients the aggregate read and write bandwidth tend to surpass that of any NFS appliance.

But for your use case, that's academic.