SAN / NAS plan out assistance needed here

[Johannes S]

I didn't know there was a bug with veeam... I'll look for that.

This is the thread: https://forum.proxmox.com/threads/veeam-silent-data-corruption.155212/

tldnr: Blockbridge (bbgeeks employer) encountered a data corruption issue during their tests, Veem published a fix but at the moment it still needs testing. There are some first positive reports though. You will want to read the whole thread to get the whole picture.

 

[bbgeek17]

We should get some results in a few days

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[geekspaz]

An update on the setup while I'm sitting in the airport to fly back home...
One of the other engineers I met at the data center apparently runs TrueNAS Scale on his home setup, and essentially (very politely and tactfully) told me to "quit stressing and worrying, it'll work great". He then went ahead and loaded TrueNAS on the storage host while I worked out some hardware issues with the other 2 servers. So for the next week we'll be testing out the performance and playing with the migration process from VMware to Proxmox to get a baseline for the steps and estimated transfer times... Amongst other things. I can give updates on what we see with TrueNAS in this setup, and if we try others out I'll provide that performance info too.

Thanks again everyone!

 

[geekspaz]

@bbgeek17 - does Blockbridge demo hardware setups? Since they have configs with Dell hardware, I can probably convince the company to give it a try as an alternative to the PowerStore...

 

[bbgeek17]

Yes, we do offer proof-of-concept installations for qualifying customers.
Please submit a ticket via https://www.blockbridge.com/proxmox using the "Contact Us" button at the bottom of the page.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[geekspaz]

It's been what, a week and a half? Update on findings thus far...
We are using TrueNAS Scale on the "storage host" Dell PowerEdge R740xd. It has the backplane installed that supports 24x 2.5" SATA/SAS drive, and 12 of those slots will also support U.2 NVMe drives. It's running 3x PCIe x16 "extension cards" to interface the NVMe drives to PCIe.
We have 10x Micron 9300 Pro NVMe drives installed, and are running RAIDZ1.
For network connections, all 3x R740xd servers (2x Proxmox hosts, plus the 1 storage host) have dual 25gbe links, but thus far we've not bonded the nics (this will be the next round because we need to).
I don't know the details on config as another guy has been doing the config and testing, but the cliff notes are:
Running disk tests on 1 VM we are getting limited by the 25gbe link, and we are seeing about 46gb throughput by running simultaneous disk tests from VM's on both Proxmox hosts with their storage on the TrueNAS host.

 

[geekspaz]

Using NFS for the VM storage, running tests on 1 VM:


After redoing the setup using iSCSI, and running VM's from both hosts, this is what the TrueNAS dashboard showed:

Edit: one additional note, we are also running dedup and LZ4 compression.

 

[LnxBil]

Edit: one additional note, we are also running dedup and LZ4 compression.

I'd be interessted in dedup ratios on your final setup. We rebuild our ZFS after having more performance problems than gain from the deduplication.

 

[geekspaz]

I'll be sure to update on it then!

 

[geekspaz]

I'd be interessted in dedup ratios on your final setup. We rebuild our ZFS after having more performance problems than gain from the deduplication.

I got an update today - dedup on zfs was causing CPU utilization at 80%+ when moving data, and freezing on any VM on that data store... So that's going away.