Storage format benchmarks

[totalimpact]

Spare hardware laying here, so I decided to benchmark some different local (not SAN) storage formats and attempt to show the pros/cons of each, help me out if i'm missing any important points.

Test bed:
Dell R730xd, H730P Raid card (array specs at the bottom)
2x Xeon e5-2683 / 64gb RAM
1x Sandisk Ultra ii pve boot drive, attached to H730, all cache disabled on single drive raid0
8x Samsung 850 Pro 512gb, RAID10 with all cache.
PVE 5.0b2, Windows 7 VM, 22 cores, 8gb RAM, nothing else is running on the host, no other data on storage.

I will compare LVM, ZFS, EXT4, raw + qcow2.

LVM thin pool (default install) = thin provisioning, only written bytes are used, fast snapshotting ~10% read perf hit vs bare LVM, supposedly slower write perf vs LVM, probably the best choice.


LVM = supposed fastest, no thin provisioning, each VM will immediately use 100% of the space you provision... not sure on write speed shown, obviously slower than LVM-thin.


ZFS = many bells and whistles, possibly unneeded complexity in tuning, fast remote snapshot diff backups, thin prov, easy expansion and more, 30-40% slower vs LVM


EXT4 formatted drive, mounted as directory, .raw file on directory This gives a ~55% read perf hit over LVM, (qcow2 known to slightly slower, but offers snapshots)


EXT4 formatted drive, mounted as directory, .qcow2 file on directory This gives a 60% read perf hit vs LVM.


Single SSD LVM-thin (pve OS drive):


Obviously this is all synthetic, take it for what you will, real world application behavior may differ.

Here are the storage specs:

 

[Giovanni]

Your ZFS and LVM screenshots show as broken links

 

[gkovacs]

Spare hardware laying here, so I decided to benchmark some different local (not SAN) storage formats and attempt to show the pros/cons of each, help me out if i'm missing any important points.

Thanks for these benchmarks.

Do I understand correctly that you put ZFS on top of a HP HW RAID array? This is not a recommended setup, therefore not much point in benchmarking IMHO. I would gladly see ZFS benchmarks though when the Proxmox installer is instructed to build a ZFS RAID10 array from the single SSDs.

 

[totalimpact]

That is correct and you are correct the best way to do ZFS is with your RAID controller or HBA in "IT mode" presenting standalone disks to the os. I did not have time to do that though, maybe on another round.

 

[totalimpact]

I have been seeing a lot of users asking about ZFS and its magic abilities, here is a crusher for those on how bad it can get if you dont pay attention every last detail.

Unfortunately the R730 is not on my bench, so today we have a Dell R720:
2x Xeon E5-2620, 32GB RAM, OS on standalone Samsung 860 M.2 on a PCIe adaptor.
A slathering of disks...... H310 flashed to LSI 9211 HBA P19 IT Mode
I have 2x Samsung 850 Pro, 512gb
6x mixed SATA 7200rpm (these are matched in pairs, but outside pair is not)
1x Intel Optane 900P 280gb

Win7 VM OS is running on LVM on the PVE OS drive, leaving a 100% bare virgin virtual drive on my zfs for the test target.

Here is a sample of how bad ZFS can get if you fudge:


What went wrong? I followed best practices - right:
1. RAID10, 6 junk 7.2k drives
2. Plenty of RAM (ARC showing 2gb in use of the 32gb)
3. RAW/Thick provisioned virtual disk, all defaults on caching, and
4. LZ4 compression on pool

Wrong: Most likely this could have been much better had I matched the pool 100%, might try that later if I can find a group matching. And I used desktop drives, a couple say "enterprise"... but go to any medium enterprise and ask if their server has 7.2k?

Lets add the entire Optane as a SLOG, and lets pick a benchmark that will pretty up this turd, Crystal Diskmark, slick like synthetic oil-


That looks almost nice... well I home your VMs dont write too much... Let run the same on Anvil a more realistic bench:


A BRUTAL 184mb write using an enterprise write cache??? what - the data slowly trickled in to the cache - we still need to tune zfs? - actually pay attention to what its doing? FYI- This drive benched at 1600/1100MB/s R/W using RAW LVM on the VM... Yep, its a great budget "Enterprise" caching card, but we didnt have that $2k to spend on 2-3 cards (you DID mirror your SLOG - right?)

Now there are several ways to clean this up and get some better performance, but as far as I know, none of them are safe for my data:
set sync=disabled on zvol or pool = POTENTIAL DATA LOSS, which gives me this:


Enable write back cache on the VM drive = POTENTIAL DATA LOSS, and totally synthetic, not realistic for real loads, gives us:

Obviously that gave us some fake read performance, and killed the writes, in other cases it may help.

Other areas like changing block sizes can help, but it locks you in to a specific data type.

How can I make this better - the proper way???
ZFS RAID10 8x matched drives (10kSAS if you need iops), 64GB RAM, Mirrored 5GB ZIL, 100-200gb L2ARC.

More to come.....