Please forgive my ignorance. What made one think I was looking to do server stuff? (my thread opening post already edited to reflect this)
I am looking for a non server workload. Just freakishly fast performance with the stuff that is available to me
You want to run VMs. Thats part of virtualization and that is server stuff. It doesn't matter that much what you are actually running (except for DBs and so on) inside the VM. The point that you run VMs at all is enough. You can run two Win10s VMs for gaming only... that is still a server workload and not made for gaming hardware. The more VMs you want to run in parallel, the worse stuff like consumer SSD will be able to handle it. And consumer hardware isn't build to survive 24/7 operation and may fail very quickly. If you want to run that computer 24/7 (and I would assume that is your plan if you want to run TrueNAS and distributed computing) that is nothing consumer hardware is build for.
This is how a gaming PC expects you to store stuff:
SSD <- onboard controller/NVMe <- ntfs <- OS
This is how Proxmox stores stuff:
SSDs <- HBA <- pool <- zvol <- cache <- virtio <- virtual disk <- filesystem <- GuestOS
Here you get write amplification with each step (ZFS, virtio, ...) between the guest OS and your physical SSD. And write amplification isn't adding up but multiplying so that it is more an exponential than an linear increase. Lets use some fictional numbers and say there is an factor 10 amplification inside the SSD (thats low for sync writes with consumer SSDs), an factor of 2 because of ZFSs ZIL, an factor of 3 because of other ZFS stuff, an factor of 4 from virtio virtualization and a factor of 2 inside the guest because of journaling. You wound't get a write amplification factor of 21 (10+2+3+4+2) but the write amplification would be a factor of 480 (10*2*3*4*2). So for each 1GB you write inside the guest your SSD would write 480GB!!!
And if you want some real numbers: With my homeserver for every 1GB written inside the VM 20 to 40 GB will be written to the NAND of the SSD because I got a write amplification of 20 for async writes and 40 for sync writes.
So if I run a windows VM and install a 100GB steam game it will write 2 to 4 TB to the SSD. If I install Windows bare metal instead and install the same game it might only write 120GB or something like that.
So if you think "600 TBW isnt that bad, i can install that game 6000 times...I will never reach that..." thats wrong. Atleast if you run it inside a VM. Because installing it 200 times inside a VM might be enough to kill the ssd. Now you know why enterprise SSDs are build for a much higher write endurance and why I sold my Samsung EVO NVMe SSDs and got some enterprise SATA SSDs instead. My 1TB enterprise SSD is now capable of writing 18000 TB and my old EVOs would have died after just 600 TB, like your SSDs.
So your 600TBW aren't that much anymore because if you write more than 20TB within 5 years inside your VM you loose your warranty and you are over the SSDs life expentation. So your SSD can only handle a constant write over 5 years of 136kb/s (20TB / (5*365*24*60*60 sec)). Now lets say 10 VMs are sharing the same SSD. Now each VM may only write with up to 14kb/s in average. If a VM writes with more than 14kb/s you would loose your warranty and the drive may die in under 5 years. 14kb/s is that low it can't even handle the logs...
Sure, you could not care about the warranty and write more like 70kb/s now your SSD will die within 1 year. Or you write with 840kb/s and the SSD might die within a month...
And you are dreaming about four SSDs in raid0 that can write with 5,300MB/s each, so 21,200 MB/s in total. Do the math yourself:
4x 600 TBW = 2,400 TB life expectation = 2,400,000,000 MB life expectation
2,400,000,000 MB life expectation / 21,200 MB/s = 113,208 sec
113,208 sec / 60 / 60 = 31.44 hours
So write with full speed for 31.44 hours and you might have killed 4 new drives. You say you don't care how long they survive, you just want maximum performance. Thats not really cheap if you need buy 4 new SSDs for 700$ every second day...
Sure, in reality your are not always writing with full speed for so long and your SSDs also wouldn't be able to handle that for more than some seconds because they are just build for short bursts and not for long sustained writes like enterprise SSDs, but it is nice to show how bad the life expectation of SSD really are. Also keep in mind: If you got a write amplification of factor 20 it's doesn't only mean that 20 times more is written to the SSDs NAND so the drive dies 20 times faster. It also means that your SSDs will be 20 times slower. If your VM writes with 1000 MB/s your SSDs will write with 20.000MB/s. So, if a raid0 can only handle 21,200 MB/s you wont see better writes than 1060MB/s inside a VM. And you are running multiple VMs that need to share that speed. So most of the time you won't even see that inside a VM.
Maybe your write amplification isn't as bad as mine but you will still get write amplification. And I'm no special case. Its just a normal homeserver thats runs local services and I am the only person why uses it. Right now it writes around 900GB a day of data while idleing and most of that are just logs/metrics. Show me a gaming PC that is writing 900GB a day while nobody is using it..if that would be normal no one would buy these Samsung QVOs anymore.
And there are alot of things like this that you probably didn't think of when building that server.
Distributed computing like BOINC, F@H and so on that are heavy scientific computations that I also would call server workloads. Especially if you want to run so many GPUs using CUDA/OpenCL in parallel.
And it sounds like you want to run a TrueNAS on it. That again is server stuff. ZFS wasn't made to be run on consumer hardware. Initially that was a file system designed by solaris to only run in datacenters on big storage servers with dozens of HDDs. No of the engineers even thought of someone might want to run that on consumer stuff. It is designed that way that it just relies on you to run it on very stable hardware with ECC RAM, UPS, redundant power supplies and so on. So even "simple" stuff like storing files is a server workload.
Look at the hardware recommendation for Proxmox and TrueNAS and they all tell you to get server and not consumer hardware.
They don't do that because they get money from HP/Dell/Lenovo for selling stuff, but because consumer hardware just isn't made to reliably run heavy workloads 24/7.
Don't ask how many consumer PCs I killed the last 20 years running BOINC 24/7...I stopped counting failed hardware...has to be atleat 8 PSUs, a dozen of RAMs, some GPUs and so on...
So if you already got that hardware thats fine and we can see what best to do with it. Atleast CPU and RAM are fine for a server and the board got alot of PCIe lanes. But just don't expect too much.
I've already seen in another thread that you complained that your EVOs are so slow. They just aren't fast SSDs for server workloads so they will be never as fast as good enterprise SSDs if it comes to advanced stuff like sync writes, heavy IO, continous IO and so on. Maybe you bought them because they got big numbers on paper and advertise with "up to 5300MB/s writes" but the reality is that you can be lucky if they reach something like 50MB/s (like a SATA enterprise SSD) on stuff like random sync writes on the host or some single digit MB/s inside the VM. What really matter in a server isn't big write/read numbers that might be achieved but instead what the SSD ensures to achieve at minimum under all conditions and at every time. And of cause how reliable and durable the SSD is.
If your SSDs just can't handle server workloads and you don't care about enterprise features and data integety you could for example try to PCI passthrough one NVMe SSD to each VM. That way you don't get virtualization overhead, no write amplification from mixed block sizes and the workload is more what they are build for. Just one OS using it without any advanced stuff in between. Then they are just like a SSDs in a normal gaming PC and should perform better.
But you would loose all the nice enterprise features like redundany, bit rot protection, block level compression, deduplication and so on and you can't run more than one VM of each SSD.
