You could check the SMART attibutes of the SSDs with something like
smartctl -a /dev/nvme0. But what you are able to see there and what not depends on the manufacturer. So they might tell you how much data you tried to write to them over their lifetime (like "Data Units Written: 285,976,046 [146 TB]") but they might not show you how much actually was written to the NAND because of the internal write amplification.And I also always install the sysstat package (
apt install sysstat). With that there is a iostat command that will low IO activity. If you run something like iostat 60 2 it will show you one statistic instantly and that statistic will how you statistics of all your drives collected since boot. If you wait 60 seconds for the command to finish it will show you similar statistics but this time they are just based on the last 60 seconds.ECC will be way slower compared to non ECC RAM. But using ECCs means your server isn't crashing that much and you can rely on it that your data is what it is supposed to be. If you compile the same code 1,000,000 times with ECC you would get 1,000,000 times the same binary. Do it without ECC and only 999,999 binaries might be the same and one of them is corrupted and will cause unexpected behavior when running that binary. But really useful it gets when your RAM is slowly dieing. Then maybe each 10th binary might be corrupted instead of each millionth binary and you are serching for bugs in that program but can't just find them...because the source code is absolutly fine and just the binary is corrupted.
If you care about your data and you want to rely on it, that your data is always correct, you should go with ECC RAM even it will be slower.
Yes, it shouldn't matter for ZFS which slot you use as long as there is no HW raid and it can access each individual drive. ZFS is also no real raid that uses arrays with a fixed stripe width or something like that. It just distributes datablocks across multiple individual drives dynamically. So it can work like a raid10 but technically it isn't a raid.
And yes, you can add pairs of drives later and stripe them so it gets one big pool. Just isn't that easy if that pool is also your boot pool.
Pveperf will only benchmark your boot pool and not other pools. "FSYNCS/SECOND: 281.94" is how much sync IOPS your boot pool can handle. All your SSDs will be terrible at handling sync writes because they all are missing the powerloss protection so don't expect good numbers there.
For random read/write you want to stripe them. So the more mirrors you stripe the better your random read/write should be.
If you stripe different 1TB SSD models all other SSDs will be slowed down to the speed of the slowest SSD. So best would be not to mix different models.
Last edited:
