PVE: 64Gb RAM, 2xSSD 128Gb zfs mirror (system + CT templates + ISOs only). VMs and CTs are stored on NAS via 10Gbe network.
NAS: 32Gb RAM, 2xSSD 128Gb zfs mirror (system only) + 4xHDD for data storage
@matpuk your test sounds like comparing tomato with potato
So what have you tested?
your PVE: contains SSD only
your NAS: has SSDs and HDDs
so you are not testing the same things!
It would be comparable if your NAS and your PVE would run only on SSDs
@ITT solving problems with killing it with hardware is one option, the better is to solve the root of the issue, which @matpuk is trying to solve.
IMHO @matpuk does a better engineering approach
systemctl disable corosync pve-ha-crm pve-ha-lrmfor example would be a good start in case you aren't clustering.
No, this is the internal database of PVE. In a cluster it is written more often, but in a single system it also written a lot. This is one of the main wear causes in PVE.After playing with iotop, I have a candidate who is writing 24/7 to a system partition every couple of seconds - pmxcfs (https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs))!
Thanks for confirmation. I've just got the same result after running iotop 60minutes on idle system:No, this is the internal database of PVE. In a cluster it is written more often, but in a single system it also written a lot. This is one of the main wear causes in PVE.
I've just discovered the issue myself after two weeks of running my pilot PVE and TrueNAS (Debian based) installs.
Why there is so HUGE difference for just system partitions?
I'm around 64kB/s for pmxcfs on a 20 nodes clusters with 1000 vms.Thanks for confirmation. I've just got the same result after running iotop 60minutes on idle system:
View attachment 43611
20Mb total of small writes every second by pmxcfs...
Well. Thanks everyone. I think I will go with 2x2.5" HDD way instead of cheap SSDs. Enterprise grade SSDs are way to costly.
Yes. It is cumulative value over 60 minutes. Sorry for my English.I'm around 64kB/s for pmxcfs on a 20 nodes clusters with 1000 vms.
Are you sure that your 20MB is not the cumulative over the 60 minutes ?
Well, then I definitely should go with 2.5" HDDsHere, TrueNAS also killed 3 of the 4 cheap consumer system disks this year (2x 120GB TLC Intenso, 1x 120GB TLC Crucial. That crucial wasn't even a year old). So best you don't buy these cheap SSDs at all when using ZFS, no matter what OS you use.
You also sometimes find good prices for enterprise SSDs. I think it's not allowed to advertise stuff here, so I won't post links, but I just bought some spare boot disks on German ebay. New and sealed from a retailer. 100GB Enterprise MLC SSD with 3.737 TB TBW (or 20 DWPD over 5 years). 25€ incl. shipping. Spinning rust wouldn't be cheaper. There is still one left for the lucky one finding itWell, then I definitely should go with 2.5" HDDs