New to Proxmox, New Installation

[Robert5150]

Hey Guys,

Hoping you can help out. I would like to setup a very basic proxmox VM Host. I am planning on using the Following:

1) Core i7-4790
2) Z97 Motherboard
3) 1 x NVME drive, 2x Sata hdd, 2x SSD
4) 32GB RAM - DDR3

I am trying to best figure out my disk layout. I am thinking this would be the best usage of my disks

NVME Boot Drive, 500 GB. 1x Disk Pool (Hot), SSDs, 1xDisk Pool (Warm) SATA Drives.

I am familiar with Linux but am really a windows guy so i want to make sure i am at least going in the right direction. I plan on setting up several windows server 2019 VMs, Including Exchange, FIle Servers, etc and maybe a few others as well.

If i allow the ProxMox installer to Use the entire nVME drive does that make sense? or will the drive just sit at 10% usage by the Proxmox install?

Thanks,

Robert

 

[Dunuin]

If i allow the ProxMox installer to Use the entire nVME drive does that make sense? or will the drive just sit at 10% usage by the Proxmox install?

Check the paragraphs "Advanced LVM Configuration Options" and "Advanced ZFS Configuration Options": https://pve.proxmox.com/wiki/Installation

By default only a small part of that NVMe will be used as your root filesystem and the bigger part of that NVMe will be used as a VM/LXC storage.

 

[Robert5150]

So i did read through that, not sure that i understood it though, to be honest. if only a small part of the drive will be used for Boot, System, Swap - can the remaining part be used for Caching? such as here "ZFS can use a dedicated drive as write cache, called the ZFS Intent Log (ZIL)."

If possible i would like to use 4x SATA SSDs in a ZFS RAID1. THis motherboard does not support using 6+SATA ports and the nVME drive at the same time.

Robert

 

[Dunuin]

So i did read through that, not sure that i understood it though, to be honest. if only a small part of the drive will be used for Boot, System, Swap - can the remaining part be used for Caching? such as here "ZFS can use a dedicated drive as write cache, called the ZFS Intent Log (ZIL)."

In theory yes. you could tell the installer to leave some of the space unallocated so you could later manually partition it and use it as a SLOG. But that doesn't make much sense in my opinion. First does a SLOG only cache sync writes and won'T help anything with async writes. If you are running alot of DBs that might make sense but usually most of your writes should be async. And you don't told us what SSD models you got. I guess they are consumer SSDs because all your other hardware is also only consumer grade. Using a SSD as a SLOG will write alot to it and your SSD might die very fast, especially if its not a enterprise SSD designed for such write intense workloads with alot of spare NAND chips and SLC/MLC instead of lesser durable TLC/QLC NAND. Lets say you use the NVMe SSD as a SLOG and it will die within a year. It isn't mirrored so as soon as your NVMe dies you will also loose your root filesystem and everything will stop working. If you want to use a SLOG I would use a dedicated SSD for it, so when the SSD dies (and it will sooner or later) your server can continue running.

Just compare recommended SLOG SSDs (like a Intel Optane SSD like the P4801X) with prosumer/consumer ones (like the Crucial P1 or Samsung 980 Pro):

	Intel P4801X M.2 100GB	Intel P3700 U.2 400GB	Micron 7300 MAX M.2 400GB	Samsung 980 Pro M.2 250GB	Crucial P1 M.2 500GB
Price per TB:	3,774 €	1,515 €	490 €	250 €	104 €
Price per TB TBW:	0.0346€ per TB TBW	0.083€ per TB TBW	0.089€ per TB TBW	0.416€ per TB TBW	0.52€ per TB TBW
TBW per TB:	109,000 TB	18,250 TB	5500 TB	600 TB	200 TB
Powerloss Protection:	yes	yes	yes	no	no
NAND type:	SLC	MLC	TLC	TLC	QLC
Class:	Enterprise	Enterprise	Enterprise	Prosumer	Consumer

Basically the cheaper the SSD is (prive per TB capacity), the faster it will die when writing to it. You pay 37 times the price for a P4801X compared to a Crucial P1 but then it is also rated for 109,000 TB of writes and not just 200 TB, so it is guaranteed to survive 545 times longer. So if you take the durability into account it makes much more sense to buy a really expensive enterprise SSD that will last magnitudes longer.
So you basically don't want to use cheap consumer SSDs for write heavy workloads like a SLOG. And another benefit. More more expensive your SSD, the lower the latency, and the faster your SLOG will be. So a 100GB P4801X for 377€ would be by far the best deal as a SLOG or DB storage drive where sync write IOPS and durability are most important.

You can also have a look at the Proxmox ZFS benchmark paper: https://www.proxmox.com/de/downloads/item/proxmox-ve-zfs-benchmark-2020

There you also see how bad the sync write performance of consumer SSDs (850 Evo and MX100) is compared to enterprise SSDs (the first 6 SSDs). The Seagate Constellation is the only HDD.

 

[Robert5150]

Thanks for the response. What i am mostly going to run are windows vms, exchange servers, client workstations (windows 10) and maybe other related items. I will not be running any Database servers (There will be no Exchange data to speak of).

I don't really need a write cache as much as i need a read cache. Read cache seems to be what fixes performance for me.

This is all consumer hardware So i dont want to trash my nvme drive (seagate ironwolf 512GB). Should i disable the ZIL and/or the ZLOG? Or does the ZIL read/write to RAM ?

Would i all together disable Sync Writes?

I am trying to read through this documentation here: https://www.45drives.com/community/articles/zfs-caching/

Thanks,

Robert

 

[Dunuin]

This is all consumer hardware So i dont want to trash my nvme drive (seagate ironwolf 512GB). Should i disable the ZIL and/or the ZLOG? Or does the ZIL read/write to RAM ?

If you don't use a SLOG (dedicated drive for your ZIL) the ZIL will be part of your normal drives. So everything is then written twice to the normal disks. First to the ZIL + RAM and later a second time from RAM to disk. Using sync writes will always cause alot of write amplification and heavy SSD wear.

Would i all together disable Sync Writes?

You could do that but that really isn't recommended. When you lie to your VM and handle secure sync writes as unsecure async writes (VM cachemode set to "Write back (unsafe)" or ZFS attribute "sync=disabled") and then the kernel will crash or a power outage occures you might loose big amounts of data. You can for example totally screw up a entire filesystem so you loose an entire partition. ZFS is writing async writes every 5 seconds to the disks. So in case you allow sync writes you should only loose the last 0-5 seconds of async data in case of an power outage/hardware failure/kernel crash.
Software developers know that sync writes cause a horrible wear and are horrible slow. If they still decide to use sync writes in their code then only because it is neccessary. So you should let the software developer choose when to use async and when sync writes. And not forbidding/faking sync writes on your own.

 

[Robert5150]

Hmm. Interesting. It sounds more and more like ZFS may not be the correct solution for me. While Hyper-V was pretty slow, it didn't trash hardware.

Based on the hardware info I have provided, what would be the best way to lay out my drives to accomplish this:

1. Not trashing my SSDs, or NVME
2. Decent Performance (primarily read) , Seems like RAM Cache handles that?
3. Redundancy - RAID - All VMs need to be on some sort of RAID Volume.
4. If i use my NVME drive, i can only use 4x SATA drives, If i skip the NVME drive i can use 6x SATA Drives.

I am not in a position to buy enterprise hardware for this project. If Proxmox/ZFS isn't a solution then i will just stick with Hyper-V.

Would it make more sense to just create:

1. 2x SSD -RAID1 - Install ProxMox here, this would be a RAID1 from the motherboard
2. 4x SATA - RAID10 - Use this for my VMs, RAID provided by motherboard
3. Use Ext4 and completely skip all the other file systems? if i did this would i still get RAM Cache benefits?

Promise last question.

Thanks,

Robert