Storage setup? Separate OS and data drive? ZFS + Slog?

[MightyMauz]

I am setting up a new proxmox node that will run as all-in-one. It will have

• database (SQL-server)
• vms / lxcs
• self-hosted AI (ollama + Blackwell 4000)

I have a mini-itx board with 1 pcie that will be populated by the GPU. It also has 2 m.2 slots that I will use for storage. What would give me the best performance? Considering that I have all kinds of drives in my inventory, I have options.
I have consumer-grade nvme drives (your regular samsungs), I have optane drives (the 16GB, 58GB and even 375GB)

To make the best use of the 2 slots, I am considering the following setups:

• os drive + data drive

16gb as OS, 2 TB nvme as data​

Keeping it separate, so that OS writes don't interfere with vm/lxc writes​

• ZFS with slog

Creating a pool with 2TB nvme + 58GB optane as slog​

Taking an already speedy drive, and supercharge it with an optane drive to minimize the impact of sync writes fo my databases​

I am leaning towards the ZFS with slog. But what would you recommend?

I am not worried about backups or redundancy, as I am running Proxmox Backup Server.

 

[news]

you need minimum 2 Devices for zfs with one vdev as mirror with flash memory.
but normal nvme m.2 don't work with zfs.
You get only about 40 - 80 MByte/s 4k random read/ write.

the nvme m.2 need DRAM and PLP, and you also need a USV.

So get a better mainboard with 6x - 8x SATA III and setup a real diy home server.

You can find my choose, the BIOSTAR B760MX2-E PRO D4 here:
See: https://forum.proxmox.com/threads/iommu-groups-for-specific-b760m-motherboard.186582/#post-870447

 

[MightyMauz]

In other words: what you have sucks, get what I have...

Back to making due with what I have. I have 2 m.2 slots, how can I get the best performance out of that?

 

[UdoB]

I have 2 m.2 slots, how can I get the best performance out of that?

That's easy: get two "Enterprise class" NVMe with PLP. (But prepare for a heart attack while looking up prices.)

Then run everything on this single mirrored vdev - on ZFS of course. You asked for "best performance" - and personally I do always recommend ZFS ;-)

And I mean it - as I would not build a new system without any reliable kind of redundancy. PLUS some levels of backups, following the 3-2-1-1-0 scheme. The only exception of both redundancy and multi-level backup is in a "lab" where the actual data has zero value...

Disclaimer: this is my personal recommendation - obviously heavily opinionated

 

[leesteken]

The 16GB Optane might be useful as a SLOG or a special device (but the redundancy must at least match the data redundancy).

I fear that the 58GB and especially the 375GB might be hybrid drives with 2 PCIe lanes connected to a small Optane and the other two lanes to a "normal" SSD, which AFAIK is not accessible with Linux and/or anything but certain Intel motherboards. The Optane part works as a (write) cache for the SSD using some form of Intel BIOS/software/fake RAID. Then again, I don't know enough to be sure, just enough to be worried, but maybe you already looked into this and have the right Intel motherboard chipset(s) and/or can correct me?

Optane was unfortunately discontinued and very expensive to get here (and unclear whether it works with Linux and/or on an AMD platform), otherwise a 16GB mirror would be perfect for swap and/or SLOG.

 

[alexskysilk]

Creating a pool with 2TB nvme + 58GB optane as slog

the slog will make absolutely no difference for this use case. it will, however, create a second spof point making your store more fragile.

 

[MightyMauz]

I fear that the 58GB and especially the 375GB might be hybrid drives with 2 PCIe lanes connected to a small Optane and the other two lanes to a "normal" SSD, which AFAIK is not accessible with Linux and/or anything but certain Intel motherboards. The Optane part works as a (write) cache for the SSD using some form of Intel BIOS/software/fake RAID. Then again, I don't know enough to be sure, just enough to be worried, but maybe you already looked into this and have the right Intel motherboard chipset(s) and/or can correct me?

They are all proper Optane drives. No hybrid nonsense. I own several, but I have not found the 16gb useful for anything else than as a dedicated Proxmox OS drive, due to its limited size. That's also why I proposed the 58gb drive as slog, and not the 16gb.

I know consumer drives are not great for random writes, but was hoping that combining it with an Optane it would limit the impact.

 

[MightyMauz]

That's easy: get two "Enterprise class" NVMe with PLP. (But prepare for a heart attack while looking up prices.)

Then run everything on this single mirrored vdev - on ZFS of course. You asked for "best performance" - and personally I do always recommend ZFS ;-)

And I mean it - as I would not build a new system without any reliable kind of redundancy. PLUS some levels of backups, following the 3-2-1-1-0 scheme. The only exception of both redundancy and multi-level backup is in a "lab" where the actual data has zero value...

Disclaimer: this is my personal recommendation - obviously heavily opinionated

It's a home lab and I already have messed up many times. PBS has been serving me great and anything lost in between backups I am willing to risk.

Performance wise, do you prefer a mirrored 2TB over 2TB with SLOG? Because it was my understanding that an Optane aiding a consumer drive will have its merits, especially for database and zvol writes

 

[leesteken]

They are all proper Optane drives. No hybrid nonsense. I own several, but I have not found the 16gb useful for anything else than as a dedicated Proxmox OS drive, due to its limited size. That's also why I proposed the 58gb drive as slog, and not the 16gb.

Good to hear, thanks for explaining this. 16G is good enough for a Proxmox installation or an SLOG or swap (since Proxmox does not support memory over-commit). Although people are not overly enthusiastic about adding an SLOG for getting better (sync write) performance. Using the 58G as a special device (which is often recommended instead of an SLOG) might work wonders. The 16G might be too small and the 375G is probably too big.

I know consumer drives are not great for random writes, but was hoping that combining it with an Optane it would limit the impact.

That's exactly what the hybrid drives do, so it might very well work. It's not about random writes but syncronous writes (which must hit the drive before anything can continue). If you can safely (BBU or large condensator) cache those, you can get better fsync/metadata performance. That's what PLP is for and excels at.

Make sure you have enough Optane drives to run a n-way mirror (for best read IOPS) to match the redundancy of (the rest of) the data. If you lose a special device, you lose everything. If you lose SLOG you lose some of the last important metadata (and the associated data).

Performance wise, do you prefer a mirrored 2TB over 2TB with SLOG? Because it was my understanding that an Optane aiding a consumer drive will have its merits, especially for database and zvol writes

A mirror does not add write performance (or endurance) and neither does RAIDz1/2/3. If you have VMs that require high write endurance (possibly using different storages for different virtual disks), you want to use Optane for those (but be sure to have enough drives for the redundancy you want/need).

 

[MightyMauz]

Hmm, I was under the impression that a special vdev wasn't particularly useful, as most data is in zvols anyway. But maybe I am wrong.

Why is ZFS so difficult? Now I get all crazy frankenstein setups in my head. Maybe divvy up the 58gb as 50% metadata and 50% slog?

 

[leesteken]

Hmm, I was under the impression that a special vdev wasn't particularly useful, as most data is in zvols anyway.

ZVOLs also use CoW (like everything else on ZFS, which also makes it relatively slow) and new metadata especially never overwrites old metadata. It's always written separately and only when it's actually on the drive (synchronously) then the old metadata is removed. That's why PLP and other sync write caching (like hardware RAID with BBU, but don't mix that with ZFS) shines. Fast metadata (like a SSD special device for a HDD storage) also makes other operations faster like listing directories.

Why is ZFS so difficult?

It's not one size fits all and ZFS has many tunable parameters. Simple storage also exists like LVM and you could get redundancy with multiple PVE nodes.

Now I get all crazy frankenstein setups in my head. Maybe divvy up the 58gb as 50% metadata and 50% slog?

Your regular data typically does not need sync writes and would therefore not benefit much of a SLOG but a special device benefits ZVOLs as much as directory (and other metadata) operations on SUBVOLs. All of this is not Proxmox specific and other interesting ZFS guides are available on the internet. and much has already been writting about ZFS on this forum as well.

I'm trying BTRFS at the momen, which is in principle a lot like ZFS in some ways but operational appearantly very different (and currently a technology preview only)...