L2ARC, PCi/SSD

[enoch85]

I'm sure this question has been asked before, but I can't decide what would be best. More RAM, and additional L2ARC - and in that case which type?
Proxmox is currently running on a HP DL380 G9 with one PCI-E slot available, so I could get a card that supports NVMe and add around 250 GB fast storage for the L2ARC, or I could get a regular SSD. I figure that PCI-e would be better as it's closer to the moterboard but a bit more hassle add (need to power off the system etc). Also do you need to mirror the L2ARC to aviod corruption when unforseen things happen, like power loss or similar?

I did some testing, and here's the summary of the arc_summary:

ZFS Subsystem Report Thu Nov 24 13:55:34 2022 ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 1.02b Mutex Misses: 388.52k Evict Skips: 388.52k ARC Size: 100.07% 125.97 GiB Target Size: (Adaptive) 100.00% 125.88 GiB Min Size (Hard Limit): 6.25% 7.87 GiB Max Size (High Water): 16:1 125.88 GiB ARC Size Breakdown: Recently Used Cache Size: 93.69% 118.01 GiB Frequently Used Cache Size: 6.31% 7.95 GiB ARC Hash Breakdown: Elements Max: 32.75m Elements Current: 70.10% 22.96m Collisions: 575.46m Chain Max: 11 Chains: 5.05m ARC Total accesses: 3.02b Cache Hit Ratio: 67.93% 2.05b Cache Miss Ratio: 32.07% 967.81m Actual Hit Ratio: 67.69% 2.04b Data Demand Efficiency: 44.46% 840.24m Data Prefetch Efficiency: 1.55% 507.09m CACHE HITS BY CACHE LIST: Most Recently Used: 24.17% 495.50m Most Frequently Used: 75.47% 1.55b Most Recently Used Ghost: 0.15% 3.05m Most Frequently Used Ghost: 0.37% 7.50m CACHE HITS BY DATA TYPE: Demand Data: 18.22% 373.56m Prefetch Data: 0.38% 7.84m Demand Metadata: 81.32% 1.67b Prefetch Metadata: 0.07% 1.45m CACHE MISSES BY DATA TYPE: Demand Data: 48.22% 466.68m Prefetch Data: 51.59% 499.25m Demand Metadata: 0.04% 373.73k Prefetch Metadata: 0.15% 1.50m DMU Prefetch Efficiency: 33.41m Hit Ratio: 25.73% 8.60m Miss Ratio: 74.27% 24.81m


Not new to ZFS, but new to both Proxmox and the use of an L2ARC. What does the above tell us? I'm asking because right now my pool is super slow and I/O intensive. I run around 30 VMs and sometimes it just freezes. FreeNAS never had this issue, but on the other hand I only used that as storage, nothing else.

Any advice is welcome!

 

[Dunuin]

More RAM, and additional L2ARC - and in that case which type?

Usually its recommended to only use L2ARC if you already maxed out your RAM and you still need more read cache. Otherwise buy more RAM first.

Also do you need to mirror the L2ARC to aviod corruption when unforseen things happen, like power loss or similar?

No, L2ARC is just read cache, so redundant data that also exists on your other data disk. No problem to lose your single L2ARC disk.

Not new to ZFS, but new to both Proxmox and the use of an L2ARC. What does the above tell us?

68% ARC hit ratio isn't that great.

In case you are running DBs or something else that does sync writes it could make sense to add a SLOG for faster writes.

Special device SSDs might also be an option to increase read+write performance, especially when using HDDs for data. With them metadata will be stored on the special device SSDs so the data disks are hit by less small IO.

 

[enoch85]

Thanks @Dunuin!

So I've already added a SLOG and will add a second set of disks today. Maybe the single mirror i slowing me down right now? On TrueNAS I never had issues like this, but there I had 8 mirrors plus a SLOG (same SLOG-disks as here) and it was flying! Even on less RAM (had ~90 GB RAM on my TrueNAS.

This are the HDDs on "mainstorage": https://www.toshiba-storage.com/products/toshiba-internal-hard-drives-n300/

NAME STATE READ WRITE CKSUM mainstorage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 wwn-0x5000039bb8d18083 ONLINE 0 0 0 wwn-0x5000039bb8d18080 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 ata-INTEL_SSDSC2BA400G4_BTHV518105PX400NGN ONLINE 0 0 0 ata-INTEL_SSDSC2BA400G4_BTHV519404N8400NGN ONLINE 0 0 0

68% ARC hit ratio isn't that great.

What isn't great? Do you mean there's no need for an L2ARC? Right now I'm steady at around 80% RAM usasge, but I'm sure it would eat more if I added it. Seems like I've hit a hard limit. The G9 can add up to around 1 TB in RAM, so no issues there.

 

[Dunuin]

Thanks @Dunuin!

So I've already added a SLOG and will add a second set of disks today. Maybe the single mirror i slowing me down right now? On TrueNAS I never had issues like this, but there I had 8 mirrors plus a SLOG (same SLOG-disks as here) and it was flying! Even on less RAM (had ~90 GB RAM on my TrueNAS.

This are the HDDs on "mainstorage": https://www.toshiba-storage.com/products/toshiba-internal-hard-drives-n300/

NAME STATE READ WRITE CKSUM mainstorage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 wwn-0x5000039bb8d18083 ONLINE 0 0 0 wwn-0x5000039bb8d18080 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 ata-INTEL_SSDSC2BA400G4_BTHV518105PX400NGN ONLINE 0 0 0 ata-INTEL_SSDSC2BA400G4_BTHV519404N8400NGN ONLINE 0 0 0


What isn't great? Do you mean there's no need for an L2ARC? Right now I'm steady at around 80% RAM usasge, but I'm sure it would eat more if I added it. Seems like I've hit a hard limit. The G9 can add up to around 1 TB in RAM, so no issues there.

My PVE server got a ARC hit ratio of 80-95%. 68% ARC hit ratio means only 2 of 3 reads will be done from RAM. Every third read hits the disks.

By default ZFS on PVE will use UP TO 50% of your hosts total RAM for its ARC. But you could limit or increase that.

And yes, the more mirrors you stripe, the better the performance will be. 4 striped mirrors got 4x the IOPS performance of a single mirror using the same disks. Especially important to stripe as much disks as possible when using HDDs. Because of the terrible IOPS performance of HDDs.

 

[enoch85]

Ok, it's settled. First add more mirrors to the pool, then check again. If still slow, add more RAM.

Thanks again! :+1:

 

[guletz]

Hi,

In your case L2arc will not help. See lines with used ghost on it. This will tell you that l2arc will be with under 1 % hit.

So, invest your money wisely, like @Dunuin @Dunusaid before: special device and/or ram.

Good luck / Bafta !

 

[enoch85]

Just added a second VDEV and changed sync=standard to sync=always to do all writes on the mirrored SLOG. IOPS are double! Both reads and write increased, reads are like 10 times faster.

 

[Dunuin]

sync=standard to sync=always

So you force all fast async writes to be handled as slow sync writes so they will use the SSD instead of the HDDs

 

[enoch85]

So you force all fast async writes to be handled as slow sync writes so they will use the SSD instead of the HDDs

That might be the case, but better safe than sorry if something happens. I also want to offload as much as possible from the HDDs. That's how my old setup was made in TrueNAS, and now it basically flies!

Made my decision based on this: https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/

Btw, I become more and more in love with Proxmox the more I use it. I mean, live data migration on a single node, wow!

 

[alexskysilk]

My 2c worth of experience:

1. l2arc vdevs almost NEVER make sense. Add ram if you have a highly cacheable read workload.
2. logs dont need more then 5 seconds worth of i/o. SO if you have a 10gbit client facing interface, and assuming 100% utilization you'd need 50gbit, or ~6.25Gbyte of slog. They also dont make much of a difference for workloads that arent highly fragmented (eg, db).
3. The vdev that makes the MOST impact for perceived performance is special. BY FAR. 5% special/data ratio is good, but larger allow you to store larger small blocks in the special device too.