I use spinning drives(HDDs) in a RAID 6 configuration using LVM thin for my VM data storage. I'm hoping to be able to add an NVME read cache(bonus if i can add a write cache). I don't see any recent posts on this topic. Has anyone done this successfully? Any recommendations?
NVME LVMcache w/ LVM Thin?
- Thread starter dawydiuk
- Start date
You say that you're using LVM-Thin, so I'm assuming you're using a hardware RAID? Then you'd need to check what your hardware RAID controller is capable of and whether it can add an explicit cache.
If you can or want to use ZFS, is there an option to use “special devices” and also a log/cache on an NVMe/SSD? That's what I do here with my HDDs, and it works very well.
If you can or want to use ZFS, is there an option to use “special devices” and also a log/cache on an NVMe/SSD? That's what I do here with my HDDs, and it works very well.
There are some myths around SLOG, L2ARC and special device(s) in ZFS which always re-appear.
1) SLOG
A SLOG is a separate storage for the ZFS Intent Log (ZIL). Synchronous writes are protocolized and written back in the ZFS pool later. For asynchronous writes (typical VM workload) SLOG doesn’t bring anything.
2) L2ARC
L2ARC is an extension of the ARC (RAM cache), not a cache in front of the storage pool. Data is only read from the L2ARC after an ARC miss. It is useful when the working set is larger than the available RAM but still fits on fast SSD/NVMe storage. If your workload already fits into RAM, L2ARC provides little or no benefit.
3) Special Devices
A special device is not a cache. It is a permanent part of the ZFS pool used to store metadata and, optionally, small data blocks (special_small_blocks). This can significantly improve random I/O performance on HDD-based pools. Since it stores live pool data, it must provide the same level of redundancy as the rest of the pool (mirror for example).
If your storage is based on a hardware RAID controller you should check (as mentioned above) if it supports additional drives as caching drives.
1) SLOG
A SLOG is a separate storage for the ZFS Intent Log (ZIL). Synchronous writes are protocolized and written back in the ZFS pool later. For asynchronous writes (typical VM workload) SLOG doesn’t bring anything.
2) L2ARC
L2ARC is an extension of the ARC (RAM cache), not a cache in front of the storage pool. Data is only read from the L2ARC after an ARC miss. It is useful when the working set is larger than the available RAM but still fits on fast SSD/NVMe storage. If your workload already fits into RAM, L2ARC provides little or no benefit.
3) Special Devices
A special device is not a cache. It is a permanent part of the ZFS pool used to store metadata and, optionally, small data blocks (special_small_blocks). This can significantly improve random I/O performance on HDD-based pools. Since it stores live pool data, it must provide the same level of redundancy as the rest of the pool (mirror for example).
If your storage is based on a hardware RAID controller you should check (as mentioned above) if it supports additional drives as caching drives.
Thank you, yes I have a RAID controller for the RAID 6 array. I thought proxmox might be able to utilitize an NVME for caching but sounds like I'm at the wrong layer my raid controller would utilize it. Thanks for pointing me in the right direction!
Beginning with ZFS 2.4.0 special devices (https://github.com/openzfs/zfs/pull/17505 ) will use a special device also for the ZIL, thus you won't need a dedicated SLOG device then (but can still add it if you consider it useful for your usecase). Beginning with ProxmoxVE 9.2 and ProxmoxBackupServer 4.2 ZFS 4.2 is included and can use this feature. It still won't bring anything for asynchronous writes but if you have workloads who might profit from a log device and don't want to add another ssd for it (or lack the needed storage slot ) this might be useful.