NAS VM on Proxmox - cache advice?

[jordanconway]

I'm planning out a new build - proxmox with a NAS OS as a VM. My current plan has 2x 1tb sata ssds and 2x 1tb nvmes and several 10tb hdds passed through to the NAS VM.
I'd like to pass the ssds or nvmes as a cache pool in the NAS VM. I'd be running more vms on proxmox and am wondering which scenario makes the most sense

1. Proxmox installed on raid1 sata ssds partitioned to also be vm storage using the nvmes as a raid1 cache for the NAS VM.
2. Proxmox installed on raid1 nvmes partitioned to also be vm storage using the sata ssds as a raid1 cache for the NAS VM

Any advice would be welcome - thanks!

 

[gea]

As you (should) use ZFS for OS and data, there is only a readcache option (L2Arc). It does not help on write and is not needed at all with enough RAM. What you can do is to use an NVMe mirror as special vdev to hold metadata, small files or all files of a filesystem. You can define this vie recsize and special small blocksize setting per filesytem. With the ZFS rewrite feature you can move data between hd and NVMe. Up from next OenZFS 2.4 you can even use the special vdev as slog.

You should separate OS and data, buy a small OS disk (SSD od M.2)

If you want a good VM performance you should place VM disks on zvols on a local Proxmox pool, much faster with lower latency compared to network shares on a storage VM. You can use Proxmox as NAS with a single ZFS pool. With a storage VM you can use a local NVMe mirror for VMs and hd+ssd special vdev for data but must provide enough RAM for 2 x ZFS on 2 x OS. Alternative is too just enable samba + acl in Proxmox. If you need a web-gui for advanced ZFS, acl and share management, look at Cockpit/Poolsman or the copy and run multi-os/multi-server napp-it cs web-gui.