For the last few years I've been running a nixos server on zfs that works both as hypervisor (sometimes, usually just containers) and a zfs storage.
Back then, I evaluated TrueNAS as a very lightweight hypervisor (didn't work too well as it was very new on linux) as well as proxmox but I couldn't find a reasonable way how to set it up. And so I ended up with a nixos server that is both services and storage.
Now my situation has changed and I'd like to re-evaluate if I can't somehow make it work with proxmox. It's important to me that there are no artificial bottlenecks, which is why I would ideally like to keep the storage close to the compute. A 10G link may be relatively fast, but it's still significantly slower than an nvme drive locally.
Currently the server has a mirrored nvme pool to boot off and where my dev containers sit. It's small, maybe 1TB or so, but fast. And a second spinning rust pool with 4×16TB drives in 2 mirrored vdevs.
The nvme pool gets snapshotted and migrated to the rust pool every few minutes. Download services use the nvme pool as scratch space iirc and they then move files to the rust pool for long term storage.
For the new setup, I'd like to use zfs as the storage tech - I've made good experiences with it and it's got various tools to improve perf plus I am fairly comfortable with the tooling by now. I'd definitely like to manage it through TrueNAS.
And I don't really see how to set this up in the most reasonable way. The NAS filer has a pretty complex hardware requirement with nvme's for fast tiers, many disks that shouldn't be virtualized, because zfs should be able to understand when the hw is flaky, etc.
If I want to run the filer as a vm, I'm getting a chicken and egg problem because the filer is a vm but it's also what should provide the storage through iSCSI(?) to proxmox.
OTOH, if I were to create a 2nd machine just for the hypervisor and keep the storage separate, then I end up in a world where perf is hard capped at network speed and that's far lower than the nvme drives actually can drive.
Any ideas how to set it up sensibly? Has anyone else here experience with a hyper converged setup that also can run a nas filer appliance?
Back then, I evaluated TrueNAS as a very lightweight hypervisor (didn't work too well as it was very new on linux) as well as proxmox but I couldn't find a reasonable way how to set it up. And so I ended up with a nixos server that is both services and storage.
Now my situation has changed and I'd like to re-evaluate if I can't somehow make it work with proxmox. It's important to me that there are no artificial bottlenecks, which is why I would ideally like to keep the storage close to the compute. A 10G link may be relatively fast, but it's still significantly slower than an nvme drive locally.
Currently the server has a mirrored nvme pool to boot off and where my dev containers sit. It's small, maybe 1TB or so, but fast. And a second spinning rust pool with 4×16TB drives in 2 mirrored vdevs.
The nvme pool gets snapshotted and migrated to the rust pool every few minutes. Download services use the nvme pool as scratch space iirc and they then move files to the rust pool for long term storage.
For the new setup, I'd like to use zfs as the storage tech - I've made good experiences with it and it's got various tools to improve perf plus I am fairly comfortable with the tooling by now. I'd definitely like to manage it through TrueNAS.
And I don't really see how to set this up in the most reasonable way. The NAS filer has a pretty complex hardware requirement with nvme's for fast tiers, many disks that shouldn't be virtualized, because zfs should be able to understand when the hw is flaky, etc.
If I want to run the filer as a vm, I'm getting a chicken and egg problem because the filer is a vm but it's also what should provide the storage through iSCSI(?) to proxmox.
OTOH, if I were to create a 2nd machine just for the hypervisor and keep the storage separate, then I end up in a world where perf is hard capped at network speed and that's far lower than the nvme drives actually can drive.
Any ideas how to set it up sensibly? Has anyone else here experience with a hyper converged setup that also can run a nas filer appliance?