When it comes to the optimal disk setup for this purpose I am unsure however. I was considering to go with 4x 4TB NAS HDDs, one nvme ssd for VMs, another nvme for read cache and the OS installed on the internal flash.
Is the internal flash a sd (mmc) card? Then you won't be able to install ProxmoxVE with the regular iso image see
https://forum.proxmox.com/threads/unable-to-get-device-for-partition-1-on-device-dev-mmcblk0.42348/
In theory you could first install Debian stable and afterwards ProxmoxVE ( see
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie for details) but unless you have a good reason I wouldn't do this. There is a good reason, why instaalling PVE to sd cards is not supported: Since the ProxmoxVE os does a lot of writes (logging and metric data plus the cluster file system database (which is also used on non-cluster single-nodes)) sd cards will get trashed soon. So: Don't do this.
Regarding nvme ssds: Usual consumer SSDs without power-loss-protection (plp) (which is true for most M2 2880 ssds) suffer the same issue as sd cards but are still more relieable than them and thanks to ZFS mirror you should be able to replace one of them in case of an failure. Alternatively get the cheapest plp-ssds (
https://geizhals.at/?cat=hdssd&xf=7... (PCIe)&offset=0&promode=true&hloc=at&hloc=de ). You could also use the boot device for VM/lxcs if you can live with the higher wearout and that OS and vms/lxcs share the same disc. In my homelab I did this since my mini-pcs just have two storage slots (one nvme, one sata) and I want to have some redundancy. If this means that in case of an reinstallation I have to restore the vms/lxcs from my backup so be it.
So in your case I would use the NVME slots for the OS. I would recommend to get two NVME sds from different brands (to minimice the chance that two will fail at the same time) if you can't afford ssds with PLP and create a ZFS mirror on them.
For the internal storage I personally wouldn't go with four hdds, but two used sata server-ssds with PLP (they are quite affordable and still have larger endurance than consumer-ssds) and two hdds with the largest capacity you can afford. Afterwards setup a mirror on the ssds for vm/lxc storage and another mirror on the hdds for raw data storage.
Alternatively you could create a mirror on the hdds and add the ssds as a special device mirror. The idea is to use the special device to store the metadata of all files and additionally very small files. By creating a custom dataset for vms/lxcs you could even use the ssds as vm/lxc storage while bulk data is still saved on the hdds. See this post by LnxBil who explains this better than me:
https://forum.proxmox.com/threads/zfs-metadata-special-device.129031/post-699290
If this sounds a little bit to eloborate to be comfortable with, stick with two ssds for VM/LXC storage and two hdds for raw data it will propably still be fine for a home server.
Regarding the read cache: How do you want to configure it ? This needs to be configured on the os level, PVE doesn't do this by default. You can do this with ZFS (
https://pve.proxmox.com/wiki/ZFS_on_Linux ) but usually it's not worth it ( having more RAM helps more). A special device on the other hand can be quite useful.
See also following old forum threads:
-
https://forum.proxmox.com/threads/zfs-ssd-pool-with-nvme-cache-zil-l2arc.45920/
-
https://forum.proxmox.com/threads/l2arc-cache-with-or-w-o-swap-on-zfs.121156/
-
https://forum.proxmox.com/threads/brauche-ich-slog-l2arc.134928/ (German, use something like deepl for translation)
Regarding write cache I have no idea what you mean, can you explain?
For backups I would recommend not to use file sharing via sshfs but instead a dedicated backup tool like kopia, restic, borg or duplicati. They all can be configured to work with a linux server ( so in case of your homeserver propably a dedicated lxc or vm would be the best option) and some of them even have a GUI, so things don't get to complicated for non-techy users.
Regarding your apps: How do you want to run email? I hope you don't plan to self-host an mail server because you will need to be quite careful to not be added to anti-spam blacklists or get abused by spammers or both. Hosting some webmail interface to an external mail provider should work though.
If you just want to host some apps please also consider to use some NAS OS with Apps (aka docker container) and VM support instead like OpenMediaVault, TrueNAS or uNRAID. OpenMediaVault could even be installed to a sd card since it has a plugin to reduce writes and extend the durability. unRAID can also be installed to SD cards as far I know. TrueNAS however doesn't support this (like PVE) though.
Imho if your homeserver main purpose is to provide network storage plus some self-hosted apps a NAS OS is way less complex and has a more novice-friendly learning curve than ProxmoxVE.
