Feedback for PVE setup for large Nextcloud installation (with other services)

[srfftc]

Hi there,

Current Situation:

We have a Nextcloud install with about 1000 users (but many are inactive). Everything is currently running on a Debian bare metal server, with caddy as a reverse proxy and Nextcloud (and other services) in Docker behind. Our current server (4 cores, 32 GB RAM, no SSDs) is aching under load, and we are therefore planning to upgrade.

Upgrade plan:

• Proxmox VE Host with 16 Cores and 128 GB ECC RAM
• ZFS Pools / Datasets:
  • 2 TB NVMe RAID1
    • Unencrypted PVE root
    • Encrypted Datasets for containers and everything except Nextcloud Files
  • 16 TB HDD RAID1
    • encrypted datasets for Nextcloud Files and other Blob Storage
• VMs: Firewall / SNI Proxy for directly Internet-facing traffic
• LXC Containers with Docker for our services (Mattermost, Collabora, Nextcloud, etc.)
• Backup: ZFS snapshots encrypted with restic and stored on Hetzner Storage box(es)

Explanation for our considerations:

• Docker within LXC: We have some experience with Docker / Docker-compose, and our current setup is heavily reliant on Docker. We didn’t want to change everything at once, so we are planning to keep Docker for now. We know that this "Containerception" (container within container) is probably not the ideal solution.
• LXC Containers because we want to run backups / snapshots via ZFS/restic from the PVE host (or within its own LXC Container) but still have Proxmox management capabilities (Resource Limitation, restart via GUI, Backup etc.).
• VM for Firewall / SNI Proxy / Reverse Proxy for better security and service isolation for Internet-facing services
• ZFS and Encryption:
  In our current setup, we have completely encrypted our HDDs with LUKS and are running ZFS on top.
  But we need to log in via the server console to unlock the encrypted disks.
  Since that is not very comfortable, we are planning to encrypt as follows:
  • PVE Root unencrypted (so we can access the Web UI / SSH PVE after startup)
  • Datasets for containers and VMs are encrypted with ZFS encryption and will have to be manually unlocked after startup

Pictures / Diagrams (some text in German):

• Proxmox host/VMs
• Storage view

Questions:

• Anybody running a similar setup, experience with PVE/LXC/Docker?
• Especially Nextcloud in Docker in LXC on ZFS?
• We would like to run SNI Proxy (via HAProxy) behind (or in?) Firewall so that we can have TLS Termination within service containers. Any recommendations or experiences to share with SNI Proxies?
• We have decided to not use VMs for our services (Nextcloud, Mattermost, etc.) but instead are going with LXC containers for our “core” services so that we can run ZFS snapshots / Backup from Host with ZFS. Anybody using Nextcloud and other services inside a VM with a good Snapshot / Backup Strategy?
• Are there any considerations regarding ZFS and Encryption? Is there maybe a better solution? What drawbacks (security-wise) do we have with an unencrypted PVE root partition?

Generally, any feedback is appreciated!