[TUTORIAL] HiveStation - Developer Workstation for Proxmox Virtual Environment

Do you have pointers to articles supporting this statement? A small size ARC should not rise problems, except performance wise...?

The practical lower limit is only reached when the ZFS internal data structures do not fit into it anymore. The concept: https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html#adaptive-replacement-cache

----
Edit: an obviously extreme example - on the *BSD side, not OpenZFS: https://wiki.freebsd.org/ZFSTuningGuide#ARC --> "vfs.zfs.arc_max="40M"" :-)

Problem encountered: RAM saturation and ZFS ARC limit exceeded, resulting in a system crash.
Test environment:

Laptop with 16 GB of RAM.
ZFS installation with the ARC limited to 2 GB (zfs_arc_max = 2 GB).
Operating system and running applications consuming approximately 4 GB of RAM.

3 virtual machines running, using approximately 8 GB of RAM in total.

Procedure triggering the incident:
Running an rsync script on the host machine to perform a backup of approximately 20 GB, while the 3 VMs were already running.
Observed result:
Despite the 2 GB limit, the ARC cache exceeded this value during the backup. The available RAM was completely consumed, the system froze (no response), and a forced reboot was required. Upon reboot, corruption was observed.

I am not the only one to have had a similar problem, it seems to me that there is an open bug that resembles this scenario.
 
Problem encountered:
Despite the 2 GB limit, the ARC cache exceeded this value during the backup.
Okay, that's strange. Fortunately I've never encountered such a situation.

(Ram is the only resource which should never be over-committed and there are more RAM consumers than the KVM processes and ARC - but I am sure you know that already.)
 
if you want to have proxmox on your laptop, you should have enough ram, but i see the problem.
(If only i had bought 64GB some time ago ;) )

Practically all my laptops are maxed out in RAM
 
Last edited:
  • Like
Reactions: aureladmin
if you want to have proxmox on your laptop, you should have enough ram, but i see the problem.
(If only i had bought 64GB some time ago ;) )

Practically all my laptops are maxed out in RAM

My main laptop (4 years old) has 32GB of RAM, and it's barely enough. My secondary laptop, for testing, is an old one (8 years old) with 16GB of RAM.
I'd like to add RAM to both, but given current prices, it's not possible.

ZFS is a very good file system for a production server, but for the HIVESTATION project, that's not necessarily the case. Users will have to choose wisely, otherwise, it's going to be a disaster.
 

HiveStation V20 - what's new since V19​


V20 is out (current build V20.5). Archive attached: HIVESTATION-V20-COMPLETE.tar.gz. Existing V15-V19 installs can move up with hivestation-upgrade.sh (no reinstall); a fresh install follows S1 -> S8 as usual. All scripts stay bilingual FR/EN, and the full guide is in the archive.

Restructuring - lib/ + payload/ (V20.0, no behaviour change). The orchestrators S1-S8 are now thin: every file installed on the target ships as a real file in payload/ (no more heredocs buried in scripts), shared code lives in lib/. The runtime interface consumed by the Plymouth theme and the widget (unit names, /usr/local/sbin binaries, state files, marker fields) is frozen and guarded by hivestation-check.sh: full manifest, anti-residue check, contract-coherence check. Same behaviour on the installed machine, much easier to audit and diff.

Encrypted RAID1 now boots with a missing disk. The classic Debian initramfs fails hard on an absent LUKS container, so a degraded encrypted RAID1 used to panic before the root mount. A new local-top hook (hs-raid1-degraded, ordered before cryptroot) detects the explicitly degraded boot, drops the durably-absent containers from the initramfs copy of crypttab, and lets the surviving disk unlock and mount degraded. The installed /etc/crypttab is untouched; a later boot with both disks reopens everything. The upgrade retrofits the hook onto existing encrypted RAID1 installs.

HiveStation Recovery family in the GRUB menu. Two adjacent recovery paths: the grub-btrfs snapshots submenu, now titled "HiveStation Recovery - Snapshots", and a "HiveStation Recovery - Proven Kernel" entry pinned to a known-good kernel. A small service counts validated boots and accumulated uptime per kernel version (2 validated boots or 6 h by default) and tags the version known-good; the kernel pruning never purges the tagged kernel.

TPM2 auto-unlock for LUKS2 (opt-in, EXPERIMENTAL). A post-install tool hivestation-tpm2.sh (status / bind / unbind) binds the LUKS2 containers to the TPM via clevis, so the disks unlock at boot without the passphrase; the passphrase slot is never removed and remains the fallback. Nothing is enrolled at install time - you run bind yourself. The tool warns about the evil-maid limitation when Secure Boot is off. Validated on a vTPM only; the encryption keeps its EXPERIMENTAL flag.

Monitoring widget v15 - least-privilege root broker. The widget no longer receives per-command NOPASSWD sudo grants (16 rules, several with loose wildcards). Every privileged read now goes through one root broker (hivestation-monitor-helper): per-verb allowlist, regex-validated arguments, a single sudoers rule, and an anchored pve-read verb instead of www-data group membership. S8 also enables the extension automatically for the target user, the Proxmox version line follows pve-manager point releases, and LUKS root devices resolve to readable /dev/mapper names.

Safer automatic disk replacement (RAID1). At boot the btrfs-auto-replace service now only detects and alerts, in every mode - it never formats anything unattended. The interactive replacement path only accepts a blank disk as candidate and asks for an explicit confirmation (type the disk name) before partitioning.

Themes. GRUB theme v2V2 and Plymouth theme v2V9: the logo now sits on a pure black background in both (a residual grey plate baked in the image had survived the V19 all-black pass), and the GRUB theme installer carries HiveStation naming end to end. The upgrade now refreshes the GRUB theme too, alongside Plymouth and the widget.

Upgrade and sizing. hivestation-upgrade.sh covers V15-V19 -> V20: target version read from the archive itself, marker VERSION= follows the release, inventory-driven retrofits (recovery entries, kernel pruning, TPM2 tool, degraded-boot hook, themes, widget), pre-upgrade restore point, --dry-run, --rollback. Note for LVM+EXT4 installs: use a disk of 64 GB or more (the root LV is sized at 25% of the volume group).

Housekeeping. Kernel command line deduplicated (parameters were written in both GRUB variables); the Proxmox enterprise repository is now disabled (Enabled: false) instead of deleted, easy to re-enable for subscribers; auto-nat-boot no longer sleeps a fixed 10 s at boot; the whole flow was re-validated end to end on NVMe devices (/dev/nvme0n1 naming, no hardcoded device names).

Full release notes are in section 8.1 of the guide; the chronological development journal is in DEVLOG-V20.md at the root of the archive. Feedback welcome.
 

Attachments