[TUTORIAL] HiveStation - Developer Workstation for Proxmox Virtual Environment

Do you have pointers to articles supporting this statement? A small size ARC should not rise problems, except performance wise...?

The practical lower limit is only reached when the ZFS internal data structures do not fit into it anymore. The concept: https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html#adaptive-replacement-cache

----
Edit: an obviously extreme example - on the *BSD side, not OpenZFS: https://wiki.freebsd.org/ZFSTuningGuide#ARC --> "vfs.zfs.arc_max="40M"" :-)

Problem encountered: RAM saturation and ZFS ARC limit exceeded, resulting in a system crash.
Test environment:

Laptop with 16 GB of RAM.
ZFS installation with the ARC limited to 2 GB (zfs_arc_max = 2 GB).
Operating system and running applications consuming approximately 4 GB of RAM.

3 virtual machines running, using approximately 8 GB of RAM in total.

Procedure triggering the incident:
Running an rsync script on the host machine to perform a backup of approximately 20 GB, while the 3 VMs were already running.
Observed result:
Despite the 2 GB limit, the ARC cache exceeded this value during the backup. The available RAM was completely consumed, the system froze (no response), and a forced reboot was required. Upon reboot, corruption was observed.

I am not the only one to have had a similar problem, it seems to me that there is an open bug that resembles this scenario.
 
Problem encountered:
Despite the 2 GB limit, the ARC cache exceeded this value during the backup.
Okay, that's strange. Fortunately I've never encountered such a situation.

(Ram is the only resource which should never be over-committed and there are more RAM consumers than the KVM processes and ARC - but I am sure you know that already.)
 
if you want to have proxmox on your laptop, you should have enough ram, but i see the problem.
(If only i had bought 64GB some time ago ;) )

Practically all my laptops are maxed out in RAM
 
Last edited:
  • Like
Reactions: aureladmin
if you want to have proxmox on your laptop, you should have enough ram, but i see the problem.
(If only i had bought 64GB some time ago ;) )

Practically all my laptops are maxed out in RAM

My main laptop (4 years old) has 32GB of RAM, and it's barely enough. My secondary laptop, for testing, is an old one (8 years old) with 16GB of RAM.
I'd like to add RAM to both, but given current prices, it's not possible.

ZFS is a very good file system for a production server, but for the HIVESTATION project, that's not necessarily the case. Users will have to choose wisely, otherwise, it's going to be a disaster.
 

HiveStation V20 - what's new since V19​


V20 is out (current build V20.5). Archive attached: HIVESTATION-V20-COMPLETE.tar.gz. Existing V15-V19 installs can move up with hivestation-upgrade.sh (no reinstall); a fresh install follows S1 -> S8 as usual. All scripts stay bilingual FR/EN, and the full guide is in the archive.

Restructuring - lib/ + payload/ (V20.0, no behaviour change). The orchestrators S1-S8 are now thin: every file installed on the target ships as a real file in payload/ (no more heredocs buried in scripts), shared code lives in lib/. The runtime interface consumed by the Plymouth theme and the widget (unit names, /usr/local/sbin binaries, state files, marker fields) is frozen and guarded by hivestation-check.sh: full manifest, anti-residue check, contract-coherence check. Same behaviour on the installed machine, much easier to audit and diff.

Encrypted RAID1 now boots with a missing disk. The classic Debian initramfs fails hard on an absent LUKS container, so a degraded encrypted RAID1 used to panic before the root mount. A new local-top hook (hs-raid1-degraded, ordered before cryptroot) detects the explicitly degraded boot, drops the durably-absent containers from the initramfs copy of crypttab, and lets the surviving disk unlock and mount degraded. The installed /etc/crypttab is untouched; a later boot with both disks reopens everything. The upgrade retrofits the hook onto existing encrypted RAID1 installs.

HiveStation Recovery family in the GRUB menu. Two adjacent recovery paths: the grub-btrfs snapshots submenu, now titled "HiveStation Recovery - Snapshots", and a "HiveStation Recovery - Proven Kernel" entry pinned to a known-good kernel. A small service counts validated boots and accumulated uptime per kernel version (2 validated boots or 6 h by default) and tags the version known-good; the kernel pruning never purges the tagged kernel.

TPM2 auto-unlock for LUKS2 (opt-in, EXPERIMENTAL). A post-install tool hivestation-tpm2.sh (status / bind / unbind) binds the LUKS2 containers to the TPM via clevis, so the disks unlock at boot without the passphrase; the passphrase slot is never removed and remains the fallback. Nothing is enrolled at install time - you run bind yourself. The tool warns about the evil-maid limitation when Secure Boot is off. Validated on a vTPM only; the encryption keeps its EXPERIMENTAL flag.

Monitoring widget v15 - least-privilege root broker. The widget no longer receives per-command NOPASSWD sudo grants (16 rules, several with loose wildcards). Every privileged read now goes through one root broker (hivestation-monitor-helper): per-verb allowlist, regex-validated arguments, a single sudoers rule, and an anchored pve-read verb instead of www-data group membership. S8 also enables the extension automatically for the target user, the Proxmox version line follows pve-manager point releases, and LUKS root devices resolve to readable /dev/mapper names.

Safer automatic disk replacement (RAID1). At boot the btrfs-auto-replace service now only detects and alerts, in every mode - it never formats anything unattended. The interactive replacement path only accepts a blank disk as candidate and asks for an explicit confirmation (type the disk name) before partitioning.

Themes. GRUB theme v2V2 and Plymouth theme v2V9: the logo now sits on a pure black background in both (a residual grey plate baked in the image had survived the V19 all-black pass), and the GRUB theme installer carries HiveStation naming end to end. The upgrade now refreshes the GRUB theme too, alongside Plymouth and the widget.

Upgrade and sizing. hivestation-upgrade.sh covers V15-V19 -> V20: target version read from the archive itself, marker VERSION= follows the release, inventory-driven retrofits (recovery entries, kernel pruning, TPM2 tool, degraded-boot hook, themes, widget), pre-upgrade restore point, --dry-run, --rollback. Note for LVM+EXT4 installs: use a disk of 64 GB or more (the root LV is sized at 25% of the volume group).

Housekeeping. Kernel command line deduplicated (parameters were written in both GRUB variables); the Proxmox enterprise repository is now disabled (Enabled: false) instead of deleted, easy to re-enable for subscribers; auto-nat-boot no longer sleeps a fixed 10 s at boot; the whole flow was re-validated end to end on NVMe devices (/dev/nvme0n1 naming, no hardcoded device names).

Full release notes are in section 8.1 of the guide; the chronological development journal is in DEVLOG-V20.md at the root of the archive. Feedback welcome.
 

Attachments

Do you accept contributions? I may want to add pre-seeding and configuration based on MB S/N (from dmidecode, etc.) for use as the FAI post-install script.
 
Do you accept contributions? I may want to add pre-seeding and configuration based on MB S/N (from dmidecode, etc.) for use as the FAI post-install script.
I accept all contributions. For your information, setting up an installation via a pre-seed script is already planned for version 23. I am currently stabilizing version 21 and am also working on version 22.
 

HiveStation V21 - what's new since V20​


V21 is out (current build V21.3.1). Archive attached: HIVESTATION-V21-COMPLETE.tar.gz. Existing V15-V20 installs can move up with hivestation-upgrade.sh (no reinstall); a fresh install follows S1 -> S8 as usual. All scripts stay bilingual FR/EN, and the full guide is in the archive.

ZFS as a first-class filesystem. Two new disk modes at S1 launch: ZFS single and ZFS mirror, at functional parity with the Btrfs stack (the Btrfs and LVM code paths are untouched). The boot chain is themed GRUB on the ESP chaining into ZFSBootMenu, which kexecs the Boot Environment - GRUB never probes the pool, and Boot Environments, snapshots and kernels are selectable at boot. The dataset layout isolates home, logs, caches and the PVE state from the rollbackable root; S1 asks for the ARC cap (default 2 GiB). S3.2 defaults to the native zfspool storage local-zfs (VM and CT snapshots out of the box); S5 installs sanoid timelines plus an APT hook that snapshots the Boot Environment before every operation - each snapshot bootable from the ZFSBootMenu menu - and a ZFS health monitor that publishes the same state file the Plymouth theme and the widget already read.

Native ZFS encryption - hybrid design. Opt-in in both ZFS modes: aes-256-gcm on root/home/var with the kernels inside the encrypted pool, and the swap as a LUKS2 partition outside the pool (a swap zvol is never used). One passphrase, typed once at a themed ZFSBootMenu prompt with your keyboard layout (a self-contained ckbcomp keymap loaded before the prompt - FR and DE layouts proven on real hardware); the booted system then loads its keys without a prompt. Use ASCII-only passphrases in ZFS modes: the initramfs LUKS prompt of the Btrfs modes accepts accented characters, the ZFSBootMenu prompt does not (tested on real hardware). Hibernation (persistent encrypted swap) works on the test bench but stays EXPERIMENTAL: some firmwares reboot on the default ACPI entry before the image is written, so the installer pins HibernateMode=shutdown.

ZFS mirror HA. GRUB is installed on each ESP (either disk boots alone) and the ZFSBootMenu images are replicated by efi-sync; a zed hooklet and a desktop alert signal degradation; a missing disk means a native degraded boot straight to the login. Disk replacement is a guided, admin-triggered run - never an unattended format; a replacement disk of the same model as the survivor is accepted (an overshooting size check rejected identical disks, found and fixed on real hardware). Boot-time missing-disk detection now goes through one dispatcher unit for both redundant modes (hivestation-auto-replace.service); the historical btrfs-auto-replace.service alias was removed in 21.3.0.

Real-hardware validation - EXPERIMENTAL flags dropped. The series was validated end to end on real hardware (UEFI firmware, hardware TPM 2.0): all four ZFS modes, ZFSBootMenu unlock with FR and DE layouts, degraded boot with a disk physically removed, automatic reintegration and guided replacement up to a reboot on the rebuilt disk, TPM2 auto-unlock, hibernate/resume on the encrypted mirror, a full encrypted Btrfs RAID1 install under active Secure Boot (signed PVE kernel chain, accented passphrase at the initramfs prompt), and an upgrade of a published V20.5 install to the current series. As a result, LUKS2 encryption (Btrfs modes), native ZFS encryption and the TPM2 auto-unlock drop their EXPERIMENTAL flags; the LVM+LUKS variant stays flagged (not yet validated on real hardware) and hibernation keeps its flag. ZFS modes need Secure Boot off: the install-time boot chain carries an unsigned DKMS-built module.

Monitoring widget v19 and Plymouth theme v2V11. The widget gains a ZFS storage panel (pool health, member devices, capacity) through direct read-only zpool/zfs calls - no new broker verb, no sudoers change - plus mode-aware preferences and a two-column layout for small screens; both consumers follow the unified auto-replace unit with a fallback for non-migrated installs. On ZFS installs, S8 applies the GRUB theme onto the ESP and Plymouth splashes after the kexec; a new S8 option installs Cockpit + the ZFS manager plugin (shipped in the archive, no download at install time).

Storage migration handles guests with snapshots (V21.2). The Btrfs storage move used to fail blind on a guest whose disk carries Proxmox snapshots. It now detects them first and asks a dedicated question (default: keep them and skip the guest); every skip, failure and accepted snapshot deletion lands in the session manifest with its cause.

Debian 13 printing/AppArmor fix (21.2.1) - all disk modes. Debian 13 ships the cupsd/cups-browsed AppArmor profiles without an abi line: printing dies silently and every session start waits out D-Bus timeouts (userspace boot in the minute-and-a-half range on affected nodes). S4 and the upgrade pin abi <abi/4.0>, and allow unix, in both profiles - printing works and the session opens in seconds.

PCIe passthrough on ZFS (21.3.1). The optional passthrough step now lands the IOMMU arguments on the Boot Environment command line (idempotent append) - previously the edit only reached /etc/default/grub, which the ZFS boot chain does not read.

Upgrade. hivestation-upgrade.sh covers V15-V20 -> V21 for Btrfs/LVM installs and refreshes an existing V21 ZFS install in place (restore point = a Boot Environment snapshot, bootable from the ZFSBootMenu menu). Adopting ZFS remains a fresh install - there is no conversion.

Full release notes are in section 8.1 of the guide; the chronological development journal is in DEVLOG-V21.md at the root of the archive. Feedback welcome.
 

Attachments

  • Like
Reactions: UdoB
I accept all contributions. For your information, setting up an installation via a pre-seed script is already planned for version 23. I am currently stabilizing version 21 and am also working on version 22.
Thanks! I'm still vacillating between your direction for proxmox and a personal spin of Talos. Any / all of which would be heavily LLM-assisted to be clear. I used to enjoy managing a horde of hosts by hand. I've aged.

Another of my funny bits is enabling authenticated Bluetooth serial access along with metrics. It's for my home machines, and that could provide other options for fixing stuff. I've used the serial aspect for embedded systems in the past. Also with BLE monitoring signals.
 
Thanks! I'm still vacillating between your direction for proxmox and a personal spin of Talos. Any / all of which would be heavily LLM-assisted to be clear. I used to enjoy managing a horde of hosts by hand. I've aged.

Another of my funny bits is enabling authenticated Bluetooth serial access along with metrics. It's for my home machines, and that could provide other options for fixing stuff. I've used the serial aspect for embedded systems in the past. Also with BLE monitoring signals.

Talos and HiveStation solve opposite problems on purpose. Talos is an immutable, API-driven, headless OS built to run a fleet of Kubernetes
nodes with no shell in sight. HiveStation is the other end: one mutable developer workstation, full desktop, running on top of Proxmox VE, where
you actually sit down and work. Neither replaces the other, and they combine well. Nothing stops you from running a Talos cluster inside HiveStation VMs. Pick per machine role rather than per preference.

On the authenticated Bluetooth serial console with BLE metrics: neat idea for your headless boxes, but it is out of scope for HiveStation and I
won't be implementing it. The project targets a workstation that has a screen, a keyboard and a physical console attached, so the recovery value
of an out-of-band radio channel is close to zero here. The cost is real though: a radio-reachable shell widens the attack surface on machines
running LUKS2 or native ZFS encryption with TPM2 auto-unlock, and it adds a Bluetooth chipset dependency to an installer that currently has none.
That trade-off doesn't fit the project.

Pre-seeding is another story, and good news there: I moved it up from V23 into V22, which is what I'm working on now. V21 is stable. Your
dmidecode-driven FAI approach is still what I have in mind for it, so if you want to weigh in on that part, now is the right moment.
 
Okay, finally I gave it a try - successfully at the end :-) - and I'd like to give some feedback:

In #1 you just say
| Boot the target machine on a Debian 13 (Trixie) Live ISO, copy the scripts over.
Which live image did you have in mind? I've tried to use https://cdimage.debian.org/debian-c...iso-hybrid/debian-live-13.6.0-amd64-gnome.iso to install option 4 = "ZFS single" on a test system.

The installation script S1-21V6.sh fails twice because of the ZFS context:
  • it tries to install zfsutils-linux. This package is in "contrib" which is not enabled in devian-live
  • then it tries to install zfs via DKMS which fails because the bootet and mounted .iso is readonly
I could install Trixie temporary onto a random device, but... that should not be necessary, right? I stopped here...

...and then I was too curious to drop it completely and so I actually did install Trixie onto another USB disk, just to get a usable installation base. Thin, no GUI. Here I could integrate "contrib" and run "apt install zfsutils-linux".

Next run:
Code:
=== Sélection des disques physiques / Physical disk selection ===
  1) /dev/disk/by-id/usb-Samsung_SSD_980_PRO_DD564190038E9-0
  2) /dev/disk/by-id/nvme-KINGSTON_OM8TAP41024K1-A00_50026B73846FF81E
Sélection du disque 1 / Disk 1 selection: 2
Disque 1 / Disk 1: /dev/nvme0n1
ZFS non chargé dans le Live - tentative d'installation / ZFS not loaded in Live - trying to install.
...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ERREUR : ZFS indisponible dans le Live (zpool/module absents).
ERROR: ZFS unavailable in the Live (missing zpool/module).

A manual try to install zfs-dkms reveals that there are no headers installed yet. "apt install linux-headers-amd64" does this.

Now I could run "apt --reinstall install zfs-dkms" to reach an important milestone:
Code:
~# zpool -V
zfs-2.3.2-2
zfs-kmod-2.3.2-2

Code:
=== Sélection des disques physiques / Physical disk selection ===
  1) /dev/disk/by-id/usb-Samsung_SSD_980_PRO_DD564190038E9-0
  2) /dev/disk/by-id/nvme-eui.00000000000000000026b73846ff81e5
Sélection du disque 1 / Disk 1 selection: 2
Disque 1 / Disk 1: /dev/nvme0n1
./S1-21V6.sh: Zeile 1107: printf: 44671.5: Ungültige Zahl.

"Ungültige Zahl" being a German error message. Let's change the locale to something generic:

Code:
# export LANG=C

...

Code:
=== Cache ZFS (ARC) / ZFS cache (ARC) ===
...
./S1-21V6.sh: line 1226: mkfs.fat: command not found

At this point I have had entered all information IP-address/Gateway/short name/fqdn etc already a zillion times. I would really wish to do this only once - the scripts should really re-use the information I entered during the previous run!
Code:
apt install dosfstools
Eleventeenth run: another time to enter all configuration data...

Now a lot of dependencies are going to get installed and the system gets successfully rebooted for the first time :-)
./S4-21V2.sh
First try:
Code:
Sélection de l'environnement graphique à installer :
Select the graphical environment to install:
  5) COSMIC
Choix / Choice: 5

machine type x86_64 is supported. Good.
missing dependency: alien
Probably I could have install "alien" manually. But now I go for KDE. This pulls in another 1500 packages...

... and the following reboot gets me to the KDE Login screen :-)


Result: it works! Thank you for the script!
 
Last edited: