[TUTORIAL] HiveStation - Developer Workstation for Proxmox Virtual Environment

Do you have pointers to articles supporting this statement? A small size ARC should not rise problems, except performance wise...?

The practical lower limit is only reached when the ZFS internal data structures do not fit into it anymore. The concept: https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html#adaptive-replacement-cache

----
Edit: an obviously extreme example - on the *BSD side, not OpenZFS: https://wiki.freebsd.org/ZFSTuningGuide#ARC --> "vfs.zfs.arc_max="40M"" :-)

Problem encountered: RAM saturation and ZFS ARC limit exceeded, resulting in a system crash.
Test environment:

Laptop with 16 GB of RAM.
ZFS installation with the ARC limited to 2 GB (zfs_arc_max = 2 GB).
Operating system and running applications consuming approximately 4 GB of RAM.

3 virtual machines running, using approximately 8 GB of RAM in total.

Procedure triggering the incident:
Running an rsync script on the host machine to perform a backup of approximately 20 GB, while the 3 VMs were already running.
Observed result:
Despite the 2 GB limit, the ARC cache exceeded this value during the backup. The available RAM was completely consumed, the system froze (no response), and a forced reboot was required. Upon reboot, corruption was observed.

I am not the only one to have had a similar problem, it seems to me that there is an open bug that resembles this scenario.
 
Problem encountered:
Despite the 2 GB limit, the ARC cache exceeded this value during the backup.
Okay, that's strange. Fortunately I've never encountered such a situation.

(Ram is the only resource which should never be over-committed and there are more RAM consumers than the KVM processes and ARC - but I am sure you know that already.)
 
if you want to have proxmox on your laptop, you should have enough ram, but i see the problem.
(If only i had bought 64GB some time ago ;) )

Practically all my laptops are maxed out in RAM
 
Last edited:
  • Like
Reactions: aureladmin
if you want to have proxmox on your laptop, you should have enough ram, but i see the problem.
(If only i had bought 64GB some time ago ;) )

Practically all my laptops are maxed out in RAM

My main laptop (4 years old) has 32GB of RAM, and it's barely enough. My secondary laptop, for testing, is an old one (8 years old) with 16GB of RAM.
I'd like to add RAM to both, but given current prices, it's not possible.

ZFS is a very good file system for a production server, but for the HIVESTATION project, that's not necessarily the case. Users will have to choose wisely, otherwise, it's going to be a disaster.
 

HiveStation V20 - what's new since V19​


V20 is out (current build V20.5). Archive attached: HIVESTATION-V20-COMPLETE.tar.gz. Existing V15-V19 installs can move up with hivestation-upgrade.sh (no reinstall); a fresh install follows S1 -> S8 as usual. All scripts stay bilingual FR/EN, and the full guide is in the archive.

Restructuring - lib/ + payload/ (V20.0, no behaviour change). The orchestrators S1-S8 are now thin: every file installed on the target ships as a real file in payload/ (no more heredocs buried in scripts), shared code lives in lib/. The runtime interface consumed by the Plymouth theme and the widget (unit names, /usr/local/sbin binaries, state files, marker fields) is frozen and guarded by hivestation-check.sh: full manifest, anti-residue check, contract-coherence check. Same behaviour on the installed machine, much easier to audit and diff.

Encrypted RAID1 now boots with a missing disk. The classic Debian initramfs fails hard on an absent LUKS container, so a degraded encrypted RAID1 used to panic before the root mount. A new local-top hook (hs-raid1-degraded, ordered before cryptroot) detects the explicitly degraded boot, drops the durably-absent containers from the initramfs copy of crypttab, and lets the surviving disk unlock and mount degraded. The installed /etc/crypttab is untouched; a later boot with both disks reopens everything. The upgrade retrofits the hook onto existing encrypted RAID1 installs.

HiveStation Recovery family in the GRUB menu. Two adjacent recovery paths: the grub-btrfs snapshots submenu, now titled "HiveStation Recovery - Snapshots", and a "HiveStation Recovery - Proven Kernel" entry pinned to a known-good kernel. A small service counts validated boots and accumulated uptime per kernel version (2 validated boots or 6 h by default) and tags the version known-good; the kernel pruning never purges the tagged kernel.

TPM2 auto-unlock for LUKS2 (opt-in, EXPERIMENTAL). A post-install tool hivestation-tpm2.sh (status / bind / unbind) binds the LUKS2 containers to the TPM via clevis, so the disks unlock at boot without the passphrase; the passphrase slot is never removed and remains the fallback. Nothing is enrolled at install time - you run bind yourself. The tool warns about the evil-maid limitation when Secure Boot is off. Validated on a vTPM only; the encryption keeps its EXPERIMENTAL flag.

Monitoring widget v15 - least-privilege root broker. The widget no longer receives per-command NOPASSWD sudo grants (16 rules, several with loose wildcards). Every privileged read now goes through one root broker (hivestation-monitor-helper): per-verb allowlist, regex-validated arguments, a single sudoers rule, and an anchored pve-read verb instead of www-data group membership. S8 also enables the extension automatically for the target user, the Proxmox version line follows pve-manager point releases, and LUKS root devices resolve to readable /dev/mapper names.

Safer automatic disk replacement (RAID1). At boot the btrfs-auto-replace service now only detects and alerts, in every mode - it never formats anything unattended. The interactive replacement path only accepts a blank disk as candidate and asks for an explicit confirmation (type the disk name) before partitioning.

Themes. GRUB theme v2V2 and Plymouth theme v2V9: the logo now sits on a pure black background in both (a residual grey plate baked in the image had survived the V19 all-black pass), and the GRUB theme installer carries HiveStation naming end to end. The upgrade now refreshes the GRUB theme too, alongside Plymouth and the widget.

Upgrade and sizing. hivestation-upgrade.sh covers V15-V19 -> V20: target version read from the archive itself, marker VERSION= follows the release, inventory-driven retrofits (recovery entries, kernel pruning, TPM2 tool, degraded-boot hook, themes, widget), pre-upgrade restore point, --dry-run, --rollback. Note for LVM+EXT4 installs: use a disk of 64 GB or more (the root LV is sized at 25% of the volume group).

Housekeeping. Kernel command line deduplicated (parameters were written in both GRUB variables); the Proxmox enterprise repository is now disabled (Enabled: false) instead of deleted, easy to re-enable for subscribers; auto-nat-boot no longer sleeps a fixed 10 s at boot; the whole flow was re-validated end to end on NVMe devices (/dev/nvme0n1 naming, no hardcoded device names).

Full release notes are in section 8.1 of the guide; the chronological development journal is in DEVLOG-V20.md at the root of the archive. Feedback welcome.
 

Attachments

Do you accept contributions? I may want to add pre-seeding and configuration based on MB S/N (from dmidecode, etc.) for use as the FAI post-install script.
 
Do you accept contributions? I may want to add pre-seeding and configuration based on MB S/N (from dmidecode, etc.) for use as the FAI post-install script.
I accept all contributions. For your information, setting up an installation via a pre-seed script is already planned for version 23. I am currently stabilizing version 21 and am also working on version 22.
 

HiveStation V21 - what's new since V20​


V21 is out (current build V21.3.1). Archive attached: HIVESTATION-V21-COMPLETE.tar.gz. Existing V15-V20 installs can move up with hivestation-upgrade.sh (no reinstall); a fresh install follows S1 -> S8 as usual. All scripts stay bilingual FR/EN, and the full guide is in the archive.

ZFS as a first-class filesystem. Two new disk modes at S1 launch: ZFS single and ZFS mirror, at functional parity with the Btrfs stack (the Btrfs and LVM code paths are untouched). The boot chain is themed GRUB on the ESP chaining into ZFSBootMenu, which kexecs the Boot Environment - GRUB never probes the pool, and Boot Environments, snapshots and kernels are selectable at boot. The dataset layout isolates home, logs, caches and the PVE state from the rollbackable root; S1 asks for the ARC cap (default 2 GiB). S3.2 defaults to the native zfspool storage local-zfs (VM and CT snapshots out of the box); S5 installs sanoid timelines plus an APT hook that snapshots the Boot Environment before every operation - each snapshot bootable from the ZFSBootMenu menu - and a ZFS health monitor that publishes the same state file the Plymouth theme and the widget already read.

Native ZFS encryption - hybrid design. Opt-in in both ZFS modes: aes-256-gcm on root/home/var with the kernels inside the encrypted pool, and the swap as a LUKS2 partition outside the pool (a swap zvol is never used). One passphrase, typed once at a themed ZFSBootMenu prompt with your keyboard layout (a self-contained ckbcomp keymap loaded before the prompt - FR and DE layouts proven on real hardware); the booted system then loads its keys without a prompt. Use ASCII-only passphrases in ZFS modes: the initramfs LUKS prompt of the Btrfs modes accepts accented characters, the ZFSBootMenu prompt does not (tested on real hardware). Hibernation (persistent encrypted swap) works on the test bench but stays EXPERIMENTAL: some firmwares reboot on the default ACPI entry before the image is written, so the installer pins HibernateMode=shutdown.

ZFS mirror HA. GRUB is installed on each ESP (either disk boots alone) and the ZFSBootMenu images are replicated by efi-sync; a zed hooklet and a desktop alert signal degradation; a missing disk means a native degraded boot straight to the login. Disk replacement is a guided, admin-triggered run - never an unattended format; a replacement disk of the same model as the survivor is accepted (an overshooting size check rejected identical disks, found and fixed on real hardware). Boot-time missing-disk detection now goes through one dispatcher unit for both redundant modes (hivestation-auto-replace.service); the historical btrfs-auto-replace.service alias was removed in 21.3.0.

Real-hardware validation - EXPERIMENTAL flags dropped. The series was validated end to end on real hardware (UEFI firmware, hardware TPM 2.0): all four ZFS modes, ZFSBootMenu unlock with FR and DE layouts, degraded boot with a disk physically removed, automatic reintegration and guided replacement up to a reboot on the rebuilt disk, TPM2 auto-unlock, hibernate/resume on the encrypted mirror, a full encrypted Btrfs RAID1 install under active Secure Boot (signed PVE kernel chain, accented passphrase at the initramfs prompt), and an upgrade of a published V20.5 install to the current series. As a result, LUKS2 encryption (Btrfs modes), native ZFS encryption and the TPM2 auto-unlock drop their EXPERIMENTAL flags; the LVM+LUKS variant stays flagged (not yet validated on real hardware) and hibernation keeps its flag. ZFS modes need Secure Boot off: the install-time boot chain carries an unsigned DKMS-built module.

Monitoring widget v19 and Plymouth theme v2V11. The widget gains a ZFS storage panel (pool health, member devices, capacity) through direct read-only zpool/zfs calls - no new broker verb, no sudoers change - plus mode-aware preferences and a two-column layout for small screens; both consumers follow the unified auto-replace unit with a fallback for non-migrated installs. On ZFS installs, S8 applies the GRUB theme onto the ESP and Plymouth splashes after the kexec; a new S8 option installs Cockpit + the ZFS manager plugin (shipped in the archive, no download at install time).

Storage migration handles guests with snapshots (V21.2). The Btrfs storage move used to fail blind on a guest whose disk carries Proxmox snapshots. It now detects them first and asks a dedicated question (default: keep them and skip the guest); every skip, failure and accepted snapshot deletion lands in the session manifest with its cause.

Debian 13 printing/AppArmor fix (21.2.1) - all disk modes. Debian 13 ships the cupsd/cups-browsed AppArmor profiles without an abi line: printing dies silently and every session start waits out D-Bus timeouts (userspace boot in the minute-and-a-half range on affected nodes). S4 and the upgrade pin abi <abi/4.0>, and allow unix, in both profiles - printing works and the session opens in seconds.

PCIe passthrough on ZFS (21.3.1). The optional passthrough step now lands the IOMMU arguments on the Boot Environment command line (idempotent append) - previously the edit only reached /etc/default/grub, which the ZFS boot chain does not read.

Upgrade. hivestation-upgrade.sh covers V15-V20 -> V21 for Btrfs/LVM installs and refreshes an existing V21 ZFS install in place (restore point = a Boot Environment snapshot, bootable from the ZFSBootMenu menu). Adopting ZFS remains a fresh install - there is no conversion.

Full release notes are in section 8.1 of the guide; the chronological development journal is in DEVLOG-V21.md at the root of the archive. Feedback welcome.
 

Attachments

  • Like
Reactions: c.dent and UdoB
I accept all contributions. For your information, setting up an installation via a pre-seed script is already planned for version 23. I am currently stabilizing version 21 and am also working on version 22.
Thanks! I'm still vacillating between your direction for proxmox and a personal spin of Talos. Any / all of which would be heavily LLM-assisted to be clear. I used to enjoy managing a horde of hosts by hand. I've aged.

Another of my funny bits is enabling authenticated Bluetooth serial access along with metrics. It's for my home machines, and that could provide other options for fixing stuff. I've used the serial aspect for embedded systems in the past. Also with BLE monitoring signals.
 
Thanks! I'm still vacillating between your direction for proxmox and a personal spin of Talos. Any / all of which would be heavily LLM-assisted to be clear. I used to enjoy managing a horde of hosts by hand. I've aged.

Another of my funny bits is enabling authenticated Bluetooth serial access along with metrics. It's for my home machines, and that could provide other options for fixing stuff. I've used the serial aspect for embedded systems in the past. Also with BLE monitoring signals.

Talos and HiveStation solve opposite problems on purpose. Talos is an immutable, API-driven, headless OS built to run a fleet of Kubernetes
nodes with no shell in sight. HiveStation is the other end: one mutable developer workstation, full desktop, running on top of Proxmox VE, where
you actually sit down and work. Neither replaces the other, and they combine well. Nothing stops you from running a Talos cluster inside HiveStation VMs. Pick per machine role rather than per preference.

On the authenticated Bluetooth serial console with BLE metrics: neat idea for your headless boxes, but it is out of scope for HiveStation and I
won't be implementing it. The project targets a workstation that has a screen, a keyboard and a physical console attached, so the recovery value
of an out-of-band radio channel is close to zero here. The cost is real though: a radio-reachable shell widens the attack surface on machines
running LUKS2 or native ZFS encryption with TPM2 auto-unlock, and it adds a Bluetooth chipset dependency to an installer that currently has none.
That trade-off doesn't fit the project.

Pre-seeding is another story, and good news there: I moved it up from V23 into V22, which is what I'm working on now. V21 is stable. Your
dmidecode-driven FAI approach is still what I have in mind for it, so if you want to weigh in on that part, now is the right moment.
 
Okay, finally I gave it a try - successfully at the end :-) - and I'd like to give some feedback:

In #1 you just say
| Boot the target machine on a Debian 13 (Trixie) Live ISO, copy the scripts over.
Which live image did you have in mind? I've tried to use https://cdimage.debian.org/debian-c...iso-hybrid/debian-live-13.6.0-amd64-gnome.iso to install option 4 = "ZFS single" on a test system.

The installation script S1-21V6.sh fails twice because of the ZFS context:
  • it tries to install zfsutils-linux. This package is in "contrib" which is not enabled in devian-live
  • then it tries to install zfs via DKMS which fails because the bootet and mounted .iso is readonly
I could install Trixie temporary onto a random device, but... that should not be necessary, right? I stopped here...

...and then I was too curious to drop it completely and so I actually did install Trixie onto another USB disk, just to get a usable installation base. Thin, no GUI. Here I could integrate "contrib" and run "apt install zfsutils-linux".

Next run:
Code:
=== Sélection des disques physiques / Physical disk selection ===
  1) /dev/disk/by-id/usb-Samsung_SSD_980_PRO_DD564190038E9-0
  2) /dev/disk/by-id/nvme-KINGSTON_OM8TAP41024K1-A00_50026B73846FF81E
Sélection du disque 1 / Disk 1 selection: 2
Disque 1 / Disk 1: /dev/nvme0n1
ZFS non chargé dans le Live - tentative d'installation / ZFS not loaded in Live - trying to install.
...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ERREUR : ZFS indisponible dans le Live (zpool/module absents).
ERROR: ZFS unavailable in the Live (missing zpool/module).

A manual try to install zfs-dkms reveals that there are no headers installed yet. "apt install linux-headers-amd64" does this.

Now I could run "apt --reinstall install zfs-dkms" to reach an important milestone:
Code:
~# zpool -V
zfs-2.3.2-2
zfs-kmod-2.3.2-2

Code:
=== Sélection des disques physiques / Physical disk selection ===
  1) /dev/disk/by-id/usb-Samsung_SSD_980_PRO_DD564190038E9-0
  2) /dev/disk/by-id/nvme-eui.00000000000000000026b73846ff81e5
Sélection du disque 1 / Disk 1 selection: 2
Disque 1 / Disk 1: /dev/nvme0n1
./S1-21V6.sh: Zeile 1107: printf: 44671.5: Ungültige Zahl.

"Ungültige Zahl" being a German error message. Let's change the locale to something generic:

Code:
# export LANG=C

...

Code:
=== Cache ZFS (ARC) / ZFS cache (ARC) ===
...
./S1-21V6.sh: line 1226: mkfs.fat: command not found

At this point I have had entered all information IP-address/Gateway/short name/fqdn etc already a zillion times. I would really wish to do this only once - the scripts should really re-use the information I entered during the previous run!
Code:
apt install dosfstools
Eleventeenth run: another time to enter all configuration data...

Now a lot of dependencies are going to get installed and the system gets successfully rebooted for the first time :-)
./S4-21V2.sh
First try:
Code:
Sélection de l'environnement graphique à installer :
Select the graphical environment to install:
  5) COSMIC
Choix / Choice: 5

machine type x86_64 is supported. Good.
missing dependency: alien
Probably I could have install "alien" manually. But now I go for KDE. This pulls in another 1500 packages...

... and the following reboot gets me to the KDE Login screen :-)


Result: it works! Thank you for the script!
 
Last edited:
  • Like
Reactions: aureladmin
Okay, finally I gave it a try - successfully at the end :-) - and I'd like to give some feedback:

In #1 you just say

Which live image did you have in mind? I've tried to use https://cdimage.debian.org/debian-c...iso-hybrid/debian-live-13.6.0-amd64-gnome.iso to install option 4 = "ZFS single" on a test system.

The installation script S1-21V6.sh fails twice because of the ZFS context:
  • it tries to install zfsutils-linux. This package is in "contrib" which is not enabled in devian-live
  • then it tries to install zfs via DKMS which fails because the bootet and mounted .iso is readonly
I could install Trixie temporary onto a random device, but... that should not be necessary, right? I stopped here...

...and then I was too curious to drop it completely and so I actually did install Trixie onto another USB disk, just to get a usable installation base. Thin, no GUI. Here I could integrate "contrib" and run "apt install zfsutils-linux".

Next run:
Code:
=== Sélection des disques physiques / Physical disk selection ===
  1) /dev/disk/by-id/usb-Samsung_SSD_980_PRO_DD564190038E9-0
  2) /dev/disk/by-id/nvme-KINGSTON_OM8TAP41024K1-A00_50026B73846FF81E
Sélection du disque 1 / Disk 1 selection: 2
Disque 1 / Disk 1: /dev/nvme0n1
ZFS non chargé dans le Live - tentative d'installation / ZFS not loaded in Live - trying to install.
...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
ERREUR : ZFS indisponible dans le Live (zpool/module absents).
ERROR: ZFS unavailable in the Live (missing zpool/module).

A manual try to install zfs-dkms reveals that there are no headers installed yet. "apt install linux-headers-amd64" does this.

Now I could run "apt --reinstall install zfs-dkms" to reach an important milestone:
Code:
~# zpool -V
zfs-2.3.2-2
zfs-kmod-2.3.2-2

Code:
=== Sélection des disques physiques / Physical disk selection ===
  1) /dev/disk/by-id/usb-Samsung_SSD_980_PRO_DD564190038E9-0
  2) /dev/disk/by-id/nvme-eui.00000000000000000026b73846ff81e5
Sélection du disque 1 / Disk 1 selection: 2
Disque 1 / Disk 1: /dev/nvme0n1
./S1-21V6.sh: Zeile 1107: printf: 44671.5: Ungültige Zahl.

"Ungültige Zahl" being a German error message. Let's change the locale to something generic:

Code:
# export LANG=C

...

Code:
=== Cache ZFS (ARC) / ZFS cache (ARC) ===
...
./S1-21V6.sh: line 1226: mkfs.fat: command not found

At this point I have had entered all information IP-address/Gateway/short name/fqdn etc already a zillion times. I would really wish to do this only once - the scripts should really re-use the information I entered during the previous run!
Code:
apt install dosfstools
Eleventeenth run: another time to enter all configuration data...

Now a lot of dependencies are going to get installed and the system gets successfully rebooted for the first time :-)

First try:
Code:
Sélection de l'environnement graphique à installer :
Select the graphical environment to install:
  5) COSMIC
Choix / Choice: 5

machine type x86_64 is supported. Good.
missing dependency: alien
Probably I could have install "alien" manually. But now I go for KDE. This pulls in another 1500 packages...

... and the following reboot gets me to the KDE Login screen :-)


Result: it works! Thank you for the script!

Thank you for testing it and for reporting each error with the exact output. Your feedback allowed me to identify several underlying bugs, including some you did not encounter, and I am currently correcting them.

The printf error at line 1107 is the most important one. It is not related to ZFS. It is the swap size calculation, which runs in all disk modes. bc always writes a decimal point and printf reads it according to the locale, so any locale with a decimal comma fails there. I only tested in French and English, so I never saw it. The calculation is being rewritten to be locale independent, and I am checking all the other numeric formatting in the project for the same problem.

For the Live environment: contrib is not enabled, the kernel headers are not installed before the DKMS build, there is no check on the writable overlay, and dosfstools is missing, which is why mkfs.fat failed during partitioning. All of this becomes a single preflight step, executed before the first question, which names the missing package instead of failing in the middle of the installation. The guide will also indicate the exact Live image to use and the RAM requirement. Can you tell me how much RAM the test machine had? It will help me set the threshold for the overlay check.

Regarding the questionnaire, you are right. S1 will save the answers of a run and offer to reuse them at the next start, without the passphrases, so a failed attempt does not restart from the first question.

The missing alien dependency for COSMIC is also on the list.

I will publish an updated V21 on Tuesday evening with all these corrections. Thank you again for the detail of your report.
 
  • Like
Reactions: UdoB

HiveStation V21.3.5 - fixes since 21.3.1​


V21.3.5 is out. Archive attached: HIVESTATION-V21-COMPLETE.tar.gz. Existing
installs move up with hivestation-upgrade.sh as usual; 21.3.2, 21.3.3 and
21.3.4 were never released, so this build carries all of them. It exists because
someone installed the suite from a stock Debian Live in a German locale and hit
four walls in a row - every one of them ours. Thanks for the detailed
report.

Locale walls. The swap size went through bc and the shell's printf,
which reads numbers according to LC_NUMERIC: in German, Spanish, Italian and
every other comma-decimal locale, S1 died right there, in every disk mode. The
calculation is integer arithmetic now - no decimal crosses a locale boundary.
Same family: the widget's Wi-Fi collector kept only nmcli lines starting with
yes: (in German they start with nein:), and the Btrfs scrub date fell over
on long-format locales. Machine-facing subprocesses now run under a forced
C.UTF-8.

Missing tools on a stock Live. The GNOME Live ships no dosfstools, so
mkfs.fat killed the run mid-partitioning, after the whole questionnaire.
parted, cryptsetup, rsync, e2fsprogs and wpasupplicant were missing
the same way. A prerequisite check now runs before the first question: each
missing package is named and installed, or the abort is explicit. Fail in ten
seconds with a diagnosis, not at minute twenty with a disk half written.

A real ZFS preflight. The old one hid everything: no contrib, no kernel
headers, verdict twenty minutes in. It now enables contrib, installs the
right headers, shows its work, verifies the module actually loaded - and runs
right after the disk-mode choice, before the questions.

The one that mattered most. An install finished with an empty ESP and the
machine would not boot - yet S1 had reported success. The chroot's exit code was
never checked, and the boot-chain safety net existed for ZFS only. Both S1 and
S2 boundaries are checked now, the net covers every disk mode, and the abort
says plainly not to reboot on that disk.

Failures that reported success. Fixing the above surfaced a family worth
naming. An empty answer at a numbered menu selected the last entry - on the
disk menu that entry was then wiped; menus now refuse empty, non-numeric or
out-of-range answers, and the disk-mode catch-all no longer means RAID1: any
unexpected value aborts before formatting. A failed grub-install during a
guided ZFS disk replacement let the one-way, destructive ESP sync run anyway;
the refusal now lives in efi-sync.sh itself, where every trigger passes. The
upgrade ticked "ZBM image regenerated" without checking; it fails by name now,
on the manifest too. None of these were introduced by this release.

A check that could not fail. The upgrade posts the ESP synchronisation
files, then verified only that the destination existed - on an upgrade it
always does. A failed install was recorded as a success, on the manifest too.
The destination is now compared with the source, and update-grub's output is
kept in a log the failure message cites. This is the fix that makes 21.3.5.

The desktop widget did not start at all. disable() destroyed the panel
but left references to the disposed boxes; asynchronous updates resumed after
deactivation and GNOME held the extension INACTIVE while it reported itself
enabled. Fixed and verified on a running node (widget v22).

Existing installs get the ESP fix too. The upgrade now refreshes the sync
primitive and its triggers on both two-ESP modes, plus the grub.d trigger on
Btrfs RAID1 - whose output goes to standard error now, since grub-mkconfig
captures a script's standard output into the grub.cfg it builds.

Also fixed. COSMIC's installer asked for alien without installing it,
and is now labelled EXPERIMENTAL; S7 no longer dies when systemd defers an
immediate service start.

Known limitation. When the ESP synchronisation refuses to run, the
monitoring widget and the boot-splash feeder still classify the last state as
healthy. The refusal itself works and is logged; the display will be corrected
in the next series.

Documentation. The guide names the validated Live image, the free space
and RAM the ZFS preflight needs, and repeats the two ZFS-mode constraints where
they bite: Secure Boot off at install time, and an ASCII passphrase.

Full release notes are in section 8.1 of the guide; the development journal is
in DEVLOG-V21.md at the root of the archive.
 

Attachments

  • Like
Reactions: UdoB

HiveStation V22 - answer files, a live image of its own, and five languages​


V22 is out, on GitLab: download HIVESTATION-V22.33.26.tar.gz from https://gitlab.com/aureladmin/hivestation/-/releases/V22.33.26. Existing installs move up with hivestation-upgrade.sh as usual - Btrfs, LVM and ZFS roots, V15 onwards. The disk layouts, the boot chains and what lands on the target are unchanged from V21; what changed is how you drive an install.

Install from an answer file. Every question of S1 to S8, of the desktop layout script and of the upgrade tool now goes through one funnel: the value comes from an answer file, else the question is asked, else the script default applies. The file is plain key=value, parsed and never executed - a line that is neither a comment nor a pair rejects the whole file. Two modes: assist, where a missing key falls back to the question, and strict, where a missing or invalid key stops before anything is done and names the exact key. Keys are semantic (disk_mode=raid1, wan_iface=ens18, disk1=/dev/disk/by-id/...), so one file carries from one machine to the next; you point at it with --preseed <path|URL>, with $HS_PRESEED, or by machine identifier in a directory - serials first, then the first NIC MAC, then default.conf. In this mode S1 refuses to touch a disk without an explicit wipe_disks=oui, and any key that no step ever consumed is reported at the end - fatal in strict on both routes. Until 22.33.20 the single-pass install only warned: a typo in a fleet answer file went through and the deployment carried on, which is the one case strict exists to stop. Ready-made files are in preseed-examples/<lang>/: seven profiles, in five languages. Only the comments differ between the language directories - every key and value is identical apart from lang_code and timezone - and a self-test compares the five sets line by line, so a key added in one language and forgotten in another cannot ship.

One questionnaire, one reboot. S0 collects the decisions of the whole chain up front and checks each answer as you type it. S1 then lays the suite and the answer file down in the target and plays S3.1 -> S3.2 -> S4 -> S5 -> S7 -> S8 inside it: state kept on disk, one log per step, a bounded timeout per step, a hard stop on failure, and --reprendre to restart at the first unfinished step. A machine that used to need seven sessions at a keyboard now needs none. The step-by-step route still works exactly as it did.

A live image of its own. tools/hivestation-live-build.sh builds a live image that runs the Proxmox kernel, and tools/hivestation-iso-build.sh refreshes an already-built one in seconds with the current tree and, optionally, an answer file. No prebuilt ISO is shipped - you build it from the archive you can read. Because the image runs the PVE kernel, its zfs module is signed with that kernel and needs no DKMS build in the live session, so zpool works there even under Secure Boot. The image carries a graphical front end over the same scripts, for people who would rather click than read a terminal.

Five languages. French, English, German, Spanish and Portuguese, from the first question to the last line of the upgrade report - the journal included, so journalctl -t hivestation speaks the installed language. The boot screens follow: Plymouth and the GRUB theme are static files, so their strings are substituted from a per-language catalogue when the theme is laid down - the project's own name and the titles that identify it stay in English everywhere, as they should. The keyboard layout you choose is honoured where it matters - at the Plymouth LUKS prompt and at the ZFSBootMenu prompt. The ZFSBootMenu prompt also names the layout it loaded, so you know before you type.

Two fixes worth naming, both found installing over a machine that had already been served. An encrypted LVM install dropped to the rescue shell on first boot: both disks carried a Btrfs volume labelled HIVBOOT - the new one and the leftover - and /boot was resolved by label with a head -1. It now comes from the volume actually mounted on the target, and the label fallback refuses when two distinct filesystems carry it. Separately, upgrading a ZFS node that had never received the Proxmox repository purged the zfs-dkms stack toward packages it could not install, leaving an initramfs with no ZFS in it; the node would not boot. The migration now lays the repository itself before touching the stack, and reads the regenerated initramfs back to check ZFS is in it before calling the step a success.

The code reads in English now. The suite is meant to be read and audited
before it is run, so its comments are no longer written only in French: its
12 101 French comment lines were moved to English, the language of the
guide. Behaviour is unchanged - not one line of code moved, and a self-test
proves it file by file.

Disk alerts and guided replacement speak the node's language. On a German
desktop the degraded-mirror dialog came out in French and English next to a
correctly translated notification, and the guided replacement run from a
terminal mixed German and bilingual lines. Those messages never went through
the catalogue. In 22.33.2 the two alert scripts and both auto-replace-*-disk.sh
do - every message, the LUKS passphrase prompt and the disk-name confirmation
included - with 95 new keys in German, Spanish and Portuguese. A yes/no
question that showed a French default next to a translated legend now takes
its default from the legend itself. The upgrade tool now re-lays the Btrfs
HA pair as it did the ZFS one, and lays the catalogues where the library reads
them first - the suite tree of the node - so a node upgraded in place gets the
fix too (22.33.3).

The desktop said the array was healthy while a disk was missing. On an
amputated Btrfs mirror the array indicator stayed green and read "healthy",
directly above a red "Health: degraded" line - two indicators side by side, two
sources, and the green one was wrong. The collector walked
/sys/fs/btrfs/<uuid>/devices/ and only flagged a member as missing when its
block node had disappeared; a disk that is genuinely pulled leaves that list
altogether, so the loop never saw it and the count stayed at zero. It counted
what is PRESENT, never what is EXPECTED. It now derives the count from
devinfo/, which is indexed by numeric devid, keeps the entry of an absent
member with its missing flag, and is readable without privilege - so no new
verb on the broker. Absent members are listed under their devid, since they no
longer have a device name, and the old counting is kept as a fallback where the
kernel does not expose devinfo. The rendering already painted a missing member
red: only the counting was blind. Widget v27 -> v28 at the time; it ships at v30.

Same bench, a finding with no fix attached: on an encrypted install the swap
disappears with its disk and the SWAP1 -> SWAP2 failover does not play. swap-failover is not enabled when the swap is encrypted:
crypttab and nofail handle it, and SWAP2 stays a reserved partition. S1 says
so to the operator and a bench requires it.

The themes and the widget stopped being laid down at all. From 22.33.5 the
Plymouth theme, then the GRUB theme, then the widget were shipped as directories
instead of tarballs, so that the repository's own checks could read inside them.
S1 copies the suite into the target from an explicit list, and the entry that
carried them was *.tar.gz - which matches no directory. Three shipped versions
therefore installed none of the three, and S8 returned a failure on every
install that asked for them. The list now names the directories, and a bench
compares what S1 lays down against what the archive carries, component by
component, so a component added to the archive and forgotten in the copy list
cannot ship again.

Three places where a state was decided by reading prose. A degraded
mirror, a cancelled ESP synchronisation and, on the desktop panel, a node that
was perfectly healthy: three symptoms, one cause. Each state was decided by
reading the PROSE of a log - words like "ERREUR" or "missing" - or a completion
marker that was also written when the operation had failed. Prose is translated,
so nothing matched on a German or Portuguese node and the fallback displayed
"active"; and a marker written on both outcomes says nothing. Every backend now
writes an ASCII verdict of its own, from a trap so that a cancelled run still
writes one, and the monitor reads that alone. The panel's own indicator was
counting the disks PRESENT rather than the disks EXPECTED, which is why a pulled
disk left the count at zero.

Error paths speak the installed language. The V22.33.4 notes said this was
not covered: the refusal paths of the shared libraries still came out in French
whatever the install language. The cause was mechanical - those messages were
built with their value already inside the string, so they matched no catalogue
entry and fell through to the French original. 306 calls now pass a template and
its values separately, and 166 new templates went into the five catalogues. What
remains is named rather than rounded off: 163 calls still build their text with
the value already inside, across ten libraries, and those keep the bilingual
French/English form. A bench counts them with a ceiling that only goes down.

Translated interfaces were losing information. Three defects of the same
family. On an English menu the option numbers were gone, because the English
catalogue entry had replaced the numbered French line and taken the number with
it - two components out of seven were selectable by the number shown. English
messages had lost their %s, so a reader saw "failed" without being told what
failed, and the confirmation that asks you to type the exact name of a disk to
be erased no longer showed that name. And three untranslated entries reached
non-French installs verbatim. Each of the three now has a check that counts the
sites and refuses to grow.

What was validated, and what was not. On real hardware, driven by an answer file: five disk modes, each driven in a different language, every one installed over an already-occupied target, chain green end to end, reboot verified. Seven modes out of seven through the graphical route. Upgrade V21 -> V22 on real hardware in ZFS mirror and Btrfs RAID1. Those figures date from the 22.33.4 series and are unchanged.

The fixes listed above were exercised since, in virtual machines rather than on metal: four installs carried to the end on both routes - from the archive in a live session and from the project's own image - in French, German and Spanish, across a ZFS mirror and encrypted Btrfs RAID1. Each one ends on the installer's own exit code, hands back the seven step logs written into the target, and was then booted from disk: unlock prompt and desktop session both come up in the installed language. Strict mode on a single-pass install was proven in both directions - a deployment that must stop does stop, one that must go through does - on all five disk families, the encrypted ZFS mirror included. Hibernation keeps its EXPERIMENTAL flag and stays firmware-dependent. Encrypted installs keep the ESP and /boot in cleartext: with Secure Boot off, TPM auto-unlock does not stop someone with brief physical access from planting a modified initramfs - only the passphrase does. The LVM run proved the mode itself but does not attest Secure Boot or TPM 2.0 - those come from the Btrfs runs.

HiveStation is under the GNU AGPL v3 - the same licence as Proxmox VE. Use it, change it, pass it on; just pass the source and the same freedoms on with it. The archive carries LICENSE, plus LICENCE-FONTS.txt for the Ubuntu Mono files embedded in the GRUB theme.

The full guide is in the archive (HIVESTATION-V22-GUIDE-EN.md), per-version detail in its §8.1, and the development journal in DEVLOG-V22.md.
 
Last edited: