Hi,
in case someone else hits this: after a reboot, guests with onboot=1 on a zfspool storage sometimes do not start, although the pool is imported and fine a minute later. Typical task log:
or
Cause
ZFSPoolPlugin::activate_storage() imports the pool on demand (with cachefile=none, so the pool is not in zpool.cache and ZFS does not import it at boot). pvestatd and pve-guests/startall both call it right after boot, about one second apart. While the first 'zpool import' is running (here ~40 s, dedup pool), the pool is invisible to both 'zpool list' and 'zpool import', so the second caller fails with "no such pool available". startall does not retry, so the guest stays stopped.
A second, independent problem: both callers then run 'zfs mount -a' and can collide ("mountpoint or dataset is busy"). 'zfs mount -a' exits non-zero if any dataset fails, so the guest start fails although the storage's own dataset is mounted. Unrelated datasets on the same pool (e.g. a backup tree with many datasets) trigger the same failure.
Affected: PVE 9.2.21 / libpve-storage-perl 9.1.13, the code is unchanged on current master. Any pool that takes more than a moment to import can hit it; whether it does depends on timing.
Fix
Filed as bug #8230 with a patch on pve-devel: import and mount are serialized per pool with a node-local lock (guest starts wait, pvestatd and API requests report "import in progress" instead of failing), and a failed 'zfs mount -a' is only fatal if the storage's own dataset is not mounted. Tested with 12 reboots on four nodes, the guests started every time.
Test package
For libpve-storage-perl 9.1.13 there is a test package attached to the bug: pve-zfspool-import-race-test-9.1.13.tar.gz
https://bugzilla.proxmox.com/attachment.cgi?id=2199
apply.sh checks the package version and that the file is unmodified, saves the original, applies the patch, checks the Perl syntax and reloads the PVE daemons. The effect shows after the next reboot; in a cluster apply it on every node with such a pool. ./revert.sh restores the original (or: apt install --reinstall libpve-storage-perl). An upgrade of libpve-storage-perl simply replaces the patched file. Details in README.txt. It is unofficial until merged.
If you test it, a reply on pve-devel with "Tested-by: Name <mail>" or a comment on the bug helps getting it merged.
Workarounds without the patch, pick one:
-
- a startall delay longer than the pool's import time:
- start the guest manually after boot
Links:
- bug #8230: https://bugzilla.proxmox.com/show_bug.cgi?id=8230
- patch on pve-devel: https://lore.proxmox.com/pve-devel/20261025213830.2432528-1-carsten.haerle@straightec.de/
in case someone else hits this: after a reboot, guests with onboot=1 on a zfspool storage sometimes do not start, although the pool is imported and fine a minute later. Typical task log:
Code:
Starting VM 124 failed: could not activate storage 'vmdata1zfs', zfs error: cannot import 'data1': no such pool availableor
Code:
could not activate storage 'vm', zfs error: cannot mount 'store/...': mountpoint or dataset is busyCause
ZFSPoolPlugin::activate_storage() imports the pool on demand (with cachefile=none, so the pool is not in zpool.cache and ZFS does not import it at boot). pvestatd and pve-guests/startall both call it right after boot, about one second apart. While the first 'zpool import' is running (here ~40 s, dedup pool), the pool is invisible to both 'zpool list' and 'zpool import', so the second caller fails with "no such pool available". startall does not retry, so the guest stays stopped.
A second, independent problem: both callers then run 'zfs mount -a' and can collide ("mountpoint or dataset is busy"). 'zfs mount -a' exits non-zero if any dataset fails, so the guest start fails although the storage's own dataset is mounted. Unrelated datasets on the same pool (e.g. a backup tree with many datasets) trigger the same failure.
Affected: PVE 9.2.21 / libpve-storage-perl 9.1.13, the code is unchanged on current master. Any pool that takes more than a moment to import can hit it; whether it does depends on timing.
Fix
Filed as bug #8230 with a patch on pve-devel: import and mount are serialized per pool with a node-local lock (guest starts wait, pvestatd and API requests report "import in progress" instead of failing), and a failed 'zfs mount -a' is only fatal if the storage's own dataset is not mounted. Tested with 12 reboots on four nodes, the guests started every time.
Test package
For libpve-storage-perl 9.1.13 there is a test package attached to the bug: pve-zfspool-import-race-test-9.1.13.tar.gz
https://bugzilla.proxmox.com/attachment.cgi?id=2199
Code:
wget -O pve-zfspool-import-race-test-9.1.13.tar.gz 'https://bugzilla.proxmox.com/attachment.cgi?id=2199'
tar xzf pve-zfspool-import-race-test-9.1.13.tar.gz
cd pve-zfspool-import-race-test
./apply.shapply.sh checks the package version and that the file is unmodified, saves the original, applies the patch, checks the Perl syntax and reloads the PVE daemons. The effect shows after the next reboot; in a cluster apply it on every node with such a pool. ./revert.sh restores the original (or: apt install --reinstall libpve-storage-perl). An upgrade of libpve-storage-perl simply replaces the patched file. Details in README.txt. It is unofficial until merged.
If you test it, a reply on pve-devel with "Tested-by: Name <mail>" or a comment on the bug helps getting it merged.
Workarounds without the patch, pick one:
-
zpool set cachefile=/etc/zfs/zpool.cache <pool> with an enabled zfs-import-cache.service, so ZFS imports the pool before any PVE service runs- a startall delay longer than the pool's import time:
pvenode config set --startall-onboot-delay 60 (timing based)- start the guest manually after boot
Links:
- bug #8230: https://bugzilla.proxmox.com/show_bug.cgi?id=8230
- patch on pve-devel: https://lore.proxmox.com/pve-devel/20261025213830.2432528-1-carsten.haerle@straightec.de/