Hi,
in case someone else hits this: after a reboot, a VM with onboot=1 on a
zfspool storage did not start, while the pool was imported and fine a
minute later. Task log:
Starting VM 124 failed: could not activate storage 'vmdata1zfs',
zfs error: cannot import 'data1': no such pool available
Cause: ZFSPoolPlugin::activate_storage() imports the pool on demand
(with cachefile=none, so the pool is not in zpool.cache and ZFS does not
import it at boot). pvestatd and pve-guests/startall both call it right
after boot, about one second apart. While the first 'zpool import' is
running (here ~40 s, dedup pool), the pool is invisible to both
'zpool list' and 'zpool import', so the second caller fails with "no
such pool available". The re-check in the code runs immediately and
does not help for imports that take longer than a few milliseconds.
startall does not retry, so the guest stays stopped.
A second, independent problem: both callers then run 'zfs mount -a' and
can collide ("cannot mount ...: mountpoint or dataset is busy").
'zfs mount -a' exits non-zero if any dataset fails, so the guest start
fails although the storage's own dataset is mounted. Unrelated datasets
on the same pool (e.g. a backup tree with hundreds of datasets) trigger
the same failure.
Affected: PVE 9.2.21 / libpve-storage-perl 9.1.13, the code is
unchanged on current master. Any pool that takes more than a moment to
import can hit it; whether it does depends on timing.
Filed as bug #8230 with a patch on pve-devel: serialize import and
mount per pool with a node-local lock (guest starts wait, pvestatd and
API requests report "import in progress" instead of failing) and treat
a failed 'zfs mount -a' as fatal only if the storage's dataset is not
mounted. Tested with 12 reboots on four nodes, the guests started every
time.
Workarounds until that lands, pick one:
- 'zpool set cachefile=/etc/zfs/zpool.cache <pool>' with an enabled
zfs-import-cache.service, so ZFS imports the pool before any PVE
service runs
- Datacenter > Options > Start/Shutdown order: a startall delay longer
than the pool's import time (timing based)
- start the guest manually after boot
The patch is attached (zfspool-import-race.patch.txt). It is
unofficial until merged, made for libpve-storage-perl 9.1.13, and any
update of that package overwrites it. To apply it on a node:
cd /usr/share/perl5
cp PVE/Storage/ZFSPoolPlugin.pm PVE/Storage/ZFSPoolPlugin.pm.orig
patch -p2 --dry-run < /root/zfspool-import-race.patch.txt # check first
patch -p2 < /root/zfspool-import-race.patch.txt
systemctl restart pvedaemon pveproxy pvestatd pvescheduler
To revert, copy the .orig file back or run
'apt install --reinstall libpve-storage-perl'. The raw patch is also
available from the pve-devel archive (append "raw" to the link below).
Links:
- bug #8230: https://bugzilla.proxmox.com/show_bug.cgi?id=8230
- patch on pve-devel: https://lore.proxmox.com/pve-devel/20261025213830.2432528-1-carsten.haerle@straightec.de/
in case someone else hits this: after a reboot, a VM with onboot=1 on a
zfspool storage did not start, while the pool was imported and fine a
minute later. Task log:
Starting VM 124 failed: could not activate storage 'vmdata1zfs',
zfs error: cannot import 'data1': no such pool available
Cause: ZFSPoolPlugin::activate_storage() imports the pool on demand
(with cachefile=none, so the pool is not in zpool.cache and ZFS does not
import it at boot). pvestatd and pve-guests/startall both call it right
after boot, about one second apart. While the first 'zpool import' is
running (here ~40 s, dedup pool), the pool is invisible to both
'zpool list' and 'zpool import', so the second caller fails with "no
such pool available". The re-check in the code runs immediately and
does not help for imports that take longer than a few milliseconds.
startall does not retry, so the guest stays stopped.
A second, independent problem: both callers then run 'zfs mount -a' and
can collide ("cannot mount ...: mountpoint or dataset is busy").
'zfs mount -a' exits non-zero if any dataset fails, so the guest start
fails although the storage's own dataset is mounted. Unrelated datasets
on the same pool (e.g. a backup tree with hundreds of datasets) trigger
the same failure.
Affected: PVE 9.2.21 / libpve-storage-perl 9.1.13, the code is
unchanged on current master. Any pool that takes more than a moment to
import can hit it; whether it does depends on timing.
Filed as bug #8230 with a patch on pve-devel: serialize import and
mount per pool with a node-local lock (guest starts wait, pvestatd and
API requests report "import in progress" instead of failing) and treat
a failed 'zfs mount -a' as fatal only if the storage's dataset is not
mounted. Tested with 12 reboots on four nodes, the guests started every
time.
Workarounds until that lands, pick one:
- 'zpool set cachefile=/etc/zfs/zpool.cache <pool>' with an enabled
zfs-import-cache.service, so ZFS imports the pool before any PVE
service runs
- Datacenter > Options > Start/Shutdown order: a startall delay longer
than the pool's import time (timing based)
- start the guest manually after boot
The patch is attached (zfspool-import-race.patch.txt). It is
unofficial until merged, made for libpve-storage-perl 9.1.13, and any
update of that package overwrites it. To apply it on a node:
cd /usr/share/perl5
cp PVE/Storage/ZFSPoolPlugin.pm PVE/Storage/ZFSPoolPlugin.pm.orig
patch -p2 --dry-run < /root/zfspool-import-race.patch.txt # check first
patch -p2 < /root/zfspool-import-race.patch.txt
systemctl restart pvedaemon pveproxy pvestatd pvescheduler
To revert, copy the .orig file back or run
'apt install --reinstall libpve-storage-perl'. The raw patch is also
available from the pve-devel archive (append "raw" to the link below).
Links:
- bug #8230: https://bugzilla.proxmox.com/show_bug.cgi?id=8230
- patch on pve-devel: https://lore.proxmox.com/pve-devel/20261025213830.2432528-1-carsten.haerle@straightec.de/
