Prevent Proxmox from discovering ZFS

[adamb]

Is there a way in proxmox 5 to stop auto discover of a zfs array?

Because we use luks+zfs we can't let proxmox handle ZFS 100%. So we have a script that runs on boot which decrypts the drives and imports the zfs array once luks is done opening the drives. This has always worked well in proxmox4. However it seems in proxmox5 once the storage is added to proxmox, proxmox consistently try's to discover the array. What makes it even worse is if I blow out the zfs cache file, proxmox will repopulate that as well. Is there anyway to get around this?

Essentially proxmox is importing the array to early, it imports the array before the l2arc cache drives are available.

 

[adamb]

As a work around, I am decrypting my l2arc disks first and it has solved the issue. Still interested to know how proxmox is keeping a copy of the zfs cache file?

 

[fabian]

you probably need to order your scripts correctly with regards to zfs importing (the import services are ordered after cryptsetup.target and systemd-udev-settle.service). the zpool.cache file handling is done by ZFS itself once you import a pool without disabling the cachefile.

I'd recommend either switching to /etc/crypttab (which should auto-generate correctly ordered systemd-cryptsetup@XXX units for each entry) or extending the zfs-import-* services to wait for your custom cryptsetup script ("systemctl edit XXX" and adding an "After=" directive should do the trick).

 

[adamb]

you probably need to order your scripts correctly with regards to zfs importing (the import services are ordered after cryptsetup.target and systemd-udev-settle.service). the zpool.cache file handling is done by ZFS itself once you import a pool without disabling the cachefile.

I'd recommend either switching to /etc/crypttab (which should auto-generate correctly ordered systemd-cryptsetup@XXX units for each entry) or extending the zfs-import-* services to wait for your custom cryptsetup script ("systemctl edit XXX" and adding an "After=" directive should do the trick).

Everything is solid and has no issues until the storage is added to proxmox. As soon as its added to proxmox this is when we start seeing the issue of auto imorting the array and re-populating the zfs cache file. Would like to know how proxmox is doing that and were it keeps a copy of the zfs cache file?

 

[fabian]

Everything is solid and has no issues until the storage is added to proxmox. As soon as its added to proxmox this is when we start seeing the issue of auto imorting the array and re-populating the zfs cache file. Would like to know how proxmox is doing that and were it keeps a copy of the zfs cache file?

if a zfspool storage is configured, the pool will be imported using a regular "zpool import -d /dev/disk/by-id POOLNAME" on the next activate_storage() call (usually by pvestatd on the next status update iteration). this will in turn update/generate the /etc/zfs/zpool.cache file. it might make sense to add '-o cachefile=none' to this import invocation (but note that any other import of a pool will still generate a cache file, unless cachefile=none is set on the pool or on the command line invocation).

if I understand you correctly, the problem is:

1. initial: zfs-import-* runs before your cryptsetup script, thus no vdevs are visible, no cachefile exists, no pools get imported
2. cryptsetup script runs, vdevs become visible
3. storage definition gets added
4. pvestatd (/any other PVE component that uses the storage) activates the storage, which runs 'zpool import', which generates a cachefile
5. subsequent boots: zfs-import-* runs before your cryptsetup script, thus no vdevs are visible, cachefile exists and references unavailable vdevs, import fails

so the proper solution would still be to order your cryptsetup script correctly / switch to crypttab, but if PVE's auto-import would set cachefile=none and you'd delete the cachefile (including regenerating initramfs if a copy is stored there), the problem would go away as well (this should be easy enough to verify)?

 

[fabian]

just sent a patch to pve-devel to make the ZFSPoolPlugin import with cachefile=none

 

[adamb]

just sent a patch to pve-devel to make the ZFSPoolPlugin import with cachefile=none

You are the man! Appreciate it.