Prevent Proxmox from discovering ZFS

Discussion in 'Proxmox VE: Installation and configuration' started by adamb, Apr 1, 2019.

  1. adamb

    adamb Member
    Proxmox Subscriber

    Joined:
    Mar 1, 2012
    Messages:
    995
    Likes Received:
    23
    Is there a way in proxmox 5 to stop auto discover of a zfs array?

    Because we use luks+zfs we can't let proxmox handle ZFS 100%. So we have a script that runs on boot which decrypts the drives and imports the zfs array once luks is done opening the drives. This has always worked well in proxmox4. However it seems in proxmox5 once the storage is added to proxmox, proxmox consistently try's to discover the array. What makes it even worse is if I blow out the zfs cache file, proxmox will repopulate that as well. Is there anyway to get around this?

    Essentially proxmox is importing the array to early, it imports the array before the l2arc cache drives are available.
     
  2. adamb

    adamb Member
    Proxmox Subscriber

    Joined:
    Mar 1, 2012
    Messages:
    995
    Likes Received:
    23
    As a work around, I am decrypting my l2arc disks first and it has solved the issue. Still interested to know how proxmox is keeping a copy of the zfs cache file?
     
  3. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,207
    Likes Received:
    498
    you probably need to order your scripts correctly with regards to zfs importing (the import services are ordered after cryptsetup.target and systemd-udev-settle.service). the zpool.cache file handling is done by ZFS itself once you import a pool without disabling the cachefile.

    I'd recommend either switching to /etc/crypttab (which should auto-generate correctly ordered systemd-cryptsetup@XXX units for each entry) or extending the zfs-import-* services to wait for your custom cryptsetup script ("systemctl edit XXX" and adding an "After=" directive should do the trick).
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. adamb

    adamb Member
    Proxmox Subscriber

    Joined:
    Mar 1, 2012
    Messages:
    995
    Likes Received:
    23
    Everything is solid and has no issues until the storage is added to proxmox. As soon as its added to proxmox this is when we start seeing the issue of auto imorting the array and re-populating the zfs cache file. Would like to know how proxmox is doing that and were it keeps a copy of the zfs cache file?
     
  5. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,207
    Likes Received:
    498
    if a zfspool storage is configured, the pool will be imported using a regular "zpool import -d /dev/disk/by-id POOLNAME" on the next activate_storage() call (usually by pvestatd on the next status update iteration). this will in turn update/generate the /etc/zfs/zpool.cache file. it might make sense to add '-o cachefile=none' to this import invocation (but note that any other import of a pool will still generate a cache file, unless cachefile=none is set on the pool or on the command line invocation).

    if I understand you correctly, the problem is:
    1. initial: zfs-import-* runs before your cryptsetup script, thus no vdevs are visible, no cachefile exists, no pools get imported
    2. cryptsetup script runs, vdevs become visible
    3. storage definition gets added
    4. pvestatd (/any other PVE component that uses the storage) activates the storage, which runs 'zpool import', which generates a cachefile
    5. subsequent boots: zfs-import-* runs before your cryptsetup script, thus no vdevs are visible, cachefile exists and references unavailable vdevs, import fails
    so the proper solution would still be to order your cryptsetup script correctly / switch to crypttab, but if PVE's auto-import would set cachefile=none and you'd delete the cachefile (including regenerating initramfs if a copy is stored there), the problem would go away as well (this should be easy enough to verify)?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,207
    Likes Received:
    498
    just sent a patch to pve-devel to make the ZFSPoolPlugin import with cachefile=none ;)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. adamb

    adamb Member
    Proxmox Subscriber

    Joined:
    Mar 1, 2012
    Messages:
    995
    Likes Received:
    23
    You are the man! Appreciate it.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice