hookscript becomes unavailable during host shutdown

[keeka]

I have a hookscript with pre-stop and post-start phases defined.
The hookscript is successfully called when I shutdown the VM via the UI or when a scheduled backup job runs.
However, if I shutdown or reboot the PVE host, when it shuts down the guest concerned, the script is not found:

hookscript error for 105 on pre-stop: script 'local:snippets/omv-guest-hookscript.pl' does not exist

The script is located on the local pve/vz volume:

hookscript: local:snippets/omv-guest-hookscript.pl

# ls -l /var/lib/vz/snippets/
total 4
-rwxr-xr-x 1 root root 1692 Jul 16 07:08 omv-guest-hookscript.pl

I thought perhaps /var/lib/vz was unmounted before the guest was shutdown (seems highly unlikely). So I attempted to call the script from somwhere on the rootfs, e.g. /usr/local/bin. But it seems only snippets directories may be be used for guest hookscripts?

I'm not sure why that script becomes unavailable so early during the host shutdown.

 

[keeka]

I added PVE storage to the root fs and moved the hookscript there.

dir: scripts
        path /usr/local/share/pve
        content snippets
        shared 0

Now, when the host shuts down, that guest's hookscript is found and runs as expected:

Question remains why the default local lvm storage seems to go unavailable before all guests have shut down. Whilst I kept the default LVM storage /var/lib/vz, all the guest disks are on a separate lvmthin volume.

 

[fabian]

please post the journal part covering the shutdown with the problem, pveversion -v output and the storage.cfg contents

 

[keeka]

Jul 19 12:49:53 pve pvesh[13463]: Stopping VM 105 (timeout = 180 seconds)
Jul 19 12:49:53 pve pve-guests[15151]: shutdown VM 105: UPID:pve:00003B2F:0000C023:64B7CDE1:qmshutdown:105:root@pam:
Jul 19 12:49:53 pve pve-guests[13482]: <root@pam> starting task UPID:pve:00003B2F:0000C023:64B7CDE1:qmshutdown:105:root@pam:
Jul 19 12:49:55 pve pvestatd[1724]: storage 'omv-backup' is not online
Jul 19 12:49:55 pve kernel:  sdc: sdc1
Jul 19 12:49:56 pve kernel: fwbr105i0: port 2(tap105i0) entered disabled state
Jul 19 12:49:56 pve kernel: fwbr105i0: port 1(fwln105i0) entered disabled state
Jul 19 12:49:56 pve kernel: vmbr0: port 6(fwpr105p0) entered disabled state
Jul 19 12:49:56 pve kernel: device fwln105i0 left promiscuous mode
Jul 19 12:49:56 pve kernel: fwbr105i0: port 1(fwln105i0) entered disabled state
Jul 19 12:49:56 pve kernel: device fwpr105p0 left promiscuous mode
Jul 19 12:49:56 pve kernel: vmbr0: port 6(fwpr105p0) entered disabled state
Jul 19 12:49:56 pve qmeventd[866]: read: Connection reset by peer
Jul 19 12:49:56 pve pve-guests[13482]: end task UPID:pve:00003B2F:0000C023:64B7CDE1:qmshutdown:105:root@pam:
Jul 19 12:49:57 pve qmeventd[15227]: Starting cleanup for 105
Jul 19 12:49:57 pve qmeventd[15227]: hookscript error for 105 on post-stop: script 'local:snippets/omv-guest-hookscript.pl' does not exist
Jul 19 12:49:57 pve qmeventd[15227]: Finished cleanup for 105

#pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-4-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.3
pve-kernel-6.2.16-4-pve: 6.2.16-5
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: 0.8.41
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

#/etc/pve/storage.cfg
dir: local
  path /var/lib/vz
  content snippets,iso,backup,vztmpl
  prune-backups keep-daily=1,keep-monthly=1,keep-weekly=1
  shared 0
lvmthin: vmdisks
  thinpool vmdisks
  vgname vmdisks
  content images,rootdir
nfs: omv-backup
  export /proxmox
  path /mnt/pve/omv-backup
  server 192.168.0.5
  content backup
  options vers=4
  prune-backups keep-daily=1,keep-monthly=1,keep-weekly=1

To which I added another local store just for snippets (with which it then worked on a host shutdown).

dir: scripts
    path /usr/local/share/pve
    content snippets
    shared 0

Whilst the journal shows the post-stop error, it does not show the pre-stop one (from tasks view @12:49:53 shown in the OP).
The guest concerned does host the omv-backup storage above. However the purpose of the hookscript is to disable and umount this prior to shutting down that guest. Which works under other circumstances.

When I get a chance, I plan to reinstate the problematic config and try a script with another guest.

 

[fabian]

the journal should start with the start of the host shutdown, not with the guest part the problem is in between those two points in time after all..

 

[keeka]

Yes, that makes sense! I'd thought it too large to post.
So I disabled all but two VMs and a container.
Lo and behold, the pve-vz volume is umounted before the host begins shutting down guests. I've no guest volumes on that store BTW.

 

[fabian]

thanks! it seems you are hitting https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991550

could you try ordering pve-storage.target after blk-availability.service? that should ensure that the blkdeactivate call happens after guest shutdown is complete..

 

[keeka]

@fabian many thanks for identifying the cause.
I'm staying with my snippets on the rootfs for time being. But if I apply the suggested change, it is presumably:

systemctl edit pve-storage.target

#/etc/systemd/system/pve-storage.target.d/override.conf
[Unit]
After=blk-availability.service

 

[fabian]

yes