hookscript becomes unavailable during host shutdown

keeka

Active Member
Dec 8, 2019
182
20
38
I have a hookscript with pre-stop and post-start phases defined.
The hookscript is successfully called when I shutdown the VM via the UI or when a scheduled backup job runs.
However, if I shutdown or reboot the PVE host, when it shuts down the guest concerned, the script is not found:

hookscript error for 105 on pre-stop: script 'local:snippets/omv-guest-hookscript.pl' does not exist

The script is located on the local pve/vz volume:

hookscript: local:snippets/omv-guest-hookscript.pl

Code:
# ls -l /var/lib/vz/snippets/
total 4
-rwxr-xr-x 1 root root 1692 Jul 16 07:08 omv-guest-hookscript.pl

I thought perhaps /var/lib/vz was unmounted before the guest was shutdown (seems highly unlikely). So I attempted to call the script from somwhere on the rootfs, e.g. /usr/local/bin. But it seems only snippets directories may be be used for guest hookscripts?

I'm not sure why that script becomes unavailable so early during the host shutdown.
 
Last edited:
I added PVE storage to the root fs and moved the hookscript there.
Code:
dir: scripts
        path /usr/local/share/pve
        content snippets
        shared 0
Now, when the host shuts down, that guest's hookscript is found and runs as expected:

Question remains why the default local lvm storage seems to go unavailable before all guests have shut down. Whilst I kept the default LVM storage /var/lib/vz, all the guest disks are on a separate lvmthin volume.
 
please post the journal part covering the shutdown with the problem, pveversion -v output and the storage.cfg contents
 
Code:
Jul 19 12:49:53 pve pvesh[13463]: Stopping VM 105 (timeout = 180 seconds)
Jul 19 12:49:53 pve pve-guests[15151]: shutdown VM 105: UPID:pve:00003B2F:0000C023:64B7CDE1:qmshutdown:105:root@pam:
Jul 19 12:49:53 pve pve-guests[13482]: <root@pam> starting task UPID:pve:00003B2F:0000C023:64B7CDE1:qmshutdown:105:root@pam:
Jul 19 12:49:55 pve pvestatd[1724]: storage 'omv-backup' is not online
Jul 19 12:49:55 pve kernel:  sdc: sdc1
Jul 19 12:49:56 pve kernel: fwbr105i0: port 2(tap105i0) entered disabled state
Jul 19 12:49:56 pve kernel: fwbr105i0: port 1(fwln105i0) entered disabled state
Jul 19 12:49:56 pve kernel: vmbr0: port 6(fwpr105p0) entered disabled state
Jul 19 12:49:56 pve kernel: device fwln105i0 left promiscuous mode
Jul 19 12:49:56 pve kernel: fwbr105i0: port 1(fwln105i0) entered disabled state
Jul 19 12:49:56 pve kernel: device fwpr105p0 left promiscuous mode
Jul 19 12:49:56 pve kernel: vmbr0: port 6(fwpr105p0) entered disabled state
Jul 19 12:49:56 pve qmeventd[866]: read: Connection reset by peer
Jul 19 12:49:56 pve pve-guests[13482]: end task UPID:pve:00003B2F:0000C023:64B7CDE1:qmshutdown:105:root@pam:
Jul 19 12:49:57 pve qmeventd[15227]: Starting cleanup for 105
Jul 19 12:49:57 pve qmeventd[15227]: hookscript error for 105 on post-stop: script 'local:snippets/omv-guest-hookscript.pl' does not exist
Jul 19 12:49:57 pve qmeventd[15227]: Finished cleanup for 105

Code:
#pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-4-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.3
pve-kernel-6.2.16-4-pve: 6.2.16-5
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: 0.8.41
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

Code:
#/etc/pve/storage.cfg
dir: local
  path /var/lib/vz
  content snippets,iso,backup,vztmpl
  prune-backups keep-daily=1,keep-monthly=1,keep-weekly=1
  shared 0
lvmthin: vmdisks
  thinpool vmdisks
  vgname vmdisks
  content images,rootdir
nfs: omv-backup
  export /proxmox
  path /mnt/pve/omv-backup
  server 192.168.0.5
  content backup
  options vers=4
  prune-backups keep-daily=1,keep-monthly=1,keep-weekly=1

To which I added another local store just for snippets (with which it then worked on a host shutdown).

Code:
dir: scripts
    path /usr/local/share/pve
    content snippets
    shared 0

Whilst the journal shows the post-stop error, it does not show the pre-stop one (from tasks view @12:49:53 shown in the OP).
The guest concerned does host the omv-backup storage above. However the purpose of the hookscript is to disable and umount this prior to shutting down that guest. Which works under other circumstances.

When I get a chance, I plan to reinstate the problematic config and try a script with another guest.
 
Last edited:
the journal should start with the start of the host shutdown, not with the guest part ;) the problem is in between those two points in time after all..
 
  • Like
Reactions: keeka
Yes, that makes sense! I'd thought it too large to post.
So I disabled all but two VMs and a container.
Lo and behold, the pve-vz volume is umounted before the host begins shutting down guests. I've no guest volumes on that store BTW.
 

Attachments

  • journal.log
    38.5 KB · Views: 2
  • Like
Reactions: leesteken and keeka
@fabian many thanks for identifying the cause.
I'm staying with my snippets on the rootfs for time being. But if I apply the suggested change, it is presumably:

systemctl edit pve-storage.target

INI:
#/etc/systemd/system/pve-storage.target.d/override.conf
[Unit]
After=blk-availability.service
 
yes
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!