virtiofs - VM can start when backing storage (ceph) is not in good state [virtioFS mnt point is empty in VM]

[scyto]

I am testing virtioFS

• I have a cephFS datastore on my proxmox nodes that is accessible via /etc/pve/docker-cephFS
• it only contains the data used for bind mounts for docker inside the VMs
• it is passed through to VM as a virtiofs device
• i happened to mess up ceph for 15 mins while i was messing with network stuff
• VM came up before ceph was ready (i assume)
• on the pve node /etc/pve/docker-cephFS has all the contents i would expect
• in the VM the passed through device seems to be empty
• a restart of the VM fixed the issue

Is there a way to make the VM only start if the storage backing the virtioFS is good (for example checking for a .something file in the root of the mnt point in the vm?)

--edit--
i forgot hookscripts exists, i can't write code from scratch to save my life, so i asked copilot

it missed the pre-start phase code on its first try, i asked it to try again and make it part of the pre-phase and it came up with this as an example, i think it will work.... how well did copilot do?

#!/bin/bash

# Path to the file you want to check
FILE_PATH="/path/to/your/file"

# VM ID
VMID=$1
PHASE=$2

# Only run the check during the pre-start phase
if [ "$PHASE" == "pre-start" ]; then
  # Check if the file exists
  if [ ! -f "$FILE_PATH" ]; then
    echo "File $FILE_PATH does not exist. VM $VMID will not start."
    exit 1
  fi

  echo "File $FILE_PATH exists. VM $VMID will start."
fi

exit 0

 

[scyto]

Well that was cool, only took 2 prompts to get something that worked (yes yes i know this is simple for the rest of you programmer scripting types)

It doesn't start when the file is missing, which is what i want, and it produces a nice error with the variables and it means i can use the same script with all 3 VMs too!

task started by HA resource agent
File /mnt/pve/docker-cephFS/.donotdelete does not exist. VM 111 will not start.
TASK ERROR: hookscript error for 111 on pre-start: command '/mnt/pve/ISOs-Templates/snippets/cephFS-hookscript.pl 111 pre-start' failed: exit code 1

and it starts when the file is present - and it adds a message to the start notification so i know the script ran - nice, this is my first hookscript - yay.

 

[scyto]

found addtional oddity

my ISOs-Templates dir is store on cephFS , it seems that local VMs are allowed to start before cephFS is ready
this means the VM failed to find the script and ha-manager locked it and i couldn't start the vm untill i had disabled it from the commandline

there are two approach i see

1. store the hookscript in /etc/pve so it is replicated around the cluster and can be used as needed
2. add the following to /etc/systemd/system/pvestatd.service.d/override.conf

[Unit]
After=pve-storage.target

any thoughts as to the 'right' approach?

 

[scyto]

hmm well method two helps because it delays the VM start so i saw no ha-manager lock, but the VM still tries to start *before* the mount is ready.....

task started by HA resource agent

TASK ERROR: hookscript error for 111 on pre-start: script 'ISOs-Templates:snippets/cephFS-hookscript.pl' does not exist

given snippets seems to be the place to store the hookscripts - this basically rules out cephFS as a place to store hookscripts......


reverting thread to usolved.