Snippets hookscript can not change CT net config

[vaka]

Hi all,

I have a two node cluster PVE 7.3-4 on different data centers with different public networks.

LXC container running on node1 and replicated to node2.

I need to change container's network settings (MAC addres, ip-addres and gw) when it migrated on node2 and change it back when it migrated back on node1.
So I created hook-scripts at snippets on both nodes that change container's network settings on pre-start phase.

Script:

#!/bin/bash

ID=$1
PHASE=$2

echo "CT HOOK: $ID, PHASE: $PHASE"

case $PHASE in
  pre-start)
    if [[ "x$ID" == "x105" ]]; then
      echo pct set $ID -net0 name=eth0,bridge=vmbr0,firewall=0,gw=1.2.3.1,hwaddr=00:00:00:01:02:03,ip=1.2.3.115/24
      sudo /usr/sbin/pct set $ID -net0 name=eth0,bridge=vmbr0,firewall=0,gw=1.2.3.1,hwaddr=00:00:00:01:02:03,ip=1.2.3.115/24
    fi
    ;;
  *)
    exit 0
    ;;
esac

User "www-data" can sudo /usr/sbin/pct set *
Script works well when I run it by hand from user "www-data" but throw an error when it called by proxmox.

Output:

GUEST HOOK: 105, PHASE: pre-start
pct set 105 -net0 name=eth0,bridge=vmbr1,firewall=0,gw=1.2.3.1,hwaddr=00:00:00:01:02:03,ip=1.2.3.115/24
trying to acquire lock...
can't lock file '/run/lock/lxc/pve-config-105.lock' - got timeout
TASK ERROR: hookscript error for 105 on pre-start: command '/var/lib/vz/snippets/pct105-net0 105 pre-start' failed: exit code 4

Have somebody any idea how to automate changing network settings on container start?

 

[fabian]

the pre-start hook is called in a locked context, so it cannot do actions that in turn require obtaining the lock..

 

[vaka]

post-start hook also called in a locked context

So there is no way to automate lxc net config changing.

 

[vaka]

Finally I done it in a hack way. But it works.
I used "post-start" phase and simply change /etc/network/interfaces inside container.

#!/bin/bash

ID=$1
PHASE=$2
IMAGE="/rpool/subvol-${ID}-disk-0"

echo "GUEST HOOK: $ID, PHASE: $PHASE"

case $PHASE in
  post-start)
    if [[ "x$ID" == "x105" ]]; then
    cat <<EOF > $IMAGE/etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 1.2.3.105/24
        gateway 1.2.3.1
        hwaddress ether 00:01:02:03:04:05
EOF
    fi
    ;;
  *)
    exit 0
    ;;
esac

 

[Mirmanium]

the pre-start hook is called in a locked context, so it cannot do actions that in turn require obtaining the lock..

Do we have any "alternative" to this behaviour? a simple "remove a passthrough drive if not connected" can not be done using hookscripts at start time

 

[Mirmanium]

Do we have any "alternative" to this behaviour? a simple "remove a passthrough drive if not connected" can not be done using hookscripts at start time

Ok, found a way. I know needs to be done carefully but if you put a
nohup ./script.sh > /dev/null 2>&1 &

code inside script.sh does not have locked context and just gets executed.

Thank you,

 

[kRowone9]

Ok, found a way. I know needs to be done carefully but if you put a
nohup ./script.sh > /dev/null 2>&1 &

code inside script.sh does not have locked context and just gets executed.

Thank you,

@Mirmanium It does not work for me. When the container has a "valid" USB device (although the wrong one), it updates it correctly and then reboots to apply changes.
However when the device ID is invalid, the LxC won't boot and won't even execute my Pre-start hookscript!

Heres my hookscript:

#!/bin/bash
 
if [ $2 == "pre-start" ]
then
     echo "Preparing VM $1 before start"
     nohup /var/lib/vz/snippets/helpers/set-usb-device.sh $1 > /var/lib/vz/snippets/helpers/latest.log 2>&1 &
fi

And here's my helper script:

#!/bin/bash

if [ $1 == 104 ] # NUT VM
then
  echo "VM $1 (NUT) is starting"
  idVendor="051d"
  idProduct="0002"
elif [ $1 == 107 ] # Frigate VM
then
  echo "VM $1 (Frigate) is starting"
  idVendor="18d1"
  idProduct="9302"
fi

currentPath="$(pct config $1 | grep dev0 | cut -d' ' -f2)"
newPath="/dev/bus/usb/$(lsusb| grep $idVendor:$idProduct | cut -d ':' -f1 | cut -d ' ' -f2,4 | sed 's| |/|g')"

if [ "$currentPath" = "$newPath" ]
then
  echo "Path $currentPath is correctly set for VM $1"
else
  echo "Updating VM $1 dev0 to $newPath"
  pct set $1 --dev0 path=$newPath
 
  #Rebooting to apply changes
  pct reboot $1
fi

I see your use case was similar (identify if passthrough device is not connected and if so remove it). Doesn't it fail for you to start before even running the pre-start?

 

[Mirmanium]

@Mirmanium It does not work for me. When the container has a "valid" USB device (although the wrong one), it updates it correctly and then reboots to apply changes.
However when the device ID is invalid, the LxC won't boot and won't even execute my Pre-start hookscript!

Heres my hookscript:

#!/bin/bash
 
if [ $2 == "pre-start" ]
then
     echo "Preparing VM $1 before start"
     nohup /var/lib/vz/snippets/helpers/set-usb-device.sh $1 > /var/lib/vz/snippets/helpers/latest.log 2>&1 &
fi

And here's my helper script:

#!/bin/bash

if [ $1 == 104 ] # NUT VM
then
  echo "VM $1 (NUT) is starting"
  idVendor="051d"
  idProduct="0002"
elif [ $1 == 107 ] # Frigate VM
then
  echo "VM $1 (Frigate) is starting"
  idVendor="18d1"
  idProduct="9302"
fi

currentPath="$(pct config $1 | grep dev0 | cut -d' ' -f2)"
newPath="/dev/bus/usb/$(lsusb| grep $idVendor:$idProduct | cut -d ':' -f1 | cut -d ' ' -f2,4 | sed 's| |/|g')"

if [ "$currentPath" = "$newPath" ]
then
  echo "Path $currentPath is correctly set for VM $1"
else
  echo "Updating VM $1 dev0 to $newPath"
  pct set $1 --dev0 path=$newPath
 
  #Rebooting to apply changes
  pct reboot $1
fi

I see your use case was similar (identify if passthrough device is not connected and if so remove it). Doesn't it fail for you to start before even running the pre-start?

What returns echo in your scripts? Mines everything o ln pre-start

 

[kRowone9]

What returns echo in your scripts? Mines everything o ln pre-start

If device is incorrect and does not exist, no echos returned at all since the pre-start hook does not even execute.

The only output I get is: TASK ERROR: Device /dev/bus/usb/001/062 does not exist



If I then manually update the device to any device such as /dev/bus/usb/001/001 (even if its not the correct one, as long as it exists) then I get a valid output and the device ID is updated to the correct device, and the container rebooted:


I'd need the hook to always execute (even if the device ID does not exist). Someone suggested in another thread to add a post-stop routine that removes the device when the container is turned off. This way, the device would always start properly and the hook would always be executed.

The problem with this solution, is that once I setup the device properly, I reboot the container. So the device would be removed in that reboot and added again, in an infinite loop.

 

[Mirmanium]

If device is incorrect and does not exist, no echos returned at all since the pre-start hook does not even execute.

The only output I get is: TASK ERROR: Device /dev/bus/usb/001/062 does not exist

View attachment 79213

If I then manually update the device to any device such as /dev/bus/usb/001/001 (even if its not the correct one, as long as it exists) then I get a valid output and the device ID is updated to the correct device, and the container rebooted:
View attachment 79215

I'd need the hook to always execute (even if the device ID does not exist). Someone suggested in another thread to add a post-stop routine that removes the device when the container is turned off. This way, the device would always start properly and the hook would always be executed.

The problem with this solution, is that once I setup the device properly, I reboot the container. So the device would be removed in that reboot and added again, in an infinite loop.

I have both hooks, post-start and pre-stop in the same hookscript.sh.
- pre-start look for the disk if exist and if so, It attach to the vm I need.
- pre-stop detach the disk from the vm.

Important to mention that only works using start or shutdown commands using VM. If you reboot within the VM, then won't be triggered.

 

[Mirmanium]

I can not confirm if its because of using USB device. My script works with a physical disk attached.

 

[kRowone9]

I have both hooks, post-start and pre-stop in the same hookscript.sh.
- pre-start look for the disk if exist and if so, It attach to the vm I need.
- pre-stop detach the disk from the vm.

Important to mention that only works using start or shutdown commands using VM. If you reboot within the VM, then won't be triggered.

The thing is that I'm trying to use it for containers. So once the LxC is on, there is no way to "hot"-attach the USB device, and once attached I have to reboot the container so that the resource change is effective and USB recognized. Because of this, if I were to put a pre-stop detach, then it would detach it also during reboot.

I'll think of something and if I come up with a solution, share it here. I have two main options being considered:

• Forget about hooks and directly place an init.d script so that the script is always executed, and before the container tries to start (so that reboot is no needed), and pct can be directly invoked without separate scripts
• When disk is assigned (needs 1 reboot), and post-stop checks if .reboot file is created and if so then it does not remove the device.

Second approach would keep using pre/post hooks, but I don't fully like it since it depends too much in everything being created/removed correctly. I'll test around.

 

[Mirmanium]

Maybe way of working with LXC is different... I can not support here. If you find any solution, would be great to know so can help others.
Regards,