proxmox cluster live migration hooks and xxx.conf files

[SimonMcNair]

Hi,
I use exec-cmds pre-start to mount virtio from an nfs share. I am finding that when I do a live migration the 106.conf does not exist on the node at the point that I need it ?

19:53:08 Action phase: pre-start
/mnt/pve/NFSShare/snippets/exec-cmds: line 46: /etc/pve/qemu-server/106.conf: No such file or directory
QEMU: kvm: -chardev socket,id=char0,path=/var/run/qemu-server/106-vhost-fs.sock: Failed to connect to '/var/run/qemu-server/106-vhost-fs.sock': No such file or directory

TASK ERROR: start failed: QEMU exited with code 1

Is this as per design please ?

 

[fiona]

Hi,
yes, the VM needs to be started on the target node while it's still running on the source node, so it can prepare and take over the state. But you should be able to access the config file via /etc/pve/nodes/<source node>/qemu-server/106.conf on the target node.

 

[SimonMcNair]

Hi,
yes, the VM needs to be started on the target node while it's still running on the source node, so it can prepare and take over the state. But you should be able to access the config file via /etc/pve/nodes/<source node>/qemu-server/106.conf on the target node.

Hi,
Thank you for the clarification. This does not sound right from a design perspective? The live folder on the running host is as per the script. The clone is running so the file should be in the folder shouldn't it ? Evaluating the host you're running on seems needlessly complicated?

Once the node is migrated I assume the file is in that directory which begs the question are the hooks being executed at the correct time on a migration please ?

 

[fiona]

Hi,
Thank you for the clarification. This does not sound right from a design perspective? The live folder on the running host is as per the script. The clone is running so the file should be in the folder shouldn't it ? Evaluating the host you're running on seems needlessly complicated?

Once the node is migrated I assume the file is in that directory which begs the question are the hooks being executed at the correct time on a migration please ?

The cluster filesystem is designed to avoid having to configuration files with the same ID. Avoiding duplicates for things that should be unique is good And the VM still resides on the source node, so the configuration should still be in the directory for the source node. It will be moved to the directory for the target node, once the migration of the state to the target is done.

 

[SimonMcNair]

I appreciate that, but that means that the pre hook will never work for a migration ?

 

[SimonMcNair]

I appreciate that, but that means that the pre hook will never work for a migration ?

Sorry, it will work but for nothing that needs the xxx.conf to exist. This seems like a bug to me ?

 

[fiona]

Sorry, it will work but for nothing that needs the xxx.conf to exist. This seems like a bug to me ?

Many hook-scripts don't need to read the config. If you do, what's the issue with accessing it via the /etc/pve/nodes/<node>/qemu-server/<ID>.conf path? I do agree that it might be a bit unintuitive, but I hope you understand why it's technically sensible to move the config file only when the migration is finished.

 

[fabian]

there are patches on the list that will add pre- and post-migration hook phases that might help with this and similar issues. but your specific issue is already trivial to solve with the existing code:

the hookscript execution in the pre- and post-start phase as part of a migration have a PVE_MIGRATED_FROM environment variable set with the source node as value, so you can just check that and load the config via the source-node path if it is set.. it's there for exactly this reason (and also used by the network scripts that are executed when starting the VM for exactly this purpose..)

 

[SimonMcNair]

Many hook-scripts don't need to read the config. If you do, what's the issue with accessing it via the /etc/pve/nodes/<node>/qemu-server/<ID>.conf path? I do agree that it might be a bit unintuitive, but I hope you understand why it's technically sensible to move the config file only when the migration is finished.

In the case of what I'm doing the script provides a link to a mount point within the vm. It needs to be attached as soon as possible.

I guess the other point is that it means that a migration doesn't run the hooks at the same point as an initial start up which could be inconsistent ?

Is it worth having a chat with an architect to get their design opinion on it ?

Cheers
Simon

 

[SimonMcNair]

there are patches on the list that will add pre- and post-migration hook phases that might help with this and similar issues. but your specific issue is already trivial to solve with the existing code:

the hookscript execution in the pre- and post-start phase as part of a migration have a PVE_MIGRATED_FROM environment variable set with the source node as value, so you can just check that and load the config via the source-node path if it is set.. it's there for exactly this reason (and also used by the network scripts that are executed when starting the VM for exactly this purpose..)

Thank you. I'll look in to this.

 

[fiona]

In the case of what I'm doing the script provides a link to a mount point within the vm. It needs to be attached as soon as possible.

I guess the other point is that it means that a migration doesn't run the hooks at the same point as an initial start up which could be inconsistent ?

The pre-start hook always runs before the VM instance is started, so where is the inconsistency? That has to happen before the state can be migrated. And that happens before the configuration is moved to the new node.

Is it worth having a chat with an architect to get their design opinion on it ?

I already gave you some rationale for the design.

 

[SimonMcNair]

Thank you for the replies. I appreciate your effort. It appears it didn't achieve any result for me anyway.

2022-09-28 10:08:47 starting VM 106 on remote node 'test'
2022-09-28 10:08:49 start remote tunnel
2022-09-28 10:08:50 ssh tunnel ver 1
2022-09-28 10:08:50 starting online/live migration on unix:/run/qemu-server/106.migrate
2022-09-28 10:08:50 set migration capabilities
2022-09-28 10:08:50 migration downtime limit: 100 ms
2022-09-28 10:08:50 migration cachesize: 256.0 MiB
2022-09-28 10:08:50 set migration parameters
2022-09-28 10:08:50 start migrate command to unix:/run/qemu-server/106.migrate
2022-09-28 10:08:50 migrate uri => unix:/run/qemu-server/106.migrate failed: VM 106 qmp command 'migrate' failed - State blocked by non-migratable device '0000:00:04.0/vhost-user-fs'
2022-09-28 10:08:51 ERROR: online migrate failure - VM 106 qmp command 'migrate' failed - State blocked by non-migratable device '0000:00:04.0/vhost-user-fs'
2022-09-28 10:08:51 aborting phase 2 - cleanup resources
2022-09-28 10:08:51 migrate_cancel
2022-09-28 10:08:54 ERROR: migration finished with problems (duration 00:00:08)

I thought that having the virtiofsd on an nfs share would enable it to move but I guess I was wrong.

Thank you for your efforts, patience, and explanation.

 

[SimonMcNair]

If you're wondering why I don't just mount the remote nfs share inside the guest there is a rationale. It is on an isolated network with no visibility of any other machine, but I want to mount some data within it for docker.

 

[hk@]

Late reply as I’m also hunting for migration hooks.
For compartmenting networking for some VMs you might consider bridges bound to only this network environment only doing layer2 for your VM which would in turn have to do its own networking.

 

[hk@]

well, thanks to @fabian - the environment variable PVE_MIGRATED_FROM is in fact a game changer for hookscripts.
this way we can detect during pre-startup if cleaning up at the former host is needed and script some ssh-calls on the former host to remove now inactive configurations. not totally trivial, but a way that helped us finally achieve live Layer 3 migration from one node to another. thx!