Can't activate LV

[hac3ru]

Hello,

I'm running a 3-node PVE 8.0.4 cluster and I have the following error when trying to migrate a VM from one host to another:

task started by HA resource agent
2023-11-21 07:18:52 use dedicated network address for sending migration traffic (10.41.199.33)
2023-11-21 07:18:52 starting migration of VM 100000 to node 'ZRH-GLT-CS03' (10.41.199.33)
2023-11-21 07:18:52 starting VM 100000 on remote node 'ZRH-GLT-CS03'
2023-11-21 07:18:53 [ZRH-GLT-CS03] can't activate LV '/dev/NAS_VG1/vm-100000-disk-0':   device-mapper: create ioctl on NAS_VG1-vm--100000--disk--0 LVM-5YfbLpbU7mwS4wUHeM1vAfSMXTyemeIOPEaddO7heGZpJyHRV5D536tNUUlduOzO failed: Device or resource busy
2023-11-21 07:18:53 ERROR: online migrate failure - remote command failed with exit code 255
2023-11-21 07:18:53 aborting phase 2 - cleanup resources
2023-11-21 07:18:53 migrate_cancel
2023-11-21 07:18:54 ERROR: migration finished with problems (duration 00:00:02)
TASK ERROR: migration problems

The VM resides on a shared storage - LVM iSCSI. I do know about the "solution" of running

dmsetup remove /dev/NAS_VG1/vm-100000-disk-0

but this is quite a pain. What I'm trying to do is to migrate about 30 - 40 VMs using the bulk migration feature. Well, most of the VMs return an error just like this one. Why is it happening, what can I do to stop it from happening and, wouldn't it be a good idea to have

dmsetup remove

automatically, when this error is triggered, and retry the power-on? I see no downside to this, but maybe I'm missing something.

Thank you.

Later edit: Some machines are using HA with a certain group. That is a group of 1 host, since I want to have certain machines on certain hosts. This causes an issue when using Bulk migration because, as soon as the VMs are moved to the target host, they come back to the host the HA dictates. I would say that this is an unwanted behavior so as a suggestion, I would love to either have the option to tick a box when using Bulk migration saying "Ignore HA Groups for this migration", which would set the HA state as ignore.

 

[Azunai333]

Why are you migrating the VMs?

If you want to do maintenance on the host, you could enable the maintenance mode of the HA service for that node via CLI:
ha-manager crm-command node-maintenance enable HOSTNAME
and to disable:
ha-manager crm-command node-maintenance disable HOSTNAME

With this all VMs with active HA are migrated elsewhere. To make this work there must be another node which the VMs can migrate to. This is defined in your HA-group.
Afterwards you can do your bulk migration of non-HA-VMs.

Edit:
If you sill want to migrate manually, the HA-group option "nofailback" might give you what you want:

nofailback: <boolean> (default = 0)
The CRM tries to run services on the node with the highest priority. If a node with higher priority comes online, the CRM migrates the service to that node. Enabling nofailback prevents that behavior.

 

[hac3ru]

Hello @Azunai333 and thank you for the reply.

Ok, that fixes the HA stuff. It's actually pretty great, I just wished this option would be available in the GUI, but I guess that'll come at some point. Any clue about the storage issues?

 

[Azunai333]

option would be available in the GUI

There is one for the nofailback in the dialog for creating or editing a HA group on the top right:

Any clue about the storage issues?

I don't have experience with iSCSI, so no, I'm afraid I can't help you with this.

 

[hac3ru]

There is one for the nofailback in the dialog for creating or editing a HA group on the top right:

I was referring to the HA maintenance mode, that it would be great to have access to it via GUI.

I don't have experience with iSCSI, so no, I'm afraid I can't help you with this.

Thank you anyways for the time spent so far. The HA Maintenance mode really helps when updates are to be performed.