crit: unable to acquire pmxcfs lock: Resource temporarily unavailable

[Dirky_uk]

Hoping somebody can point me in the right direction.

Today I upgraded the RAM in each of the 3 nodes of my cluster. I am using Proxmox 8.1.4 on 3 x HP mini PC's.
All was working well until I noticed one of the LXC was not restarting.
I realised my error, I beleive I had set it has HA but not figured the storage out correctly. It appears the container had moved but left the storage behind. When I attempted to move the container back to the original node, I received an error.
So here is what happens now. If I attempt to start the container 400 on node mary:-

()
task started by HA resource agent
TASK ERROR: no such logical volume pve/vm-400-disk-

If I then disable the ha of this container at datacenter level, then try to migrate the container to midge (which is where the storage is)
It appears to work

()
Requesting HA migration for CT 400 to node midge
TASK OK

And then it fails with

()
task started by HA resource agent
2024-05-06 19:27:53 starting migration of CT 400 to node 'midge' (192.168.1.212)
2024-05-06 19:27:53 found local volume 'local-lvm:vm-400-disk-0' (in current VM config)
2024-05-06 19:27:54 ERROR: storage migration for 'local-lvm:vm-400-disk-0' to storage 'local-lvm' failed - no such logical volume pve/vm-400-disk-0
2024-05-06 19:27:54 aborting phase 1 - cleanup resources
2024-05-06 19:27:54 ERROR: found stale volume copy 'local-lvm:vm-400-disk-0' on node 'midge'
2024-05-06 19:27:54 start final cleanup
2024-05-06 19:27:54 ERROR: migration aborted (duration 00:00:02): storage migration for 'local-lvm:vm-400-disk-0' to storage 'local-lvm' failed - no such logical volume pve/vm-400-disk-0
TASK ERROR: migration aborted

So, if I look at the volumes on midge, I see:-

Howver, if I look to see the config file for 400, I see:-


So it appears there is no config file for 400 ? I figure I will create one based on 401.conf, however it won't allow me to copy the 401.conf to 400.conf ?



So I am a bit unsure where to go from here. I do have a feeling I may have inadvertantly forced the shutdown of one of the nodes during the ram upgrades but not sure if that has broken things.
Any ideas?
Thanks!
D

 

[Dirky_uk]

Just some more info in case it helps.

 

[gfngfn256]

Don't exactly know what's up with your HW or SW, but for HA you need shared storage as per docs.
What is your storage setup? You obviously have some storage errors.

IIWY I would hope to have a full & restorable backup of the said CT400 - I'd delete it completely from cluster - & then after setting up my HA HW&SW cluster correctly, restore the CT400 from backup. Thats just my personal 2 cents. However as I said IDW what's really going on at your end.

 

[Dirky_uk]

Don't exactly know what's up with your HW or SW, but for HA you need shared storage as per docs.
What is your storage setup? You obviously have some storage errors.

IIWY I would hope to have a full & restorable backup of the said CT400 - I'd delete it completely from cluster - & then after setting up my HA HW&SW cluster correctly, restore the CT400 from backup. Thats just my personal 2 cents. However as I said IDW what's really going on at your end.

Yeah, not bothered about the ha setup currently. I'd just like to get this container running again with its storage and leave it one one of the nodes.

 

[gfngfn256]

So as I said - remove it completely - & restore it from a working backup. Once restored - don't start it - remove any HA settings etc. & then restart.