can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout

A

Akubra

New Member
Jul 30, 2023
7
0
1
I'm using Veeam 12.3 to backup VMs on Proxmox, to do that Veeam is using a worker VM. Every couple of days backup fails as the worker can't be found:
Failed to prepare the worker <WORKERNAME>: Failed to synchronize configuration settings of the worker <WORKERNAME> can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout;

If I look at Proxmox, I can see the worker VM still has a status of starting. Only way to fix that is run 'systemctl restart pvedaemon' from the Proxmox console.
Once this process is killed I can go Veeam and run initiate a test for the worker and that completes successfully and for the next couple of days backups work fine.

Reached out to Veeam but they think it is a Proxmox issue.
Have also created a new Veeam worker VM but issue does come back.

Tried:
lsof /var/lock/qemu-server/lock-104.conf
ps -f -p <PID>
but that did not give me a lot of info:
task UPID:<PROXMOXSERVER>:003A0684:16B

Not sure what next steps to take.
 
hi,

is there still a running task for that vm?

if the lock can't be aquired, then it's most likely conflicting with another action that's currently running (such as starting, backup, etc.)

Akubra said:
Reached out to Veeam but they think it is a Proxmox issue.
Click to expand...
if they think it's a proxmox ve issue, they should reach out (e.g. on our developer list/bug tracker/etc)
they have a much easier way to debug proxmox ve (since it's open source) than we have to debug veeam...
 
Thanks Dominik, this worker VM is not running 24/7. As soon as Veeam starts the backup job, it starts up the worker VM, does the backup job and shuts down the VM again. This works for a couple of days (1 backup job per day) but then it throws the error.
Regardless of Veeam responding, how can I check what else might be locking the file, preventing a VM from starting.

As mentioned, I tried (which as recommended by Veeam):
lsof /var/lock/qemu-server/lock-104.conf
which returns a <PID>

and then run:
ps -f -p <PID from previous CMD>

but that did not give me a lot of info:
task UPID:<PROXMOXSERVER>:003A0684:16B

Maybe it does but I'm not interpreting it correctly.
 
Akubra said:
task UPID:<PROXMOXSERVER>:003A0684:16B
Click to expand...
this corresponds to a worker task, and if it still is running, this should be visible on the ui (on the bottom) as a running task
if it's not running anymore, it should still be there, you can get e.g. the status from the 'upid' with

Code: 
pvenode task status <upid>

or show all running tasks on the commandline with:

Code: 
pvenode task list --source active
 
  • Like
Reactions: Akubra
Akubra said:
I'm using Veeam 12.3 to backup VMs on Proxmox, to do that Veeam is using a worker VM. Every couple of days backup fails as the worker can't be found:
Failed to prepare the worker <WORKERNAME>: Failed to synchronize configuration settings of the worker <WORKERNAME> can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout;

If I look at Proxmox, I can see the worker VM still has a status of starting. Only way to fix that is run 'systemctl restart pvedaemon' from the Proxmox console.
Once this process is killed I can go Veeam and run initiate a test for the worker and that completes successfully and for the next couple of days backups work fine.
Click to expand...

I have exact the same issue.

Code: 
systemctl restart pvedaemon

Helped that the worker-VM leave the "try to start" problem, after the service restart Veeam is abele to start the VM. But not to configure.
I still have to restart the PVE host, then it works again. for 1 to 3 the days.

Were you able to solve the problem?

Regards,
 
You must log in or register to reply here.
Top Bottom