can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout

A

Akubra

New Member
Jul 30, 2023
9
0
1
I'm using Veeam 12.3 to backup VMs on Proxmox, to do that Veeam is using a worker VM. Every couple of days backup fails as the worker can't be found:
Failed to prepare the worker <WORKERNAME>: Failed to synchronize configuration settings of the worker <WORKERNAME> can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout;

If I look at Proxmox, I can see the worker VM still has a status of starting. Only way to fix that is run 'systemctl restart pvedaemon' from the Proxmox console.
Once this process is killed I can go Veeam and run initiate a test for the worker and that completes successfully and for the next couple of days backups work fine.

Reached out to Veeam but they think it is a Proxmox issue.
Have also created a new Veeam worker VM but issue does come back.

Tried:
lsof /var/lock/qemu-server/lock-104.conf
ps -f -p <PID>
but that did not give me a lot of info:
task UPID:<PROXMOXSERVER>:003A0684:16B

Not sure what next steps to take.
 
hi,

is there still a running task for that vm?

if the lock can't be aquired, then it's most likely conflicting with another action that's currently running (such as starting, backup, etc.)

Akubra said:
Reached out to Veeam but they think it is a Proxmox issue.
Click to expand...
if they think it's a proxmox ve issue, they should reach out (e.g. on our developer list/bug tracker/etc)
they have a much easier way to debug proxmox ve (since it's open source) than we have to debug veeam...
 
Thanks Dominik, this worker VM is not running 24/7. As soon as Veeam starts the backup job, it starts up the worker VM, does the backup job and shuts down the VM again. This works for a couple of days (1 backup job per day) but then it throws the error.
Regardless of Veeam responding, how can I check what else might be locking the file, preventing a VM from starting.

As mentioned, I tried (which as recommended by Veeam):
lsof /var/lock/qemu-server/lock-104.conf
which returns a <PID>

and then run:
ps -f -p <PID from previous CMD>

but that did not give me a lot of info:
task UPID:<PROXMOXSERVER>:003A0684:16B

Maybe it does but I'm not interpreting it correctly.
 
Akubra said:
task UPID:<PROXMOXSERVER>:003A0684:16B
Click to expand...
this corresponds to a worker task, and if it still is running, this should be visible on the ui (on the bottom) as a running task
if it's not running anymore, it should still be there, you can get e.g. the status from the 'upid' with

Code: 
pvenode task status <upid>

or show all running tasks on the commandline with:

Code: 
pvenode task list --source active
 
  • Like
Reactions: Akubra
Akubra said:
I'm using Veeam 12.3 to backup VMs on Proxmox, to do that Veeam is using a worker VM. Every couple of days backup fails as the worker can't be found:
Failed to prepare the worker <WORKERNAME>: Failed to synchronize configuration settings of the worker <WORKERNAME> can't lock file '/var/lock/qemu-server/lock-104.conf' - got timeout;

If I look at Proxmox, I can see the worker VM still has a status of starting. Only way to fix that is run 'systemctl restart pvedaemon' from the Proxmox console.
Once this process is killed I can go Veeam and run initiate a test for the worker and that completes successfully and for the next couple of days backups work fine.
Click to expand...

I have exact the same issue.

Code: 
systemctl restart pvedaemon

Helped that the worker-VM leave the "try to start" problem, after the service restart Veeam is abele to start the VM. But not to configure.
I still have to restart the PVE host, then it works again. for 1 to 3 the days.

Were you able to solve the problem?

Regards,
 
BananaJoeX said:
I have exact the same issue.

Code: 
systemctl restart pvedaemon

Helped that the worker-VM leave the "try to start" problem, after the service restart Veeam is abele to start the VM. But not to configure.
I still have to restart the PVE host, then it works again. for 1 to 3 the days.

Were you able to solve the problem?

Regards,
Click to expand...
Only way this fixed it for me was to have this worker VM run 24/7 instead of start/stop being triggered by the backup job.
So far so good
 
im trying to use cron job.
0 18 * * * systemctl restart pvedaemon

runs it every day at 1800 which is about 2 hours before backups are set to start.

dunno what are the ramifications of it
 
venicius_14 said:
Hello,

I have the same problem.

Can you share how you managed to keep the worker on permanently?
Click to expand...

As a workaround I do not let Veeam control the starting and stopping of the Proxmox worker VM.
Instead I leave the VM running all times.


  1. On the Veeam Backup & Replication (VBR) server, open a command prompt as admin
  2. Open the Proxmox PVE plug-in configuration file:
    notepad 'C:\Program Files\Veeam\Plugins\PVE\Service\appsettings.json'
  3. Find the Workers section
  4. Set the KeepTurnedOn value to true
  5. Save and close the file
  6. Close the VBR console
  7. Open services.msc and restart the Veeam PVE Service
  8. Re-open the VBR console and test the backup behaviour
 
jaredfonteles said:
im trying to use cron job.
0 18 * * * systemctl restart pvedaemon

runs it every day at 1800 which is about 2 hours before backups are set to start.

dunno what are the ramifications of it
Click to expand...
running the cron jobs worked for me. i have made it so they do not run at the same time on each node though. the proxies start and stop on their own as needed by veeam. i did not need to keep them running, and i do not have to keep checking if they are still running, as they behave as intended by veeam, and torn on, run the backups and turn off
 
You must log in or register to reply here.
Top Bottom