Lock files are not removed automatically.

[mmuratusta]

Hello,

When creating or cloning a server in Proxmox, a lock file is generated under /var/lock/qemu-server. These lock files should be deleted automatically once the process is completed so that we can configure VM settings via the API. However, in our case, the lock files are not being removed automatically. This issue is occurring across almost all of our servers located in different regions even in newly installed ones.

Here are the steps I've tried:

1. qm unlock <vmid> - This command does not remove the lock files.
   
   
   

   root@pve01:/var/lock/qemu-server# ls -l
   total 0
   -rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-100.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:06 lock-101.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:25 lock-105.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:31 lock-107.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-265.conf
   
   root@pve01:/var/lock/qemu-server# qm unlock 105
   
   root@pve01:/var/lock/qemu-server# ls -l
   total 0
   -rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-100.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:06 lock-101.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:25 lock-105.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:31 lock-107.conf
   -rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-265.conf

2. qm destroy <vmid> - This command also does not remove the lock files.

root@pve01:/var/lock/qemu-server# ls -l
total 0
-rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-100.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:06 lock-101.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:25 lock-105.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:31 lock-107.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-265.conf

root@pve01:/var/lock/qemu-server# qm destroy 107
  Logical volume "vm-107-disk-0" successfully removed.

root@pve01:/var/lock/qemu-server# ls -l
total 0
-rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-100.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:06 lock-101.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:25 lock-105.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:31 lock-107.conf
-rwxr-xr-x 1 root root 0 Jun 28 03:05 lock-265.conf

I am able to manually remove lock files.

Is this a common issue in Proxmox v8.2.4? How can I resolve this problem?

Thank you for your help!

 

[t.lamprecht]

Lock files do not need to be removed, a lock is not defined by a file existing, rather it's a separate state for a file.
And using, e.g., the flock syscall one can place locks and unlock them again later on an (already existing!) file descriptor.

Note that Proxmox VE uses two separate locking mechanisms for virtual guests, one for local short-lived "atomic" operations. Here, the files you see are used, and one for longer running jobs (like backup, migrate, ...), that one is saved inside the VM config itself.
The latter helps to hedge, e.g., against a power outage, as that would drop any local locks, so marking the resource as locked in the pmxcfs clustered configuration system.

The atomic operation will always be get unlocked automatically once the action is finished.

qm unlock <vmid> - This command does not remove the lock files.

This command is removing any lock property from the VM config (content), it does not remove any files, as that's simply not required.
But it indeed uses the lockfile to safely remove that property, so as that command does not throw any error, and also uses the API, it's clear that file-locking works as expected in your case.

What is the actual issue you're seeing?

 

[mmuratusta]

Hello Thomas,

thank you for detailed information. I am trying to change VM config after cloning but I get following error. Since the error message says `can't lock file '/var/lock/qemu-server/lock-265.conf'`, I thought it might be because this file already exists.

PVE2 API: This API Request Failed. HTTP CODE: 500, HTTP ERROR: HTTP/1.1 500 can't lock file '/var/lock/qemu-server/lock-265.conf' - got timeout

I am posting this information

/nodes/pve01/qemu/265/config

Array
(
    [memory] => 2048
    [ostype] => l26
    [sockets] => 1
    [cores] => 1
    [cpu] => host
    [kvm] => 1
    [onboot] => 1
)

Source VM Config

root@pve01:/etc/pve/qemu-server# cat 100.conf
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=9.0.0,ctime=1719535713
name: test
net0: virtio=BC:24:11:16:76:BA,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: pve01.bursa:vm-100-disk-0,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=3016f205-de5c-4d8a-a0f9-acacd43fee94
sockets: 1
vmgenid: f9d48e73-cbee-4e06-8f7a-934b781a381e

Copied VM Config

root@pve01:/etc/pve/qemu-server# cat 265.conf
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=9.0.0,ctime=1719535713
name: vps265-cus2
net0: virtio=BC:24:11:AD:6A:6B,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: pve01.bursa:vm-265-disk-0,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=da8546e2-3678-4cb6-aca2-0730f417d5e7
sockets: 1
vmgenid: c6090606-a70e-4f06-a3e4-82204b48ffd4

 

[fiona]

Hi,

thank you for detailed information. I am trying to change VM config after cloning but I get following error. Since the error message says `can't lock file '/var/lock/qemu-server/lock-265.conf'`, I thought it might be because this file already exists.

PVE2 API: This API Request Failed. HTTP CODE: 500, HTTP ERROR: HTTP/1.1 500 can't lock file '/var/lock/qemu-server/lock-265.conf' - got timeout

that most likely means some other task is already holding the lock. Do you wait for the clone API call to finish? It spawns a task and returns a UPID which you can use to query the task status, i.e. /nodes/{node}/tasks/{upid}/status.