Duplicate vmid's?

[cronistar]

Hi Friends,
We have a 21-node cluster with a few thousand VMs; most are powered off (they're used for students to practice networking, computer science, etc, not actual production workloads). We do a lot of scripting to provision machines out for the students and we've noticed we're getting duplicate vmid's.

We're using proxmoxer for the majority of our provisioning, right before copying the vm, we make a call to cluster/nextid to get the next unique vmid-- but the value we're getting back isn't always unique. It tends to be the vmid of a VM we just created.



We're not cloning VMs asynchronously, we wait for the previous clone operation to finish before getting the next vmid & starting to copy the next vm. Every node in the cluster is running 8.2.0 as well. We do occasionally get a file lock issue

cfs-lock 'file-user_cfg' error: got lock request timeout

but those don't seem to correlate to the duplicate vmids.

Does anyone have advice as to the logs I could look into to see what could be causing this? It feels like the cluster is out of sync, but we aren't seeing any other major problems.

 

[sw-omit]

Probably the first things I would personally check would be the pvecm status and the system log specifically the part about corosync if it maybe goes out of sync there.
Since it is a large cluster, do you have the primary link on a completely separate port / switch / subnet (not just vlan) that you use only for that and nothing else? (Not even accessing the management-interface) And have you checked that the latency between all of the nodes to all of the nodes is below 5 ms consistently?

Also, since I don't know how proxmoxer works, does it always does the requests to 1 specific node or to "a node in the cluster"? If the former, tried changing it to one of the other nodes? If the later, is it possible/have you tried forcing it to use only 1 or 2 nodes?