VM is locked , Cannot unlock

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
Hello all ,

I've faced to problem with starting one my VM's in proxmox , before that I did turn off dedicated server (of curse before shutdown I turned off all CT's and VM's) then I returned back server and When I tried to start VM I got this error: VM is locked (backup) so I did try qm unlock VMID but I got an error with following this message:
Code:
Code:
unable to open file '/etc/pve/nodes/node1/qemu-server/102.conf.tmp.252157' - Permission denied

At first , When I run the hypervisor's node I did setup a cluster with Two-node but after that I did turned off second node and run this command for solve quorum error :
Code:
pvecm expected 1
Then server works just fine , But after that reboot I cannot start one of my VM's because of this error
VM is locked (backup) and Now I cannot backup or unlock any of my VM's becuase of this error:
Code:
Code:
[B]unable to open file '/etc/pve/nodes/node1/qemu-server/102.conf.tmp.252157' - Permission denied
[/B]
Any suggestion please? :confused::confused::confused::confused:
 

udo

Famous Member
Apr 22, 2009
5,935
185
93
Ahrensburg; Germany
Hello all ,

I've faced to problem with starting one my VM's in proxmox , before that I did turn off dedicated server (of curse before shutdown I turned off all CT's and VM's) then I returned back server and When I tried to start VM I got this error: VM is locked (backup) so I did try qm unlock VMID but I got an error with following this message:
Code:
Code:
unable to open file '/etc/pve/nodes/node1/qemu-server/102.conf.tmp.252157' - Permission denied

At first , When I run the hypervisor's node I did setup a cluster with Two-node but after that I did turned off second node and run this command for solve quorum error :
Code:
pvecm expected 1
Then server works just fine , But after that reboot I cannot start one of my VM's because of this error
VM is locked (backup) and Now I cannot backup or unlock any of my VM's becuase of this error:
Code:
Code:
[B]unable to open file '/etc/pve/nodes/node1/qemu-server/102.conf.tmp.252157' - Permission denied
[/B]
Any suggestion please? :confused::confused::confused::confused:
Hi,
are both nodes running?
Do you have quorum? is /etc/pve is writable?
Code:
touch /etc/pve/dummy
rm /etc/pve/dummy
How looks your quorum?
Code:
pvecm status
Udo
 

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
Hi,
are both nodes running?
Do you have quorum? is /etc/pve is writable?
Code:
touch /etc/pve/dummy
rm /etc/pve/dummy
How looks your quorum?
Code:
pvecm status
Udo
Hi Udo ,

No just one of nodes is running , like before.
Permission denied When I tried to make dummy file, also pvecm is stopped because cman is not running and cman cannot start because of this error:
Code:
Starting cluster: 
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... Cannot find node name in cluster.conf
Unable to get the configuration
Cannot find node name in cluster.conf
cman_tool: corosync daemon didn't start Check cluster logs for details
[FAILED]
 

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
Also as I mentioned before , I decluster servers a long time ago :D
and here is my cluster.conf now:
Code:
<?xml version="1.0"?>
<cluster name="[B]Dummy Name[/B]" config_version="4">


  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>


  <clusternodes>


  </clusternodes>


</cluster>
It's seems so empty, right? ^_^

Also It's not possible to make changes in ssh folder , same error for writing permission and cannot chand chmod of files.

Also It's my mount details:
Code:
root@node1:/etc/ssh# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                      10M     0   10M   0% /dev
tmpfs                    6.7G  428K  6.7G   1% /run
/dev/mapper/pve-root      95G   51G   40G  57% /
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                     14G   31M   14G   1% /run/shm
/dev/fuse                 30M   24K   30M   1% /etc/pve
/dev/mapper/pve-data     1.7T  300G  1.4T  19% /var/lib/vz
/dev/sdb1                1.8T  1.6T  216G  89% /sdb
/var/lib/vz/private/109   50G  1.8G   49G   4% /var/lib/vz/root/109
none                     3.0G  4.0K  3.0G   1% /var/lib/vz/root/109/dev
none                     4.0K     0  4.0K   0% /var/lib/vz/root/109/sys/fs/cgroup
none                     615M  112K  615M   1% /var/lib/vz/root/109/run
none                     5.0M     0  5.0M   0% /var/lib/vz/root/109/run/lock
none                     3.0G     0  3.0G   0% /var/lib/vz/root/109/run/shm
none                     100M     0  100M   0% /var/lib/vz/root/109/run/user
 
Last edited:

udo

Famous Member
Apr 22, 2009
5,935
185
93
Ahrensburg; Germany
Also as I mentioned before , I decluster servers a long time ago :D
and here is my cluster.conf now:
Code:
<?xml version="1.0"?>
<cluster name="[B]Dummy Name[/B]" config_version="4">


  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>


  <clusternodes>


  </clusternodes>


</cluster>
It's seems so empty, right? ^_^
...
Hi,
your "declustering" was wrong! An pve-non-cluster-system don't have an /etc/pve/cluster.conf.

The official way is an new installation... but I have seen a long time ago an thread about successfull declustering.

Udo
 

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
Hi,
your "declustering" was wrong! An pve-non-cluster-system don't have an /etc/pve/cluster.conf.

The official way is an new installation... but I have seen a long time ago an thread about successfull declustering.

Udo
So clustering in PVE is a one way road and cannot return ?
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
972
173
So clustering in PVE is a one way road and cannot return ?

if you have a single node now, just delete the cluster.conf.

> rm /etc/pve/cluster.conf
> rm -r /etc/cluster
> reboot
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
972
173
you cannot do the following?:

> rm -r /etc/cluster
 

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
you cannot do the following?:

> rm -r /etc/cluster
Here for cluster.conf
Code:
root@node1:~# rm -r /etc/pve/cluster.conf
rm: cannot remove `/etc/pve/cluster.conf': Permission denied
but rm /etc/cluster is ok and now I checked there is no /etc/cluster anymore.

Also my biggest problem is I cannot unlock the VM , I'm getting this error:
Code:
root@node1:~# qm unlock 103
unable to open file '/etc/pve/nodes/node1/qemu-server/103.conf.tmp.416266' - Permission denied
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
972
173
try to stop cman:

> service stop cman

remove the cluster.conf from /etc/pve:

> rm /etc/pve/cluster.conf

reboot.
 

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
try to stop cman:

> service stop cman

remove the cluster.conf from /etc/pve:

> rm /etc/pve/cluster.conf

reboot.
Hello ,

When I run this command:
Code:
service cman stop
There is no output , and about remove cluster.conf , I got this error:
Code:
root@node1:/etc/pve# rm -rf cluster.conf
rm: cannot remove `cluster.conf': Permission denied
As I mentioned before , I cannot append / add / remove any thing in /etc/. :(
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
972
173
Hello ,

When I run this command:
Code:
service cman stop
There is no output , and about remove cluster.conf , I got this error:
Code:
root@node1:/etc/pve# rm -rf cluster.conf
rm: cannot remove `cluster.conf': Permission denied
As I mentioned before , I cannot append / add / remove any thing in /etc/. :(

you CAN edit everything in /etc/, but not inside /etc/pve/...
read more on: http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)

as its hard to fix broken cluster via forum support and its probably faster for you to just re-install.

or you go for a support package (with remote login) so our support team can fix it for you.
 

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
you CAN edit everything in /etc/, but not inside /etc/pve/...
read more on: http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)

as its hard to fix broken cluster via forum support and its probably faster for you to just re-install.

or you go for a support package (with remote login) so our support team can fix it for you.
Yes , I'm sorry.
But I tried pmxcfs too.
Code:
root@node1:/etc/ssh# [B]pmxcfs -l[/B]
[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)

absolutely I'll get a subscription package but I have some problem(s) in here. :(
 

udo

Famous Member
Apr 22, 2009
5,935
185
93
Ahrensburg; Germany
Yes , I'm sorry.
But I tried pmxcfs too.
Code:
root@node1:/etc/ssh# [B]pmxcfs -l[/B]
[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)

absolutely I'll get a subscription package but I have some problem(s) in here. :(
Hi,
that sounds, that you have /etc/pve still mounted? Try to umount first...

Udo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!