[main] crit: unable to acquire pmxcfs lock: Resource temporarily unavailable

[kafisatz]

Hi everyone

Any idea why pmxcfs is not working?
Please see below.

root@pve1:/var/lib/pve-cluster# systemctl status corosync
● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; disabled; preset: enabled)
     Active: active (running) since Mon 2023-09-11 08:20:13 CEST; 10min ago
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview
   Main PID: 1202 (corosync)
      Tasks: 9 (limit: 76534)
     Memory: 134.7M
        CPU: 5.229s
     CGroup: /system.slice/corosync.service
             └─1202 /usr/sbin/corosync -f

Sep 11 08:20:35 pve1 corosync[1202]:   [KNET  ] rx: host: 4 link: 0 is up
Sep 11 08:20:35 pve1 corosync[1202]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Sep 11 08:20:35 pve1 corosync[1202]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 10)
Sep 11 08:20:35 pve1 corosync[1202]:   [KNET  ] pmtud: PMTUD link change for host: 4 link: 0 from 469 to 1397
Sep 11 08:20:35 pve1 corosync[1202]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Sep 11 08:20:35 pve1 corosync[1202]:   [QUORUM] Sync members[3]: 2 3 4
Sep 11 08:20:35 pve1 corosync[1202]:   [QUORUM] Sync joined[1]: 4
Sep 11 08:20:35 pve1 corosync[1202]:   [TOTEM ] A new membership (2.6f2) was formed. Members joined: 4
Sep 11 08:20:35 pve1 corosync[1202]:   [QUORUM] Members[3]: 2 3 4
Sep 11 08:20:35 pve1 corosync[1202]:   [MAIN  ] Completed service synchronization, ready to provide service.
root@pve1:/var/lib/pve-cluster# pvecm status
Cluster information
-------------------
Name:             cluster
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Mon Sep 11 08:31:05 2023
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000002
Ring ID:          2.6f2
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 10.14.15.201 (local)
0x00000003          1 10.14.15.202
0x00000004          1 10.14.15.203
root@pve1:/var/lib/pve-cluster# pmxcfs
[main] notice: resolved node name 'pve1' to '10.14.15.201' for default node IP address
[main] notice: unable to acquire pmxcfs lock - trying again
[main] crit: unable to acquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
root@pve1:/var/lib/pve-cluster#

 

[kafisatz]

I restarted the corosync service
does this look right?

Sep 11 08:37:28 pve1 systemd[1]: Started corosync.service - Corosync Cluster Engine.
Sep 11 08:37:30 pve1 corosync[8744]:   [KNET  ] rx: host: 3 link: 0 is up
Sep 11 08:37:30 pve1 corosync[8744]:   [KNET  ] link: Resetting MTU for link 0 because host 3 joined
Sep 11 08:37:30 pve1 corosync[8744]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 10)
Sep 11 08:37:30 pve1 corosync[8744]:   [KNET  ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 11 08:37:30 pve1 corosync[8744]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Sep 11 08:37:31 pve1 corosync[8744]:   [KNET  ] rx: host: 4 link: 0 is up
Sep 11 08:37:31 pve1 corosync[8744]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Sep 11 08:37:31 pve1 corosync[8744]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 10)
Sep 11 08:37:31 pve1 corosync[8744]:   [QUORUM] Sync members[3]: 2 3 4
Sep 11 08:37:31 pve1 corosync[8744]:   [QUORUM] Sync joined[2]: 3 4
Sep 11 08:37:31 pve1 corosync[8744]:   [TOTEM ] A new membership (2.704) was formed. Members joined: 3 4
Sep 11 08:37:31 pve1 corosync[8744]:   [QUORUM] This node is within the primary component and will provide service.
Sep 11 08:37:31 pve1 corosync[8744]:   [QUORUM] Members[3]: 2 3 4
Sep 11 08:37:31 pve1 corosync[8744]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 11 08:37:31 pve1 corosync[8744]:   [KNET  ] pmtud: PMTUD link change for host: 4 link: 0 from 469 to 1397
Sep 11 08:37:31 pve1 corosync[8744]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: update cluster info (cluster name  cluster, version = 7)
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: node has quorum
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: members: 2/8114, 3/1141, 4/944
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: starting data syncronisation
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: members: 2/8114, 3/1141, 4/944
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: starting data syncronisation
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: received sync request (epoch 2/8114/00000003)
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: received sync request (epoch 2/8114/00000003)
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: received all states
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: leader is 2/8114
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: synced members: 2/8114, 3/1141, 4/944
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: start sending inode updates
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: sent all (0) updates
Sep 11 08:37:33 pve1 pmxcfs[8114]: [dcdb] notice: all data is up to date
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: received all states
Sep 11 08:37:33 pve1 pmxcfs[8114]: [status] notice: all data is up to date
Sep 11 08:37:36 pve1 pvestatd[1321]: got timeout
Sep 11 08:37:36 pve1 pvestatd[1321]: status update time (5.182 seconds)
Sep 11 08:37:46 pve1 pvestatd[1321]: got timeout
Sep 11 08:37:47 pve1 pvestatd[1321]: status update time (5.188 seconds)
Sep 11 08:37:56 pve1 pvestatd[1321]: got timeout
Sep 11 08:37:56 pve1 pvestatd[1321]: status update time (5.216 seconds)
Sep 11 08:38:06 pve1 pvestatd[1321]: got timeout
Sep 11 08:38:06 pve1 pvestatd[1321]: status update time (5.187 seconds)
Sep 11 08:38:16 pve1 pvestatd[1321]: got timeout
Sep 11 08:38:16 pve1 pvestatd[1321]: status update time (5.189 seconds)