Connection failure. Network error or Proxmox VE Service not running?

[AdryMS]

I have 1 cluster and 1 of nodes got error on web gui, and error said "Connection failure. Network error or Proxmox VE Service not running?"

already tried systemctl restart pveproxy pvedaemon and pvecm updatecerts / pvecm updatecerts --force
but when after updatecert was got error (re)generate node files generate new node certificate got timeout
i still can access using ssh, but i can't access it from gui and my main cluster, already see This one

and my when i check pvedaemon.service was

root@SVR-27:~# systemctl status pvedaemon.service
● pvedaemon.service - PVE API Daemon
   Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-04-08 09:33:33 WIB; 58min ago
  Process: 1089 ExecStart=/usr/bin/pvedaemon start (code=exited, status=0/SUCCESS)
 Main PID: 1246 (pvedaemon)
    Tasks: 4 (limit: 4915)
   Memory: 144.7M
   CGroup: /system.slice/pvedaemon.service
           ├─1246 pvedaemon
           ├─1247 pvedaemon worker
           ├─1248 pvedaemon worker
           └─1249 pvedaemon worker

Apr 08 09:36:19 SVR-27 pvedaemon[1247]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!
Apr 08 09:52:41 SVR-27 pvedaemon[1249]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: got lock request timeout
Apr 08 09:53:44 SVR-27 pvedaemon[1247]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!
Apr 08 09:53:44 SVR-27 pvedaemon[1249]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!
Apr 08 09:53:44 SVR-27 pvedaemon[1248]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!
Apr 08 09:54:52 SVR-27 pvedaemon[1247]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: got lock request timeout
Apr 08 09:56:54 SVR-27 pvedaemon[1248]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!
Apr 08 10:14:29 SVR-27 pvedaemon[1249]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: got lock request timeout
Apr 08 10:21:58 SVR-27 pvedaemon[1248]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!
Apr 08 10:24:52 SVR-27 pvedaemon[1249]: authentication failure; rhost=10.10.30.1 user=root@pam msg=error during cfs-locked 'authkey' operation: no quorum!

any help please?

 

[Dominic]

It looks to me like this

no quorum!

is your problem.
What is the status of your cluster?

pvecm status

 

[AdryMS]

It looks to me like this

is your problem.
What is the status of your cluster?

pvecm status

Hey thank you staff for your reply, currently my nodes was recovery by it self (without using HA)
my pvecm status on my nodes now

root@SVR-27:~# pvecm status
Cluster information
-------------------
Name:             mycluster
Config Version:   18
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Apr  9 13:53:06 2021
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000003
Ring ID:          1.b542
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.10.30.20
0x00000002          1 10.10.30.22
0x00000003          1 10.10.30.27 (local)

May i know why my nodes like that before? i thought quorum was using voting system each nodes?

 

[Dominic]

Roughly speaking, when some nodes cannot connect to other nodes anymore. You can find some information in the administration guide https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_quorum