The server certificate /etc/pve/local/pve-ssl.pem is not yet active

L

lukasz.matys

Member
Dec 10, 2015
34
3
8
Hello.
We've setup new proxmox cluster.
When starting vhosts, we get the following error:

unning as unit 100.scope.
kvm: -vnc unix:/var/run/qemu-server/100.vnc,x509,password: Failed to start VNC server: The server certificate /etc/pve/local/pve-ssl.pem is not yet active
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 100 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=25176948-8ab7-4d4c-bbb0-9d95607f93b2' -name test -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k pl -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:59762bbfaac' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/dev/vg0/vm-100-disk-1,if=none,id=drive-ide0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=32:63:36:31:34:36,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1

Cluster is up since few days.
Can you help me?

Regards.
 
I did like here,

https://forum.proxmox.com/threads/f...ve-local-pve-ssl-pem-is-not-yet-active.26004/

But no success. I've removed specified files and I'm trying to generate new certs,

root@cp-node1:/etc/pve/priv# pvecm keygen pve-root-ca.key

Corosync Cluster Engine Authentication key generator.

Gathering 1024 bits for key from /dev/urandom.

corosync-keygen: Failed to set key file permissions to 0400: Function not implemented

command 'corosync-keygen -l -k pve-root-ca.key' failed: exit code 3

root@cp-node1:/etc/pve/priv#

What to do next?
 
I tryied like this,

root@cp-node1:~# pvecm keygen pve-root-ca.key
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/urandom.
Writing corosync key to pve-root-ca.key.
root@cp-node1:~#

root@cp-node1:~# cp pve-root-ca.key /etc/pve/priv/

root@cp-node1:~# pvecm updatecerts -f
unable to load Private Key
140472850441872:error:0906D06C:pEM routines:pEM_read_bio:no start line:pem_lib.c:703:Expecting: ANY PRIVATE KEY
generating pve root certificate failed:
command 'openssl req -batch -days 3650 -new -x509 -nodes -key /etc/pve/priv/pve-root-ca.key -out /etc/pve/pve-root-ca.pem -subj '/CN=Proxmox Virtual Environment/OU=b6b846256e4073b3171b757e035a51cd/O=PVE Cluster Manager CA/'' failed: exit code 1
root@cp-node1:~#

:/
 
you need to run the following command once:

Code: 
rm /etc/pve/priv/pve-root-ca.key /etc/pve/pve-root-ca.pem

this deletes the cluster CA certificate and private key (if they exist).

then you need to run this command on all nodes of your cluster

Code: 
pvecm updatecerts -f

this command regenerates the node certificate and key, but before that it checks if the cluster CA certificate and key exist, and if necessary, will generate a new one automatically.

finally restart the web interface to reload the new certificate (also on each node of the cluster):

Code: 
systemctl restart pveproxy

please don't run random commands that are not mentioned here!
 
  • Like
Reactions: alecmedina and chrone
Your commands match those run by lukasz.matys and myself. It seems the generated private key is invalid:

Attempting to recreate cluster:
Code: 
    systemctl stop corosync.service;
    systemctl stop pve-cluster.service;
    pmxcfs -l;
    rm -f /etc/pve/pve-root-ca.pem /etc/pve/priv/pve-root-ca.key /etc/pve/priv/pve-root-ca.srl;
    rm -f /etc/corosync/corosync.conf;
    cd /root;
    pvecm keygen pve-root-ca.key;
The following subsequently fails as the created key isn't valid:
Code: 
    cat pve-root-ca.key > /etc/pve/priv/pve-root-ca.key;
    pvecm updatecerts -force;

Output:
Code: 
  [admin@kvm5a ~]# pvecm keygen pve-root-ca.key;
    Corosync Cluster Engine Authentication key generator.
    Gathering 1024 bits for key from /dev/urandom.
    Writing corosync key to pve-root-ca.key.
  [admin@kvm5a ~]# dir
    total 4
    -r-------- 1 admin root 128 Jul 30 16:33 pve-root-ca.key
  [admin@kvm5a ~]# openssl rsa -in pve-root-ca.key -check
    unable to load Private Key
    140234217199248:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:696:Expecting: ANY PRIVATE KEY
 
you run pvecm keygen , which is not for a root ca, but for the corosync key
 
  • Like
Reactions: brucexx
Had an issue with this in the log:
Aug 03 18:35:02 pve1-weha pveproxy[1728]: proxy detected vanished client connection
Aug 03 18:35:02 pve1-weha pveproxy[1729]: '/etc/pve/nodes/pve2-weha/pve-ssl.pem' does not exist!
Aug 03 18:35:32 pve1-weha pveproxy[1729]: proxy detected vanished client connection
Aug 03 18:35:33 pve1-weha pveproxy[1729]: '/etc/pve/nodes/pve2-weha/pve-ssl.pem' does not exist!
Aug 03 18:35:37 pve1-weha pveproxy[1729]: '/etc/pve/nodes/pve2-weha/pve-ssl.pem' does not exist!

Could not connect to both nodes in my newly created test cluster , the second node was not responding to direct web requests and was timing out when looked at from the Datacenter perspective, the cluster was formed:

Date: Mon Aug 3 18:34:37 2020
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000002
Ring ID: 1.2f4
Quorate: Yes

Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 2
Flags: Quorate

The commands provided by dcsapak fixed that right away.

Thx
 
I have this same error for a new node I just added to my existing cluster. What's the correct way to generate keys?
 
please provide pvecm status and journalctl -b -u pve-cluster output..
 
Code: 
pvecm updatecerts -f
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
root@falcon:~# ls -l /etc/pve/nodes/phoenix/
total 0
drwxr-xr-x 2 root www-data 0 May 24 23:10 lxc
drwxr-xr-x 2 root www-data 0 May 24 23:10 openvz
drwx------ 2 root www-data 0 May 24 23:10 priv
drwxr-xr-x 2 root www-data 0 May 24 23:10 qemu-server
root@falcon:~# pvecm status
Cluster information
-------------------
Name:             rebel
Config Version:   32
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue May 25 04:59:35 2021
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000004
Ring ID:          1.9251c
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      3
Quorum:           3
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.1.1.16
0x00000003          1 10.1.1.12
0x00000004          1 10.1.1.13 (local)
root@falcon:~# journalctl -b -u pve-cluster
-- Logs begin at Mon 2021-05-24 23:37:05 MDT, end at Tue 2021-05-25 04:59:36 MDT. --
May 24 23:37:32 falcon systemd[1]: Starting The Proxmox VE cluster filesystem...
May 24 23:37:32 falcon pmxcfs[7192]: [quorum] crit: quorum_initialize failed: 2
May 24 23:37:32 falcon pmxcfs[7192]: [quorum] crit: can't initialize service
May 24 23:37:32 falcon pmxcfs[7192]: [confdb] crit: cmap_initialize failed: 2
May 24 23:37:32 falcon pmxcfs[7192]: [confdb] crit: can't initialize service
May 24 23:37:32 falcon pmxcfs[7192]: [dcdb] crit: cpg_initialize failed: 2
May 24 23:37:32 falcon pmxcfs[7192]: [dcdb] crit: can't initialize service
May 24 23:37:32 falcon pmxcfs[7192]: [status] crit: cpg_initialize failed: 2
May 24 23:37:32 falcon pmxcfs[7192]: [status] crit: can't initialize service
May 24 23:37:33 falcon systemd[1]: Started The Proxmox VE cluster filesystem.
May 24 23:37:38 falcon pmxcfs[7192]: [status] notice: update cluster info (cluster name  rebel, version = 32)
May 24 23:37:38 falcon pmxcfs[7192]: [dcdb] notice: members: 4/7192
May 24 23:37:38 falcon pmxcfs[7192]: [dcdb] notice: all data is up to date
May 24 23:37:38 falcon pmxcfs[7192]: [status] notice: members: 4/7192
May 24 23:37:38 falcon pmxcfs[7192]: [status] notice: all data is up to date
May 24 23:37:44 falcon pmxcfs[7192]: [status] notice: cpg_send_message retry 10
May 24 23:37:45 falcon pmxcfs[7192]: [status] notice: cpg_send_message retry 20
May 24 23:37:46 falcon pmxcfs[7192]: [status] notice: cpg_send_message retry 30
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: members: 1/4338, 3/5262, 4/7192
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: starting data syncronisation
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: members: 1/4338, 3/5262, 4/7192
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: starting data syncronisation
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: node has quorum
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: cpg_send_message retried 36 times
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: received sync request (epoch 1/4338/00000044)
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: received sync request (epoch 1/4338/00000037)
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: received all states
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: leader is 1/4338
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: synced members: 1/4338, 3/5262, 4/7192
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: all data is up to date
May 24 23:37:47 falcon pmxcfs[7192]: [dcdb] notice: dfsm_deliver_queue: queue length 2
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: received all states
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: all data is up to date
May 24 23:37:47 falcon pmxcfs[7192]: [status] notice: dfsm_deliver_queue: queue length 12
May 24 23:41:08 falcon pmxcfs[7192]: [dcdb] notice: data verification successful
May 24 23:49:26 falcon pmxcfs[7192]: [status] notice: received log
May 25 00:05:26 falcon pmxcfs[7192]: [status] notice: received log
May 25 00:21:26 falcon pmxcfs[7192]: [status] notice: received log
May 25 00:36:26 falcon pmxcfs[7192]: [status] notice: received log
May 25 00:41:04 falcon pmxcfs[7192]: [dcdb] notice: data verification successful
May 25 00:51:26 falcon pmxcfs[7192]: [status] notice: received log
May 25 01:06:23 falcon pmxcfs[7192]: [status] notice: received log
May 25 01:21:24 falcon pmxcfs[7192]: [status] notice: received log
May 25 01:37:23 falcon pmxcfs[7192]: [status] notice: received log
May 25 01:41:05 falcon pmxcfs[7192]: [dcdb] notice: data verification successful
May 25 01:42:43 falcon pmxcfs[7192]: [status] notice: cpg_send_message retried 1 times
May 25 01:52:24 falcon pmxcfs[7192]: [status] notice: received log
May 25 02:08:24 falcon pmxcfs[7192]: [status] notice: received log
May 25 02:23:24 falcon pmxcfs[7192]: [status] notice: received log
May 25 02:39:23 falcon pmxcfs[7192]: [status] notice: received log
May 25 02:41:06 falcon pmxcfs[7192]: [dcdb] notice: data verification successful
May 25 02:55:23 falcon pmxcfs[7192]: [status] notice: received log
May 25 03:11:24 falcon pmxcfs[7192]: [status] notice: received log
May 25 03:17:33 falcon pmxcfs[7192]: [status] notice: cpg_send_message retried 1 times
May 25 03:27:24 falcon pmxcfs[7192]: [status] notice: received log
May 25 03:33:50 falcon pmxcfs[7192]: [status] notice: received log
May 25 03:33:55 falcon pmxcfs[7192]: [status] notice: received log
May 25 03:41:07 falcon pmxcfs[7192]: [dcdb] notice: data verification successful
May 25 03:43:28 falcon pmxcfs[7192]: [status] notice: received log
May 25 03:59:28 falcon pmxcfs[7192]: [status] notice: received log
May 25 04:15:27 falcon pmxcfs[7192]: [status] notice: received log
May 25 04:29:43 falcon pmxcfs[7192]: [status] notice: cpg_send_message retried 1 times
May 25 04:30:28 falcon pmxcfs[7192]: [status] notice: received log
May 25 04:41:08 falcon pmxcfs[7192]: [dcdb] notice: data verification successful
May 25 04:46:28 falcon pmxcfs[7192]: [status] notice: received log
 
you're mixing nodes - please provide the output from the failing node...
 
ah, sorry. here is the failing node.
Code: 
root@phoenix:~# pvecm status
Cluster information
-------------------
Name:             rebel
Config Version:   32
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue May 25 05:19:35 2021
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000002
Ring ID:          2.93160
Quorate:          No

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      1
Quorum:           3 Activity blocked
Flags:

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 10.1.1.8 (local)
root@phoenix:~# journalctl -b -u pve-cluster
-- Logs begin at Mon 2021-05-24 23:22:34 MDT, end at Tue 2021-05-25 05:20:06 MDT. --
May 24 23:22:36 phoenix systemd[1]: Starting The Proxmox VE cluster filesystem...
May 24 23:22:36 phoenix pmxcfs[2170]: [quorum] crit: quorum_initialize failed: 2
May 24 23:22:36 phoenix pmxcfs[2170]: [quorum] crit: can't initialize service
May 24 23:22:36 phoenix pmxcfs[2170]: [confdb] crit: cmap_initialize failed: 2
May 24 23:22:36 phoenix pmxcfs[2170]: [confdb] crit: can't initialize service
May 24 23:22:36 phoenix pmxcfs[2170]: [dcdb] crit: cpg_initialize failed: 2
May 24 23:22:36 phoenix pmxcfs[2170]: [dcdb] crit: can't initialize service
May 24 23:22:36 phoenix pmxcfs[2170]: [status] crit: cpg_initialize failed: 2
May 24 23:22:36 phoenix pmxcfs[2170]: [status] crit: can't initialize service
May 24 23:22:37 phoenix systemd[1]: Started The Proxmox VE cluster filesystem.
May 24 23:22:42 phoenix pmxcfs[2170]: [status] notice: update cluster info (cluster name  rebel, version = 32)
May 24 23:22:42 phoenix pmxcfs[2170]: [dcdb] notice: members: 2/2170
May 24 23:22:42 phoenix pmxcfs[2170]: [dcdb] notice: all data is up to date
May 24 23:22:42 phoenix pmxcfs[2170]: [status] notice: members: 2/2170
May 24 23:22:42 phoenix pmxcfs[2170]: [status] notice: all data is up to date
May 24 23:22:59 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 10
May 24 23:23:00 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 20
May 24 23:23:00 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retried 20 times
May 24 23:23:09 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 10
May 24 23:23:10 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 20
May 24 23:23:11 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 30
May 24 23:23:12 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 40
May 24 23:23:13 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 50
May 24 23:23:14 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 60
May 24 23:23:15 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retried 68 times
May 24 23:23:29 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 10
May 24 23:23:30 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retried 17 times
May 24 23:23:39 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 10
May 24 23:23:40 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 20
May 24 23:23:41 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 30
May 24 23:23:42 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 40
May 24 23:23:43 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 50
May 24 23:23:44 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 60
May 24 23:23:45 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 70
May 24 23:23:46 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 80
May 24 23:23:47 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 90
May 24 23:23:48 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retry 100
May 24 23:23:48 phoenix pmxcfs[2170]: [status] notice: cpg_send_message retried 100 times
May 24 23:23:48 phoenix pmxcfs[2170]: [status] crit: cpg_send_message failed: 6
 
so that node is not part of the quorum, so it also won't be able to write to /etc/pve to regenerate any files. find out why it's not part of the quorate partition (hint: the corosync logs might contain an error pointing at the cause), fix that issue, then run the command again on the phoenix node.
 
there are definitely a bunch of errors in the syslog of the machine i'm having trouble with, but I'm not sure which one is causing the problem. i've tried searching for a few of them, but i haven't found anything that's helped yet. the issue with /etc/pve/local/pve-ssl.key seems circular to me. can't generate the keys because it doesn't have quorum, but it can't get quorum because it doesn't have the key. how do i solve this?

Code: 
May 25 19:52:50 phoenix systemd-timesyncd[1853]: Synchronized to time server for the first time 45.63.54.13:123 (2.debian.pool.ntp.org).
May 25 19:52:51 phoenix systemd[1]: Created slice User Slice of UID 0.
May 25 19:52:51 phoenix systemd[1]: Starting User Runtime Directory /run/user/0...
May 25 19:52:51 phoenix systemd[1]: Started User Runtime Directory /run/user/0.
May 25 19:52:51 phoenix systemd[1]: Starting User Manager for UID 0...
May 25 19:52:51 phoenix systemd[2849]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
May 25 19:52:51 phoenix systemd[2849]: Reached target Paths.
May 25 19:52:51 phoenix systemd[2849]: Reached target Timers.
May 25 19:52:51 phoenix systemd[2849]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
May 25 19:52:51 phoenix systemd[2849]: Listening on GnuPG cryptographic agent and passphrase cache.
May 25 19:52:51 phoenix systemd[2849]: Listening on GnuPG network certificate management daemon.
May 25 19:52:51 phoenix systemd[2849]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
May 25 19:52:51 phoenix systemd[2849]: Reached target Sockets.
May 25 19:52:51 phoenix systemd[2849]: Reached target Basic System.
May 25 19:52:51 phoenix systemd[2849]: Reached target Default.
May 25 19:52:51 phoenix systemd[2849]: Startup finished in 48ms.
May 25 19:52:51 phoenix systemd[1]: Started User Manager for UID 0.
May 25 19:52:51 phoenix systemd[1]: Started Session 1 of user root.
May 25 19:52:51 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 90
May 25 19:52:51 phoenix corosync[2278]:   [TOTEM ] A new membership (2.b5739) was formed. Members
May 25 19:52:52 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 100
May 25 19:52:52 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retried 100 times
May 25 19:52:52 phoenix pmxcfs[2171]: [status] crit: cpg_send_message failed: 6
May 25 19:52:53 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 10
May 25 19:52:53 phoenix corosync[2278]:   [TOTEM ] Token has not been received in 1726 ms
May 25 19:52:54 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 20
May 25 19:52:54 phoenix pveproxy[2844]: worker exit
May 25 19:52:54 phoenix pveproxy[2486]: worker 2844 finished
May 25 19:52:54 phoenix pveproxy[2486]: starting 2 worker(s)
May 25 19:52:54 phoenix pveproxy[2486]: worker 2892 started
May 25 19:52:54 phoenix pveproxy[2486]: worker 2909 started
May 25 19:52:54 phoenix pveproxy[2843]: worker exit
May 25 19:52:54 phoenix pveproxy[2892]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 25 19:52:54 phoenix pveproxy[2909]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 25 19:52:54 phoenix pveproxy[2486]: worker 2843 finished
May 25 19:52:54 phoenix pveproxy[2486]: starting 1 worker(s)
May 25 19:52:54 phoenix pveproxy[2486]: worker 2946 started
May 25 19:52:54 phoenix pveproxy[2946]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 25 19:52:55 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 30
May 25 19:52:55 phoenix corosync[2278]:   [TOTEM ] Token has not been received in 4051 ms
May 25 19:52:56 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 40
May 25 19:52:57 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 50
May 25 19:52:58 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retry 60
May 25 19:52:59 phoenix corosync[2278]:   [TOTEM ] A new membership (2.b574d) was formed. Members
May 25 19:52:59 phoenix corosync[2278]:   [QUORUM] Members[1]: 2
May 25 19:52:59 phoenix corosync[2278]:   [MAIN  ] Completed service synchronization, ready to provide service.
May 25 19:52:59 phoenix pmxcfs[2171]: [status] notice: cpg_send_message retried 66 times
May 25 19:52:59 phoenix pvestatd[2296]: status update time (16.845 seconds)
May 25 19:52:59 phoenix pmxcfs[2171]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/phoenix/local: -1
May 25 19:52:59 phoenix pveproxy[2909]: worker exit
May 25 19:52:59 phoenix pveproxy[2892]: worker exit
May 25 19:52:59 phoenix pveproxy[2486]: worker 2909 finished
May 25 19:52:59 phoenix pveproxy[2486]: starting 1 worker(s)
May 25 19:52:59 phoenix pveproxy[2486]: worker 2953 started
May 25 19:52:59 phoenix pveproxy[2486]: worker 2892 finished
May 25 19:52:59 phoenix pveproxy[2486]: starting 1 worker(s)
May 25 19:52:59 phoenix pveproxy[2486]: worker 2954 started
May 25 19:52:59 phoenix pveproxy[2946]: worker exit
May 25 19:52:59 phoenix pveproxy[2953]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 25 19:52:59 phoenix pveproxy[2954]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 25 19:52:59 phoenix pveproxy[2486]: worker 2946 finished
May 25 19:52:59 phoenix pveproxy[2486]: starting 1 worker(s)
May 25 19:52:59 phoenix pveproxy[2486]: worker 2955 started
May 25 19:52:59 phoenix pveproxy[2955]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737.
May 25 19:53:00 phoenix systemd[1]: Starting Proxmox VE replication runner...
May 25 19:53:00 phoenix pvesr[3015]: error during cfs-locked 'file-replication_cfg' operation: no quorum!
May 25 19:53:00 phoenix systemd[1]: pvesr.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 25 19:53:00 phoenix systemd[1]: pvesr.service: Failed with result 'exit-code'.
May 25 19:53:00 phoenix systemd[1]: Failed to start Proxmox VE replication runner.
May 25 19:53:00 phoenix corosync[2278]:   [TOTEM ] Token has not been received in 1725 ms
May 25 19:53:01 phoenix cron[2277]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
 
Last edited:
how is the network situation between the nodes? anything out of the ordinary? can you try restarting corosync and pve-cluster services?
 
yep - was just about to follow up on that. networking was the issue. all of the nodes are on a 10g switch - so on that switch i disabled IGMP snooping and on the nodes i ran
Code: 
echo 0 >/sys/class/net/vmbr0/bridge/multicast_snooping
and then on the existing node ran:
Code: 
service pve-cluster restart
service corosync restart
on the new node, i reinstalled proxmox, updated it, updated multicast_snooping, and re-joined it to the cluster. which worked immediately.
 
  • Like
Reactions: fabian
fabian said:
you need to run the following command once:

Code: 
rm /etc/pve/priv/pve-root-ca.key /etc/pve/pve-root-ca.pem

this deletes the cluster CA certificate and private key (if they exist).

then you need to run this command on all nodes of your cluster

Code: 
pvecm updatecerts -f

this command regenerates the node certificate and key, but before that it checks if the cluster CA certificate and key exist, and if necessary, will generate a new one automatically.

finally restart the web interface to reload the new certificate (also on each node of the cluster):

Code: 
systemctl restart pveproxy

please don't run random commands that are not mentioned here!
Click to expand...
This happened to me because I renamed a node to an old node name after replacing a server.

The fix was to remove all the old ssh keys then update the certs like this.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back