Cert error when join

Omer SAVAS

Member
Sep 18, 2018
6
0
21
36
Kütahya
www.omersavas.com
Hello i have a cluster with four nodes. It runned as problemless. And i want join with new node. I copied join information from first node and paste on new node. After i write password and click join button. Than screen freezed. I waited some time. But not worked. And i reloaded page. But not worked gui. I see my first machine gui. New node is visible in the cluster as red icon on first machine gui. But new node's gui not work already. I restart new node machine. After restart fitst node. new node gui never start and fist node gui is run but every node is red icon (like a closed).

i connected to new node with ssh. "journalctl -xe" response:

Code:
Sep 23 14:34:59 fiziksel5 pveproxy[7398]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Sep 23 14:35:00 fiziksel5 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: A start job for unit pvesr.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pvesr.service has begun execution.
--
-- The job identifier is 3653.
Sep 23 14:35:00 fiziksel5 pvesr[7399]: error with cfs lock 'file-replication_cfg': no quorum!
Sep 23 14:35:00 fiziksel5 systemd[1]: pvesr.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pvesr.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 2.
Sep 23 14:35:00 fiziksel5 systemd[1]: pvesr.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pvesr.service has entered the 'failed' state with result 'exit-code'.
Sep 23 14:35:00 fiziksel5 systemd[1]: Failed to start Proxmox VE replication runner.
-- Subject: A start job for unit pvesr.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pvesr.service has finished with a failure.
--
-- The job identifier is 3653 and the job result is failed.
Sep 23 14:35:01 fiziksel5 cron[1084]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [TOTEM ] A new membership (5:33756) was formed. Members
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [CPG   ] downlist left_list: 0 received
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [QUORUM] Members[1]: 5
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 23 14:35:04 fiziksel5 pveproxy[7396]: worker exit
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7396 finished
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: starting 1 worker(s)
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7410 started
Sep 23 14:35:04 fiziksel5 pveproxy[7410]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Sep 23 14:35:04 fiziksel5 pveproxy[7397]: worker exit
Sep 23 14:35:04 fiziksel5 pveproxy[7398]: worker exit
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7397 finished
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7398 finished
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: starting 2 worker(s)
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7411 started
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7412 started
Sep 23 14:35:04 fiziksel5 pveproxy[7412]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Sep 23 14:35:04 fiziksel5 pveproxy[7411]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.


After i connect to first node with ssh. i go to "/etc/pve/nodes/fiziksel1" directory and run "ls -al". I waiting, waiting and not working. And not responsed error. i thinked cert auth problem. I tryed run "pvecm updatecerts --force". not success. Responsed "got timeout". Even not working "systemctl restart pveproxy". Now i restart first machine. Not working first machine's gui too. how can i repair?
 
I have a new response with "journalctl -xe" code after "systemctl restart pve-cluster".

I runed pve-cluster restart code. its waiting but do not anything. after stop ctrl+c and runed journalctl -xe

Code:
Sep 24 11:20:08 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:10 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:13 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243388) was formed. Members
Sep 24 11:20:15 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms
Sep 24 11:20:17 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:20 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243408) was formed. Members
Sep 24 11:20:22 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:24 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv (pmxcfs.c:103:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = (null) (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv = 0x5629ff507990 (priv) (pmxcfs.c:110:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv (0) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv/jou (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv/jou (pmxcfs.c:103:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv/jou (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = jou (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv/jou = 0x5629ff507990 (priv/jou) (pmxcfs.c:110:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv/jou (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv/jou (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv/jou (-2) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv/journalctl (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv/journalctl (pmxcfs.c:103:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv/journalctl (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = journalctl (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv/journalctl = 0x5629ff507990 (priv/journalctl) (pmxcfs.c:110:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv/journalctl (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv/journalctl (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv/journalctl (-2) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243428) was formed. Members
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv (pmxcfs.c:103:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = (null) (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv = 0x5629ff507990 (priv) (pmxcfs.c:110:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv (0) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv/- (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv/- (pmxcfs.c:103:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv/- (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = - (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv/- = 0x5629ff507990 (priv/-) (pmxcfs.c:110:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv/- (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv/- (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv/- (-2) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:29 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms
Sep 24 11:20:31 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:34 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243448) was formed. Members
Sep 24 11:20:36 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms
Sep 24 11:20:39 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:41 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243468) was formed. Members
Sep 24 11:20:43 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:46 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4563 ms
Sep 24 11:20:48 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243488) was formed. Members
Sep 24 11:20:50 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:53 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4563 ms
Sep 24 11:20:55 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243508) was formed. Members
Sep 24 11:20:57 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms


and i dicovered fail machine in /etc/pve/.members:

{
"nodename": "fiziksel1",
"version": 1,
"cluster": { "name": "KOI", "version": 5, "nodes": 5, "quorate": 1 },
"nodelist": {
"fiziksel1": { "id": 1, "online": 0},
"fiziksel2": { "id": 2, "online": 0},
"fiziksel3": { "id": 3, "online": 0}
"fiziksel4": { "id": 4, "online": 0}
"fiziksel5": { "id": 5, "online": 0}
}
}


fiziksel5 is my fail node (new node). but i didnt delete. When i try delete and save fiziksel5 machine, get error : [ Error writing /etc/pve/.members: Input/output error ]