Cert error when join

Omer SAVAS

Member
Sep 18, 2018
6
0
21
36
Kütahya
www.omersavas.com
Hello i have a cluster with four nodes. It runned as problemless. And i want join with new node. I copied join information from first node and paste on new node. After i write password and click join button. Than screen freezed. I waited some time. But not worked. And i reloaded page. But not worked gui. I see my first machine gui. New node is visible in the cluster as red icon on first machine gui. But new node's gui not work already. I restart new node machine. After restart fitst node. new node gui never start and fist node gui is run but every node is red icon (like a closed).

i connected to new node with ssh. "journalctl -xe" response:

Code:
Sep 23 14:34:59 fiziksel5 pveproxy[7398]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Sep 23 14:35:00 fiziksel5 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: A start job for unit pvesr.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pvesr.service has begun execution.
--
-- The job identifier is 3653.
Sep 23 14:35:00 fiziksel5 pvesr[7399]: error with cfs lock 'file-replication_cfg': no quorum!
Sep 23 14:35:00 fiziksel5 systemd[1]: pvesr.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pvesr.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 2.
Sep 23 14:35:00 fiziksel5 systemd[1]: pvesr.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pvesr.service has entered the 'failed' state with result 'exit-code'.
Sep 23 14:35:00 fiziksel5 systemd[1]: Failed to start Proxmox VE replication runner.
-- Subject: A start job for unit pvesr.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pvesr.service has finished with a failure.
--
-- The job identifier is 3653 and the job result is failed.
Sep 23 14:35:01 fiziksel5 cron[1084]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [TOTEM ] A new membership (5:33756) was formed. Members
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [CPG   ] downlist left_list: 0 received
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [QUORUM] Members[1]: 5
Sep 23 14:35:02 fiziksel5 corosync[1086]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 23 14:35:04 fiziksel5 pveproxy[7396]: worker exit
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7396 finished
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: starting 1 worker(s)
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7410 started
Sep 23 14:35:04 fiziksel5 pveproxy[7410]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Sep 23 14:35:04 fiziksel5 pveproxy[7397]: worker exit
Sep 23 14:35:04 fiziksel5 pveproxy[7398]: worker exit
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7397 finished
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7398 finished
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: starting 2 worker(s)
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7411 started
Sep 23 14:35:04 fiziksel5 pveproxy[1129]: worker 7412 started
Sep 23 14:35:04 fiziksel5 pveproxy[7412]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.
Sep 23 14:35:04 fiziksel5 pveproxy[7411]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1688.


After i connect to first node with ssh. i go to "/etc/pve/nodes/fiziksel1" directory and run "ls -al". I waiting, waiting and not working. And not responsed error. i thinked cert auth problem. I tryed run "pvecm updatecerts --force". not success. Responsed "got timeout". Even not working "systemctl restart pveproxy". Now i restart first machine. Not working first machine's gui too. how can i repair?
 
I have a new response with "journalctl -xe" code after "systemctl restart pve-cluster".

I runed pve-cluster restart code. its waiting but do not anything. after stop ctrl+c and runed journalctl -xe

Code:
Sep 24 11:20:08 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:10 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:13 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243388) was formed. Members
Sep 24 11:20:15 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms
Sep 24 11:20:17 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:20 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243408) was formed. Members
Sep 24 11:20:22 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:24 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv (pmxcfs.c:103:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = (null) (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv = 0x5629ff507990 (priv) (pmxcfs.c:110:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv (0) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv/jou (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv/jou (pmxcfs.c:103:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv/jou (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = jou (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv/jou = 0x5629ff507990 (priv/jou) (pmxcfs.c:110:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv/jou (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv/jou (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv/jou (-2) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv/journalctl (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv/journalctl (pmxcfs.c:103:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv/journalctl (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = journalctl (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv/journalctl = 0x5629ff507990 (priv/journalctl) (pmxcfs.c:110:find_plug)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv/journalctl (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv/journalctl (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:26 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv/journalctl (-2) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243428) was formed. Members
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv (pmxcfs.c:103:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = (null) (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv = 0x5629ff507990 (priv) (pmxcfs.c:110:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv (0) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_fuse_getattr /priv/- (pmxcfs.c:127:cfs_fuse_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug start priv/- (pmxcfs.c:103:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug priv/- (cfs-plug.c:52:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: cfs_plug_base_lookup_plug name = priv new path = - (cfs-plug.c:59:cfs_plug_base_lookup_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: find_plug end priv/- = 0x5629ff507990 (priv/-) (pmxcfs.c:110:find_plug)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: enter cfs_plug_base_getattr priv/- (cfs-plug.c:84:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_plug_base_getattr priv/- (cfs-plug.c:103:cfs_plug_base_getattr)
Sep 24 11:20:27 fiziksel1 pmxcfs[1726]: [main] debug: leave cfs_fuse_getattr /priv/- (-2) (pmxcfs.c:154:cfs_fuse_getattr)
Sep 24 11:20:29 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms
Sep 24 11:20:31 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:34 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243448) was formed. Members
Sep 24 11:20:36 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms
Sep 24 11:20:39 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4564 ms
Sep 24 11:20:41 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243468) was formed. Members
Sep 24 11:20:43 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:46 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4563 ms
Sep 24 11:20:48 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243488) was formed. Members
Sep 24 11:20:50 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2213 ms
Sep 24 11:20:53 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 4563 ms
Sep 24 11:20:55 fiziksel1 corosync[2089]:   [TOTEM ] A new membership (1:243508) was formed. Members
Sep 24 11:20:57 fiziksel1 corosync[2089]:   [TOTEM ] Token has not been received in 2214 ms


and i dicovered fail machine in /etc/pve/.members:

{
"nodename": "fiziksel1",
"version": 1,
"cluster": { "name": "KOI", "version": 5, "nodes": 5, "quorate": 1 },
"nodelist": {
"fiziksel1": { "id": 1, "online": 0},
"fiziksel2": { "id": 2, "online": 0},
"fiziksel3": { "id": 3, "online": 0}
"fiziksel4": { "id": 4, "online": 0}
"fiziksel5": { "id": 5, "online": 0}
}
}


fiziksel5 is my fail node (new node). but i didnt delete. When i try delete and save fiziksel5 machine, get error : [ Error writing /etc/pve/.members: Input/output error ]
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!