Node ist nach Cluster-Join nicht mehr über das Gui erreichbar

[paddl82]

Hallo zusammen.

Ich habe ein Testsystem, welches ich um einen zweiten Node gerne erweitern wollte.
Den zweiten Node habe ich frisch aufgesetzt und upgedatet. Das hinzufügen hat auch wunderbar funktioniert, allerdings habe ich nachdem der zweite Node in den Cluster aufgenommen wurde, kein WebGui mehr. Über SSH ist dieser allerdings erreichbar.



Ich bin diesem Artikel gefolgt: https://pve.proxmox.com/wiki/Cluster_Manager

Wieso erreiche ich diesen nicht mehr?

Node1: pve

proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-8 (running version: 6.0-8/b6b80da7)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-4.15: 5.4-9
pve-kernel-5.0.21-2-pve: 5.0.21-6
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-14-pve: 4.15.18-39
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-10-pve: 4.15.18-32
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-7
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-7
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-9
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

Node2: pve-oge

proxmox-ve: 6.0-2 (running kernel: 5.0.21-3-pve)
pve-manager: 6.0-9 (running version: 6.0-9/508dcee0)
pve-kernel-5.0: 6.0-9
pve-kernel-helper: 6.0-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-7
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-7
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-9
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve1

 

[t.lamprecht]

Ich bin diesem Artikel gefolgt: https://pve.proxmox.com/wiki/Cluster_Manager

Cluster kann man seit 5.2 auch übers Webinterface erstellen und joinen, das nimmt einiges an Arbeit ab, nur so zur Info.

Über SSH ist dieser allerdings erreichbar.

OK, dann ist noch nichts verloren. Zuerst wäre es mal gut herauszufinden was den schief gelaufen ist.

systemctl status corosync pve-cluster

 

[t.lamprecht]

Von beiden nodes wäre auch

pvecm status

interessant.

 

[paddl82]

Hallo, danke für deinen Tipp.
Anbei sind die Ausgaben der beiden Node's

Node 1:

root@pve:~# pvecm status
Cannot initialize CMAP service
root@pve:~#

root@pve:~# systemctl status corosync pve-cluster
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2019-10-18 06:45:16 CEST; 3 days ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
  Process: 2456 ExecStart=/usr/sbin/corosync -f $COROSYNC_OPTIONS (code=killed, signal=KILL)
 Main PID: 2456 (code=killed, signal=KILL)

Oct 18 06:45:11 pve corosync[2456]:   [TOTEM ] Retransmit List: f 10 12 13 15 16 1b 1d 1e 1f
Oct 18 06:45:12 pve corosync[2456]:   [TOTEM ] Retransmit List: f 10 12 13 15 16 1b 1d 1e 1f
Oct 18 06:45:12 pve corosync[2456]:   [TOTEM ] Retransmit List: f 10 12 13 15 16 1b 1d 1e 1f
Oct 18 06:45:12 pve corosync[2456]:   [KNET  ] link: host: 2 link: 0 is down
Oct 18 06:45:12 pve corosync[2456]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 18 06:45:12 pve corosync[2456]:   [KNET  ] host: host: 2 has no active links
Oct 18 06:45:13 pve corosync[2456]:   [TOTEM ] Token has not been received in 36 ms
Oct 18 06:45:13 pve corosync[2456]:   [KNET  ] rx: host: 2 link: 0 is up
Oct 18 06:45:16 pve systemd[1]: corosync.service: Main process exited, code=killed, status=9/KILL
Oct 18 06:45:16 pve systemd[1]: corosync.service: Failed with result 'signal'.

● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-10-17 08:26:10 CEST; 4 days ago
  Process: 2224 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
  Process: 2350 ExecStartPost=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
 Main PID: 2239 (pmxcfs)
    Tasks: 11 (limit: 4915)
   Memory: 52.4M
   CGroup: /system.slice/pve-cluster.service
           └─2239 /usr/bin/pmxcfs

Oct 20 12:20:56 pve pmxcfs[2239]: [dcdb] crit: cpg_initialize failed: 2
Oct 20 12:20:56 pve pmxcfs[2239]: [status] crit: cpg_initialize failed: 2
Oct 20 12:21:02 pve pmxcfs[2239]: [quorum] crit: quorum_initialize failed: 2
Oct 20 12:21:02 pve pmxcfs[2239]: [confdb] crit: cmap_initialize failed: 2
Oct 20 12:21:02 pve pmxcfs[2239]: [dcdb] crit: cpg_initialize failed: 2
Oct 20 12:21:02 pve pmxcfs[2239]: [status] crit: cpg_initialize failed: 2
Oct 20 12:21:08 pve pmxcfs[2239]: [quorum] crit: quorum_initialize failed: 2
Oct 20 12:21:08 pve pmxcfs[2239]: [confdb] crit: cmap_initialize failed: 2
Oct 20 12:21:08 pve pmxcfs[2239]: [dcdb] crit: cpg_initialize failed: 2
Oct 20 12:21:08 pve pmxcfs[2239]: [status] crit: cpg_initialize failed: 2
root@pve:~#

Node 2:

root@pve-oge:~# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
root@pve-oge:~#

root@pve-oge:~# systemctl status corosync pve-cluster
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-10-18 09:15:36 CEST; 3 days ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 885 (corosync)
    Tasks: 9 (limit: 4915)
   Memory: 128.7M
   CGroup: /system.slice/corosync.service
           └─885 /usr/sbin/corosync -f

Oct 18 09:15:36 pve-oge corosync[885]:   [KNET  ] host: host: 1 has no active links
Oct 18 09:15:36 pve-oge corosync[885]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Oct 18 09:15:36 pve-oge corosync[885]:   [KNET  ] host: host: 1 has no active links
Oct 18 09:15:36 pve-oge corosync[885]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 0)
Oct 18 09:15:36 pve-oge corosync[885]:   [KNET  ] host: host: 2 has no active links
Oct 18 09:15:36 pve-oge corosync[885]:   [TOTEM ] A new membership (2.7b1c) was formed. Members joined: 2
Oct 18 09:15:36 pve-oge corosync[885]:   [CPG   ] downlist left_list: 0 received
Oct 18 09:15:36 pve-oge corosync[885]:   [QUORUM] Members[1]: 2
Oct 18 09:15:36 pve-oge corosync[885]:   [MAIN  ] Completed service synchronization, ready to provide service.
Oct 18 09:15:36 pve-oge systemd[1]: Started Corosync Cluster Engine.

● pve-cluster.service - The Proxmox VE cluster filesystem
   Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-10-21 02:54:51 CEST; 7h ago
  Process: 18975 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)

Oct 21 02:54:51 pve-oge systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Oct 21 02:54:51 pve-oge systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Oct 21 02:54:51 pve-oge systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Oct 21 02:54:51 pve-oge systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Oct 21 02:54:51 pve-oge systemd[1]: Stopped The Proxmox VE cluster filesystem.
Oct 21 02:54:51 pve-oge systemd[1]: pve-cluster.service: Start request repeated too quickly.
Oct 21 02:54:51 pve-oge systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Oct 21 02:54:51 pve-oge systemd[1]: Failed to start The Proxmox VE cluster filesystem.
root@pve-oge:~#