experiencing web login issue with the new cluster

[kingl]

To whom it may concern:

i had issue with the old cluster, i removed all settings and created a new 2 nodes cluster. everything looks ok on the cluster side.

pvecm nodes
Node Sts Inc Joined Name
1 M 40 2015-07-14 10:15:10 atlassiantools3
2 M 48 2015-07-14 10:15:13 atlassiantools2


clustat
Cluster Status for pmapcluster01 @ Tue Jul 14 10:36:22 2015
Member Status: Quorate

Member Name ID Status
------ ---- ---- ------
atlassiantools3 1 Online, Local
atlassiantools2 2 Online

pvecm status
Version: 6.2.0
Config Version: 2
Cluster Name: pmapcluster01
Cluster Id: 40289
Cluster Member: Yes
Cluster Generation: 48
Membership state: Cluster-Member
Nodes: 2
Expected votes: 2
Total votes: 2
Node votes: 1
Quorum: 2
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: atlassiantools3
Node ID: 1
Multicast addresses: 239.192.157.254
Node addresses: 132.197.63.112

i restored the user.cfg and datacenter.cfg to the /etc/pve

cat user.cfg
user:suser@pam:1:0:name:email:::
user:root@pam:1:0:::email:::

group:Admin::Admin users:



acl:1:/datacenter:suser@pam:Administrator:

however when i tried to log onto node2 with root, it would log me in then immediately log me out and ask me to log in again, it keeps doing that.

on node1 when i try to connect it shows "Secure Connection Failed"

any help is appreciated.

btw i followed this instruction to remove old cluster and create new one, http://undefinederror.org/how-to-reset-cluster-configuration-in-proxmox-2/

my proxmox version is 3.4

 

[kingl]

syslog on the node which shows "secure connection failed"

Jul 14 11:03:18 atlassiantools3 pveproxy[187649]: problem with client 132.197.41.135; rsa_padding_check_pkcs1_type_1: block type is not 01
Jul 14 11:03:19 atlassiantools3 pveproxy[187651]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jul 14 11:03:19 atlassiantools3 pveproxy[187649]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jul 14 11:03:20 atlassiantools3 pveproxy[187651]: problem with client 132.197.41.135; rsa_padding_check_pkcs1_type_1: block type is not 01
Jul 14 11:03:20 atlassiantools3 pveproxy[187649]: problem with client 132.197.41.135; rsa_padding_check_pkcs1_type_1: block type is not 01
Jul 14 11:03:21 atlassiantools3 pveproxy[187649]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jul 14 11:03:22 atlassiantools3 pveproxy[187649]: problem with client 132.197.41.135; rsa_padding_check_pkcs1_type_1: block type is not 01
Jul 14 11:03:23 atlassiantools3 pveproxy[187651]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jul 14 11:03:23 atlassiantools3 pveproxy[187649]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jul 14 11:03:25 atlassiantools3 pveproxy[187649]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.

syslog on node which keeps asking log in

Jul 14 10:42:04 atlassiantools2 pvedaemon[526318]: <root@pam> successful auth for user 'root@pam'
Jul 14 10:51:48 atlassiantools2 rrdcached[3896]: flushing old values
Jul 14 10:51:48 atlassiantools2 rrdcached[3896]: rotating journals
Jul 14 10:51:48 atlassiantools2 rrdcached[3896]: started new journal /var/lib/rrdcached/journal/rrd.journal.1436885508.228957
Jul 14 10:51:48 atlassiantools2 rrdcached[3896]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1436878308.228925
Jul 14 10:55:30 atlassiantools2 pvedaemon[526318]: <root@pam> successful auth for user 'root@pam'
Jul 14 10:55:58 atlassiantools2 pvedaemon[526317]: ipcc_send_rec failed: Transport endpoint is not connected
Jul 14 10:56:30 atlassiantools2 pvedaemon[526318]: <root@pam> successful auth for user 'root@pam'
Jul 14 10:56:42 atlassiantools2 pveproxy[526319]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.

 

[kingl]

for whatever reason i can log into the node atlassiantools3, however it shows "server offline" and datacenter on the left hand side is empty.

the other node atlassiantools2 still keep asking for log in.

 

[kingl]

i did a bit more digging,

i can log into the web interface on node atlassiantools2, datacenter shows 2 nodes all green on the left hand side, i can click on any of the vm on node atlassiantools2, whenever i try to click on any vm on the other node atlassiantools3 it asks me to log in.

if i tried to log into the web interface on atlassiantools3, then it would give me connection error, server offline msg.

it looks like cert issue.

ls -lrt /etc/pve/local/
total 1
drwxr-x--- 2 root www-data 0 Jul 14 09:21 qemu-server
drwx------ 2 root www-data 0 Jul 14 09:21 priv
drwxr-x--- 2 root www-data 0 Jul 14 09:21 openvz
-rw-r----- 1 root www-data 1383 Jul 14 09:21 pve-ssl.pem
-rw-r----- 1 root www-data 1675 Jul 14 09:21 pve-ssl.key

i tried

pvecm updatecerts

restart pve-cluster and pveproxy

whenever i log onto the atlassiantools3 i see this msg in the syslog

Jul 14 12:47:25 atlassiantools3 pveproxy[218215]: problem with client xxx.xxx.xxx.xxx; rsa_padding_check_pkcs1_type_1: block type is not 01
Jul 14 12:47:28 atlassiantools3 pveproxy[218214]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.

also see this is syslog

Jul 14 12:58:15 atlassiantools3 pveproxy[218215]: ipcc_send_rec failed: Transport endpoint is not connected
Jul 14 12:58:15 atlassiantools3 pveproxy[218214]: ipcc_send_rec failed: Transport endpoint is not connected

 

[udo]

Hi,
your clock is on all nodes ok?

Udo

 

[kingl]

yes clock is ok, ntp is set.

i solved the problem by deleting the cluster setting then creating a new cluster on the fresh node, adding the other 2 nodes to the new cluster, lastly restore the nodes info.

thank you for your reply.