[Error: Authentication Failed] attempting to connect to VM from different Node

[discontented]

Hello there, we are beginning to see a strange problem that only occurred for the first time yesterday. Up until yesterday morning, any user could connect to any VM console from any Node. Yesterday afternoon, a few users started to see the error above when trying to view the console of a VM on host 5 when they have logged in to host 1 (for instance).

This now happens across our 20 box cluster. I have checked the time is right and made sure the SSH keys are all working as I thought this was a cluster problem. I have not as yet run the the command:
pvecm -f updatecerts and this is why:

Upon looking at the syslog on the Node, I see this error:

Jan 3 09:41:05 engvmcltr17 pmxcfs[2073]: [status] notice: received log
Jan 3 09:44:00 engvmcltr17 pmxcfs[2073]: [status] notice: received log
Jan 3 09:44:00 engvmcltr17 pmxcfs[2073]: [status] notice: received log
Jan 3 09:44:03 engvmcltr17 pvedaemon[815971]: authentication failure; rhost= user=root@pam msg=Authentication failure

Very strange, root user Auth failure? So I tried an LDAP auth user (full privs):

Jan 3 09:45:05 engvmcltr17 pmxcfs[2073]: [status] notice: received log
Jan 3 09:45:05 engvmcltr17 pmxcfs[2073]: [status] notice: received log
Jan 3 09:45:08 engvmcltr17 pvedaemon[818028]: authentication failure; rhost= user=USERNAME@DOMAINmsg=80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 52e, vece#000
Jan 3 09:45:08 engvmcltr17 pmxcfs[2073]: [status] notice: received log

LDAP auth is definitely working as I logged in successfully. I've checked the DC too, acct not locked etc. Any ideas???

UPDATED:

/var/log/apache2/error.log

[Thu Jan 03 06:25:02 2013] [warn] RSA server certificate CommonName (CN) `engvmcltr17.xxx.xxx.com' does NOT match server name!?
[Thu Jan 03 06:25:02 2013] [warn] RSA server certificate CommonName (CN) `engvmcltr17.xxx.xxx.com' does NOT match server name!?

Cert error?

I very much look forward to your reply,

Rob

 

[dietmar]

What is the output of

# pvecm status

 

[discontented]

Here you go Dietmar:

root@engvmcltr17:~# pvecm status
Version: 6.2.0
Config Version: 26
Cluster Name: NAME
Cluster Id: 39860
Cluster Member: Yes
Cluster Generation: 708
Membership state: Cluster-Member
Nodes: 20
Expected votes: 20
Total votes: 20
Node votes: 1
Quorum: 11
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: engvmcltr17
Node ID: 14
Multicast addresses: 239.192.155.80
Node addresses: 192.168.8.27

This is the same on all nodes with the exception of the node name / ID

Rob

 

[dietmar]

This is the same on all nodes with the exception of the node name / ID

That looks correct. Does it help if you restart apache2 and pvedaemon?

# service apache2 restart
# service pvedaemon restart

 

[discontented]

Thanks for the reply.

The system is under heavy use at the moment, I'll organise some down time for this evening and restart the services. If this doesn't work I'll restart the server, if that fails I'll try updating the certs and complete another restart. I'll get back to you tomorrow with the results.

Thanks for the help so far.

 

[discontented]

Good afternoon,

Restarting the services, restarting the Node and running pvecm updatecerts (with and without -f) has not worked. We are still having problems. Ideas?

 

[werdisturbed]

I am having this same issue. Were you able to find a resolution?

An additional note, when accessing server a to server b, I am able to interact with the VMs, changing their settings and starting, but unable to shutdown or stop and unable to connect to the console (with the authentication failure; rhost= user=root@pam msg=Authentication failure error). When accessing from server b to server a, it constantly requests that I login again. This happens with both a local account and an AD account.

 

[discontented]

Not at all. We've upgraded to the latest sw, re-keyed everything, stopped / re-started services to no avail. In the end we told our users to log into the node the vm is hosted on. We really don't have the time to fault-find.

 

[pixelplumber]

I'm also having this issue, I have to be logged in on the node the VM is on for the console to work otherwise I get Error:Authentication Failed in the console window.

Having said this though it is only happening for KVM VMs for me I can still use the console (on remote nodes) for OpenVZ CTs.

Have tried rebooting a node, pvecm updatecerts -f and restarting various services. Will try restarting whole cluster out of hours.

 

[werdisturbed]

Yeah that's the same. Everything works to control the remote nodes, just the console won't load with auth failures.

 

[pixelplumber]

Any change with this behaviour after an upgrade to 2.3? I have to wait a bit for a quiet weekend to make the move.

 

[werdisturbed]

Same issue after upgrading to 2.3-12.

 

[bruno.oliveira]

Had the same issue here.
Syncronizing the nodes datetime fixed it (vide http://forum.proxmox.com/threads/2994-problem-sync-time).

Regards,
Bruno

 

[pixelplumber]

Excellent, that's fixed it for me too.

Thanks for the tip Bruno.

Had the same issue here.
Syncronizing the nodes datetime fixed it (vide http://forum.proxmox.com/threads/2994-problem-sync-time).

Regards,
Bruno

 

[werdisturbed]

I had to find an internal ntp server to point to as I can't route out, but that fixed it for me as well. It's strange that I didn't see the errors I used to hinting to that. Originally I had seen the error on post 14 here (http://forum.proxmox.com/threads/9830-Authentication-failed-when-migrating) but haven't seen it in some time.

 

[pixelplumber]

I also think I'll need to look at the ntpd.conf file again as well. Got it pointed to the main PVE server, but it must have drifted and not corrected.

 

[RodinM]

Yes, the reason was in time differerence between hosts. I had this problem a long time. Now, after I corrected the time on my hosts and set up ntp, it is finally working.
Strange that the time difference (about 5 minutes) was no problem when I created the cluster.