[SOLVED]2 Node Cluster, Unable to use Console on VM's hosted on other node.

[riptide_wave]

Hello, I am having a strange issue with the Console feature in proxmox. I currently have a 2 node cluster, node srv-1-02 and srv-1-03. When I signin to the web GUI on srv-1-02 I am able to access the console on all VM's on that node, as well as the shell for srv-1-03, but if I try to access a VM on srv-1-03 it hangs on Status: Plain Authentication, but then goes to Error: Authentication failed. This is consistent even with logging out and back into the proxmox web management. The same thing happens when I sign into the srv-1-03 web portal, and try to access a VM on srv-1-02. Any ideas? Everything else works fine

srv-1-02
pveversion -v

root@srv-1-02:~# pveversion -vpve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-93
pve-kernel-2.6.32-19-pve: 2.6.32-93
pve-kernel-2.6.32-18-pve: 2.6.32-88
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-18
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-8
ksm-control-daemon: 1.1-1

srv-1-03
pveversion -v

root@srv-1-03:~# pveversion -vpve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-93
pve-kernel-2.6.32-19-pve: 2.6.32-93
pve-kernel-2.6.32-18-pve: 2.6.32-88
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-18
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-8
ksm-control-daemon: 1.1-1

pvecm nodes

root@srv-1-02:~# pvecm nodesNode  Sts   Inc   Joined               Name
   1   M    676   2013-03-24 09:45:32  srv-1-02
   2   M    716   2013-03-24 09:46:04  srv-1-03

pvecm status

root@srv-1-02:~# pvecm statusVersion: 6.2.0
Config Version: 2
Cluster Name: Cluster-1
Cluster Id: 45403
Cluster Member: Yes
Cluster Generation: 716
Membership state: Cluster-Member
Nodes: 2
Expected votes: 2
Total votes: 2
Node votes: 1
Quorum: 2  
Active subsystems: 5
Flags: 
Ports Bound: 0  
Node name: srv-1-02
Node ID: 1
Multicast addresses: 239.192.177.13 
Node addresses: 10.1.6.2

 

[tom]

Re: 2 Node Cluster, Unable to use Console on VM's hosted on other node.

Restart cman and pve-cluster on both nodes and try again.

service cman restart

service pve-cluster restart

 

[riptide_wave]

Re: 2 Node Cluster, Unable to use Console on VM's hosted on other node.

Just tried, still the same error. There were no problems with the restart though on either nodes.

root@srv-1-02:~# service cman restartStopping cluster: 
   Stopping dlm_controld... [  OK  ]
   Stopping fenced... [  OK  ]
   Stopping cman... [  OK  ]
   Waiting for corosync to shutdown:[  OK  ]
   Unloading kernel modules... [  OK  ]
   Unmounting configfs... [  OK  ]
Starting cluster: 
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Unfencing self... [  OK  ]
root@srv-1-02:~# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster.

root@srv-1-03:~# service cman restartStopping cluster: 
   Stopping dlm_controld... [  OK  ]
   Stopping fenced... [  OK  ]
   Stopping cman... [  OK  ]
   Waiting for corosync to shutdown:[  OK  ]
   Unloading kernel modules... [  OK  ]
   Unmounting configfs... [  OK  ]
Starting cluster: 
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Unfencing self... [  OK  ]
root@srv-1-03:~# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster.

 

[tom]

Re: 2 Node Cluster, Unable to use Console on VM's hosted on other node.

See http://forum.proxmox.com/threads/11...on-t-open-when-opened-from-the-different-node

 

[riptide_wave]

Re: 2 Node Cluster, Unable to use Console on VM's hosted on other node.

Thanks, was able to get it working. Mark this as solved!

For others, the quickest way to fix this is to sync time on the servers then restart the services, or do a reboot (what I did)
to sync time, I had to use ntpdate -u for some reason, but it worked!

ntpdate -u pool.ntp.org