ProxMox ">_ Console" doesn't work across the cluster, in some cases

[TasMot]

I have a 3 node ProxMox cluster. All updated to:
Kernel Version : Linux 4.15.18-26-pve #1 SMP PVE 4.15.18-54 (Sat, 15 Feb 2020 15:34:24 +0100)

PVE Manager Version : pve-manager/5.4-13/aee6f0ec

I use the current version of Chrome running on Windows 10 to access and manage the ProxMox cluster.

When I am logged on to ProxMox1 or ProxMox3 (I know very original names), I cannot access the console of the VMs on ProxMox2. However; when I am logged on to ProxMox2 I can access the console for all 3 members of the cluster. I know this can be fixed because I had the problem before (believe it or not when I only had ProxMox1 and ProxMox3). BUT, I forgot to take a note of how to solve the problem and I can't find that article again. There is no fancy configuration, all SSL is still at the default.

All ProxMox hosts have a static IP for IPv4 and IPv6.

Thanks in advance for any help you can provide,

Tom
P.S. I will keep a note this time because I should have #4 coming on line soon (as soon as I can have email down for a couple hours while I convert from XenServer to ProxMox).

 

[oguz]

hi,

When I am logged on to ProxMox1 or ProxMox3 (I know very original names), I cannot access the console of the VMs on ProxMox2. However; when I am logged on to ProxMox2 I can access the console for all 3 members of the cluster

what does it tell you when you try to access console of VMs on PMX2 from PMX1-3 ? can you access the node shell of PMX2 from the other nodes?
if you post the output of pvecm status from all nodes it could help.

i also encourage you to upgrade your cluster to the latest PVE 6.1 whenever you can.

 

[TasMot]

hi,



what does it tell you when you try to access console of VMs on PMX2 from PMX1-3 ? can you access the node shell of PMX2 from the other nodes?
if you post the output of pvecm status from all nodes it could help.

i also encourage you to upgrade your cluster to the latest PVE 6.1 whenever you can.

root@ProxMox1:~# pvecm status
Quorum information
------------------
Date: Fri Mar 20 12:11:34 2020
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 3/26248
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000003 1 2001:470:1f07:da3::221 (local)
0x00000002 1 2001:470:1f07:da3::223
0x00000001 1 2001:470:1f07:da3::242
root@ProxMox1:~#

root@ProxMox2:~# pvecm status
Quorum information
------------------
Date: Fri Mar 20 12:14:25 2020
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 3/26248
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000003 1 2001:470:1f07:da3::221
0x00000002 1 2001:470:1f07:da3::223
0x00000001 1 2001:470:1f07:da3::242 (local)
root@ProxMox2:~#

root@ProxMox3:~# pvecm status
Quorum information
------------------
Date: Fri Mar 20 12:16:51 2020
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000002
Ring ID: 3/26248
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000003 1 2001:470:1f07:da3::221
0x00000002 1 2001:470:1f07:da3::223 (local)
0x00000001 1 2001:470:1f07:da3::242
root@ProxMox3:~#


This is the console when I try to connect to 2 from 1 or 3:

 

[oguz]

-> can you connect to ProxMox2 shell? (click ProxMox2 on the left menu and choose 'Shell')

-> what do you see in the tasklog (bottom) after connection fails? can you post the output of that task?

 

[TasMot]

From 1 and 3 when I click on the ProxMox2 Shell, I have to enter the root password, but it connects.

 

[oguz]

can you ssh from all nodes to all the others? (ex. pmx1 -> pmx3/pmx2 and vice versa for all of them)
it should work without any user interaction.
if it doesn't, then check the /etc/hosts of all nodes. make sure hostname --ip-address returns the correct IP for all nodes.

 

[TasMot]

can you ssh from all nodes to all the others? (ex. pmx1 -> pmx3/pmx2 and vice versa for all of them)
it should work without any user interaction.
if it doesn't, then check the /etc/hosts of all nodes. make sure hostname --ip-address returns the correct IP for all nodes.

No, I can't accessing ProxMox2 always asks for a password.
However; I think I know what is going on now.

A while back I noticed that in a moment of plain dumb, I assigned ProxMox2 the same IP as my main switch (an older 3COM 48 port switch). Since I couldn't get into the switch to change it's IP, I changed the IP for ProxMox2.

I have since changed the IP for the switch.

So, now the question is, "will it fix everything if I change the IP for ProxMox2 back, or am I totally hosed?" If that is not the solution, how do I proceed?

Thanks,
Tom

 

[oguz]

No, I can't accessing ProxMox2 always asks for a password.

it should not ask a password. please check the /etc/hosts and hostname like i told you

 

[TasMot]

it should not ask a password. please check the /etc/hosts and hostname like i told you

I do have a fully functional Active Directory integrated DNS system, so I only have minimal info in the hosts file. Here are the 3 hosts files:

root@ProxMox1:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
2001:470:1f07:da3::221 ProxMox1.corp.TasMot.com ProxMox1 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@ProxMox1:~#

root@ProxMox2:~/.ssh# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
2001:470:1f07:da3::242 ProxMox2.corp.TasMot.com ProxMox2 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@ProxMox2:~/.ssh#

root@ProxMox3:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
2001:470:1f07:da3::223 ProxMox3.corp.TasMot.com ProxMox3 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@ProxMox3:~#


Thanks,

Tom

 

[TasMot]

can you ssh from all nodes to all the others? (ex. pmx1 -> pmx3/pmx2 and vice versa for all of them)
it should work without any user interaction.
if it doesn't, then check the /etc/hosts of all nodes. make sure hostname --ip-address returns the correct IP for all nodes.

Yes all hostname commands return the correct IP using the DNS system.

 

[TasMot]

After months of working on this on my own and then back in March finally asking for help, I have finally found the solution to this problem. The answer was because of a slightly different problem that caused a different line of searching for an answer. In any case, the problem was bad permissions on the /etc directory. Here is the explanation of the way I found the solution: https://forum.proxmox.com/threads/vm-migration-failed.72544/#post-324309.