SSH connecting to wrong host, began after a problem adding qdevice

[ctrlbrk]

Yesterday was great. But I knew I needed to replace the physical node 3 of my cluster with a qdevice. Today was that day.

I had some small problem somewhere along the way of adding the qdevice, but it worked easily enough.

Then suddenly my replication jobs from node 'pve' to 'pve2' started failing with SSH errors.

I went down the rabbit hole here:
https://forum.proxmox.com/threads/s...to-bypass-ssh-known_hosts-bug-s.137809/page-2

@esi_y

So as of now, the issue is from pve when issuing 'ssh pve2' I am connected with pve again, not with pve2. This worked fine yesterday. Not sure at what point exactly that it stopped.

The cluster has quorum.

Praying that @esi_y will help me over the last hurdle here, my head hurts!

 

[ctrlbrk]

I checked hosts file, ping, and .ssh/config. I cannot find any error!

Where else would ssh figure out what IP address to use for a named alias?

 

[esi_y]

Hi! Idk I felt comfy with the ssh internals, now you ask about routing and all.

So can you elaborate on the "replace node 3 with qdevice" - did you e.g. reuse some IP address. I understand you now have 2 nodes (pve and pve2) and you have a qdevice (but mind you it's not casting votes from what you posted before!)...

I would wonder:

1) Can you describe the network setup, I understand it's some datacentre but is it virtual to you or are those supposed to be 3 dedicated physical machines with a proper switch etc?

2) How exactly did you go about adding that QDevice ... if you can find history (not just commands but shown output) it would help. Be precise, which node did you run which command on. What did you install where.

3) From now on, connect to both nodes only from workstation by direct SSH connection, just to avoid whatever shenanigans.

4) Post (from both pve, pve2 ... maybe even the qdevice): cat /etc/network/interfaces

5) Post output of routel (from both nodes, even from qdevice), you may need to apt install iproute2.

 

[ctrlbrk]

So I just tried ssh to my qdevice, and guess what, it is redirecting me back to pve just like pve2 is. I am really confused.

 

[esi_y]

I checked hosts file, ping, and .ssh/config. I cannot find any error!

Where else would ssh figure out what IP address to use for a named alias?

It doesn't work that way, ssh connects to the IP, the alias is provided just to check for the right key.

 

[ctrlbrk]

This for example makes no sense to me. Asking for .253 but early on getting .251

04:00 PM [pve]~ root # "ssh" -vvv 192.168.1.253
OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 2: Applying options for *
debug1: /root/.ssh/config line 16: Applying options for 192.168.1.253
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.251 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.1.251 [192.168.1.251] port 212.

 

[esi_y]

So I just tried ssh to my qdevice, and guess what, it is redirecting me back to pve just like pve2 is. I am really confused.

Also whenever from now you refer to "tried to ssh", it is really important to mention from where to where, easiest to just show ouputs like pve# this command ... pve2# that command ... workstation# command ...

 

[ctrlbrk]

host "pve" is 192.168.1.251
host "pve2" is 192.168.1.252
host "pbs-ifire" is 192.168.1.253

I am just using posts to debug for a second. When I am on pve, and ssh to pve2 or pbs-ifire, early on in the process it "resolves" the IP to .251 even though I am explicitly using an IP (and not a name) on the command line, ie:

[pve] # ssh -vvv 192.168.1.253
...
Connecting to 192.168.1.251.

I am going to check the files listed on the text of post 6.

 

[esi_y]

Are you positive this is SSH only issue?

Anyhow the truth is that default install has /root/.ssh/config a single liner and /etc/ssh/ssh_config quite limited how come you have it populated?

 

[pille99]

debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *

check this 2 lines. something is wrong there.

 

[ctrlbrk]

Solved! ChatGPT found the error right away. LOL

My .ssh/config had entry 'HostName' instead of 'Host' when I added qdevice that happened.

I am so thankful for your time @esi_y -- if you can give me a link to your favorite charity, I will make a donation in your honor.

Thank you!

 

[esi_y]

Solved! ChatGPT found the error right away. LOL

My .ssh/config had entry 'HostName' instead of 'Host' when I added qdevice that happened.

Hilarious - but you know your QD does not seem to be casting votes, correct?

I am so thankful for your time @esi_y -- if you can give me a link to your favorite charity, I will make a donation in your honor.

Just help others when they need it, wherever applicable!

 

[ctrlbrk]

but you know your QD does not seem to be casting votes, correct?

Yikes, no I didn't. Any ideas? lol

 

[esi_y]

The other thing! Why do you need Host-level config entries for your nodes in the root's config file?

 

[ctrlbrk]

# pvecm nodes

Membership information
----------------------
Nodeid Votes Qdevice Name
1 1 A,NV,NMW pve (local)
2 1 NR pve2
0 0 Qdevice (votes 1)
04:19 PM [pve]~ root # pvecm status
Cluster information
-------------------
Name: ifire
Config Version: 11
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Sun Mar 3 16:19:42 2024
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000001
Ring ID: 1.4bd
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate Qdevice

Membership information
----------------------
Nodeid Votes Qdevice Name
0x00000001 1 A,NV,NMW 192.168.1.251 (local)
0x00000002 1 NR 192.168.1.252
0x00000000 0 Qdevice (votes 1)

 

[esi_y]

Yikes, no I didn't. Any ideas? lol

So can you tell us more about how you set it up? My post #3 questions 1 and 2.

 

[ctrlbrk]

The other thing! Why do you need Host-level config entries for your nodes in the root's config file?

Perhaps my logic error. I use ProxyJump when connecting to VM's so I jump from pve node to vm's. the .ssh/config has those defined.

it had definitions for IP address as well with custom port, not sure at what point that became so but somewhere I was landing on port 22 and I needed it to switch to my port 212.

 

[ctrlbrk]

on node [pbs-ifire] # apt install corosync-qnetd

That was to setup qdevice on pbs-ifire. It's a PBS install and running qdevice outside of that. IP is 192.168.1.253.

Then I installed on both cluster nodes: #
# apt install corosync-qdevice corosync-qnetd
# pvecm qdevice setup 192.168.1.253

Just ran journalctl on [pve] and seeing this:



There is no firewall between .251 and .253 (or .252) they are on a private network at the datacenter. The 100.x.x.x IP address is tailscale showing up. Not sure why, there is no special routing or exit node or subnet tail setup with tailscale. But it seems proxmox somewhere is hearing from a tailscale IP instead of the expected one.

 

[ctrlbrk]

On the qdevice node:

Mar 03 16:26:26 pbs-ifire.ifire.net corosync-qnetd[1157]: Unhandled error when reading from client. Disconnecting client (-12271): SSL peer cannot verify your certificate.

Repeated in logs.

 

[esi_y]

Perhaps my logic error. I use ProxyJump when connecting to VM's so I jump from pve node to vm's. the .ssh/config has those defined.

it had definitions for IP address as well with custom port, not sure at what point that became so but somewhere I was landing on port 22 and I needed it to switch to my port 212.

If you absolutely must to this, I would suggest having a separate user on the nodes with the user-level config for this.