No matching key exchange method found.

[inDane]

Hey all,
one of my nodes is causing my cluster to break sync every now and then. When that is happening, it usually does not want me to access it via SSH and gives this error:

Unable to negotiate with 10.168.61.21 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1

Can anyone explain what is happening here and how to solve it? Is this a common thing?

Usually a reboot fixes this temporarily.

Best

 

[bbgeek17]

Unable to negotiate with 10.168.61.21 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1

the error literally means that ssh server is able to negotiate with the above listed methods and your client has them disabled (most likely). This symptom is quite generic and there is no single cause or fix. I'd start with trying to enable one of the offered methods to be able to login and examining the logs.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[inDane]

that is odd, they are allowed and its only one out of 8 identical installations, that shows this behavior. And also, it is not permanently, sometimes it works, sometimes it doesn't.

 

[bbgeek17]

that is odd, they are allowed and its only one out of 8 identical installations, that shows this behavior. And also, it is not permanently, sometimes it works, sometimes it doesn't.

there is no way to solve it without getting your hands dirty. Off the wall thought - you have an IP conflict.
Beyond that, increase verbosity of the ssh client, ie in Linux "ssh -vvvv", to get more details. Examine the log files (journal, auth, etc). Increase sshd debug level on startup and examine the logs.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[inDane]

Thanks for your input!

there is no way to solve it without getting your hands dirty. Off the wall thought - you have an IP conflict.
Beyond that, increase verbosity of the ssh client, ie in Linux "ssh -vvvv", to get more details. Examine the log files (journal, auth, etc). Increase sshd debug level on startup and examine the logs.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

there is no way to solve it without getting your hands dirty. Off the wall thought - you have an IP conflict.

Yes, this is what I initially thought too. But no, unfortunately not. At this point I'm glancing towards broken tranceivers/nics/switchports...

I've had this problem before with the same node. I did a fresh reinstall and the same problem is coming up again.
I am going to re-evaluate the switch config again tomorrow and see if I made some mistake there, if not, I probably need to switch cables as a first test...

 

[bbgeek17]

another thought - MTU mismatch could wreck all kind of havoc thats not easy to trace back sometimes.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[inDane]

i cant really narrow down to what it is.
We've redone the switch config and the local network configs.

Sometimes ssh just works and other times it doesnt.

When it works, its says

debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1


when it does not, it says:
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-ctr,aes256-cbc,aes192-ctr,aes192-cbc,aes128-ctr,aes128-cbc,3des-cbc
debug2: ciphers stoc: aes256-ctr,aes256-cbc,aes192-ctr,aes192-cbc,aes128-ctr,aes128-cbc,3des-cbc
debug2: MACs ctos: hmac-sha2-256,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: MACs stoc: hmac-sha2-256,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96

Without changing anything. It just alternates on its own.

It is driving me nuts.

EDIT: i've got a hot trace that might actually lead to the firewall. Im investigating further. Yet, it is absolutely not clear, why it is only showing this effect only on one node ..

 

[inDane]

it was an ip address conflict...
It was a IP on the switch itself for debugging purposes in that network, that hasnt been removed by the network-admin after the debugging....