[SOLVED] SSH 'refused connect' | PVE

[alasdairc]

Hi there,

I may have broken SSH on my PVE host by doing `systemctl stop sshd` a while back in the shell and not the web GUI - ever since, I've not been able to connect to the host over SSH. It's showing as running, and even accepts incoming SSH connections, however all connection attempts are rejected


Client side (-vvv verbose SSH logging):

OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "myhost.domain.com" port 60252
debug2: ssh_connect_direct
debug1: Connecting to myhost.domain.com [X.X.X.X] port 60252.
debug1: Connection established.
debug1: identity file /home/XXXX/.ssh/id_rsa type -1
debug1: identity file /home/XXXX/.ssh/id_rsa-cert type -1
debug1: identity file /home/XXXX/.ssh/id_dsa type -1
debug1: identity file /home/XXXX/.ssh/id_dsa-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519 type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519_sk type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/XXXX/.ssh/id_xmss type -1
debug1: identity file /home/XXXX/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4

(client username obscured)

Server side (auth.log):

Feb  1 22:30:29 lon-dc01 sshd[2410673]: refused connect from X.X.X.X (X.X.X.X)
Feb  1 22:31:35 lon-dc01 sshd[2411094]: refused connect from X.X.X.X (X.X.X.X)
Feb  1 22:47:17 lon-dc01 sshd[2417111]: refused connect from X.X.X.X (X.X.X.X)

(client IP obscured)

Fail2Ban was enabled but that has now been disabled - there are no banned IP's with fail2ban-client or in iptables

I have tried SSH'ing from different clients both on the LAN and elsewhere, however all get rejected


Thanks

EDIT: Everything in sshd.conf is default (that came with Proxmox), only the port has been changed

 

[oguz]

hi,

have you tried restarting the SSH service? try: systemctl restart sshd

could you also diff the original sshd config with yours and post it here?

and just in case, what is your output from pveversion -v?

 

[alasdairc]

hi,

have you tried restarting the SSH service? try: systemctl restart sshd

could you also diff the original sshd config with yours and post it here?

and just in case, what is your output from pveversion -v?

Hi there,

Thanks for your response

Yes I have tried restarting SSHD and also stopping and starting

My SSHD config:

#       $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

Port 60252
#AddressFamily any
#ListenAddress 
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none


# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem       sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server

pveversion -v:

root@lon-dc01:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-10
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.4.151-1-pve: 5.4.151-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.16.0-1
pve-zsync: 2.2.1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Thank you

 

[oguz]

is the ListenAddress in the ssh conf correct?

also:

proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-10
pve-kernel-5.13.19-3-pve: 5.13.19-7

did you reboot after making package upgrades? (you have an older kernel running than the one installed, if that's not intentional please do a reboot)

 

[alasdairc]

The listenaddress is correct, I've also tried with commenting that out

The host has been up for 60+ days so I will definitely give that a shot

Thank you

 

[oguz]

The host has been up for 60+ days so I will definitely give that a shot

let us know if that changes anything

 

[alasdairc]

let us know if that changes anything

Hey,

Apologies about the delay, I had to schedule downtime for the host to be rebooted

I have patched the host and given it a reboot - unfortunately still having the same issue


Thank you for your time on this

 

[oguz]

Apologies about the delay, I had to schedule downtime for the host to be rebooted

I have patched the host and given it a reboot - unfortunately still having the same issue

i've tested the same /etc/ssh/sshd_config file here and it seems to work fine for me.

do you have any other files in /etc/ssh/sshd_config.d/ directory? (those are being included in your config file, so worth taking a look)

also please check your journalctl for any SSH related errors when you restart the service.

btw, is this a single standalone node, or is it a part of a cluster?

 

[alasdairc]

i've tested the same /etc/ssh/sshd_config file here and it seems to work fine for me.

do you have any other files in /etc/ssh/sshd_config.d/ directory? (those are being included in your config file, so worth taking a look)

also please check your journalctl for any SSH related errors when you restart the service.

btw, is this a single standalone node, or is it a part of a cluster?

Hey there,

Nothing in /etc/ssh/sshd_config.d/

I have restarted the service and done:

systemctl restart sshd ; journalctl -xe

- there are no errors, all green, and the service starts up fine

This is a standalone node


Thank you

 

[oguz]

thanks.

have you made any other hardening on the host?

on a hunch, could you check the /etc/hosts.allow and /etc/hosts.deny files as well?

 

[alasdairc]

thanks.

have you made any other hardening on the host?

on a hunch, could you check the /etc/hosts.allow and /etc/hosts.deny files as well?

WHAT!!!

/etc/hosts.deny had deny all for SSHD!

I have absolutely no idea where that came from, I definitely didn't do it!

I am now able to SSH onto the box... definitely confused though!

Thank you so much for your time and intellect, I really appreciate it

 

[oguz]

WHAT!!!

/etc/hosts.deny had deny all for SSHD!
I have absolutely no idea where that came from, I definitely didn't do it!

neither did i
and it's not set by default

I am now able to SSH onto the box... definitely confused though!

Thank you so much for your time and intellect, I really appreciate it

gladly! please mark the thread as [SOLVED] so others with the same issue can also know