[SOLVED] SSH 'refused connect' | PVE

alasdairc

New Member
Jun 25, 2021
25
0
1
25
Hi there,

I may have broken SSH on my PVE host by doing `systemctl stop sshd` a while back in the shell and not the web GUI - ever since, I've not been able to connect to the host over SSH. It's showing as running, and even accepts incoming SSH connections, however all connection attempts are rejected


Client side (-vvv verbose SSH logging):
Code:
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "myhost.domain.com" port 60252
debug2: ssh_connect_direct
debug1: Connecting to myhost.domain.com [X.X.X.X] port 60252.
debug1: Connection established.
debug1: identity file /home/XXXX/.ssh/id_rsa type -1
debug1: identity file /home/XXXX/.ssh/id_rsa-cert type -1
debug1: identity file /home/XXXX/.ssh/id_dsa type -1
debug1: identity file /home/XXXX/.ssh/id_dsa-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/XXXX/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519 type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519-cert type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519_sk type -1
debug1: identity file /home/XXXX/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/XXXX/.ssh/id_xmss type -1
debug1: identity file /home/XXXX/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
(client username obscured)

Server side (auth.log):
Code:
Feb  1 22:30:29 lon-dc01 sshd[2410673]: refused connect from X.X.X.X (X.X.X.X)
Feb  1 22:31:35 lon-dc01 sshd[2411094]: refused connect from X.X.X.X (X.X.X.X)
Feb  1 22:47:17 lon-dc01 sshd[2417111]: refused connect from X.X.X.X (X.X.X.X)
(client IP obscured)

Fail2Ban was enabled but that has now been disabled - there are no banned IP's with fail2ban-client or in iptables

I have tried SSH'ing from different clients both on the LAN and elsewhere, however all get rejected


Thanks

EDIT: Everything in sshd.conf is default (that came with Proxmox), only the port has been changed
 
Last edited:

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
694
118
hi,

have you tried restarting the SSH service? try: systemctl restart sshd

could you also diff the original sshd config with yours and post it here?

and just in case, what is your output from pveversion -v?
 

alasdairc

New Member
Jun 25, 2021
25
0
1
25
hi,

have you tried restarting the SSH service? try: systemctl restart sshd

could you also diff the original sshd config with yours and post it here?

and just in case, what is your output from pveversion -v?

Hi there,

Thanks for your response

Yes I have tried restarting SSHD and also stopping and starting

My SSHD config:
Code:
#       $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

Port 60252
#AddressFamily any
#ListenAddress 
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none


# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem       sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server

pveversion -v:
Code:
root@lon-dc01:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-10
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.4.151-1-pve: 5.4.151-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.4-1
proxmox-backup-file-restore: 2.1.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.16.0-1
pve-zsync: 2.2.1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1


Thank you
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
694
118
is the ListenAddress in the ssh conf correct?

also:
Code:
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-10
pve-kernel-5.13.19-3-pve: 5.13.19-7

did you reboot after making package upgrades? (you have an older kernel running than the one installed, if that's not intentional please do a reboot)
 

alasdairc

New Member
Jun 25, 2021
25
0
1
25
The listenaddress is correct, I've also tried with commenting that out

The host has been up for 60+ days so I will definitely give that a shot

Thank you
 

alasdairc

New Member
Jun 25, 2021
25
0
1
25
let us know if that changes anything

Hey,

Apologies about the delay, I had to schedule downtime for the host to be rebooted

I have patched the host and given it a reboot - unfortunately still having the same issue


Thank you for your time on this
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
694
118
Apologies about the delay, I had to schedule downtime for the host to be rebooted

I have patched the host and given it a reboot - unfortunately still having the same issue
i've tested the same /etc/ssh/sshd_config file here and it seems to work fine for me.

do you have any other files in /etc/ssh/sshd_config.d/ directory? (those are being included in your config file, so worth taking a look)

also please check your journalctl for any SSH related errors when you restart the service.

btw, is this a single standalone node, or is it a part of a cluster?
 

alasdairc

New Member
Jun 25, 2021
25
0
1
25
i've tested the same /etc/ssh/sshd_config file here and it seems to work fine for me.

do you have any other files in /etc/ssh/sshd_config.d/ directory? (those are being included in your config file, so worth taking a look)

also please check your journalctl for any SSH related errors when you restart the service.

btw, is this a single standalone node, or is it a part of a cluster?

Hey there,

Nothing in /etc/ssh/sshd_config.d/

I have restarted the service and done:
SQL:
systemctl restart sshd ; journalctl -xe
- there are no errors, all green, and the service starts up fine

This is a standalone node


Thank you
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
694
118
thanks.

have you made any other hardening on the host?

on a hunch, could you check the /etc/hosts.allow and /etc/hosts.deny files as well?
 

alasdairc

New Member
Jun 25, 2021
25
0
1
25
thanks.

have you made any other hardening on the host?

on a hunch, could you check the /etc/hosts.allow and /etc/hosts.deny files as well?

WHAT!!!

/etc/hosts.deny had deny all for SSHD!

I have absolutely no idea where that came from, I definitely didn't do it!

I am now able to SSH onto the box... definitely confused though!

Thank you so much for your time and intellect, I really appreciate it :D
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
694
118
WHAT!!!

/etc/hosts.deny had deny all for SSHD!
I have absolutely no idea where that came from, I definitely didn't do it!
neither did i ;)
and it's not set by default :)
I am now able to SSH onto the box... definitely confused though!

Thank you so much for your time and intellect, I really appreciate it :D
gladly! please mark the thread as [SOLVED] so others with the same issue can also know :)
 
  • Like
Reactions: alasdairc

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!