SSH connection error port 22: Connection refused immediately following successful connect

[cactusphone]

Hi, whenever I am logged in via SSH to the host, I've noticed strange behavior where if I disconnect, and try to reconnect there is a window of about 90 seconds where I get a port 22: Connection refused immediately following. Following that ~90 second window I can login. The pattern is the same for the root user as well as a second Linux PAM user I set up. Firewall is turned off. I set up fail2ban based on the docs provided by proxmox and added the following rules (much longer ban for ssh than 90 seconds)

[proxmox]
enabled = true
port = https,http,8006
filter = proxmox
backend = systemd
maxretry = 3
findtime = 2d
bantime = 3h

[sshd]
port = ssh
logpath = %(sshd_log)s
enabled = true
maxretry = 3
bantime = 3h

I login with a second session ahead of time and look at the fail2 ban jail status after the failed login and confirm no one is banned there

fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:    0
|  |- Total failed:    3
|  `- Journal matches:    _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:    0
   |- Total banned:    0
   `- Banned IP list:

I made no changes to sshd_config, here it is

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/usr/bin:/bin:/usr/games

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile    .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
KbdInteractiveAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the KbdInteractiveAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via KbdInteractiveAuthentication may bypass
# the setting of "PermitRootLogin prohibit-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and KbdInteractiveAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem    sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#    X11Forwarding no
#    AllowTcpForwarding no
#    PermitTTY no
#    ForceCommand cvs server

I added an SSH key for the second user but did not for the root user. The behavior is the same for the second user whether I login with that key or password.

Anyone know what might be causing this? I tried checking journalctl -u ssh.service but only believe I'm seeing records for the successful connections.

 

[bbgeek17]

#LogLevel INFO

You can change the logging level to DEBUG and retry your testing after sshd restart: https://manpages.debian.org/testing...~:text=Gives the verbosity,is not recommended.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[cactusphone]

Thank you, logs do add additional debug info but show no new records while tailing the log during the failed connections.

Safe to assume if nothing is there in the logs the issue is outside of the host machine and something with my internal network?

 

[bbgeek17]

Safe to assume if nothing is there in the logs the issue is outside of the host machine and something with my internal network?

I wouldnt say that its safe to assume anything. It is possible that this is some sort of weird manifestation of duplicate IP, wrongly cached ARP or something similar.
Your PVE installation is sufficiently divergent from vanilla install, that anything is possible. Make sure your ARPs are in order, get tcpdump and watch the traffic/connections.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox