Port number 2230 cannot be used for SSH?

[Sami Vakkuri]

I ask this just for curiosity, since the problem can be avoided by using for example port number 2231 instead...

Can someone explain why exaclty port number 2230 cannot be used to preroute SSH traffic to VM's port 22? I cannot find any reference that this particular port number would be used by some service on PVE or even assigned as standard port for some service.

So I have added rules like this to PVE's /etc/network/interfaces so that I can connect directly to a VM by using same domain name (server.example.com) and different port number for each VM:

#SERVICE NAME, Description
post-up   iptables -t nat -A PREROUTING -p tcp -d [PUBLIC-IP-ADDRESS] --dport 2230 -i vmbr0 -j DNAT --to-destination [INTERNAL-IP-ADDRESS]:22
post-down iptables -t nat -D PREROUTING -p tcp -d [PUBLIC-IP-ADDRESS] --dport 2230 -i vmbr0 -j DNAT --to-destination [INTERNAL-IP-ADDRESS]:22

After changing this file, I run systemctl restart networking on PVE

I have my own naming-scheme for port numbers to keep everything under control (basically number will start with '2' and then VM-ID will follow), so this is why I have "strange" port numbers for services. I have tens of such rules in the interfaces file and I have used same method every time I have created a new VM. However port number 2230 is now the first that just does not work: Connecting with SSH just times out.

Firewall cannot be an issue: there is rules only on VM level and those allow connection from limited public IP's (to destination port 22). Connection starts to work if I change the rule above to use --dport 2231 (or someting else) instead - and breaks again if I change --dport 2230.

I tried to use puTTY and WinSCP for connection from outside (to server.example.com), I also tried telnetting that port and that also results as time-out. I am able to connect to VM internal IP address with port 22 all the time (from other VM on the same network), so this must be an issue with PVE?

I don't have another PVE server setup like this one so I cannot test does this problem occur always in PVE.

 

[ggoller]

PVE shouldn't use port 2230, you could maybe check ss -tulpn if there is a service running on that port.

 

[Sami Vakkuri]

Thank you for quick reply!
Command results:

Netid         State          Recv-Q         Send-Q                 Local Address:Port                  Peer Address:Port         Process                                                                                                                         
udp           UNCONN         0              0                            0.0.0.0:111                        0.0.0.0:*             users:(("rpcbind",pid=860,fd=5),("systemd",pid=1,fd=37))                                                                       
udp           UNCONN         0              0                          127.0.0.1:323                        0.0.0.0:*             users:(("chronyd",pid=1038,fd=5))                                                                                               
udp           UNCONN         0              0                               [::]:111                           [::]:*             users:(("rpcbind",pid=860,fd=7),("systemd",pid=1,fd=39))                                                                       
udp           UNCONN         0              0                              [::1]:323                           [::]:*             users:(("chronyd",pid=1038,fd=6))                                                                                               
tcp           LISTEN         0              128                          0.0.0.0:22                         0.0.0.0:*             users:(("sshd",pid=1157,fd=3))                                                                                                 
tcp           LISTEN         0              100                          0.0.0.0:25                         0.0.0.0:*             users:(("master",pid=1382,fd=13))                                                                                               
tcp           LISTEN         0              4096                         0.0.0.0:111                        0.0.0.0:*             users:(("rpcbind",pid=860,fd=4),("systemd",pid=1,fd=36))                                                                       
tcp           LISTEN         0              4096                       127.0.0.1:85                         0.0.0.0:*             users:(("pvedaemon worke",pid=1945363,fd=6),("pvedaemon worke",pid=1944395,fd=6),("pvedaemon worke",pid=1937126,fd=6),("pvedaemon",pid=1489,fd=6))
tcp           LISTEN         0              4096                               *:8006                             *:*             users:(("pveproxy worker",pid=1945248,fd=6),("pveproxy worker",pid=1941276,fd=6),("pveproxy worker",pid=1939515,fd=6),("pveproxy",pid=1503,fd=6))
tcp           LISTEN         0              4096                               *:3128                             *:*             users:(("spiceproxy work",pid=1699417,fd=6),("spiceproxy",pid=1509,fd=6))                                                       
tcp           LISTEN         0              128                             [::]:22                            [::]:*             users:(("sshd",pid=1157,fd=4))                                                                                                 
tcp           LISTEN         0              100                             [::]:25                            [::]:*             users:(("master",pid=1382,fd=14))                                                                                               
tcp           LISTEN         0              4096                            [::]:111                           [::]:*             users:(("rpcbind",pid=860,fd=6),("systemd",pid=1,fd=38))

So, if I understand this correctly, only port in use (that technically could be used by my naming scheme) would be 3128 - but 2230 is not listed. Actually 3128 port seems to be used by SPICE console, and it seems to stay same even if I reopen SPICE. Quick Googling and that is actually default port for SPICE, so there is no possibility that SPICE could have used 2230 when I was troubleshooting this...

 

[KevinS]

Have you tried checking with tcpdump where the packet is getting lost? (for example: tcpdump -i any host <source-ip>)

I'm pretty sure there is some NAT or firewall issue, but we would need a full output to confirm that.

 

[Sami Vakkuri]

I ran tcpdump on PVE but I needed to limit dst port 2230 for tcpdump since there was so much traffic (over 3000 packets per second) without limitation that I was not able to spot this on connection try in output:

root@server-name:~# tcpdump -i any host [REMOTE-IP] and dst port 2230
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:20:23.310656 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.56101 > server-name.example.com.2230: Flags [S], seq 1046470395, win 65535, options [mss 1400,nop,wscale 7,nop,nop,sackOK], length 0
18:20:24.325383 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.56101 > server-name.example.com.2230: Flags [S], seq 1046470395, win 65535, options [mss 1400,nop,wscale 7,nop,nop,sackOK], length 0
18:20:26.318760 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.56101 > server-name.example.com.2230: Flags [S], seq 1046470395, win 65535, options [mss 1400,nop,wscale 7,nop,nop,sackOK], length 0
[...]

So here we have connections coming from my WinSCP when it tried to connect to 2230. However, I don't know how tcpdump could reveal where these packages are sent? As an comparison, If I connect using 2231, the first dump looks like the same, but following differ because VM actually replies for first package:

root@server-name:~# tcpdump -i any host [REMOTE-IP] and dst port 2231
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
10:23:29.133585 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.61881 > server-name.example.com.2231: Flags [S], seq 1909574109, win 65535, options [mss 1400,nop,wscale 7,nop,nop,sackOK], length 0
10:23:29.201660 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.61881 > server-name.example.com.2231: Flags [.], ack 2800853180, win 32768, length 0
10:23:29.234613 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.61881 > server-name.example.com.2231: Flags [P.], seq 0:30, ack 1, win 32768, length 30
10:23:29.320241 enp33s0f0np0 In  IP 1-2-3-4.isp.domain.com.61881 > server-name.example.com.2231: Flags [.], ack 44, win 32767, length 0
[...]

 

[KevinS]

That's why I mentioned only limiting the IP-Address

But in that case limit the port to 22, you will see if the package hits the tap interface of the VM or not.