TCP connections from guest VM to host are refused

[Erik Horn]

I have set up a test cluster with four main servers and a witness. Guest networking is EVPN and routed via BGP to our production network. The firewall is disabled.

A windows guest is running in this environment and is able to communicate with everything except for the host it is running on. The guest can communicate with other cluster members. If I migrate the guest to a different host, the situation is the same, it can not communicate with the host it's running on.

This seems to affect TCP connections. Ping works fine. The connections are refused and not timing out suggesting that the packets are getting to the host and not being routed into the bit bucket.

I've tested using ssh, the webgui, and nc -l. All fail equally.

To illustrate the issue, I ran the following from the guest:

PS C:\Users\redacted> ping -n 1 10.6.112.102

Pinging 10.6.112.102 with 32 bytes of data:
Reply from 10.6.112.102: bytes=32 time<1ms TTL=64

Ping statistics for 10.6.112.102:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
PS C:\Users\redacted> ssh 10.6.112.102
ssh: connect to host 10.6.112.102 port 22: Connection refused

A packet capture from the produced these results:

# tcpdump -npi any host 10.7.1.99 and \( port 22 or icmp \)
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:01:28.560770 tap103i0 P   IP 10.7.1.99 > 10.6.112.102: ICMP echo request, id 1, seq 13394, length 40
15:01:28.560776 fwln103i0 Out IP 10.7.1.99 > 10.6.112.102: ICMP echo request, id 1, seq 13394, length 40
15:01:28.560777 fwpr103p0 P   IP 10.7.1.99 > 10.6.112.102: ICMP echo request, id 1, seq 13394, length 40
15:01:28.560777 Buildup In  IP 10.7.1.99 > 10.6.112.102: ICMP echo request, id 1, seq 13394, length 40
15:01:28.560800 Buildup Out IP 10.6.112.102 > 10.7.1.99: ICMP echo reply, id 1, seq 13394, length 40
15:01:28.560802 fwpr103p0 Out IP 10.6.112.102 > 10.7.1.99: ICMP echo reply, id 1, seq 13394, length 40
15:01:28.560803 fwln103i0 P   IP 10.6.112.102 > 10.7.1.99: ICMP echo reply, id 1, seq 13394, length 40
15:01:28.560806 tap103i0 Out IP 10.6.112.102 > 10.7.1.99: ICMP echo reply, id 1, seq 13394, length 40

15:01:45.402911 tap103i0 P   IP 10.7.1.99.63131 > 10.6.112.102.22: Flags [SEW], seq 272630743, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
15:01:45.402920 fwln103i0 Out IP 10.7.1.99.63131 > 10.6.112.102.22: Flags [SEW], seq 272630743, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
15:01:45.402921 fwpr103p0 P   IP 10.7.1.99.63131 > 10.6.112.102.22: Flags [SEW], seq 272630743, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
15:01:45.402921 Buildup In  IP 10.7.1.99.63131 > 10.6.112.102.22: Flags [SEW], seq 272630743, win 62720, options [mss 8960,nop,wscale 8,nop,nop,sackOK], length 0
15:01:45.402946 Buildup Out IP 10.6.112.102.22 > 10.7.1.99.63131: Flags [R.], seq 0, ack 272630744, win 0, length 0
15:01:45.402948 fwpr103p0 Out IP 10.6.112.102.22 > 10.7.1.99.63131: Flags [R.], seq 0, ack 1, win 0, length 0
15:01:45.402949 fwln103i0 P   IP 10.6.112.102.22 > 10.7.1.99.63131: Flags [R.], seq 0, ack 1, win 0, length 0
15:01:45.402952 tap103i0 Out IP 10.6.112.102.22 > 10.7.1.99.63131: Flags [R.], seq 0, ack 1, win 0, length 0

I truncated the packet trace after the first connection attempt. The ssh connection goes on to retry several times with differing tcp options, all of which fail.

I haven't been able to determine the reason the connections are refused, especially since ping is working.

If additional information is needed, I'm happy to provide it.

Thanks,

Erik

 

[shanreich]

It seems like the connection 'suceeds', but the host immediately answers with a RST. Are you sure there is no firewall running on the host? Have you checked via iptables-save ? Is SSH set up to only allow certain IP ranges?

 

[Erik Horn]

The ssh config hasn't been modified. It's not a problem with IP addresses, because migrating the VM to another cluster member allows it to communicate with the host that it was previously on. The only thing that I know of that is changing is the path the packets take to get to the host.

The output from iptables-save:

# iptables-save
# Generated by iptables-save v1.8.9 on Fri Sep 13 08:11:29 2024
*raw
:PREROUTING ACCEPT [118526239:149805297261]
:OUTPUT ACCEPT [114780072:208240813870]
COMMIT
# Completed on Fri Sep 13 08:11:29 2024
# Generated by iptables-save v1.8.9 on Fri Sep 13 08:11:29 2024
*filter
:INPUT ACCEPT [35964667:52390023502]
:FORWARD ACCEPT [564721:560787620]
:OUTPUT ACCEPT [35222068:81014411660]
COMMIT
# Completed on Fri Sep 13 08:11:29 2024

This issue also affects the web interface on port 8006. I used nc to set up a temporary service on port 2345 to troubleshoot, just in case it was a security policy in ssh and pveproxy. It shows the same issue.

Thanks,

Erik