[SOLVED] intervlan ssh connection gets closed

Urbaman · Jan 24, 2023

Hi,

I'm trying to ssh form my PC (in vlan100) to proxmox port in vlan10. The connection gets closed after some seconds.
If I ssh to the proxmox port in vlan100, the connection stays open.
Is there a reason why?

Thank you.

Urbaman · Jan 24, 2023

For further inspection, I get

Code:

sshd[36338]: error: kex_exchange_identification: Connection closed by remote host

on the host, while

Code:

client_loop: send disconnect: Connection reset

on the client.

cave · Jan 24, 2023

Urbaman said:
I'm trying to ssh form my PC (in vlan100) to proxmox port in vlan10.

Is there any router in between to route traffic from VLAN100 to VLAN10?

Urbaman · Jan 24, 2023

Actually pfSense.
From openvpn (also on pfSense, so on dedicated network) I can ssh on both vlans without problems if this helps.

Urbaman · Jan 24, 2023

Ok,

Tested the firewall rules on pfSense, can't reach the rightly unreacable vlans (like ceph storage, pve cluster, ...), while can reach the others.
Ssh still getting closed connection after some seconds if I ssh cross-network (cross-vlan).

Ping works seamlessly, seems.

cave · Jan 24, 2023

Urbaman said:
Is there a reason why?

Yes.

## ## ##

dare to draw your setup here? https://asciiflow.com/

with IP's, netmasks, with Nic's, with router, with vpn, with firewalls, switches, proxmox network settings (nics, bridges, vlans, which port is tagged, which is an access port), and finally your vlans. sshd config.

### ### ###

anyhow, sounds to me like a pfSense/firewall/network/routing issue not really related to virtualization & Proxmox & PVE .
Probably asking that question on a board focusing on that topic to solving that topic, is more helpful.

Urbaman · Jan 24, 2023

Hi

I'll get back with the network design and settings as soon as I get it done.

Just also saying that I can ssh really fine to other devices on vlan100 (pbs, switch...) with stable connections, just ssh to pve gets closed with that error on pve's syslog, that's why I'm focusing on pve not router/firewall/switch

Thank you very much.

Urbaman · Jan 24, 2023

Hi, here I am with the network diagram, and more informations on the system.

ISP modem/router goes to proxmox, bridged to pfSense.
All other proxmox ports go to the switch, also bridged to pfSense.
Each port has its network/vlan.

Salient Proxmox info:

Routing:

Code:

root@pvenode1:~# ip route
default via 10.0.100.3 dev vmbr9 proto kernel onlink
10.0.10.0/24 dev vmbr1 proto kernel scope link src 10.0.10.11
10.0.20.0/24 dev vmbr2 proto kernel scope link src 10.0.20.11
10.0.30.0/24 dev vmbr3 proto kernel scope link src 10.0.30.11
10.0.40.0/24 dev vmbr4 proto kernel scope link src 10.0.40.11
10.0.50.0/24 dev vmbr5 proto kernel scope link src 10.0.50.11
10.0.60.0/24 dev vmbr6 proto kernel scope link src 10.0.60.11
10.0.70.0/24 dev vmbr7 proto kernel scope link src 10.0.70.11
10.0.80.0/24 dev vmbr8 proto kernel scope link src 10.0.80.11
10.0.100.0/24 dev vmbr9 proto kernel scope link src 10.0.100.11
169.254.0.0/16 dev idrac proto kernel scope link src 169.254.0.2
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.11

Network:

Code:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno2 inet manual
#wan0

iface eno1 inet manual
        mtu 9000
#vlan10 - LAN

iface eno4 inet manual
        mtu 9000
#vlan20 - Proxmox Cluster

iface eno3 inet manual
        mtu 9000
#vlan30 - Ceph Storage

iface enp5s0f3 inet manual
        mtu 9000
#vlan40 - Ceph Public

iface enp5s0f2 inet manual
        mtu 9000
#vlan50 - VMs LAN

iface enp5s0f1 inet manual
        mtu 9000
#vlan60 - pfSense Sync

iface enp5s0f0 inet manual
        mtu 9000
#vlan70 - Shared Storage

iface enp129s0f1 inet manual
        mtu 9000
#vlan80

iface enp129s0f0 inet manual
        mtu 9000
#vlan100 - Management

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.11/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.0.10.11/24
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr2
iface vmbr2 inet static
        address 10.0.20.11/24
        bridge-ports eno4
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr3
iface vmbr3 inet static
        address 10.0.30.11/24
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr4
iface vmbr4 inet static
        address 10.0.40.11/24
        bridge-ports enp5s0f3
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr5
iface vmbr5 inet static
        address 10.0.50.11/24
        bridge-ports enp5s0f2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr6
iface vmbr6 inet static
        address 10.0.60.11/24
        bridge-ports enp5s0f1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr7
iface vmbr7 inet static
        address 10.0.70.11/24
        bridge-ports enp5s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr8
iface vmbr8 inet static
        address 10.0.80.11/24
        bridge-ports enp129s0f1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr9
iface vmbr9 inet static
        address 10.0.100.11/24
        gateway 10.0.100.3
        bridge-ports enp129s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

Symptoms:
Can ping and ssh from PC1 to PC2 or switch, can ping and ssh from PC1 to proxmox on 10.0.10.11.
Can ping and ssh from PC1 to proxmox on 10.0.100.11 but ssh session gets closed...

cave · Jan 24, 2023

@Urbaman
what type is your switch?
where is the VLAN Tagging done?

cave · Jan 24, 2023

pls output from "route" & "route -n" instead "ip route"

Urbaman · Jan 24, 2023

Hi,

I think pfSense should be tagging the vlans, the switch is Cisco Nexus 3172tq.

See the other thread about not being able to reach proxmox/pfsense from PC1 or PC2 when trying to manage vlans on proxmox network.

Urbaman · Jan 24, 2023

Code:

root@pvenode1:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.100.3      0.0.0.0         UG    0      0        0 vmbr9
10.0.10.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
10.0.20.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr2
10.0.30.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr3
10.0.40.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr4
10.0.50.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr5
10.0.60.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr6
10.0.70.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr7
10.0.80.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr8
10.0.100.0      0.0.0.0         255.255.255.0   U     0      0        0 vmbr9
link-local      0.0.0.0         255.255.0.0     U     0      0        0 idrac

Code:

root@pvenode1:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.100.3      0.0.0.0         UG    0      0        0 vmbr9
10.0.10.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
10.0.20.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr2
10.0.30.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr3
10.0.40.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr4
10.0.50.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr5
10.0.60.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr6
10.0.70.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr7
10.0.80.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr8
10.0.100.0      0.0.0.0         255.255.255.0   U     0      0        0 vmbr9
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 idrac

cave · Jan 24, 2023

the port connecting to enp129s0f0 is setup on the switch with which vlan tag?

same for the tag eno1.

cave · Jan 24, 2023

enp129s0f0/vmbr9 is your gateway?
isnt that your vlan100 mgmt network?

wan/vmbr0 is missing in the route?

Urbaman · Jan 24, 2023

Well, quite sure they're correctly tagged right, 10 (eno1) and 100 (enp129s0f0).
If I connect a PC to a similarly tagged port, I get the right IP on the right network from pfSense.
I also can't access other vlans/networks due to psSense rules, but 10 and 100 should be communicating (and thay are, as I can communicate fine with other devices on those networks).

I'll double check the cables, but all of the proxmox ports are connected to access tagged ports of the switch In the right way.

I should mention that I also have other vlans that I'm planning to use for clustering, ceph and so on, but let's keep it simple for the moment.

cave · Jan 24, 2023

Hmm, still i think it's a pfsense issue.

and to be frank, your VLAN setup confuses me.
i have no clue about your guest vlan model.

Looks like you do subnetting on your pfsense and switchsplitting on your cisco Switch.

1 port = 1 vlan = 1 subnet translates to 1 port = 1 subnet
No Need to vlan between switch and proxmox if there is anyhow only a 1 to 1 match - patch cable to vlan.

pc2 is connected to an access port on the switch?

from PC2, you try to ssh root@10.0.100.11

check on your proxmox host if sshd binds to 10.0.100.11 (output netstat -tulpn)

hmm

Urbaman · Jan 24, 2023

I'll try that, and I'll also doublecheck pfSense' settings for interfaces and vlans (already thoguht I didn't need vlan-aware bridges).

I'll also try a Linux machine instead of the two windows machines I'm using as clients.

I'll get back with the output.

Urbaman · Jan 24, 2023

netstat output (did not change while the other PC is in ssh):

Code:

root@pvenode1:~# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      5936/master         
tcp        0      0 127.0.0.1:85            0.0.0.0:*               LISTEN      5997/pvedaemon     
tcp        0      0 127.0.0.1:61000         0.0.0.0:*               LISTEN      593629/kvm         
tcp        0      0 127.0.0.1:61001         0.0.0.0:*               LISTEN      591814/kvm         
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      5575/sshd: /usr/sbi
tcp6       0      0 ::1:25                  :::*                    LISTEN      5936/master         
tcp6       0      0 :::8006                 :::*                    LISTEN      6072/pveproxy       
tcp6       0      0 :::22                   :::*                    LISTEN      5575/sshd: /usr/sbi
tcp6       0      0 :::1311                 :::*                    LISTEN      4998/dsm_om_connsvc
tcp6       0      0 :::3128                 :::*                    LISTEN      6087/spiceproxy     
udp        0      0 127.0.0.1:161           0.0.0.0:*                           5558/snmpd         
udp        0      0 127.0.0.1:323           0.0.0.0:*                           5567/chronyd       
udp6       0      0 ::1:161                 :::*                                5558/snmpd         
udp6       0      0 ::1:323                 :::*                                5567/chronyd

Urbaman · Jan 24, 2023

I'm also posting on Netgate (pfSense) forums, let's see if I can find out the culprit

Urbaman · Jan 26, 2023

Ok, probably they got it explained from Netgate forum.

As I got your setup, Proxmox is multi-homed. It has an IP in vlan10 and also one in vlan100.
This will result into asymmetric routing issues. The request goes from vlan10 though pfSense out to vlan100 to the Proxmox IP. But Proxmox sends its response back to the source IP, which is vlan10 and which it has an interface IP within. So the response goes back the "short way", out on the vlan10 interface and hence doesn't pass pfSense.

This may work for ICMP packets (ping), since they not statefull, but TCP traffic fails.

So you should remove the vlan10 IP from Proxmox.

So, cross-vlan to proxmox node is going to be tricky, I'll need to give only the management IP and the "internal networks" (cluster, ceph) IPs, and contact only to management.

Thank you very much.

[SOLVED] intervlan ssh connection gets closed

Member

Member

Well-Known Member

Member

Member

Well-Known Member

Member

Member

Well-Known Member

Well-Known Member

Member

Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Member

Member

Member