ssh issue between two clusters

[RobFantini]

Hello

I've 2 clusters with these networks 10.1.10.0/24 and 10.1.15.0/24

for some unknown reason I can no longer ssh from one pve cluster to another. they can ping. both can access vm's on any cluster and internet.

Probably this is a managed switch issue [ I've been fine tuning our fully managed switches and probably did something..] While I am 99% certain it is something with switching I'd figure after working on and off at this for hours I'd ask here for a suggestion . I have not done any configuration of pve firewall.

does anyone have a suggestion to fix this?

PS: this kind of thing does not get solved until after a post here.

best regards, Roberto.

 

[RobFantini]

removing 'mtu 9000' from vmbr0 fixed this issue.

I have no idea why .

 

[dfenwick]

One side or the other probably has the DF (don't fragment) bit set. Your routing infrastructure may be blocking path MTU. Essentially, it either can't discover what the maximum MTU of the path is, path MTU is disabled, you're blocking path MTU, or you have don't fragment enabled somewhere. 9000 bytes won't fit through a 1500 byte hole.

 

[RobFantini]

thank you that makes sense.

Can I ask you some other managed switch questions?

 

[dfenwick]

Of course. Fire away.

 

[RobFantini]

for vlan membership a port can have T or U . I use T to pass on a vlan to VM's .

some basic questions:
Is it good practice to use only one vlan U per port?

More info:
I use vlan 8 for all our switches.

on our main stack of 3 Netgear M5300 series, the IP address of the switch is stuck at 169.254.100.100/255.255.0.0 . I assume that becomes the network for vlan 1 . after recent fine tuning I found the I had to add vlan 8 Untagged to all lag ports for all connections to work . So some ports have 2 Untagged vlans. of course only one pvid.

 

[dfenwick]

U stands for untagged. An untagged interface assumes the vlan tagging of the vlan it's associated with on the switch itself, provided it's not supplying a vlan tag at the interface itself. Many switches support dual mode tagging as well, meaning you can have an untagged and a tagged interface. The untagged traffic will belong to whatever vlan the switchport is a member of, and any other tagged traffic (traffic tagged by the end device interfaces) will also be allowed to traverse the switchport.

Having said that, as far as I know you can only ever have a single untagged vlan associated with an interface. I'm sure there are some odd edge cases out there (I seem to remember a weird configuration at one point where we had computers pigtailed off VoIP phones that required some monkeying around) but yes, you can typically *only* have an untagged interface associated with a single vlan.

You're doing it right.

 

[RobFantini]

Thank you for that answer.

regarding the switch's IP:
I assume that the switch IP 169.254.100.100/255.255.0.0 is unreachable from our vlans.

My question:
Is it normal practice for a layer 3 switch to have an address like that?

to me it seems like something that should be dealt with. however as the switch is set up to do vlan routing , the gateway address for each vlan can be left able to access the switch using ssh or telnet. so we have plenty of management addresses.

 

[dfenwick]

169.254.100.100 is the default address for the M5300. Did you change the management addresses on those switches?

 

[RobFantini]

169.254.100.100 is the default address for the M5300. Did you change the management addresses on those switches?

using the latest firmware on our model - gsm7353sv2 the IP can not be on a vlan routing interface. or a similiar warning , and trying to change it may result in the stack rebooting which is not easy as we are open 24/7

anyway my question is this - does leaving that at 169.254.100.100 cause operation issues? I think it is why I needed to put vlan 8U to all interfaces.

 

[dfenwick]

using the latest firmware on our model - gsm7353sv2 the IP can not be on a vlan routing interface. or a similiar warning , and trying to change it may result in the stack rebooting which is not easy as we are open 24/7

anyway my question is this - does leaving that at 169.254.100.100 cause operation issues? I think it is why I needed to put vlan 8U to all interfaces.

It really depends on your environment. I never want anything to sit in what I consider a fail mode on my network. 169.254.x.x leaps out as a broken address in my environments, so I just make sure it's configured right in the first place.

To be fair, I have never touched a M5300. I *have* seen switches that won't let you have your management (aka loopback) interface in the same vlan as other interfaces, but I've never seen a switch that didn't allow you to put it in a routeable vlan with an IP address. That seems a little broken to me.

169.254.100.100 isn't a specific problem, but that's also the IP address range (169.254.0.0/16) that Microsoft puts clients that can't obtain a DHCP address. If you're ok with that, I can't think of any reason why you can't just leave it alone.

 

[RobFantini]

It really depends on your environment. I never want anything to sit in what I consider a fail mode on my network. 169.254.x.x leaps out as a broken address in my environments, so I just make sure it's configured right in the first place.

To be fair, I have never touched a M5300. I *have* seen switches that won't let you have your management (aka loopback) interface in the same vlan as other interfaces, but I've never seen a switch that didn't allow you to put it in a routeable vlan with an IP address. That seems a little broken to me.

169.254.100.100 isn't a specific problem, but that's also the IP address range (169.254.0.0/16) that Microsoft puts clients that can't obtain a DHCP address. If you're ok with that, I can't think of any reason why you can't just leave it alone.

yea that has bothered me like a itch on my nose for 2 months. starting out i had a good address [ cli tool ezconfig was used ]. but there were issues along the way and dhcp no longer get an ack to the switch. the m4300 series had recent upgrades that made this easy to fix. so i'll wait to see if this series gets same feature in an upgrade. then again i'll probably try to change some saturday afternoon when i've a few hours in case needed.