[SOLVED] issue with one vlan

[RobFantini]

we have a vlan that has connection issues a couple of times per week. vlan 3 has most of our server vm's like ldap, dhcp, nextcloud and 20 others.

we can run traceroute to all vlans addresses except vlan 3 . examples next:

# vlan 3
# traceroute mail
traceroute to mail (10.1.3.14), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
...
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *


# OK
# traceroute pve2
traceroute to pve2 (10.1.10.2), 30 hops max, 60 byte packets
 1  _gateway (10.1.3.1)  0.808 ms  1.223 ms  2.888 ms
 2  pve2.fantinibakery.com (10.1.10.2)  0.166 ms  0.139 ms  0.112 ms

# traceroute alex
traceroute to alex (10.1.37.81), 30 hops max, 60 byte packets
 1  _gateway (10.1.3.1)  1.365 ms  1.618 ms  1.932 ms
 2  alex.fantinibakery.com (10.1.37.81)  0.458 ms  0.514 ms  0.479 ms

I know it is impossible to debug our issue without switch and cluster network config.

all vlans are tagged at the netgear m5300 .

we use bridged network at pve using lacp bond.

at pve we use just vmbr0

auto vmbr0
iface vmbr0 inet static
        address 10.1.10.3/24
        gateway 10.1.10.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr3
iface vmbr3 inet static
        address 10.1.130.3/24
        bridge-ports bond3
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

we do not have an separate entries per vlan like vmbr0.3 . perhaps those are needed?


any suggestions on a direction to go to try to fix?

 

[RobFantini]

so we just use vmbr0 and tak vms like this:

net0: virtio=F6:BC:CB:79:8D:3E,bridge=vmbr0,tag=3

 

[RobFantini]

using -I on traceroute works :

# traceroute -I mail
traceroute to mail (10.1.3.14), 30 hops max, 60 byte packets
 1  mail.fantinibakery.com (10.1.3.14)  0.315 ms * *

 

[Stoiko Ivanov]

we can run traceroute to all vlans addresses except vlan 3 . examples next:

Where do you run those traceroutes from?
- what path do the packets take? (`ip route get 10.1.3.14` run on the same prompt should tell you)

using -I on traceroute works :

if traceroute default fails and traceroute -I succeeds - I would take a look at the router/switch/firewall for potential rules regarding UDP packets (traceroute without options should send udp datagrams to ports 33434 and above - see `man traceroute`)

we have a vlan that has connection issues a couple of times per week

if the issue does not happen always but only every now and then - my guess would be that something on the way might get overloaded and thus drop packets - sadly those issues are harder to debug than, a hard failure ....

we do not have an separate entries per vlan like vmbr0.3 . perhaps those are needed?

not necessarily - especially if ICMP packets (traceoute -I) work

I hope this helps!

 

[RobFantini]

Hello Stoiko,
thank you for the reply!

'- what path do the packets take? (`ip route get 10.1.3.14` run on the same prompt should tell you)' :

from a pve node

# ip route get 10.1.3.14
10.1.3.14 via 10.1.10.1 dev vmbr0 src 10.1.10.3 uid 0

from a desktop

# ip route get 10.1.3.14
10.1.3.14 via 10.1.37.1 dev eno1 src 10.1.37.80 uid 0
    cache

It is good that pve interfaces do not need to be changed.

I'll look at firewalls next.

 

[RobFantini]

disabling UFW at the target fixes traceroute .

network does seem to be the cause of overload causes service issues. we are in the process of upgrading hardware from 1G to Mellanox 25G nics and 40G switch. so until the upgrade is done , we'll limit some of the high bandwith clients.

thanks again for pointing the way out of the network maze.

 

[Stoiko Ivanov]

glad you found the reason for the not working traceroute!

please mark the thread as 'SOLVED' - this helps other users who run into similar problems

Thanks!