Network timeouts

[mangelot]

Hello,

I wonder if someone could help me finding the cause off our problem. I'm using the latest version of Proxmox.

This is my situation:

eno1 has ipv4 and ipv6 network
eno2 has a different subnet ipv4 (vlan25 untagged on switch)

vmbr0 > slave eno1 added IPv4 address 187.xx.31.xx/32 gateway 187.xx.31.1 (Proxmox Gui is working on port 8006)
vmbr1 > slave eno2 this interface is connected to the different subnet, we don't need proxmox gui on this interface so no static ip is set on the host for this.

LXC container 101
(ID:net0 name:eth0 Bridge:vmbr0 static ip: 187.xx.31.10/32 Gateway ipv4 187.xx.31.1)
(ID:net0 name:eth1 Bridge:vmbr0 static ip: 2a00:xxxx:x:31::10/32 Gateway IPV6 2a00:xxxx:x:31::1)

LXC container 102
(ID:net0 name:eth0 Bridge:vmbr0 static ip: 187.xx.31.11/32 Gateway ipv4 187.xx.31.1)
(ID:net0 name:eth1 Bridge:vmbr0 static ip: 2a00:xxxx:x:31::11/32 Gateway IPV6 2a00:xxxx:x:31::1)

LXC container 103
(ID:net0 name:eth0 Bridge:vmbr0 static ip: 187.xx.31.12/32 Gateway ipv4 187.xx.31.1)
(ID:net0 name:eth1 Bridge:vmbr0 static ip: 2a00:xxxx:x:31::12/32 Gateway IPV6 2a00:xxxx:x:31::1)

LXC container 104 (using differrent subnet 25)
(ID:net0 name:eth0 Bridge:vmbr1 static ip: 187.xx.25.10/32 Gateway ipv4 187.xx.25.1)
(ID:net0 name:eth1 Bridge:vmbr0 static ip: 2a00:xxxx:31:25::10/32 Gateway IPV6 2a00:xxxx:x:31::1)

Host /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno2 inet manual

iface enp65s0 inet manual

iface enp65s0d1 inet manual

iface eno1 inet manual


auto vmbr0
iface vmbr0 inet static
    address 187.xxx.31.5/32
    gateway 187.xxx.31.1
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0

iface vmbr0 inet6 static
    address 2a00:xxxx:1:31::5/32
    gateway 2a00:xxxx:1:31::1

auto vmbr1
iface vmbr1 inet manual
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

route inside containers ipv4:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway.DC    0.0.0.0         UG    100    0        0 eth0
default         gateway.DC    0.0.0.0         UG    101    0        0 eth1
gateway.DC 0.0.0.0         255.255.255.255 UH    100    0        0 eth0
gateway.DC 0.0.0.0         255.255.255.255 UH    101    0        0 eth1

Kernel IPv6 routing table

Destination                    Next Hop                   Flag Met Ref Use If
localhost/128                  [::]                       U    256 2      0 lo
2a00:xxxx::/32                 [::]                       U    102 19      0 eth2
2a00:xxxx::/32                 [::]                       U    103 1      0 eth3
fe80::/64                      [::]                       U    102 1      0 eth2
fe80::/64                      [::]                       U    103 1      0 eth3
[::]/0                         _gateway                   UG   102 19      0 eth2
[::]/0                         _gateway                   UG   103 1      0 eth3
localhost/128                  [::]                       Un   0   19      0 lo
srv101.hostname.com/128         [::]                       Un   0   26      0 eth2
domainname.com/128        [::]                       Un   0   21      0 eth3
srv101.hostname.com/128         [::]                       Un   0   26      0 eth2
srv101.hostname.com/128         [::]                       Un   0   3      0 eth3
ff00::/8                       [::]                       U    256 26      0 eth2
ff00::/8                       [::]                       U    256 25      0 eth3
[::]/0                         [::]                       !n   -1  1      0 lo

From inside the containers we can ping ipv4 and ipv6 in and out, no problems ;-)
If we use WINSCP sometimes the containers are unreachable and slow when refreshing the file structures, every now and then with some connections lost and we must reconnect, and everything is then just fine for some minutes

One of the containers (104) is running Bind nameserver and ipv4 resolves okay, but IPv6 timeouts sometimes, not always btw.
I tried with Firewalls off and on, but no difference

Any ideas whats causing these slow network en timeouts.. ??

 

[aaron]

Did you check if there is packet loss during these times?

You can check the counters for example with netstat -s and look out for lines with retransmit*

That would be one explanation. Could be caused by problematic hardware like a somewhat faulty cable for example.

 

[mangelot]

 netstat -s
Ip:
    Forwarding: 2
    9372408 total packets received
    1129061 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    8957 incoming packets delivered
    6850 requests sent out
    55 outgoing packets dropped
    42 dropped because of missing route
    242784 reassemblies required
    121392 packets reassembled ok
    121392 fragments received ok
    242784 fragments created
Icmp:
    772 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
        destination unreachable: 116
        echo requests: 656
    772 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 116
        echo replies: 656
IcmpMsg:
        InType3: 116
        InType8: 656
        OutType0: 656
        OutType3: 116
Tcp:
    39 active connection openings
    192 passive connection openings
    3 failed connection attempts
    2 connection resets received
    1 connections established
    8446 segments received
    7874 segments sent out
    3 segments retransmitted
    0 bad segments received
    0 resets sent
Udp:
    36 packets received
    116 packets to unknown port received
    0 packet receive errors
    152 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    3 resets received for embryonic SYN_RECV sockets
    89 TCP sockets finished time wait in fast timer
    4 delayed acks sent
    1602 packet headers predicted
    2802 acknowledgments not containing data payload received
    1789 predicted acknowledgments
    Detected reordering 1 times using SACK
    TCPTimeouts: 1
    TCPLossProbes: 2
    TCPDSACKRecv: 1
    TCPDSACKIgnoredNoUndo: 1
    TCPSackShiftFallback: 2
    IPReversePathFilter: 495051
    TCPRcvCoalesce: 72
    TCPOFOQueue: 2
    TCPAutoCorking: 4
    TCPSynRetrans: 1
    TCPOrigDataSent: 5830
    TCPHystartTrainDetect: 7
    TCPHystartTrainCwnd: 408
    TCPDelivered: 5865
IpExt:
    OutMcastPkts: 34
    InBcastPkts: 669237
    InOctets: 1122141630
    OutOctets: 6294138
    OutMcastOctets: 1360
    InBcastOctets: 371979180
    InNoECTPkts: 9372359
    InECT1Pkts: 3
    InECT0Pkts: 46

This is the output (notice only 3 segments retransmitted) but lots of (with invalid addresses) is this the problem?