[SOLVED] Possible MTU misconfiguration detected

T

TechHome

Active Member
Apr 12, 2020
40
1
28
Hello,
I changed the MTU on both nodes to 8988, and I got the likely full bandwidth. But after some minutes all breaks. ISCSI won't work and in the syslog at both nodes is something like this:
un 28 16:22:03 pangolin corosync[2211]: [KNET ] pmtud: possible MTU misconfiguration detected. kernel is reporting MTU: 8988 bytes for host 2 link 1 but the other node is not acknowledging packets of this size. Jun 28 16:22:03 pangolin corosync[2211]: [KNET ] pmtud: This can be caused by this node interface MTU too big or a network device that does not support or has been misconfigured to manage MTU of this size, or packet loss. knet will continue to run but performances might be affected
 
Yes. I have the UniFi Switch PRO 24 POE and enabled jumbo frame. There are also some new errors:

Code: 
Jun 29 17:14:20 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:23 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:26 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:29 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:29 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:29 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2101248 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:29 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:29 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2103296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:29 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:29 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2101248 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:29 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:29 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2103296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:32 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:35 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:38 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:39 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:39 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:39 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2101248 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:39 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:39 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2103296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:40 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:40 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2101248 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:40 pangolin kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 29 17:14:40 pangolin kernel: blk_update_request: I/O error, dev sda, sector 2103296 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Jun 29 17:14:41 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:44 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)
Jun 29 17:14:47 pangolin iscsid[2012]: connect to 192.168.1.46:3260 failed (Connection refused)

EDIT: rebooted and iscsid issues were away. But possible MTU misconfiguration detected is still present.
 
Last edited:
I get this on boot:
Code: 
Jun 30 01:02:30 pangolin corosync[2308]:   [KNET  ] pmtud: PMTUD link change for host: 2 link: 1 from 469 to 6021
Jun 30 01:02:30 pangolin corosync[2308]:   [KNET  ] pmtud: Global data MTU changed to: 6021
Jun 30 01:02:34 pangolin pmxcfs[2297]: [status] notice: received log
Jun 30 01:02:44 pangolin pmxcfs[2297]: [status] notice: received log
Jun 30 01:03:00 pangolin systemd[1]: Starting Proxmox VE replication runner...
Jun 30 01:03:01 pangolin systemd[1]: pvesr.service: Succeeded.
Jun 30 01:03:01 pangolin systemd[1]: Started Proxmox VE replication runner.
Jun 30 01:04:00 pangolin systemd[1]: Starting Proxmox VE replication runner...
Jun 30 01:04:01 pangolin systemd[1]: pvesr.service: Succeeded.
Jun 30 01:04:01 pangolin systemd[1]: Started Proxmox VE replication runner.
Jun 30 01:04:04 pangolin corosync[2308]:   [KNET  ] pmtud: possible MTU misconfiguration detected. kernel is reporting MTU: 8988 bytes for host 2 link 1 but the other node is not acknowledging packets of this size.
Jun 30 01:04:04 pangolin corosync[2308]:   [KNET  ] pmtud: This can be caused by this node interface MTU too big or a network device that does not support or has been misconfigured to manage MTU of this size, or packet loss. knet will continue to run but performances might be affected.
 
as your logs show, the link practically only supports an MTU of 6021.
 
fabian said:
as your logs show, the link practically only supports an MTU of 6021.
Click to expand...

Which link? I have only two nodes. On the node with the warnings I have 2x 10Gbit activated at 9000MTU. Intel says that Intel Corporation Ethernet Connection X557 have jumbo frames....
 
Last edited:
spirit said:
can you send your /etc/network/interfaces config file && results of "ip addr" ?
Click to expand...
Code: 
root@pangolin:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network# configuration from sourced files, so do not attempt to move any of# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno3 inet manual
        mtu 9000

iface eno1 inet manual
        mtu 9000

iface eno2 inet manual
        mtu 9000

iface eno4 inet manual
        mtu 9000

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.100/24
        gateway 192.168.1.1
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0
        mtu 9000

auto vmbr1
iface vmbr1 inet static
        address 192.168.2.100/24
        bridge-ports eno4
        bridge-stp off
        bridge-fd 0
        mtu 9000

root@pangolin:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ac:1f:6b:6a:5b:2e brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ac:1f:6b:6a:5b:2f brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether ac:1f:6b:6a:5f:3e brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether ac:1f:6b:6a:5f:3f brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:6a:5f:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.100/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe6a:5f3e/64 scope link
       valid_lft forever preferred_lft forever
7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:6a:5f:3f brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.100/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe6a:5f3f/64 scope link
       valid_lft forever preferred_lft forever
8: tap108i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 7e:3d:5e:85:dc:5a brd ff:ff:ff:ff:ff:ff
9: tap108i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether f2:e6:cd:b2:f5:a2 brd ff:ff:ff:ff:ff:ff
10: tap120i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 36:d8:aa:b4:ab:a2 brd ff:ff:ff:ff:ff:ff
11: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 02:37:3c:2c:d3:b1 brd ff:ff:ff:ff:ff:ff
12: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 12:45:b6:4a:3e:63 brd ff:ff:ff:ff:ff:ff
13: tap106i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 0a:14:51:5a:07:e7 brd ff:ff:ff:ff:ff:ff
14: tap114i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 32:3e:db:65:82:57 brd ff:ff:ff:ff:ff:ff
15: veth107i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:cb:71:4e:fe:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
16: veth110i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:8a:e1:e9:02:6a brd ff:ff:ff:ff:ff:ff link-netnsid 1
root@pangolin:~#
 
Code: 
Jun 30 01:02:30 pangolin corosync[2308]:   [KNET  ] pmtud: PMTUD link change for host: 2 link: 1 from 469 to 6021

here you can see that knet reached a max MTU of 6021 (via PMTUD, which basically boils down to sending packets of various sizes and seeing which of those reach the other end).
 
your proxmox nodes config seem to be fine. Sound like really a physical switch problem.


you can also verify with a ping like

from node1:

ping -M do -s 8988 <node2>

try to lower the value until you don't have any error like "ping: sendmsg: Message too long"
 
could you try to do an iperf between both nodes ?

node2 : "iperf -s"

node1: iperf -c <ipofnode2> -t 300

(this will made an iperf for 5min)


if it was a corosync bug, this is strange than you also have problems with iscsi.
 
but that's only one link, it's likely the other one that's making problems (links are usually number 0, 1, ..)
 
It's an older network card. It can't do more. But how can I hide this warning for the link 2 on node 2?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back