infiniband voes

[mir]

Infibind drivers in kernel 3.10 floods the log with this error:
ib0: packet len 2049 (> 2048) too long to send, dropping

Which causes this:
Mar 31 19:17:30 esx2 pmxcfs[3477]: [status] notice: cpg_send_message retry 80
Mar 31 19:17:31 esx2 pmxcfs[3477]: [status] notice: cpg_send_message retry 90
Mar 31 19:17:32 esx2 pmxcfs[3477]: [status] notice: cpg_send_message retry 100
Mar 31 19:17:32 esx2 pmxcfs[3477]: [status] notice: cpg_send_message retried 100 times
Mar 31 19:17:32 esx2 pmxcfs[3477]: [status] crit: cpg_send_message failed: 6


Anybody knows how to solve this?

 

[dietmar]

I guess you need to set a larger MTU size on ib0.

 

[dietmar]

There are some multicast notes in:

https://pve.proxmox.com/wiki/Multicast_notes

Did you configure netmtu in cluster.conf?

 

[mir]

auto ib0
iface ib0 inet static
address 10.0.0.2
netmask 255.255.255.0
pre-up echo connected > /sys/class/net/ib0/mode
mtu 65520

 

[mir]

<totem netmtu="2044" token="54000"/>

 

[mir]

removing netmtu fron cluster.conf solved the problem

 

[dietmar]

Please check if the MTU setting is correctly applied:

# cat /sys/class/net/ib0/mtu

 

[mir]

I have found the fail. Since multicast adds a 5 bytes header the max netmtu size is 2043 and not 2044 as explained in the wiki. Using a netmtu size of 2043 works as intended.

 

[term]

I updated the wiki with the corrected value of netmtu 2043.

 

[mir]

but maybe this is a kernel 3.10 issue since I have used 2044 for netmtu on 2.6.32 for a long time without problems. But since 2044 works on 2.6.32 2043 will also work so maybe we should leave it with 2043 to be on the safe side.