tg3 networking module: multicast trouble

[geomaster]

HN

#dmidecode -t baseboard# dmidecode 2.9
SMBIOS 2.3 present.


Handle 0x0029, DMI type 10, 6 bytes
On Board Device Information
        Type: Video
        Status: Enabled
        Description: Intel SVGA Video Graphics Controller


Handle 0x002A, DMI type 10, 6 bytes
On Board Device Information
        Type: Ethernet
        Status: Enabled
        Description: Broadcom5704 Ethernet Device


Handle 0x002B, DMI type 10, 6 bytes
On Board Device Information
        Type: SCSI Controller
        Status: Enabled
        Description: LSI53C10 SCSI Controller


Handle 0x0033, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Intel
        Product Name: Main Board
        Version: d22177-553
        Serial Number: QSHM63302707
        Asset Tag: SN104232
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: Not Specified
        Chassis Handle: 0x0034
        Type: Motherboard
        Contained Object Handles: 0

For NICs kernel load tg3 module. With kernel pve-kernel-2.6.32-13-pve multicast is OK

# asmping 239.192.2.34 vzhost3.kmz
asmping joined (S,G) = (*,239.192.2.234)
pinging 192.168.10.90 from 192.168.10.40
  unicast from 192.168.10.90, seq=1 dist=0 time=0.243 ms
  unicast from 192.168.10.90, seq=2 dist=0 time=0.189 ms
multicast from 192.168.10.90, seq=2 dist=0 time=0.204 ms
  unicast from 192.168.10.90, seq=3 dist=0 time=0.169 ms
multicast from 192.168.10.90, seq=3 dist=0 time=0.183 ms
  unicast from 192.168.10.90, seq=4 dist=0 time=0.143 ms
multicast from 192.168.10.90, seq=4 dist=0 time=0.157 ms
  unicast from 192.168.10.90, seq=5 dist=0 time=0.148 ms
multicast from 192.168.10.90, seq=5 dist=0 time=0.162 ms
  unicast from 192.168.10.90, seq=6 dist=0 time=0.176 ms
multicast from 192.168.10.90, seq=6 dist=0 time=0.189 ms

--- 192.168.10.90 statistics ---
6 packets transmitted, time 5407 ms
unicast:
   6 packets received, 0% packet loss
   rtt min/avg/max/std-dev = 0.143/0.178/0.243/0.033 ms
multicast:
   5 packets received, 0% packet loss since first mc packet (seq 2) recvd
   rtt min/avg/max/std-dev = 0.157/0.179/0.204/0.017 ms

Inspect dmesg:

#cat /var/log/dmesg | grep tg3tg3.c:v3.119 (May 18, 2011)
tg3 0000:0c:02.0: PCI INT A -> GSI 48 (level, low) -> IRQ 48
tg3 0000:0c:02.0: eth0: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:133MHz:64-bit) MAC address 00:0e:0c:43:71:95
tg3 0000:0c:02.0: eth0: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:0c:02.0: eth0: RXcsums[0] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
tg3 0000:0c:02.0: eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
tg3 0000:0c:02.1: PCI INT B -> GSI 49 (level, low) -> IRQ 49
tg3 0000:0c:02.1: eth1: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:133MHz:64-bit) MAC address 00:0e:0c:43:71:96
tg3 0000:0c:02.1: eth1: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:0c:02.1: eth1: RXcsums[0] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
tg3 0000:0c:02.1: eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
tg3 0000:0c:02.0: firmware: requesting tigon/tg3_tso.bin
tg3 0000:0c:02.0: eth0: Link is up at 1000 Mbps, full duplex
tg3 0000:0c:02.0: eth0: Flow control is off for TX and off for RX

Found: tg3 0000:0c:02.0: firmware: requesting tigon/tg3_tso.bin
Firmware is loaded.

OK. Upgrade to proxmox 2.2 with kernel pve-kernel-2.6.32-16-pve. Reboot. And... No quorum, no cluster.
Check dmesg:

# cat /var/log/dmesg | grep tg3tg3.c:v3.122 (December 7, 2011)
tg3 0000:0c:02.0: PCI INT A -> GSI 48 (level, low) -> IRQ 48
tg3 0000:0c:02.0: eth0: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:133MHz:64-bit) MAC address 00:0e:0c:43:71:95
tg3 0000:0c:02.0: eth0: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:0c:02.0: eth0: RXcsums[0] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[0]
tg3 0000:0c:02.0: eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
tg3 0000:0c:02.1: PCI INT B -> GSI 49 (level, low) -> IRQ 49
tg3 0000:0c:02.1: eth1: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:133MHz:64-bit) MAC address 00:0e:0c:43:71:96
tg3 0000:0c:02.1: eth1: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:0c:02.1: eth1: RXcsums[0] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
tg3 0000:0c:02.1: eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
tg3 0000:0c:02.0: eth0: Link is up at 1000 Mbps, full duplex
tg3 0000:0c:02.0: eth0: Flow control is off for TX and off for RX

Found: version for tg3 is changed (3.119 -> 3.122), no firmware loaded.
Check multicast - only unicast. No multicast -> no quorum -> no cluster.

Temporary solution: reboot to kernel pve-kernel-2.6.32-13-pve. Multicast is OK, cluster is OK.

It's magic...
How I can to solve this problem?

 

[dietmar]

Found: version for tg3 is changed (3.119 -> 3.122), no firmware loaded.

And logs in dmesg?

 

[geomaster]

And logs in dmesg?

tg3.c:v3.122 (December 7, 2011)tg3 0000:0c:02.0: PCI INT A -> GSI 48 (level, low) -> IRQ 48
tg3 0000:0c:02.0: eth0: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:133MHz:64-bit) MAC address 00:0e:0c:43:71:95
tg3 0000:0c:02.0: eth0: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:0c:02.0: eth0: RXcsums[0] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[0]
tg3 0000:0c:02.0: eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
tg3 0000:0c:02.1: PCI INT B -> GSI 49 (level, low) -> IRQ 49
tg3 0000:0c:02.1: eth1: Tigon3 [partno(BCM95704A6) rev 2100] (PCIX:133MHz:64-bit) MAC address 00:0e:0c:43:71:96
tg3 0000:0c:02.1: eth1: attached PHY is 5704 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0])
tg3 0000:0c:02.1: eth1: RXcsums[0] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
tg3 0000:0c:02.1: eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
tg3 0000:0c:02.0: eth0: Link is up at 1000 Mbps, full duplex
tg3 0000:0c:02.0: eth0: Flow control is off for TX and off for RX

 

[geomaster]

http://pastebin.com/y5Z0K8YU - full dmesg for kernel pve-kernel-2.6.32-16-pve