HP Ethernet 10GB 530T

A

adamb

Famous Member
Proxmox Subscriber
Mar 1, 2012
1,329
77
113
We are looking to utilize the HP Ethernet 10GB 530T within our clusters (From the looks of it, it is just a broadcom card). After a fresh install this card is not recognized. I have no issues seeing this card with centos6 or rhel6. Is there something special which I need to do?

I would imagine it uses the bnx2x driver, but unsure. I attempted to "modprobe bnx2x" without any luck.

It looks like the kernel is loading the module, but something just isn't right.

Oct 1 14:58:44 testprox1 kernel: bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.74.22 ($DateTime: 2012/12/19 23:09:04 $)
Oct 1 14:58:44 testprox1 kernel: bnx2x: msix capability found
Oct 1 14:58:44 testprox1 kernel: bnx2x: about to allocate ethernet device with tx_count 31 rx_count 32
Oct 1 14:58:44 testprox1 kernel: bnx2x 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
Oct 1 14:58:44 testprox1 kernel: bnx2x 0000:24:00.0: setting latency timer to 64
Oct 1 14:58:44 testprox1 kernel: bnx2x 0000:24:00.0: part number 0-0-0-0

Not sure if the enterprise repository would help in this situation, but I plan on purchasing a subscription for this cluster once we pin down how we will make the payment (paypal/bank account is quite odd for a company to pay with)
 
Last edited:
adamb said:
Oct 1 14:58:44 testprox1 kernel: bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.74.22 ($DateTime: 2012/12/19 23:09:04 $)
Oct 1 14:58:44 testprox1 kernel: bnx2x: msix capability found
Oct 1 14:58:44 testprox1 kernel: bnx2x: about to allocate ethernet device with tx_count 31 rx_count 32
Oct 1 14:58:44 testprox1 kernel: bnx2x 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
Oct 1 14:58:44 testprox1 kernel: bnx2x 0000:24:00.0: setting latency timer to 64
Oct 1 14:58:44 testprox1 kernel: bnx2x 0000:24:00.0: part number 0-0-0-0
Click to expand...
Obviously the card works in Linux since you can already see it in CentOS. The HP site also has list of drivers for SuSE and RedHat but there does not seem to be any drivers for Debian. Did you try to install Debian or Ubuntu and see your card is recognized?

adamb said:
Not sure if the enterprise repository would help in this situation, but I plan on purchasing a subscription for this cluster once we pin down how we will make the payment (paypal/bank account is quite odd for a company to pay with)
Click to expand...
Dont think Enterprise repo will do any good since drivers are included in all versions of PVE i believe.
PayPal does support payment by Credit Card without needing any paypal account which is very safe. May be try your company Credit Card.
 
symmcom said:
Obviously the card works in Linux since you can already see it in CentOS. The HP site also has list of drivers for SuSE and RedHat but there does not seem to be any drivers for Debian. Did you try to install Debian or Ubuntu and see your card is recognized?


Dont think Enterprise repo will do any good since drivers are included in all versions of PVE i believe.
PayPal does support payment by Credit Card without needing any paypal account which is very safe. May be try your company Credit Card.
Click to expand...

That's where I get confused, proxmox is based on the rhel kernel, so it should have similar drivers. I did find the source for this card on HP's site so maybe I will give it a try in the AM.

The system is seeing the card and loading bnx2x but for reason its just not working. I even noticed that I have entries in "/etc/udev/rules.d/70-persistent-net.rules" with eth2 and eth3 with the bnx2x driver.

# PCI device 0x14e4:/sys/devices/pci0000:00/0000:00:09.0/0000:24:00.0 (bnx2x)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="2c:44:fd:8e:df:90", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x14e4:/sys/devices/pci0000:00/0000:00:09.0/0000:24:00.1 (bnx2x)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="2c:44:fd:8e:df:94", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"


Good point, I complexly forgot we don't need a paypal account to make a payment. Hoping to get this taken care of in the AM also.
 
Alternatively you can swap Intel 10G SFP+ NICs in there. The broadcom drivers did not seem to have a good history with older kernels when I researched this... so we specifically went with Intel 10GBase-T (the copper ones even!).
 
briceburg said:
Alternatively you can swap Intel 10G SFP+ NICs in there. The broadcom drivers did not seem to have a good history with older kernels when I researched this... so we specifically went with Intel 10GBase-T (the copper ones even!).
Click to expand...

I have to disagree, there were some issue in the beginning which I was able to work through. I have 7 clusters using the actual broadcom card, there was a driver issue in the beggining but that was fixed. Now I am trying to use a HP version of this card and it was giving me trouble. I was able to compile it from the source provided on HP's site. I am good to go now, but it doesn't explain why the cards won't work out of the box as I feel it should. I appreciate the input!
 
adamb said:
I have to disagree, there were some issue in the beginning which I was able to work through. I have 7 clusters using the actual broadcom card, there was a driver issue in the beginning but that was fixed. Now I am trying to use a HP version of this card and it was giving me trouble. I was able to compile it from the source provided on HP's site. I am good to go now, but it doesn't explain why the cards won't work out of the box as I feel it should. I appreciate the input!
Click to expand...

I wanted to update this. Looks like the driver works out of the box but I am unable to see the card in dmesg after a fresh install which is what made me think it wasn't being recognized. The default driver seems to work. Now I just need to work through the NMI error one of the nodes is having when disconnecting a cable from the 10GB card. Appreciate the input!
 
the thing is not about the card not working at all - it is about the bnx2x driver generating gigs of log entries (in 3 separate files just to be sure) with these:

Code: 
------------[ cut here ]------------
WARNING: at net/core/dev.c:1758 skb_gso_segment+0x220/0x310() (Tainted: GW  ---------------   )
Hardware name: ProLiant BL460c Gen8
: caps=(0xffffffffc0001020, 0x0) len=2548 data_len=1248 ip_summed=1
Modules linked in: nbd vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nf_conntrack vz dquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_len gth xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limi onfigfs fuse vzevent ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core is csi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc bonding 8021q garp ipv6 ext4 jbd2 snd_pcsp snd_pcm snd_page_alloc snd_timer snd ioatdma iTCO_wdt video soundcore serio_raw hpilo dca hpwdt                                                iTCO_vendor_support output power_meter shpchp ext3 mbcache jbd dm_round_robin dm                                               _multipath ses enclosure sg qla2xxx hpsa scsi_transport_fc scsi_tgt bnx2x mdio [                                               last unloaded: nbd]
Pid: 5868, comm: rsyslogd veid: 0 Tainted: G        W  ---------------    2.6.32-25-pve #1
Call Trace:
 <IRQ>  [<ffffffff8106f757>] ? warn_slowpath_common+0x87/0xe0
 [<ffffffff8106f866>] ? warn_slowpath_fmt+0x46/0x50
 [<ffffffff814d62e5>] ? inet_gso_segment+0x105/0x2a0
 [<ffffffff81469f20>] ? skb_gso_segment+0x220/0x310
 [<ffffffff814756e6>] ? neigh_resolve_output+0x116/0x2f0
 [<ffffffff8146c6c9>] ? dev_hard_start_xmit+0x1a9/0x610
 [<ffffffff81506800>] ? br_dev_queue_push_xmit+0x0/0xa0
 [<ffffffff8146ce38>] ? dev_queue_xmit+0x308/0x530
 [<ffffffff8150686c>] ? br_dev_queue_push_xmit+0x6c/0xa0
 [<ffffffff815068f8>] ? br_forward_finish+0x58/0x60
 [<ffffffff81506bfb>] ? __br_forward+0xab/0xd0
 [<ffffffff814984dc>] ? nf_hook_slow+0xac/0x120
 [<ffffffff81507af0>] ? br_handle_frame_finish+0x0/0x2f0
 [<ffffffff81506d8d>] ? br_forward+0x5d/0x70
 [<ffffffff81507ce7>] ? br_handle_frame_finish+0x1f7/0x2f0
 [<ffffffff81507f8a>] ? br_handle_frame+0x1aa/0x250
 [<ffffffff8146d364>] ? __netif_receive_skb+0x244/0x750
 [<ffffffff8146d9b8>] ? netif_receive_skb+0x58/0x60
 [<ffffffff8145e755>] ? __alloc_skb+0x95/0x260
 [<ffffffff8146dad0>] ? napi_skb_finish+0x50/0x70
 [<ffffffff8151614c>] ? vlan_gro_receive+0x8c/0xa0
 [<ffffffffa003f97a>] ? bnx2x_rx_int+0xefa/0x16b0 [bnx2x]
 [<ffffffff8145ff3a>] ? consume_skb+0x3a/0x80
 [<ffffffffa003d998>] ? bnx2x_free_tx_pkt+0x1b8/0x2a0 [bnx2x]
 [<ffffffffa00401d4>] ? bnx2x_poll+0xa4/0x2e0 [bnx2x]
 [<ffffffff8146e331>] ? net_rx_action+0x1a1/0x3b0
 [<ffffffff810794bb>] ? __do_softirq+0x11b/0x260
 [<ffffffff8100c32c>] ? call_softirq+0x1c/0x30
 [<ffffffff8100de95>] ? do_softirq+0x75/0xb0
 [<ffffffff81079795>] ? irq_exit+0xc5/0xd0
 [<ffffffff8154a382>] ? do_IRQ+0x72/0xe0
 [<ffffffff8100bb13>] ? ret_from_intr+0x0/0x11
 <EOI>  [<ffffffff810707a8>] ? do_syslog+0x218/0x670
 [<ffffffff810707d5>] ? do_syslog+0x245/0x670
 [<ffffffff810bc809>] ? do_futex+0x159/0xb10
 [<ffffffff81218143>] ? kmsg_read+0x43/0x60
 [<ffffffff8120c4eb>] ? proc_reg_read+0x7b/0xc0
 [<ffffffff8154099c>] ? thread_return+0xbc/0x870
 [<ffffffff8119fcde>] ? vfs_read+0x9e/0x190
 [<ffffffff8119fe1a>] ? sys_read+0x4a/0x90
 [<ffffffff8100b182>] ? system_call_fastpath+0x16/0x1b

this happens in both 2.6.32-23 and 2.6.32-25 because apparantly they contain outdated bnx2x drivers (again). Manually updating them fixes this tho
 
Last edited:
well Im afraid I have bad news:

Code: 
Oct 24 04:16:22 prox2 kernel: ------------[ cut here ]------------
Oct 24 04:16:22 prox2 kernel: WARNING: at net/core/dev.c:1758 skb_gso_segment+0x220/0x310() (Tainted: G        W  ---------------   )
Oct 24 04:16:22 prox2 kernel: Hardware name: ProLiant BL460c Gen8
Oct 24 04:16:22 prox2 kernel: tun: caps=(0x80000049, 0x0) len=2800 data_len=1380 ip_summed=1
Oct 24 04:16:22 prox2 kernel: Modules linked in: vzethdev vznetdev pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vzrst nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 vzcpt nf_conntrack vzdquota vzmon vzdev ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp vhost_net tun macvtap macvlan ipt_REJECT kvm_intel ip_tables kvm dlm configfs fuse vzevent ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc bonding 8021q garp ipv6 ext4 jbd2 snd_pcsp snd_pcm snd_page_alloc snd_timer snd soundcore ioatdma hpilo serio_raw iTCO_wdt hpwdt iTCO_vendor_support dca video output shpchp power_meter ext3 mbcache jbd dm_round_robin dm_multipath ses enclosure sg qla2xxx hpsa scsi_transport_fc scsi_tgt bnx2x mdio [last unloaded: scsi_wait_scan]
Oct 24 04:16:22 prox2 kernel: Pid: 0, comm: swapper veid: 0 Tainted: G        W  ---------------    2.6.32-26-pve #1
Oct 24 04:16:22 prox2 kernel: Call Trace:
Oct 24 04:16:22 prox2 kernel: <IRQ>  [<ffffffff8106f757>] ? warn_slowpath_common+0x87/0xe0
Oct 24 04:16:22 prox2 kernel: [<ffffffff8106f866>] ? warn_slowpath_fmt+0x46/0x50
Oct 24 04:16:22 prox2 kernel: [<ffffffff81469f20>] ? skb_gso_segment+0x220/0x310
Oct 24 04:16:22 prox2 kernel: [<ffffffff810640b2>] ? default_wake_function+0x12/0x20
Oct 24 04:16:22 prox2 kernel: [<ffffffff8146c6c9>] ? dev_hard_start_xmit+0x1a9/0x610
Oct 24 04:16:22 prox2 kernel: [<ffffffff8148ac3a>] ? sch_direct_xmit+0x16a/0x1d0
Oct 24 04:16:22 prox2 kernel: [<ffffffff8146cec8>] ? dev_queue_xmit+0x398/0x530
Oct 24 04:16:22 prox2 kernel: [<ffffffff8150686c>] ? br_dev_queue_push_xmit+0x6c/0xa0
Oct 24 04:16:22 prox2 kernel: [<ffffffff815068f8>] ? br_forward_finish+0x58/0x60
Oct 24 04:16:22 prox2 kernel: [<ffffffff81506bfb>] ? __br_forward+0xab/0xd0
Oct 24 04:16:22 prox2 kernel: [<ffffffff814984dc>] ? nf_hook_slow+0xac/0x120
Oct 24 04:16:22 prox2 kernel: [<ffffffff81507af0>] ? br_handle_frame_finish+0x0/0x2f0
Oct 24 04:16:22 prox2 kernel: [<ffffffff81506d8d>] ? br_forward+0x5d/0x70
Oct 24 04:16:22 prox2 kernel: [<ffffffff81507ce7>] ? br_handle_frame_finish+0x1f7/0x2f0
Oct 24 04:16:22 prox2 kernel: [<ffffffff81507f8a>] ? br_handle_frame+0x1aa/0x250
Oct 24 04:16:22 prox2 kernel: [<ffffffff8146d364>] ? __netif_receive_skb+0x244/0x750
Oct 24 04:16:22 prox2 kernel: [<ffffffff8146d9b8>] ? netif_receive_skb+0x58/0x60
Oct 24 04:16:22 prox2 kernel: [<ffffffff8145e755>] ? __alloc_skb+0x95/0x260
Oct 24 04:16:22 prox2 kernel: [<ffffffff8146dad0>] ? napi_skb_finish+0x50/0x70
Oct 24 04:16:22 prox2 kernel: [<ffffffff8151614c>] ? vlan_gro_receive+0x8c/0xa0
Oct 24 04:16:22 prox2 kernel: [<ffffffffa0040bd3>] ? bnx2x_rx_int+0xf03/0x17a0 [bnx2x]
Oct 24 04:16:22 prox2 kernel: [<ffffffff8145ff3a>] ? consume_skb+0x3a/0x80
Oct 24 04:16:22 prox2 kernel: [<ffffffff811b0001>] ? __link_path_walk+0x321/0xf30
Oct 24 04:16:22 prox2 kernel: [<ffffffff8104ef43>] ? __wake_up+0x53/0x70
Oct 24 04:16:22 prox2 kernel: [<ffffffffa0041514>] ? bnx2x_poll+0xa4/0x2d0 [bnx2x]
Oct 24 04:16:22 prox2 kernel: [<ffffffff81089ead>] ? send_sigqueue+0xfd/0x1a0
Oct 24 04:16:22 prox2 kernel: [<ffffffff8146e331>] ? net_rx_action+0x1a1/0x3b0
Oct 24 04:16:22 prox2 kernel: [<ffffffff810794bb>] ? __do_softirq+0x11b/0x260
Oct 24 04:16:22 prox2 kernel: [<ffffffff8100c32c>] ? call_softirq+0x1c/0x30
Oct 24 04:16:22 prox2 kernel: [<ffffffff8100de95>] ? do_softirq+0x75/0xb0
Oct 24 04:16:22 prox2 kernel: [<ffffffff81079795>] ? irq_exit+0xc5/0xd0
Oct 24 04:16:22 prox2 kernel: [<ffffffff8154a382>] ? do_IRQ+0x72/0xe0
Oct 24 04:16:22 prox2 kernel: [<ffffffff8100bb13>] ? ret_from_intr+0x0/0x11
Oct 24 04:16:22 prox2 kernel: <EOI>  [<ffffffff812dc26b>] ? intel_idle+0xdb/0x160
Oct 24 04:16:22 prox2 kernel: [<ffffffff812dc249>] ? intel_idle+0xb9/0x160
Oct 24 04:16:22 prox2 kernel: [<ffffffff81435cd4>] ? cpuidle_idle_call+0x94/0x130
Oct 24 04:16:22 prox2 kernel: [<ffffffff81009219>] ? cpu_idle+0xa9/0x100
Oct 24 04:16:22 prox2 kernel: [<ffffffff8151c831>] ? rest_init+0x85/0x94
Oct 24 04:16:22 prox2 kernel: [<ffffffff81c34cd6>] ? start_kernel+0x40b/0x417
Oct 24 04:16:22 prox2 kernel: [<ffffffff81c3433b>] ? x86_64_start_reservations+0x126/0x12a
Oct 24 04:16:22 prox2 kernel: [<ffffffff81c34436>] ? x86_64_start_kernel+0xf7/0x106
Oct 24 04:16:22 prox2 kernel: ---[ end trace 1b84d4beb921b055 ]---

Code: 
root@prox2:/etc/pve/openvz# uname -a
Linux prox2 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 GNU/Linux
root@prox2:/etc/pve/openvz# pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-8
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-17
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-4
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1
 
What version of bnx2x is in the new kernel? I am running version:1.76.54 without any issues on 2.6.32-25-pve. I have been on this for a week or to now, and its been stressed pretty hard.
 
root@prox2:~# modinfo bnx2x
filename: /lib/modules/2.6.32-26-pve/kernel/drivers/net/bnx2x/bnx2x.ko
version: 1.76.54
license: GPL
description: Broadcom NetXtreme II 10G/20G Ethernet Driver
author: Eliezer Tamir
srcversion: 7074158BE540690CF35B82D
 
adamb said:
What version of bnx2x is in the new kernel? I am running version:1.76.54 without any issues on 2.6.32-25-pve. I have been on this for a week or to now, and its been stressed pretty hard.
Click to expand...

the same.

Code: 
root@pve:~# uname -a
Linux pve 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 GNU/Linux

root@pve :~# modinfo bnx2x
filename:       /lib/modules/2.6.32-26-pve/kernel/drivers/net/bnx2x/bnx2x.ko
version:        1.76.54
license:        GPL
description:    Broadcom NetXtreme II 10G/20G Ethernet Driver
 
mo_ said:
root@prox2:~# modinfo bnx2x
filename: /lib/modules/2.6.32-26-pve/kernel/drivers/net/bnx2x/bnx2x.ko
version: 1.76.54
license: GPL
description: Broadcom NetXtreme II 10G/20G Ethernet Driver
author: Eliezer Tamir
srcversion: 7074158BE540690CF35B82D
Click to expand...

Any specific you were doing to cause this?
 
its just regular day to day workflow happening: people using the fileserver, CMSses, mailserver, groupware and such. no backups/migrations/large host-based file/image-transfers

In the meantime I found this: http://forum.broadcom.com/showthread.php?38533-Debian-6-0-7-2-6-32-19-GSO-Errors

And while this is for an older kernel version, it appears to not have been completely fixed. Ill try the workaround provided in the very last post there
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back