[SOLVED] 10G ethernet config questions

[rmflint]

I am using 3 different cards between my R710, 620 and R720 servers:
1) Built in card for Dell R620 ( C63DV 0C63DV DELL X520/I350 DAUGHTER CARD 10GBE NETWORK )
2) Add in Intel card: Intel X520-DA2 10Gb 10Gbe 10 Gigabit Network Adapter
3) Add in Mellanox card: Mellanox ConnectX-3 EN CX312A Dual Port 10 Gigabit (Also you can use the single port connect-x3)

Thank you so much! It’s refreshing to get a response which is to the point without a bunch of what if’s and maybes.

 

[rmflint]

I will probably go with the Dell x520/i350 daughter card per vshaulsk’s experience but I am curious if anyone has successfully used the Qlogic 57800 sfp+ with Proxmox? The Qlogic card is compatible with my r620 and there is currently a good deal on eBay with the sfp+ transceivers included.

 

[vshaulsk]

What I found is that if the card is compatible with the Dell Server and works good with Debian .... you should not have problems with Proxmox

My entire setup is using dell servers:
R710, R620 and R620 as the proxmox cluster (Running about 15 vm's & 7 containers)
R720 as my Freenas head unit / SAN & NAS (Disk shelf for ISCSI housing SSD's) (Netapp disk shelf housing 24x4TB Sata drives for general NAS use)

- Unifi and Mikrotik 10GB switches
(2 x Mikrotik 10gb 4-port for ISCSI multipath)
(Unifi 16 port 10gb switch for LAN & a couple of VLAN)

I use some standard gigabit switches for corosync and another small 8 port unifi switch for my DMZ network segment

I use all 10Gtek DAC cables marked either ubiquiti, intel or generic and everything seems to work fine @ full 10gb speeds

 

[rmflint]

What I found is that if the card is compatible with the Dell Server and works good with Debian .... you should not have problems with Proxmox

My entire setup is using dell servers:
R710, R620 and R620 as the proxmox cluster (Running about 15 vm's & 7 containers)
R720 as my Freenas head unit / SAN & NAS (Disk shelf for ISCSI housing SSD's) (Netapp disk shelf housing 24x4TB Sata drives for general NAS use)

- Unifi and Mikrotik 10GB switches
(2 x Mikrotik 10gb 4-port for ISCSI multipath)
(Unifi 16 port 10gb switch for LAN & a couple of VLAN)

I use some standard gigabit switches for corosync and another small 8 port unifi switch for my DMZ network segment

I use all 10Gtek DAC cables marked either ubiquiti, intel or generic and everything seems to work fine @ full 10gb speeds

Thanks again for sharing your setup which is basically what I am trying to accomplish. I went ahead and ordered the Intel X520-DA2 for my Supermicro NAS and the X520/I350 for my Dell R620 Proxmox server.

Cheers

 

[rmflint]

What I found is that if the card is compatible with the Dell Server and works good with Debian .... you should not have problems with Proxmox

My entire setup is using dell servers:
R710, R620 and R620 as the proxmox cluster (Running about 15 vm's & 7 containers)
R720 as my Freenas head unit / SAN & NAS (Disk shelf for ISCSI housing SSD's) (Netapp disk shelf housing 24x4TB Sata drives for general NAS use)

- Unifi and Mikrotik 10GB switches
(2 x Mikrotik 10gb 4-port for ISCSI multipath)
(Unifi 16 port 10gb switch for LAN & a couple of VLAN)

I use some standard gigabit switches for corosync and another small 8 port unifi switch for my DMZ network segment

I use all 10Gtek DAC cables marked either ubiquiti, intel or generic and everything seems to work fine @ full 10gb speeds

I just tried the X520/I350 in my r620 but I’m not sure if it’s a bad card or I am missing something. It has power but does not work on the 1g ports.

I get the following error on boot and I don’t see it in the device configuration menu:

Plug & Play Configuration Error
IRQ Allocations
Bus#01/...

Do you think it’s a bad card or possibly a bios issue? I tried reseating it and still same problem. When I put my old 4-port 1g nick card back in that card works fine. I figured I’d wait to see if anyone chimes in before I request a return.

Thanks

 

[vshaulsk]

I have not had issues with the R620/R720 built in NIC cards. I have purchased 3 x C63DV 0C63DV DELL X520/I350 DAUGHTER CARD from Ebay with no issues.

1) I would double check that it is plugged in correctly ..... sounds silly, but from experience I have been burnt by silly things like this more than I would like to admit

2) Typically after install, I have to go through the lifecycle controller and update the NIC firmware to the latest.

Try another card?
Mine have all been recognized by the system once fully plugged in and system booted and completed new hardware check

 

[rmflint]

I have not had issues with the R620/R720 built in NIC cards. I have purchased 3 x C63DV 0C63DV DELL X520/I350 DAUGHTER CARD from Ebay with no issues.

1) I would double check that it is plugged in correctly ..... sounds silly, but from experience I have been burnt by silly things like this more than I would like to admit

2) Typically after install, I have to go through the lifecycle controller and update the NIC firmware to the latest.

Try another card?
Mine have all been recognized by the system once fully plugged in and system booted and completed new hardware check

Launching the lifecycle controller seemed to fix the problem. Unfortunately, the firmware update failed even though it successfully list the available updates via dell ftp server. Apparently this is not working for some reason. I guess I might try downloading the updates to a usb drive and trying to run this through the lifecycle controller instead.

Have you tried updating firmware from dells ftp site in lifecycle controller recently? It appears others are having the same issue.

 

[vshaulsk]

I also have had the same issue when updating the firmware through DELL.
The way I have resolved this (not the only way), was to setup the following:
VM running Dell Openmanage Enterprise - I use this to monitor all my dell servers
On a windows 10 VM I have DELL EMC Repository Manager running.

THe EMC repository manager allows me to download the correct firmware
The Dell Openmanage VM keeps track of the firmware installed within the dell servers as compared to the local repository

Allows me to update all the servers easily....

 

[bferrell]

Let me see.... I'm using
Dell BROADCOM 57810S
Intel X520-T Dual Port

 

[rmflint]

I also have had the same issue when updating the firmware through DELL.
The way I have resolved this (not the only way), was to setup the following:
VM running Dell Openmanage Enterprise - I use this to monitor all my dell servers
On a windows 10 VM I have DELL EMC Repository Manager running.

THe EMC repository manager allows me to download the correct firmware
The Dell Openmanage VM keeps track of the firmware installed within the dell servers as compared to the local repository

Allows me to update all the servers easily....

Can you provide any pointers on how to install the OpenManage Enterprise vm on Proxmox?

Also, I installed emc OpenManage on a Windows vm but have no idea what the login host name, username, and password would be?

Any help would be appreciated!

 

[rmflint]

Can you provide any pointers on how to install the OpenManage Enterprise vm on Proxmox?

Also, I installed emc OpenManage on a Windows vm but have no idea what the login host name, username, and password would be?

Any help would be appreciated!

After lots of research and trial and error, I finally figured how to create a Dell OpenManage Enterprise VM.

Here are the steps which worked for me:

1. Register and download Dell OpenManage Enterprise OVF installers (openmanage_enterprise_ovf_format_3.2.zip)

2. Extract OVF folder and copy to Proxmox host's iso directory (/var/lib/vz/template/iso/)

3. Run Proxmox command 'qm importovf [any available VM id] /var/lib/vz/ovf/template/iso/openmanage_enterprise.x86_64.0.0.1.ovf [your storage device]

ex> 'qm importovf 105 /var/lib/vz/ovf/template/iso/openmanage_enterprise.x86_64.0.0.1.ovf datapool

4. Before starting new vm you will need to make a couple of additions to the setup:

4.1 Add network device and change SCSI Controller to 'VMware PVSCSI' from the 'Hardware' tab
4.2 Enable QEMU agent from the 'Options' tab

5. Start VM and setup password and proper network parameters for your network from the textual configurator via the Console

6. Now you should be able to login to the Dell OpenManage Enterprise interface from your browser (note: default username is 'admin'). The ip address is 'https://[previously configured ip from step 5]

ex> https://192.168.0.20