Dedicated NIC for PBS

[ufoludek]

Hey, I have 2 servers sitting next to each other, one is running PVE the other one is running PBS, I had 2 SFP+ cards lying around so I wanted to connect both of them, I installed the cards and everything, they are visible both from the BIOS and the OS itself.
How can I tell PVE and PBS to use the 10 gig card between them? I want to use the 1 gig card on PVE and PBS for internet and updates but use use 10 gig card for backups and restores, any way I can achieve that?
Thanks in advance for the help.

 

[Lukas Wagner]

Hi,

simply configure a separate network for this connection. For example, if your main network is 192.168.10.0/24, the direct connection could be 192.168.11.0/24. In PVE, you can add the PBS under the configured IP in the second network.

 

[ufoludek]

Hi,
Let's say I currently have 10.0.10.6 for PVE and 10.0.10.7 for PBS, can I do it with one network or I need a separate network for the 10 gig nics?

 

[bigun89]

If you want a dedicated connection, I'd personally use a network that leaves no room for guesswork.

PVE:
128.50.50.1 / 255.255.255.252

PBS:
128.50.50.2 / 255.255.255.252

No gateway necessary.

 

[ufoludek]

I tried your method @bigun89 but it looks like it's not working.

This is PVE

root@pve:~# ip -br a | grep enp
enp65s0f0        UP             128.50.50.1/30 fe80::a236:9fff:fe1e:ffa4/64
enp65s0f1        DOWN           
root@pve:~# ip link | grep enp
    altname enp193s0f0
    altname enp193s0f1
4: enp65s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
5: enp65s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
root@pve:~# ip route
default via 10.18.6.1 dev vmbr0 proto kernel onlink 
10.18.6.0/24 dev vmbr0 proto kernel scope link src 10.18.6.7 
128.50.50.0/30 dev enp65s0f0 proto kernel scope link src 128.50.50.1

This is PBS

root@pbs:~# ip -br a | grep enp
enp2s0f0         UP             128.50.50.2/30 fe80::a236:9fff:fe98:15a4/64
enp2s0f1         DOWN           
root@pbs:~# ip link | grep enp
    altname enp3s0f0
    altname enp3s0f1
4: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
5: enp2s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
root@pbs:~# ip route
default via 10.18.6.1 dev eno1 proto kernel onlink 
10.18.6.0/24 dev eno1 proto kernel scope link src 10.18.6.9 
128.50.50.0/30 dev enp2s0f0 proto kernel scope link src 128.50.50.2

Ping or anything else won't go through, any idea?

 

[bigun89]

Can I see the /etc/network/interfaces files from both servers?

Also, what models are these SFP+ cards? How are they connected? Twinax? Fiber? If fiber, what models are the modules? What kind of fiber is being used?

I can see the OS is detecting a link, but in the world of SFP+ I've seen that not mean much.

 

[ufoludek]

Can I see the /etc/network/interfaces files from both servers?

Also, what models are these SFP+ cards? How are they connected? Twinax? Fiber? If fiber, what models are the modules? What kind of fiber is being used?

I can see the OS is detecting a link, but in the world of SFP+ I've seen that not mean much.

Sure.

root@pve:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto enp65s0f0
iface enp65s0f0 inet static
    address 10.18.7.1/30
#10gig

iface enp65s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.18.6.7/24
    gateway 10.18.6.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

root@pbs:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
    address 10.18.6.9/24
    gateway 10.18.6.1

iface eno2 inet manual

iface enp2s0f1 inet manual

auto enp2s0f0
iface enp2s0f0 inet static
    address 10.18.7.2/30
#10gig

The cables are DAC SFP+

 

[bigun89]

The configuration looks fine. My configuration is somewhat similar, except my PBS install is virtual and is accessing a SAN and has a dedicated NIC for it - nothing I can add there.

I'm starting to bank on some incompatibility in the hardware - especially since you're using twinax. Look over the twinax cable, it should have a model on it. I need the model on it and on both of your cards.

 

[ufoludek]

The configuration looks fine. My configuration is somewhat similar, except my PBS install is virtual and is accessing a SAN and has a dedicated NIC for it - nothing I can add there.

I'm starting to bank on some incompatibility in the hardware - especially since you're using twinax. Look over the twinax cable, it should have a model on it. I need the model on it and on both of your cards.

The cables are FS 10G SFP+ Passive Direct Attach Copper Twinax Cable SFPP-PC03

The NICs are DELL's Intel 10G 2P X520

 

[bigun89]

100% it's the cable. That's an FS brand and I've always had trouble out of them.

Intel has a compatibility chart that lists the DAC's that will work with their cards. I'm sure you can find one that will work outside of the compatibility chart, but it's a crapshoot at best and it may cause you problems down the line if it does work. Stick with what the manufacturer says when it comes to SFP+.

The compatible cable will run you in the neighborhood of about $12 USD.

 

[bigun89]

Can we get a status update? Was I right?

 

[ufoludek]

Unfortunately I'm not able to replace the cable because the whole infrastructure it's not hosted by us but outsourced, the hosting refuses to change the cable because they say "it works" on other servers.