10Gbps Netzwerkkarte

[CV_Chris]

Hallo zusammen,

ich habe einen QNAP-TS-932X direkt (ohne Switch) mit SFP+ 10Gbps an den Proxmox-Server angeschlossen.
Ziel ist es, Netzwerklaufwerke für die VM´s zu integrieren.
NAS: 192.168.10.5/19
Server: 192.168.10.3/19

Über das normale LAN (1Gbit/s) können Sie sich unterhalten.
Die 10Gbit/s-Schnittstelle des NAS habe ich auf die 192.168.8.5/19 gesetzt, allerdings habe ich dann keinen Zugriff mehr auf die Freigegebenen Ordner (via NFS) für Backups vom Proxmox aus. Zudem wird mir die verbaute Netzwerkkarte zwar angezeigt, aber niemals als "aktiv" deklariert.
Ich habe bisher einige verschiedene Konstellationen getestet - leider erfolglos.

Was mache ich falsch?


Oder liegt mein Problem bei den Treiber ( Intel® 82599ES )?

 

[Stefan_R]

Auf der Netzwerkkarte wurde auf PVE Seite keine IP adresse vergeben? So kann der Zugriff nicht klappen.

Zum Umstellen der bisherigen IP (von vmbr0) die bridge bearbeiten und als slave device den Namen der 10G Karte eintragen ("enpXYZ"). Alternativ eine neue Bridge hinzufügen (vmbr1, unter "Erstellen" "Linux Bridge" auswählen) und dort den slave eintragen, dann eine andere IP als auf der vmbr0 vergeben.

 

[CV_Chris]

Sobald ich das mache sind die angebundenen Storages nicht mehr erreichbar und kein Kontakt zum QNAP möglich.
Setting:
QNAP-NAS:
1GB LAN: 192.168.10.5/19
10GB: 192.168.10.17/19

PVE:
1GB-LAN: 192.168.10.3/19
10GB (enp1s0f0 & enp1s0f1 - Bridge): 192.168.10.16/19

Allerdings wird die Netzwerkkarte immer noch nicht als Aktiv angezeigt...




Ich befürchte es liegt tatsächlich an dem Treiber der im Debian initialisiert werden muss. Oder inkompatibel ist?

Danke für deine Hilfe

 

[DoubleD]

Q1: Are you trying to use 1g NIC for management (setup QNAP via WebUI), and use the 10g NIC for NFS traffic?

Q2: Are you trying to bond the 1g NIC and the NIC 10g? or otherwise have them setup as a working-protect (fail-over) scenario?

Q3a: Do you happen to have a 10g switch or other device SFP to see if you can get link light (active)?
Q3b: Do you happen to have another SFP + cable (or even an SFP cable) to verify that you have a good cable?
Q3c: are you using DAC or fiber to connect your SFP + ports?
DAC (direct attached copper) are generally made for certain vendors (Cisco, qnap, etc ...) and are not universal.
Fiber, on the other hand, you need to get SFP + transceiver (insert) for your make (QNAP for QNAP unit) but the fiber is universal.

Q4: Do you see link in the QNAP WebUI?


Q5: In QNAP WebUI, do you have NFS service bound to the NIC 10g?

 

[DoubleD]

Presumably, you have a pair of 10GB NICs on QNAP and a pair of 10GB NICs on PVE host.
1. Set static IP address for all 410GB NICs (not assigned to a bond or bridge) (e.g. 10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4)
2. Use a single SFP + cable to connect QNAP port 1 to QNAP port 2. Do you get link light on QNAP?
2. Use a single SFP + cable to connect PVEhost port 1 to PVEhost port 2. Do you get link light on PVE?

That should tell you:
A. If you have a good cable
B. If you can use that cable with each end point (PVE, QNAP) ... or not.

 

[CV_Chris]

Hi DoubleD,
You definitely asked the right questions.
It works so far now. One more thing would be: I like to connect the NAS and the server directly with 2 cables.
However, it still runs the detour over the entire LAN.
How do I explain to the server that it should only address the NAS directly?



I assigned a static route to the NAS for the 10Gbps network card.

Tanks a lot - Chris

 

[DoubleD]

You have accidentally created a network loop. This is a faux pas in the networking world.
The only reason did your network has not collapsed is because vSwitches in hypervisors (like VMware and Proxmox) have an extra feature set called Spanning Tree Protocol (STP) which intelligently breaks the network loop correctly.


You need to change the IP range for the 10GB network (i.e. 192.168.40.1 and 192.168.40.2).
Benefit # 1: Fixes your network loop (Star network (normal/healthy) + a direct connect on different network for storage traffic)
Benefit # 2: You can now Directly address NFS / QNAP over 10GB

Note: I'd recommend using LACP/802.3ad (need to set at both ends of 10GB connection) for the 10GB pair as it give you max throughput (not that you'd saturate that pipe) and max redundancy. Otherwise, Bind with Round-Robin (or some other NIC binding). VMBR (vSwitch) is typically intended to host multiple devices (IP addresses) behind it. If you were direct connecting two PC's would you place a pair of physical switches between them?

 

[DoubleD]

Sorry, my last project was creating storage network with at pair of Proxmox hosts in a cluster and setup with PVE Storage Management (one-to-one). Not the same as your game end. You want multiple VM's to access multiple NFS shares on a QNAP SAN over a 10GB cross-over (many-to-many).

I know that normally Accessing NFS (or SMB) shares on a NAS means staying on production network (192.168.10.x), and that works on the star network topology (NAS is a spoke on the "hub" / switch). What you have drawn is a network loop.

Plan A: would be to keep everything on 192.168.10.x in star topology and connect to central switch (that happens to be upgraded to a 10GB switch).

Plan B: direct connect PVE & QNAP on 192.168.40.x network
Note1: you'll need a 2nd vNIC on each VM and a 2nd VMBR on PVE (to connect all of those secondary vNICs to storage network)
Note2: NFS shares will have to be on the 192.168.40.x network
Only put NFS on those QNAP 10GB NICs (in pic below, you can see that QNAP allows for service binding to certain NICs)

Note: personally, I'd try to get it working with a single cross-over first and after that works, figure out how to bring in 2nd cross-over for redundancy (and possibly better speed).

Plan C: use IP tables perhaps, but I'm not savvy enough to give good advice on shaping traffic to use cross-over for NFS traffic ONLY.

 

[DoubleD]

Plan C: use IP tables perhaps, but I'm not savvy enough to give good advice on shaping traffic to use cross-over for NFS traffic ONLY, and PREVENT NFS from traversing on the 1GB network.