Intel X710 and Mikrotik CHR problem

[Deepcuts]

Hello,

I am testing Mikrotik CHR on a MS-01 from Minisforum with Proxmox 8.1.5.
The machine has two SFP+ with Intel X710, two RJ-45 on an additional Intel X550-T2 on PCIe and two Intel I226.
I use one I226-V nic for the management bridge.

I have passed as raw devices one port from X550 and one port from X710 to the Mikrotik VM. X550 as WAN and X710 as LAN.
X710 SFP+ port is connected to a Mikrotik switch in a SFP+ port via a short 1m DAC XS+DA0001
X550 port is connected to an ONT via ethernet.



The issue
When powering on the VM, the X710 nic is not reachable from the internal network connected in the switch, no matter how long I wait.
At 1st, I thought I messed something with CHR configuration, but the same issue is present with a CHR without any configuration whatsoever.
After many "tweaks" and tries, I have discovered that once the VM in up, I have to unplug the DAC from X710 for a couple of seconds and plug it back to have a working connection to the CHR VM. Or I have to disable the interface on the switch for a couple of seconds.
The switch is rulled out as the same issue is present if I connect the X710 into my PC, without any switch in the middle.
I have tried with another DAC and with a RJ-45 module and ethernet cable, to no avail.
All network cards have the latest available firmware installed.

Anyone with a X710 nic or a MS-01 workstation tried using it with Mikrotik CHR and maybe observed the same behavior?

later edit: temporary "workaround" found here

 

[Maverick87Shaka]

Actually my situation with CHR inside proxmox with MS-01 is even worst I believe. Even if I assign the whole interface X710 to the CM with PCIe PT, I'm not seeing them inside CHR as SFP module, I just see them as ethernet port, so not able to check anything regarding SFP status.

Did you actually manage to actually see and manage SFP from CHR directly?
Thanks

 

[Deepcuts]

Actually my situation with CHR inside proxmox with MS-01 is even worst I believe. Even if I assign the whole interface X710 to the CM with PCIe PT, I'm not seeing them inside CHR as SFP module, I just see them as ethernet port, so not able to check anything regarding SFP status.

Did you actually manage to actually see and manage SFP from CHR directly?
Thanks

I gave up on this project for now. Waiting for some magic updates that might fix this.

From CHR with passthrough, the interface is reported as ethernet. No sign of any SPF details. (I have to disable and enable the interface when CHR 1st boots to be able to access it. Weirdly enough, no link down/up reported in CHR logs)



On the switch side I can see the module and some info

 

[emunt6]

hi!

Why not use "virtio" network interface for VM?

You create "bridge interface" on Proxmox host, after you add network card to the bridge, finnally virtio for VMs ( bridged):

vmbr10
  > enps0p1 - X550
  > ether1 - VM


vmbr20
  > enps1p1 -  X710
  > ether2 - VM

You need the following packages:
( You need to use OpenVswitch, because the "ip link" bridge will not work correctly)

$> apt-get install openvswitch-switch
$> apt-get install ifupdown2


/etc/network/interfaces

allow-ovs vmbr10
iface vmbr10 inet manual
ovs_type OVSBridge
ovs_ports enps0p1

allow-ovs vmbr20
iface vmbr20 inet manual
ovs_type OVSBridge
ovs_ports enps1p1

# X550
allow-vmbr10 enps0p1
iface enps0p1 inet manual
ovs_bridge vmbr10
ovs_type OVSPort

# X710
allow-vmbr20 enps1p1
iface enps1p1 inet manual
ovs_bridge vmbr20
ovs_type OVSPort



Manually:
$> ovs-vsctl addbr vmbr10
$> ovs-vsctl addbr vmbr20
$> ovs-vscctl add-port vmbr10 enps0p1
$> ovs-vscctl add-port vmbr20 enps1p1


VMs config:
ethernet1 - virtio, bridge: vmbr10
ethernet2 - virtio, bridge: vmbr20

 

[Deepcuts]

Hi emunt6,

Thank you for your help.

The main reason I do not want to use bridges is because this machine will NAT two 10 Gb/s networks.
From my brief testing, bridge mode is 2.5-3x slower than passthrough. By slower I mean it uses 2.5-3x more CPU resources.
Another reason, albeit not that big of a deal, is that this machine will only run two instances of Mikrotik CHR for routing and I really don't like using a bridge only for one interface. Might be OCD, but I try to stay away from bridges as far as I can while using RouterOS

 

[Maverick87Shaka]

I gave up on this project for now. Waiting for some magic updates that might fix this.

From CHR with passthrough, the interface is reported as ethernet. No sign of any SPF details. (I have to disable and enable the interface when CHR 1st boots to be able to access it. Weirdly enough, no link down/up reported in CHR logs)


On the switch side I can see the module and some info
View attachment 65951

Thanks for the confirmation, I thought since it was in passthrough I should have to see SFP tab (like on physical device) .

So far didn't have module info inside CHR is actually what make me uncomfortable with the CHR inside Proxmox VM. Currently since I have just a single 2,5GB Fiber connection, I'm using just the two ETH port forts, and everything works, but I would like to add some 10GB at some point, and so far I'm not really happy to have sfp+ ports passed to CHR without see stuff on that side.

I don't know what to do......I've mixed feeling about it, I'm also thinking to come back on physical RB5009 router.

 

[Deepcuts]

Thanks for the confirmation, I thought since it was in passthrough I should have to see SFP tab (like on physical device) .

So far didn't have module info inside CHR is actually what make me uncomfortable with the CHR inside Proxmox VM. Currently since I have just a single 2,5GB Fiber connection, I'm using just the two ETH port forts, and everything works, but I would like to add some 10GB at some point, and so far I'm not really happy to have sfp+ ports passed to CHR without see stuff on that side.

I don't know what to do......I've mixed feeling about it, I'm also thinking to come back on physical RB5009 router.

Make sure you are not running a DHCP server on I226-LM, the one in the middle.
Will cause big issues because of Intel vPro.

 

[Maverick87Shaka]

Make sure you are not running a DHCP server on I226-LM, the one in the middle.
Will cause big issues because of Intel vPro.

I've already lost 3 days on that, I'm fine. not using the 226-LM anymore.
The problem is now with the X710 to be honest, I would like to use also that port inside CHR, but even in PT, i see them as a standard ethernet port, unable to managing and see SFP module details.

 

[Deepcuts]

I found a strange "workaround" for this issue.
I was pretty sure this was an issue with Mikrotik CHR, but it seems it has nothing to do with it.

Youtube short recording with the issue and "workaround": https://www.youtube.com/watch?v=Dt97NDDTiU0

Short description of what happens in the video:

Machine was just started.
SSH to Proxmox host and do an ip a to show the interface is there.
Login to Proxmox web.
Start Mikrotik CHR VM.
Open VM's console and check ethernet for link.
Result: no link.

Reboot machine.
SSH to Proxmox host and issue ethtool enp3s0f0 (with or without watch -n1 does not matter)
Login to Proxmox web.
Start Mikrotik CHR VM.
Open VM's console and check ethernet for link.
Result: link.

I have repeated this multiple times to make sure it was not just a fluke.

Now, I have absolutely no clue why just issuing ethtool enp3s0f0 before starting the VM fixes this problem.
There is nothing changed on the nic, just displaying some info about it.
Issuing the command after the VM has been started will throw some error that it cannot find the interface and does not fix the issue.

Maybe anyone more versed can shed a light on this?

 

[Sobte]

Hi, I found a perfect solution

tx/rx flow control must select auto

If you choose on or off, you will still have the above old problems.


Now, reboot.
The problem is solved.

PS: Installed Version 7.14.2

 

[Deepcuts]

Hi, I found a perfect solution

If you choose on or off, you will still have the above old problems.

View attachment 66672
Now, reboot.
The problem is solved.

PS: Installed Version 7.14.2
View attachment 66674

Played with rx and tx FC a bit and noticed some improvements also.
Just when I was about to say I got this fixed, with both set to auto, after a reboot the link was still down.
So in my experience, not a 100% fix or I just messed something up while testing. WIll redo the test.
In the meantime, I ordered some genuine Intel SFP+ modules (INTEL E10GSFPSR) and 1m FO patch cables to test.

later edit:
Setup rx and tx FC to Auto on both ends. Tested a bit and so far so good. No problems so far. Will test for a couple of days to be sure.

 

[Deepcuts]

So far so good. Setting rx and tx to auto on both ends seems to do the trick.
Also, for others that might have this issue and think the problem is with the DAC cable, I replaced it now with an INTEL E10GSFPSR transceiver (SR optics) and a 2m OM5 patch (LC/LC) and I have the same problem. So don't bother.