Limited pcie slots, 10Gb SPF+ OR NVME + 2.5Gb

[Mistborn]

Have a low budget Proxmox build I've been running for a few months.

Acer Aspire i3-10105, 16gb ram (non ECC, MB doesn't support ECC)

8TB WD Red x2 zfs mirror for main storage

2TB x2 Teamgroup m34 nvme ZFS mirror for PE install and VMs

This MB only has one pcie 16x slot, and one pcie 1x. Currently have pcie nvme using the 16 slot, 1x is empty.

Uses: OMV lxc, Plex lxc, windows VM with homeseer and blue iris (8 wifi cams so far), pi hole lxc, soon to add omada controller lxc

I am upgrading my network from deco x20s to Omada gear. Switch has 8x 2.5Gbe and 2x 10Gb SPF+.

What would you do?
Just add 2.5Gb pciex1 card. Easiest option but doesn't take advantage of the new SPF+ links

Add 10Gb SPF card. Would have to remove the pcie nvme. Either running the remaining drive without a mirror or changing hardware. I don't think I could do zfs mirror the nvme to a USB nvme drive? What other options?

Internet is not limited by 1Gbe connections. So the 10Gbe speed will only help internal traffic. Right now I get connection issues and buffering from Plex streams and the IP cameras, but the deco units are less than stellar and all being replaced by omada APs.

Leaning towards just using a 2.5 Gbe card for the server and I will have the 10Gb links for down the road if I ever upgrade to a proper server and/or get faster internet. Maybe I can use the 1Gbe connector for just my IP cams and the 2.5 for everything else, but I have blueiris and homeseer in the same VM so not sure I could separate them like that without adding a separate windows VM.

Proxmox has been a fun adventure so far, I knew absolutely zero about Linux when starting so it's been a big learning process.

 

[Mistborn]

So I ended up cutting the back off my 1x pcie port so I can stick the nvme adapter or 10gb SPF+ pcie card in there and run it at the reduced 1x speed. SPF+ card in order, Mellanox MCX311A-XCAT ConnectX-3, which I believe is pcie3.0, should run at about 8gbps on the 1x slot which is pcie 3.0.
(The 16x slot is pcie 4.0 and the 1x is 3.0)
Decided to test the speed of the nvme in the 1x slot. Found that the system couldn't detect the drive in the 1x slot, but the drive in the nvme slot is only reading 800Mb/s! With an nvme in the pcie4.0 16x slot it is reading 2500Mb/s.

So no idea why the nvme slot is running at what looks like pcie 2.0 speed. But since I can't run the nvme adapter in the 1x slot, looks like I will leave the nvmes as is and put the SPF+ adapter in the 1x slot, hopefully that will work.

 

[Mistborn]

Also I tested the nvme drive from Ubuntu live USB, same speed issues and it also could not detect the drive in the 1x slot, so not a Proxmox problem, but probably motherboard, BIOS or Linux problem?

 

[leesteken]

Decided to test the speed of the nvme in the 1x slot. Found that the system couldn't detect the drive in the 1x slot, but the drive in the nvme slot is only reading 800Mb/s! With an nvme in the pcie4.0 16x slot it is reading 2500Mb/s.

Mbits or MBytes? PCIe 3.0 is about 985MBytes/s, so 800 might not too far off (and the x1 probably needs to share bandwidth with other devices via the motherboard chipset).

 

[Mistborn]

Mbits or MBytes? PCIe 3.0 is about 985MBytes/s, so 800 might not too far off (and the x1 probably needs to share bandwidth with other devices via the motherboard chipset).

Sorry those numbers should all be MBytes/s!

With nothing in either pcie slots, the nvme in the motherboard m2 nvme slot, I get 800MB/s.
With the adapter in the pcie 16x slot I get 2500MB/s on the same nvme drive.

The drive won't work in the 1x slot

 

[Mistborn]

Decided to use the 10Gb Mellanox MCX311A-XCAT CX311A ConnectX-3.
Put it in the 1x slot after cutting out the back of the slot. Proxmox detecting it and it is working fine. I don't know what speed it is running, have it connected to my omada switch with a cheap DAC. Not sure how I would test the speed as those are my only 2 SPF+ devices so far. I would guess it would top out at 8gbps on the 1x slot.

Still no idea why my motherboad nvme slot is so slow.
–---------------
On another note installed Omada SDN on Proxmox. For the life of me I could not get it to work on an LXC. Tried installing it 8 different times following slightly different tutorials and 3 different Ubuntu versions. Appears to install, when I run tpeap status it says that it is running. Can ping the LXC but can't access the web interface. Firewalls all turned of to trouble shoot it. But seems like it is not really running right because if I stop it and restart it it does not restart. I could not find anything obvious in the logs to help me with the issue.

Gave up and installed it on Ubuntu 20.04 VM and it worked right away. Wasting resources that way but unless somebody has an idea on the LXC issue, I guess I will leave it as a VM.

I thought I could manage VLANs directly on the omada switch but after trying it it is very limited and I had to switch vlan management back to the router.

Pretty happy with the omada stuff, but the configuration is somewhat limited if you are not using their router, which seems like most people think their routers are marginal. My first time with VLANs and have various VLANs passed from openwrt router to Omada switch and 4 APs. Using my 1 Gb Ethernet adapter on Proxmox host to send all IOT vlan traffic to my windows VM for blueiris (IP cams) and homeseer (home automation). I guess I could setup VLANs in Proxmox to pass them all on the SPF+ adapter, but this seemed cleaner and I don't know how to do VLANs in Proxmox yet!

 

[Mistborn]

I tested motherboard nvme slot on another one of these Acer computers I have at our office and it is also limited to about 800MB/s. Stock nvme driver some model of WD, blue I think. Strange I read quite a bit of reviews on this PC and nobody mentioned the severely limited nvme slot.