Unusually Bad Network Performance with 40 Gb NICs


New Member
Nov 14, 2023
Hey all, hate to do this cause I know that there is a constant stream of "my NIC isn't as fast as I thought it would be" posts, but what I'm seeing goes well beyond any sort of CPU overhead or packet size issues. I haven't been able to find a solution anywhere that works yet, so figured I'd take a stab here. I'm still kind of a noob so if you're going to call me stupid, please at least include what I'm doing wrong as well so I can learn while you do so.

I have two separate servers I've been running for a number of months now, one running TrueNAS Core and the other running Proxmox 7.4-3. The TrueNAS box has a 3.4Ghz six core Xeon, and Proxmos has dual ten core Xeons, both have over 64GB Ram. Proxmox has 4 SSDs in RAIDZ, and TrueNAS has 21 spinners across 4 vdevs in a single pool. They both have their own built in 1 GB Nics that I've been using connected to a switch with no issues.

I recently acquired two Mellanox Connect X4 NICs for a steal. Each have dual 40Gb QFSP+, and I have both ports connected with DACs bridging directly between the two servers. However the transfers speeds I'm seeing don't/barely clear speed I would expect from a 1Gb NIC.

I first tried setting up an NFS share on TrueNAS that pointed at a 10 GB video file. I set up the NIC in TrueNAS with a seperate IP on a differnt subnet than the usual one, and in Proxmox a Linux Bridge for one of the ports in the same subnet. I created a new Ubuntu Container in proxmox and pinged the TrueNAS NIC as a sanity check, it worked fine. However when I tried to connect to the NFS share (after switching the container to a privileged one after it wouldn't let me mount NFS), the speeds were so slow that it even just directory browsing was unusable slow. I eventually was able to start a copy to the local storage but basically nothing was happening. The Interface widget on TrueNAS was report output speeds in the kilobytes.

I then figured that NFS probably isn't really the best fit with containers, so then I tried spinning up an up-to-date version of Ubuntu desktop in another Prox VM. I got it connected to the NFS share, and was able to start a copy to local storage. Better than the container, but it was topping out at around 60 MB/s(as reported by both Ubuntu's file manager and the TrueNAS utility, as well as just paying attention to time to complete). I tried the same thing using SMB, and it was averaging about 80MB/s. I then finally tried one last time in an Ubuntu Server VM with NFS, and it was only getting around 110MB/s.

All of the VMs I mentioned were given "8" cores and 12 GB of RAM, and never showed more than a few dozen percent usage in Proxmox GUI. I tried changing all the MTU settings to 9000, and played around with VM multiqueue settings with values of 2 and 4. This bumped all the speeds by only 20-40 MB/s, except for Ubuntu Server which got up to around 200MB/s for a few spikes. At least according to the TrueNAS Interface GUI widgets, all the transfers have indeed gone through the 40Gb NIC and not the 1Gb one. I have tried playing around with setting up the NICs on both sides as just one port or using all of them in Round Robin or LAGG, always the same performance. I ran ethtool {devicename} in a shell for proxmox, and it showed the "speed" parameter as 40000. Truenas was also reporting 40Gb Media in its interface description.

I knew I was never going to get the full fat 5 GB a second theoretical speeds of 40Gb, but I was certainly expecting more than this. Clearly I'm doing something wrong, but I don't really know what to do next. I don't even know if the problem lay in truenas or in proxmox, and I don't know of a good way to isolate which one it is short of taking both of them out of my rack and trying them both with my personal computer. However that is both a ton of work and also not conclusive because it introduces a bunch more variables anyways.

If any of the wiser here than me are willing to spare me some of your time and insight, I would be greatly appreciative.
Last edited:
Oh and one other thing I forgot to mention, this isn't exclusive to VMs or just the input direction. I tried setting up both NFS and SMB within the proxmox Datacenter as VZDump backup storage, and both of them were around 80MB/s when I tried upload a backup of a VM to truenas.
Last edited:


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!