Hey all, hate to do this cause I know that there is a constant stream of "my NIC isn't as fast as I thought it would be" posts, but what I'm seeing goes well beyond any sort of CPU overhead or packet size issues. I haven't been able to find a solution anywhere that works yet, so figured I'd take a stab here. I'm still kind of a noob so if you're going to call me stupid, please at least include what I'm doing wrong as well so I can learn while you do so.
I have two separate servers I've been running for a number of months now, one running TrueNAS Core and the other running Proxmox 7.4-3. The TrueNAS box has a 3.4Ghz six core Xeon, and Proxmos has dual ten core Xeons, both have over 64GB Ram. Proxmox has 4 SSDs in RAIDZ, and TrueNAS has 21 spinners across 4 vdevs in a single pool. They both have their own built in 1 GB Nics that I've been using connected to a switch with no issues.
I recently acquired two Mellanox Connect X4 NICs for a steal. Each have dual 40Gb QFSP+, and I have both ports connected with DACs bridging directly between the two servers. However the transfers speeds I'm seeing don't/barely clear speed I would expect from a 1Gb NIC.
I first tried setting up an NFS share on TrueNAS that pointed at a 10 GB video file. I set up the NIC in TrueNAS with a seperate IP on a differnt subnet than the usual one, and in Proxmox a Linux Bridge for one of the ports in the same subnet. I created a new Ubuntu Container in proxmox and pinged the TrueNAS NIC as a sanity check, it worked fine. However when I tried to connect to the NFS share (after switching the container to a privileged one after it wouldn't let me mount NFS), the speeds were so slow that it even just directory browsing was unusable slow. I eventually was able to start a copy to the local storage but basically nothing was happening. The Interface widget on TrueNAS was report output speeds in the kilobytes.
I then figured that NFS probably isn't really the best fit with containers, so then I tried spinning up an up-to-date version of Ubuntu desktop in another Prox VM. I got it connected to the NFS share, and was able to start a copy to local storage. Better than the container, but it was topping out at around 60 MB/s(as reported by both Ubuntu's file manager and the TrueNAS utility, as well as just paying attention to time to complete). I tried the same thing using SMB, and it was averaging about 80MB/s. I then finally tried one last time in an Ubuntu Server VM with NFS, and it was only getting around 110MB/s.
All of the VMs I mentioned were given "8" cores and 12 GB of RAM, and never showed more than a few dozen percent usage in Proxmox GUI. I tried changing all the MTU settings to 9000, and played around with VM multiqueue settings with values of 2 and 4. This bumped all the speeds by only 20-40 MB/s, except for Ubuntu Server which got up to around 200MB/s for a few spikes. At least according to the TrueNAS Interface GUI widgets, all the transfers have indeed gone through the 40Gb NIC and not the 1Gb one. I have tried playing around with setting up the NICs on both sides as just one port or using all of them in Round Robin or LAGG, always the same performance. I ran ethtool {devicename} in a shell for proxmox, and it showed the "speed" parameter as 40000. Truenas was also reporting 40Gb Media in its interface description.
I knew I was never going to get the full fat 5 GB a second theoretical speeds of 40Gb, but I was certainly expecting more than this. Clearly I'm doing something wrong, but I don't really know what to do next. I don't even know if the problem lay in truenas or in proxmox, and I don't know of a good way to isolate which one it is short of taking both of them out of my rack and trying them both with my personal computer. However that is both a ton of work and also not conclusive because it introduces a bunch more variables anyways.
If any of the wiser here than me are willing to spare me some of your time and insight, I would be greatly appreciative.
I have two separate servers I've been running for a number of months now, one running TrueNAS Core and the other running Proxmox 7.4-3. The TrueNAS box has a 3.4Ghz six core Xeon, and Proxmos has dual ten core Xeons, both have over 64GB Ram. Proxmox has 4 SSDs in RAIDZ, and TrueNAS has 21 spinners across 4 vdevs in a single pool. They both have their own built in 1 GB Nics that I've been using connected to a switch with no issues.
I recently acquired two Mellanox Connect X4 NICs for a steal. Each have dual 40Gb QFSP+, and I have both ports connected with DACs bridging directly between the two servers. However the transfers speeds I'm seeing don't/barely clear speed I would expect from a 1Gb NIC.
I first tried setting up an NFS share on TrueNAS that pointed at a 10 GB video file. I set up the NIC in TrueNAS with a seperate IP on a differnt subnet than the usual one, and in Proxmox a Linux Bridge for one of the ports in the same subnet. I created a new Ubuntu Container in proxmox and pinged the TrueNAS NIC as a sanity check, it worked fine. However when I tried to connect to the NFS share (after switching the container to a privileged one after it wouldn't let me mount NFS), the speeds were so slow that it even just directory browsing was unusable slow. I eventually was able to start a copy to the local storage but basically nothing was happening. The Interface widget on TrueNAS was report output speeds in the kilobytes.
I then figured that NFS probably isn't really the best fit with containers, so then I tried spinning up an up-to-date version of Ubuntu desktop in another Prox VM. I got it connected to the NFS share, and was able to start a copy to local storage. Better than the container, but it was topping out at around 60 MB/s(as reported by both Ubuntu's file manager and the TrueNAS utility, as well as just paying attention to time to complete). I tried the same thing using SMB, and it was averaging about 80MB/s. I then finally tried one last time in an Ubuntu Server VM with NFS, and it was only getting around 110MB/s.
All of the VMs I mentioned were given "8" cores and 12 GB of RAM, and never showed more than a few dozen percent usage in Proxmox GUI. I tried changing all the MTU settings to 9000, and played around with VM multiqueue settings with values of 2 and 4. This bumped all the speeds by only 20-40 MB/s, except for Ubuntu Server which got up to around 200MB/s for a few spikes. At least according to the TrueNAS Interface GUI widgets, all the transfers have indeed gone through the 40Gb NIC and not the 1Gb one. I have tried playing around with setting up the NICs on both sides as just one port or using all of them in Round Robin or LAGG, always the same performance. I ran ethtool {devicename} in a shell for proxmox, and it showed the "speed" parameter as 40000. Truenas was also reporting 40Gb Media in its interface description.
I knew I was never going to get the full fat 5 GB a second theoretical speeds of 40Gb, but I was certainly expecting more than this. Clearly I'm doing something wrong, but I don't really know what to do next. I don't even know if the problem lay in truenas or in proxmox, and I don't know of a good way to isolate which one it is short of taking both of them out of my rack and trying them both with my personal computer. However that is both a ton of work and also not conclusive because it introduces a bunch more variables anyways.
If any of the wiser here than me are willing to spare me some of your time and insight, I would be greatly appreciative.
Last edited: