feel free to post it in the comments, i don't need this on my HW to hit 27gbps, nice find y'all - this is awesome
what hardware are you running where this is needed?
Last edited:
Intel Nuc 13 Pro Thunderbolt Ring Network Ceph Cluster
- Thread starter l-koeln
- Start date
what hardware are you running where this is needed?
feel free to post it in the comments, i don't need this on my HW to hit 27gbps, nice find y'all - this is awesome
what hardware are you running where this is needed?
I've got 3x minisforum ms-01 with a 13900h. Even with the --bidir flag you have no problem with iperf hitting 27?
Hello,
Hoping that somebody may be able to give me some advice. I have an MS-01 and I've connected to a QNAP NAS using a thunderbolt 3 cable. On a Windows host, I am able to easily get 10gbps transfer speeds.
I followed the guide here to get thunderbolt networking enabled: https://gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570
I didn't need the mesh, so I just added the adapter to a bridge and connected to a Windows VM using the VirtIO driver.
I have an SFP+ connection added to the VM also and I get near 10gbps. The Thunderbolt connection is stuck around 150mbps.
Is there any advice on why this may be slow slow? I can use the same cable and connect to a Windows host and get > 10gbps.
Thanks!
Hoping that somebody may be able to give me some advice. I have an MS-01 and I've connected to a QNAP NAS using a thunderbolt 3 cable. On a Windows host, I am able to easily get 10gbps transfer speeds.
I followed the guide here to get thunderbolt networking enabled: https://gist.github.com/scyto/67fdc9a517faefa68f730f82d7fa3570
I didn't need the mesh, so I just added the adapter to a bridge and connected to a Windows VM using the VirtIO driver.
I have an SFP+ connection added to the VM also and I get near 10gbps. The Thunderbolt connection is stuck around 150mbps.
Is there any advice on why this may be slow slow? I can use the same cable and connect to a Windows host and get > 10gbps.
Thanks!
Hey there.. Question about following Scyto's guide - can I only use IPv4 on the Thunderbolt ring? Are there any issues with that? I've got one NUC 13 Pro right now and I just bought the second one. I'll get the third one in a couple months. I'm guessing I can do Thunderbolt networking between the two NUCs with Ceph until I get my third one?
I'm running only ipv4, there is nothing wrong with just running that. If you have no use for ipv6 there is no need to set it up.
Thanks for the reply. I'll be setting up the two nodes for now once I get my hardware. I'll put all the lines of code in the files, but will just have the ones for the third node commented out for now until I get it and get it installed.
Was messing with configuring ceph just now and noticed quite a big regression in the 6.8.4-2-pve and 6.8.4-3-pve kernel, I'm not yet sure what is the issue but setup is exactly the same and the only difference is newer kernel. Tested lowering the MTU to 1500 and that did help alot and i got speeds around 20Gbps on the new kernel but with high retries and tons of interrupts on the thunderbolt interface.
I have tested it and confimed it to be the kernel when i pinn to the previous verions (
I have tested it and confimed it to be the kernel when i pinn to the previous verions (
proxmox-boot-tool kernel pin 6.5.13-5-pve) and run the test results are again at around 26Gbps.
Code:
Connecting to host 10.0.0.7, port 5201
[ 5] local 10.0.0.6 port 55548 connected to 10.0.0.7 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.99 GBytes 25.7 Gbits/sec 4 1.75 MBytes
[ 5] 1.00-2.00 sec 3.02 GBytes 26.0 Gbits/sec 1 1.75 MBytes
[ 5] 2.00-3.00 sec 3.02 GBytes 25.9 Gbits/sec 32 1.81 MBytes
[ 5] 3.00-4.00 sec 2.99 GBytes 25.6 Gbits/sec 1 1.81 MBytes
[ 5] 4.00-5.00 sec 2.98 GBytes 25.6 Gbits/sec 0 1.81 MBytes
[ 5] 5.00-6.00 sec 3.01 GBytes 25.9 Gbits/sec 7 2.37 MBytes
[ 5] 6.00-7.00 sec 3.03 GBytes 26.0 Gbits/sec 0 2.37 MBytes
[ 5] 7.00-8.00 sec 3.00 GBytes 25.8 Gbits/sec 0 2.37 MBytes
[ 5] 8.00-9.00 sec 2.96 GBytes 25.5 Gbits/sec 2 2.37 MBytes
[ 5] 9.00-10.00 sec 3.04 GBytes 26.2 Gbits/sec 0 2.37 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 30.1 GBytes 25.8 Gbits/sec 47 sender
[ 5] 0.00-10.00 sec 30.1 GBytes 25.8 Gbits/sec receiver
Code:
Connecting to host 10.0.0.7, port 5201
[ 5] local 10.0.0.6 port 45246 connected to 10.0.0.7 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.28 GBytes 11.0 Gbits/sec 492 1.56 MBytes
[ 5] 1.00-2.00 sec 1.79 GBytes 15.3 Gbits/sec 553 1.50 MBytes
[ 5] 2.00-3.00 sec 1.76 GBytes 15.2 Gbits/sec 617 1.50 MBytes
[ 5] 3.00-4.00 sec 1.61 GBytes 13.8 Gbits/sec 482 1.56 MBytes
[ 5] 4.00-5.00 sec 664 MBytes 5.57 Gbits/sec 197 1.50 MBytes
[ 5] 5.00-6.00 sec 1012 MBytes 8.49 Gbits/sec 379 1.94 MBytes
[ 5] 6.00-7.00 sec 1.47 GBytes 12.6 Gbits/sec 520 1.37 MBytes
[ 5] 7.00-8.00 sec 1.50 GBytes 12.9 Gbits/sec 566 1.31 MBytes
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-8.04 sec 11.0 GBytes 11.8 Gbits/sec 3806 sender
[ 5] 0.00-8.04 sec 0.00 Bytes 0.00 bits/sec receiver
Last edited:
Hey, I have not set this up yet - but - I do remember seeing something about interrupts causing issues and somehow they had to set the affinity to a single core and that solved it. Do you recall reading this in this thread somewhere?
Here is the thread: https://forum.proxmox.com/threads/i...etwork-ceph-cluster.131107/page-7#post-656320
Looks like they were able to solve the retries problem in the post linked above (in this thread).
Last edited:
I'm reading everything I can about Ceph before I get all my nodes in place. It sounds like you can set Ceph up in multiple ways. For these 3 NUC w/Thunderbolt mesh setups go, how are you configuring Ceph? I'm reading where people are saying Ceph doesn't make sense with less than 5 nodes. Can anyone that has this three NUC Thunderbolt mesh setup speak to how they have Ceph configured and how it is performing for them?
Thank you!
Thank you!
Let me start off by saying thanks to all who have figured most of this out. Without you I wouldn't even know where to start.
I have a NUC13 i7-1360P 3 node cluster. PVE 8.2.2. BIOS ANRPL357.0031.2024.0207.1420. Addon 2.5GbE and the 0.3 OWC TB4 cables.
I've spent the past week and half learning and following everyone's advice. I ended up ditching OpenFabric and dual stack, and have a working IPv6 OSPFv3 config. IPerf3 gets about 26Gbps but gets interrupts averaging about 400 per second. I don't know that this even matters because when I run iperf3 with -b10000M for example, the interrupt count is negligible, so it only jumps when pushed to the limit. What's a realistic ceph usage on these anyway?! I've tried the smp_affinity_list script restricting thunderbolt to cores 0-7 which are the P cores on these, and saw no difference whatsoever in the interrupt count.
Here's the big question. I see you all posting your iperf results point-to-point. When i pull the thunderbolt cable between nodes, so there's no longer a direct link and it has to route through the 3rd NUC, the results are abysmal. It goes from 26Gbps to 20Mbps. What are your results?
Also, CPU usage. I noticed the sending node doesn't use much CPU when maxing out iperf, but the receive side has one CPU pegged at 100%. Dropping the iperf bandwidth test down to 10Gb brings it down to a more reasonable 15%. What are you seeing using OSPFv2, OSPFv3, or OpenFabric?
I have a NUC13 i7-1360P 3 node cluster. PVE 8.2.2. BIOS ANRPL357.0031.2024.0207.1420. Addon 2.5GbE and the 0.3 OWC TB4 cables.
I've spent the past week and half learning and following everyone's advice. I ended up ditching OpenFabric and dual stack, and have a working IPv6 OSPFv3 config. IPerf3 gets about 26Gbps but gets interrupts averaging about 400 per second. I don't know that this even matters because when I run iperf3 with -b10000M for example, the interrupt count is negligible, so it only jumps when pushed to the limit. What's a realistic ceph usage on these anyway?! I've tried the smp_affinity_list script restricting thunderbolt to cores 0-7 which are the P cores on these, and saw no difference whatsoever in the interrupt count.
Here's the big question. I see you all posting your iperf results point-to-point. When i pull the thunderbolt cable between nodes, so there's no longer a direct link and it has to route through the 3rd NUC, the results are abysmal. It goes from 26Gbps to 20Mbps. What are your results?
Also, CPU usage. I noticed the sending node doesn't use much CPU when maxing out iperf, but the receive side has one CPU pegged at 100%. Dropping the iperf bandwidth test down to 10Gb brings it down to a more reasonable 15%. What are you seeing using OSPFv2, OSPFv3, or OpenFabric?
Last edited:
It's not in the gist because it's irrelevant to Thunderbolt or Ceph which is the topic of that gist and this thread. FWIW I went with ZFS.
I use BTRFS as I can also put VMs there and use snapshots in the GUI.
Hey, that’s good to know. Thanks for sharing that info!
Have you tried rolling back to kernel 6.5.13-5-pve? Theres one person in this thread that current kernels cause lower speeds but going back to that version ran a solid 26Gb.
I just have a 2-node cluster right now - I'll add the third one in a couple months then enable and move everything over to CephFS.
I just went straight for that kernel version that has the good speeds and was getting 26Gb with iPerf3. Worked great. So far, plug/unplugs of the Thunderbolt cable recovers without any trouble, and full shutdown/restarts of the entire cluster bring the interface back up. Working great so far! Thanks to everyone in this thread and a big thanks to @scyto for his gists that he wrote up. That was a huge help.
I have a few small retries during the test which isn't a big deal at all. I have an OWC Thunderbolt cable coming today so I'll give that one a try and see if the errors go away.
root@pve-node01:~# iperf3 -c 10.0.0.82
Connecting to host 10.0.0.82, port 5201
[ 5] local 10.0.0.81 port 38028 connected to 10.0.0.82 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.07 GBytes 26.4 Gbits/sec 9 1.62 MBytes
[ 5] 1.00-2.00 sec 3.07 GBytes 26.4 Gbits/sec 2 1.62 MBytes
[ 5] 2.00-3.00 sec 3.08 GBytes 26.5 Gbits/sec 0 1.75 MBytes
[ 5] 3.00-4.00 sec 3.07 GBytes 26.4 Gbits/sec 9 2.37 MBytes
[ 5] 4.00-5.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
[ 5] 5.00-6.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
[ 5] 6.00-7.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
[ 5] 7.00-8.00 sec 3.08 GBytes 26.4 Gbits/sec 0 2.37 MBytes
[ 5] 8.00-9.00 sec 3.08 GBytes 26.4 Gbits/sec 0 2.37 MBytes
[ 5] 9.00-10.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 30.8 GBytes 26.4 Gbits/sec 20 sender
[ 5] 0.00-10.00 sec 30.8 GBytes 26.4 Gbits/sec receiver
iperf Done.
root@pve-node01:~#
Connecting to host 10.0.0.82, port 5201
[ 5] local 10.0.0.81 port 38028 connected to 10.0.0.82 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.07 GBytes 26.4 Gbits/sec 9 1.62 MBytes
[ 5] 1.00-2.00 sec 3.07 GBytes 26.4 Gbits/sec 2 1.62 MBytes
[ 5] 2.00-3.00 sec 3.08 GBytes 26.5 Gbits/sec 0 1.75 MBytes
[ 5] 3.00-4.00 sec 3.07 GBytes 26.4 Gbits/sec 9 2.37 MBytes
[ 5] 4.00-5.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
[ 5] 5.00-6.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
[ 5] 6.00-7.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
[ 5] 7.00-8.00 sec 3.08 GBytes 26.4 Gbits/sec 0 2.37 MBytes
[ 5] 8.00-9.00 sec 3.08 GBytes 26.4 Gbits/sec 0 2.37 MBytes
[ 5] 9.00-10.00 sec 3.08 GBytes 26.5 Gbits/sec 0 2.37 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 30.8 GBytes 26.4 Gbits/sec 20 sender
[ 5] 0.00-10.00 sec 30.8 GBytes 26.4 Gbits/sec receiver
iperf Done.
root@pve-node01:~#
What I really want to know is what iperf shows when you have at least a 3 node setup, and you fail a link so it no longer has a direct connection to the adjacent node.
For example, disconnect the link between PVE01 and PVE02 so it no longer has a direct connection. The run iPerf3 from PVE01 > PVE02 so it is forced to route through PVE03. I was only getting 20Mbit in this scenario.
For example, disconnect the link between PVE01 and PVE02 so it no longer has a direct connection. The run iPerf3 from PVE01 > PVE02 so it is forced to route through PVE03. I was only getting 20Mbit in this scenario.
Here's an iperf3 test with a down link between pve01 and pve02, so the traffic is routed over pve03
I'd test that but I only have 2 nodes at this point. Will have to wait a couple months to get the 3rd NUC 13 Pro. With the system, ram, and drives, it about $950 so it'll take a bit.
Thanks for this - are these the normal speeds you get when all the cables are up/plugged in? Just curious if the speeds dropped or not when a link is down.
With all cables attached (direct test) I reach the full 26 Gbit/s