Intel Nuc 13 Pro Thunderbolt Ring Network Ceph Cluster

scyto · Aug 20, 2023

I think i cracked it.... (though possibly fatal last words)
- one Windows Server VM and one CT installed on node 3
- all VM and CT storage on ceph
- HA configured
- pulled the power on node 3
- VM went to node 2 and was restarted
- CT went to node 1 and was restarted

took about 3 or 4 mins from when power was pulled.... and no errors once i supplied the power again...

I will write up my final steps in the next few days, plus some tips and tricks i found after hitting issues caused by orphaned OSDs on the nvme, sshs errors during migration (i think all caused by a lot of the tinkering i did).

ualex · Aug 20, 2023

scyto said:
Well what you posted above looks perfect to me... so maybe its the filenames? I really don't know! let me know how you get on.

BTW if you have same model as I do i wonder if BIOS changes behaviour? If you do i am on BIOS `ANRPL357.0026.2023.0314.1458`
FYI one machines BIOS got updated to `ANRPL357.0027.2023.0607.1754 by windows 11 i had done a test install on - i reverted to the 0026 variant when i was having issues with the add in ethernet board.... they seemed to go away...

I am also running "ANRPL357.0027.2023.0607.1754 ", I updated all firmware before I began with the rest. What type of issues did you notice with the ethernet board?

ualex · Aug 20, 2023

scyto said:
I think i cracked it.... (though possibly fatal last words)
- one Windows Server VM and one CT installed on node 3
- all VM and CT storage on ceph
- HA configured
- pulled the power on node 3
- VM went to node 2 and was restarted
- CT went to node 1 and was restarted

took about 3 or 4 mins from when power was pulled.... and no errors once i supplied the power again...

View attachment 54470

I will write up my final steps in the next few days, plus some tips and tricks i found after hitting issues caused by orphaned OSDs on the nvme, sshs errors during migration (i think all caused by a lot of the tinkering i did).

Thanks for all the testing and experimenting! I will wait on your write-up and continue with my setup.

scyto · Aug 20, 2023

ualex said:
Thanks for all the testing and experimenting! I will wait on your write-up and continue with my setup.

done, added new items in list of steps

things to note:
1. i can't get migration to be reliable over the 10.x network, so still using 'default' at this time - live migration is still darn quick over 2.5gbe

2. ceph and the cluster are very fragile if one mucks with networking too much (i broke a lot and learnt things like how to purge ceph and reset it, what to do when you can't remove a ghost monitor, ghost osds and more). So defintely try and setup ceph only once you are sure your network is 100% reliable and can survive node down or cable loss.

scyto · Aug 20, 2023

ualex said:
I am also running "ANRPL357.0027.2023.0607.1754 ", I updated all firmware before I began with the rest. What type of issues did you notice with the ethernet board?

Its was weird and i don't know how i solved it.

When the machine was powered off on node 3 i could see the green link lights, however during boot the port would disable itself - there was no real reason in dmesg, ip a just showed no carrier.

so i swapped the io card from one NUC to another (thinking it was faulty card) - nope
so i swapped cables thinking bad cables - nope
so i did two things:

downloaded older variant of firmware and installed via F7 key at boot
blew away proxmox OS and installed windows 11
booted into windows and it was green link lights (but worked)
installed official drivers from intels site and it went into 2.5gbe mode (one yellow light).
blew away windows and reinstalled proxmox

Its been reliable ever since. i don't really know what fixed it. it's possible i just borked the proxmox install.

But i see no reason to move firmware at this point - i should note that when i had node 3 on the newer firmware i don't recall seeing the thunderbolt PCI device id change between ports but perhaps it was and that was causing some of the weird inconsistencies.

I don't plan to run production VMs on this until i have proved a few weeks of stability

ualex · Aug 23, 2023

@scyto I tested with MTU size on the thunderbolt, but it is better to configure 4000.

MTU 9000 gives:

Code:

Accepted connection from 10.0.0.2, port 32842
[  5] local 10.0.0.3 port 5201 connected to 10.0.0.2 port 32852
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.30 GBytes  11.2 Gbits/sec                
[  5]   1.00-2.00   sec  1.31 GBytes  11.3 Gbits/sec                
[  5]   2.00-3.00   sec  1.31 GBytes  11.2 Gbits/sec                
[  5]   3.00-4.00   sec  1.30 GBytes  11.2 Gbits/sec                
[  5]   4.00-5.00   sec  1.31 GBytes  11.3 Gbits/sec                
[  5]   5.00-6.00   sec  1.31 GBytes  11.3 Gbits/sec                
[  5]   6.00-7.00   sec  1.31 GBytes  11.3 Gbits/sec                
[  5]   7.00-8.00   sec  1.31 GBytes  11.2 Gbits/sec                
[  5]   8.00-9.00   sec  1.31 GBytes  11.3 Gbits/sec                
[  5]   9.00-10.00  sec  1.32 GBytes  11.3 Gbits/sec                
[  5]  10.00-10.00  sec  1.43 MBytes  11.4 Gbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  13.1 GBytes  11.3 Gbits/sec                  receiver

MTU 4000 gives:

Code:

Accepted connection from 10.0.0.2, port 42494
[  5] local 10.0.0.3 port 5201 connected to 10.0.0.2 port 42508
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.41 GBytes  20.7 Gbits/sec                
[  5]   1.00-2.00   sec  2.49 GBytes  21.4 Gbits/sec                
[  5]   2.00-3.00   sec  2.46 GBytes  21.1 Gbits/sec                
[  5]   3.00-4.00   sec  2.49 GBytes  21.4 Gbits/sec                
[  5]   4.00-5.00   sec  2.47 GBytes  21.2 Gbits/sec                
[  5]   5.00-6.00   sec  2.48 GBytes  21.3 Gbits/sec                
[  5]   6.00-7.00   sec  2.48 GBytes  21.3 Gbits/sec                
[  5]   7.00-8.00   sec  2.49 GBytes  21.4 Gbits/sec                
[  5]   8.00-9.00   sec  2.48 GBytes  21.3 Gbits/sec                
[  5]   9.00-10.00  sec  2.49 GBytes  21.4 Gbits/sec                
[  5]  10.00-10.00  sec  1.20 MBytes  21.1 Gbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  24.7 GBytes  21.3 Gbits/sec                  receiver

Almost double speed ;-)

Just for the sake of it, MTU 1500:

Code:

[  5]   0.00-1.00   sec  2.52 GBytes  21.6 Gbits/sec                 
[  5]   1.00-2.00   sec  2.58 GBytes  22.1 Gbits/sec                 
[  5]   2.00-3.00   sec  2.59 GBytes  22.3 Gbits/sec                 
[  5]   3.00-4.00   sec  2.59 GBytes  22.2 Gbits/sec                 
[  5]   4.00-5.00   sec  2.59 GBytes  22.3 Gbits/sec                 
[  5]   5.00-6.00   sec  2.58 GBytes  22.2 Gbits/sec                 
[  5]   6.00-7.00   sec  2.59 GBytes  22.2 Gbits/sec                 
[  5]   7.00-8.00   sec  2.58 GBytes  22.2 Gbits/sec                 
[  5]   8.00-9.00   sec  2.59 GBytes  22.2 Gbits/sec                 
[  5]   9.00-10.00  sec  2.57 GBytes  22.1 Gbits/sec                 
[  5]  10.00-10.00  sec  1.99 MBytes  19.9 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  25.8 GBytes  22.1 Gbits/sec                  receiver

scyto · Aug 24, 2023

That makes total sense to me, reading the source code it is clear the TB frame size is -4000, it was annotated to say larger shouldn’t be an issue… nice find! I will update my guide. Does this mean you are happy with your network setup now?

scyto · Aug 24, 2023

Interesting that MTU 4000 isn’t any better than MTU1500 I wonder if we need something better aligned to the TB frame size.

scyto · Aug 24, 2023

might want to test a 64k MTU as that’s the maximum allowable… according to the source… (I assume an MTU of 65536?.). Also I wonder if the lo MTU is in play here…

scyto · Aug 24, 2023

MTU 65520 (i tested no other MTUs)

I would call 26.5 Gbits/sec a win ....

Code:

- - - - - - - - - - - - - - - - - - - - - - - - -
iperf3 -P 10 -c 10.0.0.81
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.08 GBytes  2.65 Gbits/sec    7             sender
[  5]   0.00-10.00  sec  3.08 GBytes  2.65 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  2.97 GBytes  2.55 Gbits/sec    3             sender
[  7]   0.00-10.00  sec  2.96 GBytes  2.54 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.24 GBytes  2.79 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  3.24 GBytes  2.78 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  3.18 GBytes  2.73 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  3.18 GBytes  2.73 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  2.92 GBytes  2.51 Gbits/sec    5             sender
[ 13]   0.00-10.00  sec  2.92 GBytes  2.50 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  2.93 GBytes  2.52 Gbits/sec   13             sender
[ 15]   0.00-10.00  sec  2.93 GBytes  2.51 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  3.23 GBytes  2.78 Gbits/sec    1             sender
[ 17]   0.00-10.00  sec  3.23 GBytes  2.78 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  3.16 GBytes  2.71 Gbits/sec    1             sender
[ 19]   0.00-10.00  sec  3.15 GBytes  2.71 Gbits/sec                  receiver
[ 21]   0.00-10.00  sec  2.96 GBytes  2.54 Gbits/sec    1             sender
[ 21]   0.00-10.00  sec  2.96 GBytes  2.54 Gbits/sec                  receiver
[ 23]   0.00-10.00  sec  3.16 GBytes  2.72 Gbits/sec    0             sender
[ 23]   0.00-10.00  sec  3.16 GBytes  2.71 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  30.8 GBytes  26.5 Gbits/sec   31             sender
[SUM]   0.00-10.00  sec  30.8 GBytes  26.5 Gbits/sec                  receiver

scyto · Aug 24, 2023

seems pretty consistent wether its 10 connections or 1

Code:

root@pve2:~# iperf3 -P 1 -c 10.0.0.81
Connecting to host 10.0.0.81, port 5201
[  5] local 10.0.0.10 port 57038 connected to 10.0.0.81 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.08 GBytes  26.5 Gbits/sec    3   2.25 MBytes      
[  5]   1.00-2.00   sec  3.09 GBytes  26.5 Gbits/sec    0   2.25 MBytes      
[  5]   2.00-3.00   sec  3.08 GBytes  26.5 Gbits/sec    0   2.25 MBytes      
[  5]   3.00-4.00   sec  3.08 GBytes  26.5 Gbits/sec   16   2.31 MBytes      
[  5]   4.00-5.00   sec  3.07 GBytes  26.4 Gbits/sec    0   2.31 MBytes      
[  5]   5.00-6.00   sec  3.02 GBytes  25.9 Gbits/sec   11   1.62 MBytes      
[  5]   6.00-7.00   sec  3.07 GBytes  26.4 Gbits/sec    0   1.62 MBytes      
[  5]   7.00-8.00   sec  3.06 GBytes  26.3 Gbits/sec    0   1.62 MBytes      
[  5]   8.00-9.00   sec  3.07 GBytes  26.4 Gbits/sec   21   1.69 MBytes      
[  5]   9.00-10.00  sec  3.08 GBytes  26.5 Gbits/sec    1   1.69 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  30.7 GBytes  26.4 Gbits/sec   52             sender
[  5]   0.00-10.00  sec  30.7 GBytes  26.4 Gbits/sec                  receiver

ualex · Aug 24, 2023

scyto said:
That makes total sense to me, reading the source code it is clear the TB frame size is -4000, it was annotated to say larger shouldn’t be an issue… nice find! I will update my guide. Does this mean you are happy with your network setup now?

Yes, I found the 4000 MTU somewhere in the TB source code, hence the test around it.

I am almost happy with my setup, but I think will not build the Ceph storage in my end. I normally do not keep 3 nodes up-and-running, so there is no super benefit for me (my Home Assistant node is connected to USB, so those cannot be live-migrated either). But I am happy you did it all, I picked a lot items from your setup into my setup.
Just waiting on my second NUCIOALUWS, but it seems to be sold out everywhere ... and delivery date shift every day by +1

scyto · Aug 24, 2023

ualex said:
it seems to be sold out everywhere

I am guessing you are not in the US given the times you post.

Mouser in the US has had them for a couple of weeks (they currently have 24 left on hand), have you tried mouser in your region?

I also saw a few on ebay where people had taken them out of the vPro version of the NUCs.

scyto · Aug 24, 2023

ualex said:
Home Assistant node is connected to USB

yeah, because i have Hyper-V as my production platform i separated z-wave and frigate off on to raspberry pi's - for zwave / zigbee that meant i could put the pi in the center of the house, powered via PoE - made my zwave and zigbee mesh much more reliable

i took a quick look at the usb over ip solutions linux has to see if that might work for VMs on proxmox - so i can keep the usb in the center of the house but have all the processing done directly in the hass VM

ualex · Aug 24, 2023

scyto said:
I am guessing you are not in the US given the times you post.

Mouser in the US has had them for a couple of weeks (they currently have 24 left on hand), have you tried mouser in your region?

I also saw a few on ebay where people had taken them out of the vPro version of the NUCs.

Correct, I am from the Netherlands. Good tip for mousser. If my current supplier cannot deliver it in a week orso, I will cancel and order there.

I am using "usbip" to connect the USB devices inside the Proxmox VMs, because USB device via the hardware is slow and not stable. Technically I can move the USB devices to a Pi

scyto · Aug 24, 2023

ualex said:
Correct, I am from the Netherlands. Good tip for mousser. If my current supplier cannot deliver it in a week orso, I will cancel and order there.

I am using "usbip" to connect the USB devices inside the Proxmox VMs, because USB device via the hardware is slow and not stable. Technically I can move the USB devices to a Pi

oh interesting, might dm you later if i can't figure that out

scyto · Aug 24, 2023

I can give you one (trivial) reason to do ceph - a cephfs store for your ISOs and Templates
put them on any node once and then they are on all nodes....
https://gist.github.com/scyto/941b24efd1ac0bf9b3cd30c3fb1e5341

ualex · Aug 25, 2023

scyto said:
I can give you one (trivial) reason to do ceph - a cephfs store for your ISOs and Templates
put them on any node once and then they are on all nodes....
https://gist.github.com/scyto/941b24efd1ac0bf9b3cd30c3fb1e5341

I tried a build with Ceph, but did not get CephFS directly to work (gave up quickly, I know) ... but your steps are super helpful, so I will give it another try soon.

ualex · Aug 25, 2023

scyto said:
oh interesting, might dm you later if i can't figure that out

The only frustrating part is to automatic bind on start-up, it can be hacked into a service. I made another bash script which does it from, including status, logging, etc.

I uploaded it to my gist:
https://gist.github.com/ualex73/e6d6088120840a10e126d62fe4061079

ualex · Aug 30, 2023

scyto said:
oh interesting, might dm you later if i can't figure that out

I cancelled my original order, it continue to shift the date. I ordered on Monday from Mouser (EU) and it got shipped out on Monday and it arrived today

I will install it tonight in my second NUC13.

BTW - i did a bit more testing, but NUC8 TB3 <-> NUC13 TB4 is not really super reliable, so i will give up on that. Only stick to NUC13.

Intel Nuc 13 Pro Thunderbolt Ring Network Ceph Cluster

Active Member

Member

Member

Active Member

Active Member

Member

Active Member

Active Member

Active Member

Active Member

Active Member

Member

Active Member

Active Member

Member

Active Member

Active Member

Member

Member

Member