Intel Nuc 13 Pro Thunderbolt Ring Network Ceph Cluster
- Thread starter l-koeln
- Start date
Hi @Kollskegg, I have having the same issue, and tried a few globs. I can not get it to work, are you having any success?
Sorry no, couldn't get a successful match using the full-length path. The MAC addresses per interface/port have been stable between reboots so far, so I resolved it by using those in the match instead. The downside to this is that I've seen the MAC addresses change between re-installs and firmware upgrades. Some see the MAC addresses changing with cable removal/insertion but for me they don't.
thats why i suggest globbing - it allows you do to partial matches, expressions etc - i suggest one of you use chatgpt, if you carefully craft your question i give it an 8 in 10 chance it will give you the correct information
I am sure I am missing something here.
I have the following two thunderbolt net devices:
For the systemd link file, I have tried a bunch of different routes, but should something like the following work?
I have the following two thunderbolt net devices:
Code:
P: /devices/pci0000:00/0000:00:0d.2/domain0/0-0/0-1/0-1.0
M: 0-1.0
R: 0
J: +thunderbolt:0-1.0
U: thunderbolt
T: thunderbolt_service
V: thunderbolt-net
E: DEVPATH=/devices/pci0000:00/0000:00:0d.2/domain0/0-0/0-1/0-1.0
E: SUBSYSTEM=thunderbolt
E: DEVTYPE=thunderbolt_service
E: DRIVER=thunderbolt-net
E: MODALIAS=tbsvc:knetworkp00000001v00000001r00000001
P: /devices/pci0000:00/0000:00:0d.2/domain0/0-0/0-3/0-3.0
M: 0-3.0
R: 0
J: +thunderbolt:0-3.0
U: thunderbolt
T: thunderbolt_service
V: thunderbolt-net
E: DEVPATH=/devices/pci0000:00/0000:00:0d.2/domain0/0-0/0-3/0-3.0
E: SUBSYSTEM=thunderbolt
E: DEVTYPE=thunderbolt_service
E: DRIVER=thunderbolt-net
E: MODALIAS=tbsvc:knetworkp00000001v00000001r00000001For the systemd link file, I have tried a bunch of different routes, but should something like the following work?
Code:
[Match]
Property=DEVPATH=/devices/pci0000:00/0000:00:0d.2/domain0/0-0/0-3/0-3.0
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=ent3I have never tested using the property match
have you tried enclosing it in quotes incase there are escaping issues on the / symbols
you could also try using globbing (property accepts * and ?) if chatgpt is not hallucinating something like
make sure you use udevadms monitor function to see what happens when you remove and add the thunderbolt cables as that will show you what is and isntt getting matched
have you tried enclosing it in quotes incase there are escaping issues on the / symbols
you could also try using globbing (property accepts * and ?) if chatgpt is not hallucinating something like
Property=DEVPATH=*/0-3 though of course that could still have the issue if the / needs to be escaped in some form - i recommend this approach because it may not be the full end point that udevadm matches - thats why you dont see a full path match in my pcie path reference version, to avoid issues like thatmake sure you use udevadms monitor function to see what happens when you remove and add the thunderbolt cables as that will show you what is and isntt getting matched
Last edited:
evidently you didn't eve read the last 10 posts, if you had you would have realised why the wya you do it doesn't work for these other people - the bus address is not unique for each port for them, so what you posted is unhelpful to them as it wont solve their issue and redunant as it is no different to the base instructions
Question for others who might be using a TB for Ceph. Is your
In the config snippets I have seen both here and online it looks as though most are running the public network from a physical link/VLAN, which makes sense if you need to share CEPH on the network with VMs, K8S for storage etc, but I assume this is going to be at a 1GB or 2.5GB network speed rather than the speed of the TB mesh?
I have taken the approach of using a shared public and cluster network, but when doing so, although Ceph replication, VMs etc is all healthy, some of the Proxmox GUI (Datacenter > Ceph and ceph > osd > manage global settings) returns HTTPS 500s. While using a separate public network all is ok.
Keen to understand how others are using it and if they have experienced similar?
public_network shared with your cluster_network?In the config snippets I have seen both here and online it looks as though most are running the public network from a physical link/VLAN, which makes sense if you need to share CEPH on the network with VMs, K8S for storage etc, but I assume this is going to be at a 1GB or 2.5GB network speed rather than the speed of the TB mesh?
I have taken the approach of using a shared public and cluster network, but when doing so, although Ceph replication, VMs etc is all healthy, some of the Proxmox GUI (Datacenter > Ceph and ceph > osd > manage global settings) returns HTTPS 500s. While using a separate public network all is ok.
Keen to understand how others are using it and if they have experienced similar?
The documentation recommends following:
So for me this looks like having just a 1/2.5 Gbit public network isn't suitable.
So for me this looks like having just a 1/2.5 Gbit public network isn't suitable.
Agree... though in my setup it's ~30Gbit over TB. In a homelab with NUCS (which this thread is largely related to) for testing it sounds as though people are doing it and its probably fine with understood risks.
In any case, that doesn't appear to be related to my issue.
Last edited:
I'm running with a shared
For external access i use direct host routes to the PVE hosts over IPv6.
Here's an example:
I had to disable
With this approach I also added a 4th (absolutely uncritical) Proxmox host to the cluster with no direct TB connection but access to the Ceph cluster.
cluster_network and public_network on the TB link.For external access i use direct host routes to the PVE hosts over IPv6.
Here's an example:
I had to disable
multicast_snooping on the Proxmox vmbr0 interface to have IPv6 NDP, DaD and so on IPv6 itself to work as expected.With this approach I also added a 4th (absolutely uncritical) Proxmox host to the cluster with no direct TB connection but access to the Ceph cluster.
Last edited:
Thanks for the info. That makes sense and was looking at a direct route option for my k3s once I’m confident of the stability as I want to ditch my longhorn setup. Good to see it’s working for you.
I assume your Proxmox Ceph GUI doesn’t have the issues that I’m experiencing with 500 status timeouts when clicking the datacentre > ceph option. Clicking the node > ceph option is ok. I have a similar error when I click the OSD manage globals button. If I try to click create button in the Ceph > Pools menu in the GUI, I also see a timeout before the modal even pops up, however I’ve just created several new pools via the CLI without any issue.
I might try to switch my OpenFabric to IPv6 and see if it’s any different.
I assume your Proxmox Ceph GUI doesn’t have the issues that I’m experiencing with 500 status timeouts when clicking the datacentre > ceph option. Clicking the node > ceph option is ok. I have a similar error when I click the OSD manage globals button. If I try to click create button in the Ceph > Pools menu in the GUI, I also see a timeout before the modal even pops up, however I’ve just created several new pools via the CLI without any issue.
I might try to switch my OpenFabric to IPv6 and see if it’s any different.
Last edited:
Thanks!! Good to know. Not sure what's going on on my side then as it all seems to be perfect otherwise.
I'm gonna switch out IPv4 to IPv6 and see if it makes any difference.
Thanks for the confirmation, much appreciated!!
