[SOLVED] Ceph says osds are not reachable

[adhammuhseen]

Hello all,

I have a 3-node cluster set up using the guide here: https://packetpushers.net/blog/proxmox-ceph-full-mesh-hci-cluster-w-dynamic-routing/
Everything was working fine when using Ceph Quincy and Reef. However, after updating to Squid, I now get this error in the health status:

HEALTH_ERR: 12 osds(s) are not reachable
osd.0's public address is not in 'fc00::/64' subnet
osd.1's public address is not in 'fc00::/64' subnet
osd.2's public address is not in 'fc00::/64' subnet
osd.3's public address is not in 'fc00::/64' subnet
osd.4's public address is not in 'fc00::/64' subnet
osd.5's public address is not in 'fc00::/64' subnet
osd.6's public address is not in 'fc00::/64' subnet
osd.7's public address is not in 'fc00::/64' subnet
osd.8's public address is not in 'fc00::/64' subnet
osd.9's public address is not in 'fc00::/64' subnet
osd.10's public address is not in 'fc00::/64' subnet
osd.11's public address is not in 'fc00::/64' subnet

I am also seeing these errors in the logs:

osd.6 8528 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory

I have tried restarting Ceph as well as destroying and recreating an osd.
Ceph seems to be functioning fine, aside from the error message.

Thanks,
Adham

 

[Neobin]

For the messages regarding IPv6, see:

B

Post in thread 'Ceph 19.2 Squid Stable Release and Ceph 17.2 Quincy soon to be EOL'

Nov 23, 2024

Just a heads up if anyone is using IPv6 for the ceph network - seems one of the recent merges for health checking only checks IPv4. Result is a fully functioning ceph cluster but the state will be flagged as HEALTH_ERR.

https://tracker.ceph.com/issues/67517

root@pve01:~# ceph health detail
HEALTH_ERR 13 osds(s) are not reachable
[ERR] OSD_UNREACHABLE: 13 osds(s) are not reachable
osd.0's public address is not in 'fc00::81/125' subnet
osd.1's public address is not in 'fc00::81/125' subnet
osd.2's public address is not in 'fc00::81/125'...

• butwelld

• 

 

[adhammuhseen]

For the messages regarding IPv6, see:

B

Post in thread 'Ceph 19.2 Squid Stable Release and Ceph 17.2 Quincy soon to be EOL'

Nov 23, 2024

Just a heads up if anyone is using IPv6 for the ceph network - seems one of the recent merges for health checking only checks IPv4. Result is a fully functioning ceph cluster but the state will be flagged as HEALTH_ERR.

https://tracker.ceph.com/issues/67517

root@pve01:~# ceph health detail
HEALTH_ERR 13 osds(s) are not reachable
[ERR] OSD_UNREACHABLE: 13 osds(s) are not reachable
osd.0's public address is not in 'fc00::81/125' subnet
osd.1's public address is not in 'fc00::81/125' subnet
osd.2's public address is not in 'fc00::81/125'...

• butwelld

• 

Thank you. That post did not turn up in my search.
I will ignore the error for now.