Ceph flapping OSDs - OSDs only using public network not the cluster one

justinclift

justinclift

Well-Known Member
Apr 1, 2024
664
188
53
Having a super weird issue with Ceph (squid, 19.2.4) today. The OSDs are flapping (ie not talking to each other) and I have no idea why.

I changed the PCIe network card in each node of my small 3 node Proxmox cluster to be a 10GbE network card today, which is probably what triggered this.

Each node is a HP Microserver, with 4x 3.5" drives and 3 network interfaces:
  1. 1GbE (public network) using IP address range 10.1.1.x/16
  2. 1GbE (corosync network) using IP address range 10.200.1.x/24
  3. 10GbE (storage network) using IP address range 10.199.10.x/24
It's been running fine for months with a 40GbE PCIe card for the storage network (same address range), but today after swapping the storage network card to a 10GbE one this strangeness happens.

The super bizarre thing is the OSDs seem to be trying to do health checks against the public IP addresses of the other OSDs. They're not supposed to be. There's literally a cluster network defined for them, which is working, and each of the OSDs is binding to it. So why are they trying to ping/healthcheck the address on the public network?

Anyway, my ceph configuration:

Code: 
# cat /etc/pve/ceph.conf
[global]
        auth_client_required = cephx
        auth_cluster_required = cephx
        auth_service_required = cephx
        fsid = 05ba3e61-6d82-4207-a683-15f6521ca672
        mon_allow_pool_delete = true
        mon_host = 10.1.1.121
        ms_bind_ipv4 = true
        ms_bind_ipv6 = false
        osd_pool_default_min_size = 2
        osd_pool_default_size = 3
        public_network = 10.1.1.0/16
        cluster_network = 10.199.10.0/24

[client]
        keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
        keyring = /etc/pve/ceph/$cluster.$name.keyring

[mds]
        keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mon.backup1]
        public_addr = 10.1.1.121

Meanwhile, the logs of the various OSDs is showing heartbeat_check failures. For example:
Code: 
# journalctl -u ceph-osd@1.service -n 40 -f
Jul 05 07:56:42 backup3 ceph-osd[2487]: 2026-07-05T07:56:42.702+0000 7fd9bb00f6c0 -1 osd.1 6872 heartbeat_check: no reply from 10.1.1.121:6814 osd.6 ever on either front or back, first ping sent 2026-07-05T07:56:22.060949+0000 (oldest deadline 2026-07-05T07:56:42.060949+0000)
Jul 05 07:56:42 backup3 ceph-osd[2487]: 2026-07-05T07:56:42.702+0000 7fd9bb00f6c0 -1 osd.1 6872 heartbeat_check: no reply from 10.1.1.124:6802 osd.8 ever on either front or back, first ping sent 2026-07-05T07:56:22.060949+0000 (oldest deadline 2026-07-05T07:56:42.060949+0000)
Jul 05 07:56:42 backup3 ceph-osd[2487]: 2026-07-05T07:56:42.702+0000 7fd9bb00f6c0 -1 osd.1 6872 heartbeat_check: no reply from 10.1.1.124:6814 osd.9 ever on either front or back, first ping sent 2026-07-05T07:56:22.060949+0000 (oldest deadline 2026-07-05T07:56:42.060949+0000)
Jul 05 07:56:42 backup3 ceph-osd[2487]: 2026-07-05T07:56:42.702+0000 7fd9bb00f6c0 -1 osd.1 6872 heartbeat_check: no reply from 10.1.1.121:6810 osd.11 ever on either front or back, first ping sent 2026-07-05T07:56:22.060949+0000 (oldest deadline 2026-07-05T07:56:42.060949+0000)

This is making no sense at all to me. Note that I've updated the packages today (no-subscription repo), and rebooted many times after changing configuration settings, but nothing seems to get the OSDs to use the correct network for the heartbeat.

Stuff I've tried includes overrides of the cluster_addr for each of the OSDs (as per the docs) in the above ceph.conf. That didn't do anything, so I removed those entries).

Anyone have ideas?
 
Last edited:
As a workaround for now, I've set the "nodown" and "noout" flags. That's allowing things to actually work "ok".

A real solution would be preferred though.
 
Yeah, I'm aware the cluster network is for replication. That's why I have it.

None of the nodes have any firewall settings. "iptables -nL" shows no entries.

Any idea why the OSD are trying to do heartbeats on the public network, when there's a cluster network defined and the OSDs are even binding to it?
 
Last edited:
I must be misunderstanding something?

This exact same configuration used to work fine with 40GbE nics. Those nics have now been swapped to 10GbE nics (same manufacturer, Mellanox) and the problems ensured.

The 3 Proxmox nodes have two 1GbE physical ports (one for the public network, one for the corosync heartbeat network) and a higher bandwidth (was 40GbE, now 10GbE) port for replication. The nodes are able to communicate with no issues whatsoever between each other on their respective networks.

ie comms between nodes on the public interfaces work fine

Note - The higher bandwidth physical network (40GbE -> 10GbE) is not able to communicate with the public network. These networks are physically isolated on purpose and there is no gateway between them. The Ceph docs even recommend this approach for security.
 
justinclift said:
I must be misunderstanding something?

This exact same configuration used to work fine with 40GbE nics. Those nics have now been swapped to 10GbE nics (same manufacturer, Mellanox) and the problems ensured.

The 3 Proxmox nodes have two 1GbE physical ports (one for the public network, one for the corosync heartbeat network) and a higher bandwidth (was 40GbE, now 10GbE) port for replication. The nodes are able to communicate with no issues whatsoever between each other on their respective networks.

ie comms between nodes on the public interfaces work fine

Note - The higher bandwidth physical network (40GbE -> 10GbE) is not able to communicate with the public network. These networks are physically isolated on purpose and there is no gateway between them. The Ceph docs even recommend this approach for security.
Click to expand...
cluster network is for osd->osd replication.
osd->monitor && osd->client is done on public network.
They are differents hearbeats, osd hearbeat between osd (on clusternetwork)
hearbeat from monitor-> osd (on public network)

https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/


so your osd should listen on both networks



BTW, isn't a little bit strange to have only 1gbit/s for your public network ? (I mean, this is the network used by the vm to read/write to your ceph osd, so it's the "storage" network).
The cluster_network for ceph is optional, it's really here if you want to offload replication of osds on another network.

IMOH, you should use your 10Gb/s for public network, and don't defined any cluster_network.
 
Last edited:
  • Like
Reactions: justinclift, gurubert and Johannes S
aderumier said:
cluster network is for osd->osd replication.
osd->monitor && osd->client is done on public network.
They are differents hearbeats, osd hearbeat between osd (on clusternetwork)
hearbeat from monitor-> osd (on public network)

https://docs.ceph.com/en/reef/rados/configuration/network-config-ref/

so your osd should listen on both networks
Click to expand...
Thanks, that helps explain it a bit. The OSDs are indeed listening on both networks according to `ss` and similar:
Code: 
# ss -nltp
State       Recv-Q      Send-Q            Local Address:Port             Peer Address:Port      Process                           
LISTEN      0           512                  10.1.1.124:6815                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=23))
LISTEN      0           512                  10.1.1.124:6814                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=22))
LISTEN      0           512                  10.1.1.124:6813                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=19))
LISTEN      0           512                  10.1.1.124:6812                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=18))
LISTEN      0           512                  10.1.1.124:6811                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=23))
LISTEN      0           512                  10.1.1.124:6810                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=22))
LISTEN      0           512                  10.1.1.124:6809                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=19))
LISTEN      0           512                  10.1.1.124:6808                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=18))
LISTEN      0           512                  10.1.1.124:6807                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=23))
LISTEN      0           512                  10.1.1.124:6806                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=22))
LISTEN      0           512                  10.1.1.124:6805                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=19))
LISTEN      0           512                  10.1.1.124:6804                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=18))
LISTEN      0           512                  10.1.1.124:6803                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=23))
LISTEN      0           512                  10.1.1.124:6802                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=22))
LISTEN      0           512                  10.1.1.124:6801                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=19))
LISTEN      0           512                  10.1.1.124:6800                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=18))
LISTEN      0           512                  10.1.1.124:6789                  0.0.0.0:*          users:(("ceph-mon",pid=17709,fd=28))
LISTEN      0           128                     0.0.0.0:22                    0.0.0.0:*          users:(("sshd",pid=1364,fd=6))
LISTEN      0           4096                    0.0.0.0:111                   0.0.0.0:*          users:(("rpcbind",pid=1078,fd=4),("systemd",pid=1,fd=149))
LISTEN      0           512                10.199.10.20:6811                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=25))
LISTEN      0           512                10.199.10.20:6810                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=24))
LISTEN      0           512                10.199.10.20:6809                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=21))
LISTEN      0           512                10.199.10.20:6808                  0.0.0.0:*          users:(("ceph-osd",pid=258031,fd=20))
LISTEN      0           512                10.199.10.20:6815                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=25))
LISTEN      0           512                10.199.10.20:6814                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=24))
LISTEN      0           512                10.199.10.20:6813                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=21))
LISTEN      0           512                10.199.10.20:6812                  0.0.0.0:*          users:(("ceph-osd",pid=258026,fd=20))
LISTEN      0           512                10.199.10.20:6803                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=25))
LISTEN      0           512                10.199.10.20:6802                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=24))
LISTEN      0           512                10.199.10.20:6801                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=21))
LISTEN      0           512                10.199.10.20:6800                  0.0.0.0:*          users:(("ceph-osd",pid=258015,fd=20))
LISTEN      0           512                10.199.10.20:6807                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=25))
LISTEN      0           512                10.199.10.20:6806                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=24))
LISTEN      0           512                10.199.10.20:6805                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=21))
LISTEN      0           512                10.199.10.20:6804                  0.0.0.0:*          users:(("ceph-osd",pid=258072,fd=20))
LISTEN      0           512                  10.1.1.124:3300                  0.0.0.0:*          users:(("ceph-mon",pid=17709,fd=27))
LISTEN      0           100                   127.0.0.1:25                    0.0.0.0:*          users:(("master",pid=1528,fd=13))
LISTEN      0           4096                  127.0.0.1:85                    0.0.0.0:*          users:(("pvedaemon worke",pid=2079,fd=6),("pvedaemon worke",pid=2078,fd=6),("pvedaemon worke",pid=2077,fd=6),("pvedaemon",pid=2076,fd=6))
LISTEN      0           4096                          *:3128                        *:*          users:(("spiceproxy work",pid=616939,fd=6),("spiceproxy",pid=2663,fd=6)) 
LISTEN      0           128                        [::]:22                       [::]:*          users:(("sshd",pid=1364,fd=7))
LISTEN      0           4096                       [::]:111                      [::]:*          users:(("rpcbind",pid=1078,fd=6),("systemd",pid=1,fd=151))
LISTEN      0           4096                          *:8006                        *:*          users:(("pveproxy worker",pid=616942,fd=6),("pveproxy worker",pid=616941,fd=6),("pveproxy worker",pid=616940,fd=6),("pveproxy",pid=2644,fd=6))
LISTEN      0           100                       [::1]:25                       [::]:*          users:(("master",pid=1528,fd=14))

I can open TCP connections from any of the OSD hosts to the other OSDs ports, including the ones the heartbeat checks say aren't responding. The network layer seems fine.

For example, grabbing the logs for OSD 2 just now, it complains that heartbeat checks to OSD 10 is failing:

Code: 
root@backup2:~# journalctl -u ceph-osd@2 -n 1
Jul 08 08:56:54 backup2 ceph-osd[258026]: 2026-07-08T08:56:54.041+0000 7bf361cd96c0 -1 osd.2 7133 heartbeat_check: no reply from 10.1.1.193:6806 osd.10 ever on either front or back, first ping sent 2026-07-08T08:50:05.631450+0000 (oldest deadline 2026-07-08T08:50:25.631450+0000)

Yet, logged opening a tcp connection to the exact IP and port it says is failing - from the host that OSD 2 is running on - seems to be working fine:
Code: 
root@backup2:~# telnet 10.1.1.193 6806
Trying 10.1.1.193...
Connected to 10.1.1.193.
Escape character is '^]'.
ceph v2

I have no idea why on earth it's saying the heartbeat check is failing (and for all of the nodes), when the network itself is clearly working. :(

Any ideas? The only thing I can think of would be if the OSD code itself is somehow sending heartbeat checks to the public interface from the cluster network interface. And that would indeed fail as there's no route between them.

aderumier said:
BTW, isn't a little bit strange to have only 1gbit/s for your public network ? (I mean, this is the network used by the vm to read/write to your ceph osd, so it's the "storage" network).
The cluster_network for ceph is optional, it's really here if you want to offload replication of osds on another network.

IMOH, you should use your 10Gb/s for public network, and don't defined any cluster_network.
Click to expand...
At present the 10GbE switch is a single switch that nothing else is connected to, just the three nodes.

After I get some more 10GbE network cards and hook it up to the rest of the homelab, then sure I can have the other things talking to it. But at the moment that's not how things are connected. There's only really a single VM that runs much on these nodes, and it's hosted directly on one of the boxes and has low throughput needs. The main purpose of the ceph network is to exist to be a replication target for my TrueNAS box which is currently connected via a 1GbE switch anyway. And yeah, that's a candidate for future upgrading to 10GbE in the ~nearish future. :)
 
Last edited:
  • Like
Reactions: Johannes S
tschmidt said:
No idea what the consequences are but 10.1.1.0/16 isn't a network. For a proper network all host bits should be zero. Use 10.1.0.0/16.
Click to expand...
Interesting. I thought the bits not covered by the CIDR mask were ignored, so 10.1.1.0/16 and 10.1.0.0/16 would be the same. Sounds like that might not be the case, so I'll update my configs from here on. :)
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom
Back