Proxmox cluster listening on all interfaces

U

unsichtbarre

Member
Oct 1, 2024
77
21
8
Pertaining to: https://forum.proxmox.com/threads/how-to-change-ceph-internal-cluster-network.132513/ I already had 'cluster_network = 10.26.22.101/24' defined, now I am wondering why 'ceph-osd' is listening on the management interface (10.26.20.101) as well?

In the results below, I notice that 'ceph-osd' only is listening on the cluster_network, however 'ceph-osd' and 'ceph-mon' and 'ceph-mds' are listening on the 'public_network' as well. Can I restrict the OSDs to only the 'cluster_network' and would that be advantageous?

THX,
-JB

Code: 
root@pve101:~# ss -tulpn | grep ceph
tcp   LISTEN 0      512     10.26.22.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6800      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6801      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:3300      0.0.0.0:*    users:(("ceph-mon",pid=1632,fd=27))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6817      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6816      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6801      0.0.0.0:*    users:(("ceph-mds",pid=1630,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6800      0.0.0.0:*    users:(("ceph-mds",pid=1630,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6789      0.0.0.0:*    users:(("ceph-mon",pid=1632,fd=28))
 
Additionally, I have specified a second 'cluster_network =' in ceph.conf, restarted all OSDs (rebooted whole node as well) and none of the OSDs seem to be listening on the additional network. Is that correct?

THX,
-JB

Code: 
root@pve101:~# cat /etc/pve/ceph.conf     
[global]
        auth_client_required = cephx
        auth_cluster_required = cephx
        auth_service_required = cephx
        cluster_network = 10.26.22.101/24, 10.26.23.101/24
        fsid = 5a1d704b-24e1-44e4-98c5-828819c08cc0
        mon_allow_pool_delete = true
        mon_host = 10.26.20.101 10.26.20.102 10.26.20.103
        ms_bind_ipv4 = true
        ms_bind_ipv6 = false
        osd_pool_default_min_size = 2
        osd_pool_default_size = 3
        public_network = 10.26.20.101/24

[client]
        keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
        keyring = /etc/pve/ceph/$cluster.$name.keyring

[mds]
        keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.pve101]
        host = pve101
        mds_standby_for_name = pve

[mds.pve102]
        host = pve102
        mds_standby_for_name = pve

[mds.pve103]
        host = pve103
        mds_standby_for_name = pve

[mon.pve101]
        public_addr = 10.26.20.101

[mon.pve102]
        public_addr = 10.26.20.102

[mon.pve103]
        public_addr = 10.26.20.103

Code: 
root@pve101:~# ss -tulpn | grep ceph
tcp   LISTEN 0      512     10.26.20.101:6816      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6817      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6800      0.0.0.0:*    users:(("ceph-mds",pid=1628,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6801      0.0.0.0:*    users:(("ceph-mds",pid=1628,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6789      0.0.0.0:*    users:(("ceph-mon",pid=1630,fd=28))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:3300      0.0.0.0:*    users:(("ceph-mon",pid=1630,fd=27))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=24))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6801      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6800      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=24))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=24))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=24))
 
spirit said:
Osd need to talk to monitor and clients (vm) which are running in public network, so they always use both public and private network
Click to expand...
Thanks! Answered and explained.

Question remains, why are the OSDs not also using the second cluster_network?
 
You must log in or register to reply here.
Top Bottom
Back