Proxmox cluster listening on all interfaces

U

unsichtbarre

Member
Oct 1, 2024
76
21
8
Pertaining to: https://forum.proxmox.com/threads/how-to-change-ceph-internal-cluster-network.132513/ I already had 'cluster_network = 10.26.22.101/24' defined, now I am wondering why 'ceph-osd' is listening on the management interface (10.26.20.101) as well?

In the results below, I notice that 'ceph-osd' only is listening on the cluster_network, however 'ceph-osd' and 'ceph-mon' and 'ceph-mds' are listening on the 'public_network' as well. Can I restrict the OSDs to only the 'cluster_network' and would that be advantageous?

THX,
-JB

Code: 
root@pve101:~# ss -tulpn | grep ceph
tcp   LISTEN 0      512     10.26.22.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=25))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=24))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6800      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=20))                                                                                                             
tcp   LISTEN 0      512     10.26.22.101:6801      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=21))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:3300      0.0.0.0:*    users:(("ceph-mon",pid=1632,fd=27))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6817      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6816      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6801      0.0.0.0:*    users:(("ceph-mds",pid=1630,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6800      0.0.0.0:*    users:(("ceph-mds",pid=1630,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2452,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2453,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=23))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2454,fd=22))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=19))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2451,fd=18))                                                                                                             
tcp   LISTEN 0      512     10.26.20.101:6789      0.0.0.0:*    users:(("ceph-mon",pid=1632,fd=28))
 
Additionally, I have specified a second 'cluster_network =' in ceph.conf, restarted all OSDs (rebooted whole node as well) and none of the OSDs seem to be listening on the additional network. Is that correct?

THX,
-JB

Code: 
root@pve101:~# cat /etc/pve/ceph.conf     
[global]
        auth_client_required = cephx
        auth_cluster_required = cephx
        auth_service_required = cephx
        cluster_network = 10.26.22.101/24, 10.26.23.101/24
        fsid = 5a1d704b-24e1-44e4-98c5-828819c08cc0
        mon_allow_pool_delete = true
        mon_host = 10.26.20.101 10.26.20.102 10.26.20.103
        ms_bind_ipv4 = true
        ms_bind_ipv6 = false
        osd_pool_default_min_size = 2
        osd_pool_default_size = 3
        public_network = 10.26.20.101/24

[client]
        keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
        keyring = /etc/pve/ceph/$cluster.$name.keyring

[mds]
        keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.pve101]
        host = pve101
        mds_standby_for_name = pve

[mds.pve102]
        host = pve102
        mds_standby_for_name = pve

[mds.pve103]
        host = pve103
        mds_standby_for_name = pve

[mon.pve101]
        public_addr = 10.26.20.101

[mon.pve102]
        public_addr = 10.26.20.102

[mon.pve103]
        public_addr = 10.26.20.103

Code: 
root@pve101:~# ss -tulpn | grep ceph
tcp   LISTEN 0      512     10.26.20.101:6816      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6817      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6800      0.0.0.0:*    users:(("ceph-mds",pid=1628,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6801      0.0.0.0:*    users:(("ceph-mds",pid=1628,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=18))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=19))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=22))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=23))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:6789      0.0.0.0:*    users:(("ceph-mon",pid=1630,fd=28))                                                                                                     
tcp   LISTEN 0      512     10.26.20.101:3300      0.0.0.0:*    users:(("ceph-mon",pid=1630,fd=27))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6805      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6804      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6807      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6806      0.0.0.0:*    users:(("ceph-osd",pid=2484,fd=24))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6801      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6800      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6803      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6802      0.0.0.0:*    users:(("ceph-osd",pid=2483,fd=24))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6813      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6812      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6815      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6814      0.0.0.0:*    users:(("ceph-osd",pid=2482,fd=24))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6809      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=21))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6808      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=20))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6811      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=25))                                                                                                     
tcp   LISTEN 0      512     10.26.22.101:6810      0.0.0.0:*    users:(("ceph-osd",pid=2485,fd=24))
 
You must log in or register to reply here.
Top Bottom
Back