[SOLVED] No ceph commands working on node 2 and node 3

[jsterr]

I wanted to install ceph from cli without interacting with the gui and did the following:

• pveceph install on pve-01 pve-02 pve-03
• pveceph init --network 10.5.5.221/24 --cluster-network 10.6.6.221/24 on pve-01
• pveceph mon create on pve-01

Now pveceph works well on pve-01 but pve-02 and pve-03 are not working.

• I cant create mgr/mons on pve-02/pve-03 they just timeout on every single pveceph command im using
• 500 on gui on crushmap and configuration database
• I can use ceph -v on pve-02 and pve-03 but not other ceph commands like ceph or ceph auth (no output)

Spoiler: ceph.conf

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.6.6.221/24
     fsid = 3016b0e9-0420-40db-a34a-2f683c7e1fe6
     mon_allow_pool_delete = true
     mon_host = 10.5.5.221
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 10.5.5.221/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.pve-01]
     public_addr = 10.5.5.221

pveceph status (pve-01): https://pastebin.com/YeQer2kb

Cant u tell, what was wrong in this process?

 

[Alwin]

• I cant create mgr/mons on pve-02/pve-03 they just timeout on every single pveceph command im using
• 500 on gui on crushmap and configuration database
• I can use ceph -v on pve-02 and pve-03 but not other ceph commands like ceph or ceph auth (no output)

Sounds like network issues. Check that all nodes can ping each other and that the interfaces all share the same MTU.

 

[jsterr]

Sounds like network issues. Check that all nodes can ping each other and that the interfaces all share the same MTU.

Thanks you nailed it! I used jumbo frames in config, but my switches lags forgot about 9000 framesize. normal ping worked well:

PING 10.5.5.221 (10.5.5.221) 56(84) bytes of data.
64 bytes from 10.5.5.221: icmp_seq=1 ttl=64 time=0.188 ms
64 bytes from 10.5.5.221: icmp_seq=2 ttl=64 time=0.148 ms

but jumbo-frame ping did not:

root@pve-01:/etc/ceph# ping -M do -s 9000 10.6.6.223
PING 10.6.6.223 (10.6.6.223) 9000(9028) bytes of data.
ping: local error: Message too long, mtu=9000

After setting framesize on switch again everything worked well and pveceph commands are working now