Proxmox 4.x cluster With Ceph cluster

[Manish saini]

Hi,

we have a proxmox cluster on network

192.168.2.x

and ceph cluster at network

10.10.2.x

proxmox and ceph are connected with another NIC with network 10.10.5.x.

both clusters are able to ping each other and no firewall restrictions.
When we mounting ceph storage at proxmox, its says communication failure and another storages like NFS and local which mounted on proxmox gets a freeze. when remove entry from storage.cfg its work fine for NFS and local.
Main concern is that ceph is not working on proxmox cluster when we mount with error communication failure.

with name of storage pool keyring file already exist at proxmox vm /etc/pve/priv/ceph/storagename.keyring

Please sugggest if there anything missed or something needs to be changed at new versions for mounting as RBD.

Thanks in advance!!!

 

[udo]

Hi,

we have a proxmox cluster on network

192.168.2.x

and ceph cluster at network

10.10.2.x

proxmox and ceph are connected with another NIC with network 10.10.5.x.

Hi,
is the routing correct?
can you ping all mon- and all osd-hosts from the pve-nodes?
Do you use the right mtu? If you use jumboframes, can you ping with jumboframes too? Like "ping -M do -s 8700 ip.of.mon.node"
Are there a firewall between proxmox-hosts and mon/osd-hosts?

Udo

 

[Manish saini]

Yes, Mon and OSD hosts are on those three nodes itself, Each nodes has 1TBX 6 HDD. yes they are in ping with each others. No firewalls between them, also proxmox firewall disabled.

Ceph cluster:

Node 1:
10.10.2.17 (eth1)
10.10.5.17 (eth2)

Node 2: >>>>>>>Master Node
10.10.2.12 (eth1)
10.10.5.12 (eth2)

Node 3:
10.10.2.161 (eth1)
10.10.5.161 (eth2)

Note: 10.10.2.x is internal ceph network and 10.10.5.x internal network for ceph-proxmox connection

Proxmox Cluster:

Node 1: >> Master Node
10.10.5.32 (eth1)
192.168.2.32(eth2)

Node 2:
10.10.5.56 (eth1)
192.168.2.56 (eth2)

Node 3:
10.10.5.37 (eth1)
192.168.2.37 (eth2)

Note: 192.168.2.x is internal proxmox network and 10.10.5.x internal network for ceph-proxmox connection.


Only with 10.10.5.x network ip's they can ping each other and ceph and proxmox has their private networks which can not ping each other.

Monitor hosts are 10.10.2.17:6789; 10.10.2.161:6789; 10.10.2.12:6789.

Let me know if any additional information needs.

Thanks in advance!!!

 

[udo]

Yes, Mon and OSD hosts are on those three nodes itself, Each nodes has 1TBX 6 HDD. yes they are in ping with each others. No firewalls between them, also proxmox firewall disabled.

...
Note: 10.10.2.x is internal ceph network and 10.10.5.x internal network for ceph-proxmox connection

Proxmox Cluster:
...
Note: 192.168.2.x is internal proxmox network and 10.10.5.x internal network for ceph-proxmox connection.


Only with 10.10.5.x network ip's they can ping each other and ceph and proxmox has their private networks which can not ping each other.

Monitor hosts are 10.10.2.17:6789; 10.10.2.161:6789; 10.10.2.12:6789.

Let me know if any additional information needs.

Thanks in advance!!!

Hi,
you mixed the naming.
Your mon are in the ceph-public_network which must be reachable from all clients (and OSDs too).
The ceph cluster network must be accesible from the OSD-nodes only.

You put the pve-host in the wrong network (ceph cluster_network). Separating the two networks for ceph, makes sense, to split the network between clients/MONs/OSDs and the replica-transfer between OSD-Nodes.

If you have an chance, change the network from the pve-hosts from 10.10.5.0/24 to 10.10.2.0/24! If not, use simply host routing on all pve-hosts like

ip route add 10.10.2.17 via 10.10.5.17
ip route add 10.10.2.161 via 10.10.5.161
ip route add 10.10.2.12 via 10.10.5.12

Udo