mixing iscsi & ceph storage

[benoitc]

My current setup is the following; I have 3 nodes with each 2 10GbE NIC. On each I setup ceph and an iscsi storahe. Isci is handled on main cluster NIC (shared with proxmox sync network) while the ceph data network is handled on another NIC. Each NIC is connected on a different switch. The NAS has 2x5GbE and 1 10 GbeE port. I am using multipath to balance the load between each port to the NAS:



I am wondering if i shouldn't put one port of the nas (the 10G) on the other switch and access it via the same NIC handling CEPH. Is someone doing it? I wonder if it would be OK to not let alone the CEPH network with such traffic. Thoughts?

 

[Alwin]

You need to benchmark this. When disaster strikes then Ceph will start recovery (refilling the recovered node) and may max out bandwidth.

 

[benoitc]

You need to benchmark this. When disaster strikes then Ceph will start recovery (refilling the recovered node) and may max out bandwidth.

hmm ok. The 2 switches have a non-blocking throughput of 120 Gbps, switching capacity of 240 Gbps and forwarding rate of 178 Mpps so it's probably enough but indeed i will test. I guess also using a separate vlan for iscsi may be needed insuch case though not sure since they are on separate network and hardwares/nics.

I will test what happen when I re-add a node.