Network Usage for Ceph and iSCSI

[el_vagokz]

Good afternoon,

Currently, we have a dedicated bond1 interface for Proxmox Ceph traffic and a separate bond2 interface for VM traffic.

Would it be possible to use bond1 for iSCSI connectivity as well, without any negative impact on performance or stability?

 

[LnxBil]

Would it be possible to use bond1 for iSCSI connectivity as well, without any negative impact on performance or stability?

Short anwser: no, you could always impact performance and stability.

Longer answer: it depends on a lot of factors and you may need to tune a lot of things (e.g. QoS / traffic shaping). This is not easy, adding another bond is always the better solution. The optimal configuration to use in a CEPH cluster is 4 dedicated networks (VM, Corosync, CEPH private, CEPH public).

 

[Blueloop]

Good afternoon,

Currently, we have a dedicated bond1 interface for Proxmox Ceph traffic and a separate bond2 interface for VM traffic.

Would it be possible to use bond1 for iSCSI connectivity as well, without any negative impact on performance or stability?

Short answer: yes. When you mix traffic types on a NIC or interface collection, you need to manage it yourself and that is fine.

Some of the considerations are:

• What throughput is required for each traffic stream
• How fast are the interfaces involved
• MTU/jumbo frames
• Are the bonds optimally configured
• Is QoS indicated
• Firewall

You don't give any idea about any requirements so I won't make any detailed recommendations. At an absolute minimum we need:

• Number of hosts in the cluster
• Number of NICs available and speed on the hosts and the iSCSI SAN
• Some basic SAN info
• Some basic switch info

With that we can give you some ideas to start off with.

 

[el_vagokz]

We have 17 nodes. Each node has a bond consisting of 25 Gbps network interfaces configured in active-standby mode. The iSCSI SAN operates at 25 Gbps.

Most likely, we will go with Fibr Channel to avoid taking any unnecessary risks.