CEPH Use High Bandwidth Utilization

[tapaskrmahato]

I have two node CEPH Cluster. But recently I observed that it utilize high bandwidth. How to resolve this issue. please help me out.

 

[sg90]

How much is high? CEPH will use plenty of bandwidth during I/O operations.

 

[tapaskrmahato]

we have 5 virtual machine running on CEPH Cluster. Cluster are running two geographical location using 10Mbps P2P dedicated bandwidth. past few days its consumes near about 25Mbps Bandwidth.

 

[sg90]

we have 5 virtual machine running on CEPH Cluster. Cluster are running two geographical location using 10Mbps P2P dedicated bandwidth. past few days its consumes near about 25Mbps Bandwidth.

25Mbps really isn't a lot, CEPH will have to send IO across between each host for every write and read.

Most people run a CEPH cluster on 10Gbps NIC's.

 

[tapaskrmahato]

25Mbps really isn't a lot, CEPH will have to send IO across between each host for every write and read.

Most people run a CEPH cluster on 10Gbps NIC's.

Thank you for Your Support. Is there any way to control bandwidth? My bandwidth will increase for very certain time then it get normal. please check the below attachment and give me the cause and solution.


2019-10-21 13:17:33.436684 mon.node001 mon.0 10.10.10.50:6789/0 78343 : cluster [WRN] Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
2019-10-21 13:17:38.436921 mon.node001 mon.0 10.10.10.50:6789/0 78344 : cluster [WRN] Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,3 (REQUEST_SLOW)
2019-10-21 13:17:41.949723 mon.node001 mon.0 10.10.10.50:6789/0 78346 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 1)
2019-10-21 13:17:41.949763 mon.node001 mon.0 10.10.10.50:6789/0 78347 : cluster [INF] Cluster is now healthy
2019-10-21 13:17:52.240678 mon.node001 mon.0 10.10.10.50:6789/0 78349 : cluster [WRN] Health check failed: 6 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
2019-10-21 13:17:58.441370 mon.node001 mon.0 10.10.10.50:6789/0 78350 : cluster [WRN] Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
2019-10-21 13:18:03.441695 mon.node001 mon.0 10.10.10.50:6789/0 78352 : cluster [WRN] Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
2019-10-21 13:18:06.193845 mon.node001 mon.0 10.10.10.50:6789/0 78353 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 1)
2019-10-21 13:18:06.193901 mon.node001 mon.0 10.10.10.50:6789/0 78354 : cluster [INF] Cluster is now healthy

 

[sg90]

No, as previously said CEPH is not made to run on such a small amount of bandwidth.

Most people run CEPH on 10Gbps, as even 1Gbps can have performance issues, you will also have issues if 25Mbps is all you have.

I would highly suggest reviewing and changing your setup.

 

[spirit]

for the vm/ct read/write, you can limit iops/bandwith on vm disk advanced options. (but with so low bandwith, you'll emulate the speed of an usbkey)



But if ceph need to rebuild an osd after a failure, I don't think it's possible to limit the bandwith.