CEPH Use High Bandwidth Utilization

tapaskrmahato

Member
Oct 21, 2019
7
0
6
35
I have two node CEPH Cluster. But recently I observed that it utilize high bandwidth. How to resolve this issue. please help me out.
 

sg90

Well-Known Member
Sep 21, 2018
287
42
48
32
How much is high? CEPH will use plenty of bandwidth during I/O operations.
 

tapaskrmahato

Member
Oct 21, 2019
7
0
6
35
we have 5 virtual machine running on CEPH Cluster. Cluster are running two geographical location using 10Mbps P2P dedicated bandwidth. past few days its consumes near about 25Mbps Bandwidth.
 

sg90

Well-Known Member
Sep 21, 2018
287
42
48
32
we have 5 virtual machine running on CEPH Cluster. Cluster are running two geographical location using 10Mbps P2P dedicated bandwidth. past few days its consumes near about 25Mbps Bandwidth.

25Mbps really isn't a lot, CEPH will have to send IO across between each host for every write and read.

Most people run a CEPH cluster on 10Gbps NIC's.
 

tapaskrmahato

Member
Oct 21, 2019
7
0
6
35
25Mbps really isn't a lot, CEPH will have to send IO across between each host for every write and read.

Most people run a CEPH cluster on 10Gbps NIC's.
Thank you for Your Support. Is there any way to control bandwidth? My bandwidth will increase for very certain time then it get normal. please check the below attachment and give me the cause and solution.
Bandwidth lan.png

2019-10-21 13:17:33.436684 mon.node001 mon.0 10.10.10.50:6789/0 78343 : cluster [WRN] Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)
2019-10-21 13:17:38.436921 mon.node001 mon.0 10.10.10.50:6789/0 78344 : cluster [WRN] Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 1,3 (REQUEST_SLOW)
2019-10-21 13:17:41.949723 mon.node001 mon.0 10.10.10.50:6789/0 78346 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 1)
2019-10-21 13:17:41.949763 mon.node001 mon.0 10.10.10.50:6789/0 78347 : cluster [INF] Cluster is now healthy
2019-10-21 13:17:52.240678 mon.node001 mon.0 10.10.10.50:6789/0 78349 : cluster [WRN] Health check failed: 6 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
2019-10-21 13:17:58.441370 mon.node001 mon.0 10.10.10.50:6789/0 78350 : cluster [WRN] Health check update: 3 slow requests are blocked > 32 sec. Implicated osds 1,5 (REQUEST_SLOW)
2019-10-21 13:18:03.441695 mon.node001 mon.0 10.10.10.50:6789/0 78352 : cluster [WRN] Health check update: 1 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)
2019-10-21 13:18:06.193845 mon.node001 mon.0 10.10.10.50:6789/0 78353 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 1 slow requests are blocked > 32 sec. Implicated osds 1)
2019-10-21 13:18:06.193901 mon.node001 mon.0 10.10.10.50:6789/0 78354 : cluster [INF] Cluster is now healthy
 

sg90

Well-Known Member
Sep 21, 2018
287
42
48
32
No, as previously said CEPH is not made to run on such a small amount of bandwidth.

Most people run CEPH on 10Gbps, as even 1Gbps can have performance issues, you will also have issues if 25Mbps is all you have.

I would highly suggest reviewing and changing your setup.
 
  • Like
Reactions: Alwin

spirit

Famous Member
Apr 2, 2010
5,865
702
133
www.odiso.com
for the vm/ct read/write, you can limit iops/bandwith on vm disk advanced options. (but with so low bandwith, you'll emulate the speed of an usbkey)



But if ceph need to rebuild an osd after a failure, I don't think it's possible to limit the bandwith.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!