Bandwidth very low - 2,3 MB/sec

ssaman

ssaman

Active Member
Proxmox Subscriber
Oct 28, 2015
38
2
28
Hello together,

we have a big problem with our ceph configuration.
Since 2 weeks the Bandwidth dropped extreme low.

Code: 
Total time run:         101.809625
Total writes made:      60
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     2.35734
Stddev Bandwidth:       2.64243
Max bandwidth (MB/sec): 16
Min bandwidth (MB/sec): 0
Average IOPS:           0
Stddev IOPS:            0
Max IOPS:               4
Min IOPS:               0
Average Latency(s):     23.4775
Stddev Latency(s):      14.9854
Max latency(s):         71.8323
Min latency(s):         7.56157
Cleaning up (deleting benchmark objects)
Removed 60 objects
Clean up completed and total clean up time :0.350491

Has anybody an idea how we can fix this?
 
Which setting do you need?
Code: 
[global]
         auth client required = cephx
         auth cluster required = cephx
         auth service required = cephx
         cluster network = 10.200.1.0/25
         fsid = e999c2ba-bd91-41d1-92b1-c7874b4b2b40
         keyring = /etc/pve/priv/$cluster.$name.keyring
         mon allow pool delete = true
         osd journal size = 5120
         osd pool default min size = 2
         osd pool default size = 3
         public network = 10.200.1.0/25

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.node3]
         host = node3
         mon addr = 10.200.1.3:6789

[mon.node2]
         host = node2
         mon addr = 10.200.1.2:6789

[mon.node1]
         host = node1
         mon addr = 10.200.1.1:6789
Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: node1-corosync
  }
  node {
    name: node2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: node2-corosync
  }
  node {
    name: node3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: node3-corosync
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster-6
  config_version: 3
  interface {
    bindnetaddr: 10.200.0.0
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}
Code: 
{
"nodename": "node1",
"version": 13,
"cluster": { "name": "cluster-6", "version": 3, "nodes": 3, "quorate": 1 },
"nodelist": {
  "node1": { "id": 1, "online": 1, "ip": "10.200.0.1"},
  "node2": { "id": 2, "online": 1, "ip": "10.200.0.2"},
  "node3": { "id": 3, "online": 1, "ip": "10.200.0.3"}
  }
}
Code: 
ID  CLASS WEIGHT   REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE NAME
-1       75.09892        - 75.1TiB 9.82TiB 65.3TiB 13.08 1.00   - root default
-3       31.43779        - 31.4TiB 3.40TiB 28.0TiB 10.80 0.83   -     host node1
  0   hdd  7.27730  1.00000 7.28TiB  797GiB 6.50TiB 10.69 0.82 124         osd.0
  1   hdd  7.27730  1.00000 7.28TiB  805GiB 6.49TiB 10.80 0.83 125         osd.1
  2   hdd  7.27730  1.00000 7.28TiB  829GiB 6.47TiB 11.13 0.85 129         osd.2
  3   hdd  7.27730  1.00000 7.28TiB  862GiB 6.44TiB 11.57 0.88 134         osd.3
  4   ssd  1.45540  1.00000 1.46TiB 95.9GiB 1.36TiB  6.43 0.49   0         osd.4
  5   ssd  0.87320  1.00000  894GiB 87.8GiB  806GiB  9.82 0.75   0         osd.5
-7       21.83057        - 21.8TiB 3.21TiB 18.6TiB 14.73 1.13   -     host node2
  6   hdd  5.45740  1.00000 5.46TiB  819GiB 4.66TiB 14.66 1.12 127         osd.6
  7   hdd  5.45740  1.00000 5.46TiB  802GiB 4.67TiB 14.36 1.10 125         osd.7
  8   hdd  5.45789  1.00000 5.46TiB  784GiB 4.69TiB 14.03 1.07 122         osd.8
  9   hdd  5.45789  1.00000 5.46TiB  886GiB 4.59TiB 15.86 1.21 138         osd.9
-10       21.83057        - 21.8TiB 3.21TiB 18.6TiB 14.71 1.12   -     host node3
10   hdd  5.45740  1.00000 5.46TiB  843GiB 4.63TiB 15.08 1.15 131         osd.10
11   hdd  5.45789  1.00000 5.46TiB  774GiB 4.70TiB 13.85 1.06 121         osd.11
12   hdd  5.45740  1.00000 5.46TiB  898GiB 4.58TiB 16.06 1.23 140         osd.12
13   hdd  5.45789  1.00000 5.46TiB  774GiB 4.70TiB 13.85 1.06 120         osd.13
                      TOTAL 75.1TiB 9.82TiB 65.3TiB 13.08
MIN/MAX VAR: 0.49/1.23  STDDEV: 2.65
Code: 
{
    "mon": {
        "ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)": 3
    },
    "mgr": {
        "ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)": 4
    },
    "osd": {
        "ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)": 14
    },
    "mds": {},
    "overall": {
        "ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)": 21
    }
}
Code: 
  cluster:
    id:     e999c2ba-bd91-41d1-92b1-c7874b4b2b40
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum node1,node2,node3
    mgr: node1(active), standbys: c6-node1, node3, node2
    osd: 14 osds: 14 up, 14 in

  data:
    pools:   1 pools, 512 pgs
    objects: 889.36k objects, 3.21TiB
    usage:   9.82TiB used, 65.3TiB / 75.1TiB avail
    pgs:     512 active+clean

  io:
    client:   157KiB/s rd, 688KiB/s wr, 27op/s rd, 64op/s wr
Code: 
ID  CLASS WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       75.09892 root default
-3       31.43779     host node1
  0   hdd  7.27730         osd.0      up  1.00000 1.00000
  1   hdd  7.27730         osd.1      up  1.00000 1.00000
  2   hdd  7.27730         osd.2      up  1.00000 1.00000
  3   hdd  7.27730         osd.3      up  1.00000 1.00000
  4   ssd  1.45540         osd.4      up  1.00000 1.00000
  5   ssd  0.87320         osd.5      up  1.00000 1.00000
-7       21.83057     host node2
  6   hdd  5.45740         osd.6      up  1.00000 1.00000
  7   hdd  5.45740         osd.7      up  1.00000 1.00000
  8   hdd  5.45789         osd.8      up  1.00000 1.00000
  9   hdd  5.45789         osd.9      up  1.00000 1.00000
-10       21.83057     host node3
10   hdd  5.45740         osd.10     up  1.00000 1.00000
11   hdd  5.45789         osd.11     up  1.00000 1.00000
12   hdd  5.45740         osd.12     up  1.00000 1.00000
13   hdd  5.45789         osd.13     up  1.00000 1.00000
Code: 
10 hdd_mainpool,
Code: 
POOL_NAME    USED    OBJECTS CLONES COPIES  MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS    RD      WR_OPS      WR
hdd_mainpool 3.21TiB  889363  71830 2668089                  0       0        0 501915168 12.6TiB 10673148708 85.4TiB

total_objects    889363
total_used       9.82TiB
total_avail      65.3TiB
total_space      75.1TiB
Code: 
replicated_rule
replicated_ssd
replicated_hdd
Code: 
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class ssd
device 5 osd.5 class ssd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
device 9 osd.9 class hdd
device 10 osd.10 class hdd
device 11 osd.11 class hdd
device 12 osd.12 class hdd
device 13 osd.13 class hdd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host node1 {
    id -3        # do not change unnecessarily
    id -2 class hdd        # do not change unnecessarily
    id -5 class ssd        # do not change unnecessarily
    # weight 31.438
    alg straw2
    hash 0    # rjenkins1
    item osd.0 weight 7.277
    item osd.1 weight 7.277
    item osd.2 weight 7.277
    item osd.3 weight 7.277
    item osd.4 weight 1.455
    item osd.5 weight 0.873
}
host node2 {
    id -7        # do not change unnecessarily
    id -8 class hdd        # do not change unnecessarily
    id -9 class ssd        # do not change unnecessarily
    # weight 21.831
    alg straw2
    hash 0    # rjenkins1
    item osd.6 weight 5.457
    item osd.7 weight 5.457
    item osd.8 weight 5.458
    item osd.9 weight 5.458
}
host node3 {
    id -10        # do not change unnecessarily
    id -11 class hdd        # do not change unnecessarily
    id -12 class ssd        # do not change unnecessarily
    # weight 21.831
    alg straw2
    hash 0    # rjenkins1
    item osd.10 weight 5.457
    item osd.13 weight 5.458
    item osd.12 weight 5.457
    item osd.11 weight 5.458
}
root default {
    id -1        # do not change unnecessarily
    id -4 class hdd        # do not change unnecessarily
    id -6 class ssd        # do not change unnecessarily
    # weight 75.099
    alg straw2
    hash 0    # rjenkins1
    item node1 weight 31.438
    item node2 weight 21.831
    item node3 weight 21.831
}

# rules
rule replicated_rule {
    id 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type host
    step emit
}
rule replicated_ssd {
    id 1
    type replicated
    min_size 1
    max_size 10
    step take default class ssd
    step chooseleaf firstn 0 type host
    step emit
}
rule replicated_hdd {
    id 2
    type replicated
    min_size 1
    max_size 10
    step take default class hdd
    step chooseleaf firstn 0 type host
    step emit
}

# end crush map
 
Last edited:
sg90 said:
1/ What is the network your using for CEPH? 1/10Gbps?
Click to expand...
We use a 10Gbps - exact model is Prosafe XS708E
sg90 said:
2/ What model are the HDD disks your using?
Click to expand...
Our disks are from 3 different Manufacturers.

node1
HUH721008AL5200

node2
HUS726060ALE610

node3
wd6002ffwx-68tz4n0


Some of them are connected directly to the Motherboard, the other ones are connected through a LSI MR 9250 4i (non-RAID).

sg90 said:
3/ From where are you running the benchmark command?
Click to expand...
It is from 1 of our 3 nodes
 
We had some issues with some OSDs before . They randomly dropped to "off". So we removed them and included them back to the Pool again.

Code: 
proxmox-ve: 5.4-2 (running kernel: 4.15.18-12-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-12
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-10-pve: 4.15.18-32
ceph: 12.2.12-pve1
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-56
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-41
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom