Ceph - Using jemalloc

[David Herselman]

It appears PVE 5.1 provides support for both tcmalloc and jemalloc:

[admin@kvm5b ~]# dpkg -l '*malloc*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                               Version                Architecture           Description
+++-==================================-======================-======================-=========================================================================
ii  libjemalloc1                       3.6.0-9.1              amd64                  general-purpose scalable concurrent malloc(3) implementation
ii  libtcmalloc-minimal4               2.5-2.2                amd64                  efficient thread-caching malloc

Current processes appears to be using tcmalloc:

[admin@kvm5b ~]# readelf -d `which ceph-osd` | grep malloc
 0x0000000000000001 (NEEDED)             Shared library: [libtcmalloc.so.4]

Are packages compiled to provide support for both tcmalloc and jemalloc? It appears one can remove a compile time option to allow loading jemalloc at run time:
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1702777


System packages:

[admin@kvm5b ~]# pveversion -v
proxmox-ve: not correctly installed (running kernel: 4.15.15-1-pve)
pve-manager: 5.1-51 (running version: 5.1-51/96be5354)
pve-kernel-4.15: 5.1-3
pve-kernel-4.15.15-1-pve: 4.15.15-6
ceph: 12.2.4-pve1
corosync: 2.4.2-pve4
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-30
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-18
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 2.1.1-3
lxcfs: 2.0.8-2
novnc-pve: 0.6-4
openvswitch-switch: 2.8.1+dfsg1-6
proxmox-widget-toolkit: 1.0-15
pve-cluster: 5.0-25
pve-container: 2.0-21
pve-docs: 5.1-17
pve-firewall: 3.0-8
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-2
qemu-server: 5.0-25
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.7-pve1~bpo9

 

[fabian]

I don't think there is a way to compile Ceph to support both tcmalloc and jemalloc (it is possible this has changed since the last time I checked - if you have concrete changes that are known to work, feel free to propose them on pve-devel). since jemalloc increases memory usage of each OSD instance by a lot, we don't want to enable it by default (see https://pve.proxmox.com/pipermail/pve-devel/2017-May/026212.html for the last time this came up).

 

[aderumier]

note that since luminous + bluestore, jemalloc don't work well. (because of rocksdb)

Ceph devs said that tcmalloc if fine now, since they have switched to async messenger.