Proxmox VE 5.1 or 5.2 connected to external Ceph Mimic cluster?

[Arcanum]

I have a small Proxmox VE 5.1 cluster (it will be upgraded to 5.2 this summer). I use an external Ceph cluster for storage, currently on Luminous.

Has anyone upgraded from Luminous to Mimic with Proxmox 5.1 or 5.2 utilizing the Ceph cluster? If so, did you run into any problems?

I wouldn't be too worried, except that when I upgraded to Luminous, the older version of Proxmox I was running at the time got very, very angry. I believe it was only an issue due to changes in the formatting of Ceph's status output, which I don't expect to be a problem this time, but I thought I'd ask anyway.

 

[udo]

Hi,
ceph don't support mimic on stretch (need an newer toolchain).
But perhaps you can run an external mimic cluster, if you don't set the crush tunables to optimal...
and perhaps "ceph osd set-require-min-compat-client jewel/luminous"

Udo

 

[Arcanum]

Finally getting back to this. Busy week....

Yeah, Proxmox and libvirt would be purely acting as a client. Looking at the Ceph Mimic documentation, it doesn't appear the tunables have changed, so that shouldn't be an issue.

I'm not seeing anything in the Mimic release notes that would indicate incompatibility with Jewel or Luminous librbd clients, either.

So it should work fine. Then again, the problem I encountered when I upgraded to Luminous had nothing to do with tunables or client versions, and had everything to do with the format of ceph status changing.

I was just hoping that someone here had already tried it. Thanks.

 

[elurex]

I am running PVE 5.2 with external Ceph Mimic Cluster and you need to upgrade librbd to 13.2.0-1 (ceph mimic) or otherwise it will fail to connect.

root@pve1:~# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
pve-manager: 5.2-3 (running version: 5.2-3/785ba980)
pve-kernel-4.15: 5.2-3
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-34
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9
root@pve1:~# dpkg -l |grep librbd
ii  librbd1                              13.2.0-1                       amd64        RADOS block device client library
ii  python-rbd                           13.2.0-1                       amd64        Python 2 libraries for the Ceph librbd library
root@pve1:~# rbd ls
test-disk-1
vm-911-disk-1
vm-911-disk-2
vm-912-disk-1
vm-912-disk-2
vm-913-disk-1
vm-913-disk-2
vm-914-disk-1
vm-914-disk-2
root@pve1:~# rbd showmapped
id pool image         snap device
0  rbd  vm-911-disk-1 -    /dev/rbd0
1  rbd  vm-911-disk-2 -    /dev/rbd1
2  rbd  vm-912-disk-1 -    /dev/rbd2
3  rbd  vm-913-disk-1 -    /dev/rbd3
4  rbd  vm-913-disk-2 -    /dev/rbd4
5  rbd  vm-914-disk-1 -    /dev/rbd5
6  rbd  vm-912-disk-2 -    /dev/rbd6
7  rbd  vm-914-disk-2 -    /dev/rbd7

 

[Arcanum]

I am running PVE 5.2 with external Ceph Mimic Cluster and you need to upgrade librbd to 13.2.0-1 (ceph mimic) or otherwise it will fail to connect.

Ahh, interesting, thanks. That's probably why udo mentioned the "ceph osd set-require-min-compat-client jewel/luminous" command. I wouldn't have expected Mimic to block older clients by default.

How did you get the Mimic library on PVE 5.2?

 

[spirit]

How did you get the Mimic library on PVE 5.2?

it's not available on debian stretch, as it need gcc8 to compile.

but you can use luminous client, on a mimic cluster, no problem.

 

[Arcanum]

it's not available on debian stretch, as it need gcc8 to compile.

Yeah, that was my understanding based on what I found while searching, which is why I was scratching my head at elurex's post.

but you can use luminous client, on a mimic cluster, no problem.

Cool, thanks.

 

[elurex]

I can confirm pve's luminous client will not work with ceph 13 mimic, that's why I went through all the trouble to find upgrade path to ceph mimic

wget -q -O- 'https://static.croit.io/keys/release.asc' | apt-key add -
echo 'deb https://static.croit.io/debian-mimic/ stretch main' >> /etc/apt/sources.list
apt update && apt upgrade

from paul.emmerich at croit.io

Caution: this is not yet qualified for production use by us, use at your
own risk. The mirror
also contains a newer version of libc because we are currently building
against libc 2.27.

However, if you just use it for librbd client... it's pretty rock solid

 

[Alwin]

To note: The foreign repository is not needed and will void support.

The above aside. It should be sufficient to install the luminous packages provided by us, as on a default Ceph Mimic setup, the client compatibility is set to luminous. The default packages shipped with Debian are Jewel.

I can confirm pve's luminous client will not work with ceph 13 mimic, that's why I went through all the trouble to find upgrade path to ceph mimic

What does not work with the Luminous client, when connecting to a Mimic cluster? Is the client compatibility set to Luminous?

 

[elurex]

@Alwin

it is only testing in lab, when client set to Luminous

ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it

monclient: hunting for new mon
2018-07-06 15:27:59.835155 7f14bc69d700  0 will not decode message of type 41 version 4 because compat_version 4 > supported
 version 3
2018-07-06 15:27:59.835255 7f14befa3700  0 monclient: hunting for new mon

also in my post already stated it's not meant for production and I understand this test pve instance does not get any support at all