[SOLVED] Unable to start VM on Ceph RBD External After Recent Update

[chrone]

Hi Proxmox,

I am unable to start VM on Ceph RBD external after recent update. From inside Proxmox, I could perform `rbd ls`. What just went wrong?

kvm: -drive file=rbd:rbd/vm-10092-disk-1:mon_host=10.100.100.111,10.100.100.112,10.100.100.113:auth_supported=cephx:id=admin:keyring=/etc/pve/priv/ceph/ceph-proxmox.keyring,if=none,id=drive-scsi0,cache=writeback,format=raw,aio=threads,detect-zeroes=on: Block format 'raw' does not support the option '10.100.100.113:auth_supported'

root@proxmox6:~# pveversion -v
proxmox-ve: 5.1-43 (running kernel: 4.15.17-1-pve)
pve-manager: 5.1-52 (running version: 5.1-52/ba597a64)
pve-kernel-4.13: 5.1-44
pve-kernel-4.15: 5.1-4
pve-kernel-4.15.17-1-pve: 4.15.17-8
pve-kernel-4.15.15-1-pve: 4.15.15-6
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.4-1-pve: 4.13.4-26
ceph: 12.2.5-1~bpo90+1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-30
libpve-guest-common-perl: 2.0-15
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-19
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-2
lxcfs: 3.0.0-1
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-15
pve-cluster: 5.0-26
pve-container: 2.0-22
pve-docs: 5.1-17
pve-firewall: 3.0-8
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-3
qemu-server: 5.0-25
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.8-pve1~bpo9

rbd: ceph-proxmox
        content images
        krbd 0
        monhost 10.100.100.111 10.100.100.112 10.100.100.113
        pool rbd
        username admin

 

[chrone]

Ok, I found the culprit. It was caused by the latest libpve-storage-perl 5.0-19 for Proxmox 5.1.x. Downgrading to libpve-storage-perl 5.0-18 makes the Ceph RBD external volume bootable again.

 

[wolfgang]

You use the ceph packages from ceph and not from Proxmox.

 

[chrone]

You use the ceph packages from ceph and not from Proxmox.

Oh yes, because I use Proxmox as Ceph OSD server as well. The Ceph MON, MDS, and RGW are running as Proxmox VMs. I've been using this setup since Proxmox 4.x with Ceph Jewel and luckily so far so good.

 

[chrone]

@wolfgang libpve-storage-perl 5.0-20 fixed this issue. Thanks!

Does Proxmox require to use its own Ceph packages? Or my setup with using Proxmox host for OSD and VM for MON, MDS, and RGW is still supported in the future?

 

[Alwin]

Does Proxmox require to use its own Ceph packages? Or my setup with using Proxmox host for OSD and VM for MON, MDS, and RGW is still supported in the future?

You can use the setup as long as it works. The Ceph services work together irrespective of the repository, as long as the versions are within Ceph's supported range.

But we only support the upgrade path and Ceph management, if our PVE and our Ceph packages are installed (specially important for support subscription).

For running Ceph services in KVMs, the clock cycles are varying (host cpu load) and this can influence the time on the ceph cluster (eg. clock skew; latency). Better alternative is to use LXC containers or to run it on the bare metal.

 

[chrone]

I see, thanks for the explanation, @Alwin

I had bad experience when running Ceph MON/MDS/RGW on LXC, it hung the host if kernel hung tasks occurs inside LXC, but perhaps that was just one time at the very early of Proxmox 5.x and Ceph 12.2.x.

 

[Alwin]

I had bad experience when running Ceph MON/MDS/RGW on LXC, it hung the host if kernel hung tasks occurs inside LXC, but perhaps that was just one time at the very early of Proxmox 5.x and Ceph 12.2.x.

This may happen, as the container limits access to the host system, rather then being abstracted, in general.