[SOLVED] Failed to add unused disk from external ceph storage

[semira uthsala]

Hi All,

I'm using 3x pve 7.1-10 nodes as a cluster. I already connected synology box as shared storage. all features are working properly at the moment.

I'm working on moving some workloads from failing redhat openstack cluster with ceph storage to the pve cluster. As I did many times before, I connected redhat ceph cluster to pve using rbd.

below the steps I did

Create ceph user with called proxmox

[client.proxmox]
        key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        caps mon = "allow *"
        caps osd = "allow *"

crate new foder on /etc/pve/prive/ called "ceph"

created new keyring file on /etc/pve/prive/ceph called rh-ceph.keyring

added rbd storage from gui. datacenter -> storage

rbd: rh-ceph
        content images
        krbd 0
        monhost 10.10.80.2,10.10.80.3,10.10.80.4
        nodes pve-01
        pool volumes
        username proxmox

after this step I can see the rh-ceph storage attached to the pve-01. I can check the space utilization and all the images inside it.

Then I do rbd copy of openstack vm image and with the proper name that proxmox can search

rbd cp volume/openstack-image-name volume/vm-103-sidk-0

then I resacn the vmid using qm rescan

qm rescan --vmid 103

After this step I can see new unreferenced disk added to vm103 with rh-ceph storage backend

but when I try to add this disk to the vm it giving me below error

failed to update VM 103: rbd error: rbd: info: (95) Operation not supported (500)

I followed this thread that suggest to remove the keyring file from the location /etc/pve/priv/ceph . but after I remove the keyring file ceph storage wont work anymore.

root@pve-01:/etc/pve/priv/ceph# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-10
pve-kernel-5.13: 7.1-7
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

 

[aaron]

Are there other Proxmox VE hosts connected to that Ceph cluster as well?
What version is that Ceph cluster running?

 

[semira uthsala]

Hi Aaron,

Are there other Proxmox VE hosts connected to that Ceph cluster as well?

No. the pve cluster I'm trying to connect got 3 nodes. but when I add RBD storage I only added to one node (pve-01)

but this ceph cluster is part of redhat openstack which used by 6 other compute node from openstack cluster. The pool I tried to connect to pve is not used by openstack.

What version is that Ceph cluster running?

ceph version 0.94.5-14.el7cp

 

[semira uthsala]

Any Idea what is the actual problem and how to fix this? I'm stuck with VM migration to pve due to this

 

[aaron]

ceph version 0.94.5-14.el7cp

Is that actually Hammer? (according to https://docs.ceph.com/en/latest/releases/)

If so, what PVE and Ceph versions do the other PVE nodes run that are able to use that cluster?

The node we are talking about is running 15.2.14-pve1 (Octopus) according to the pveversion -v output. Could it be, that the version difference is just too large to be fully compatible?

 

[semira uthsala]

Thanks for your response.

what PVE and Ceph versions do the other PVE nodes run that are able to use that cluster?

There is no other PVE nodes connected to this ceph cluster. only RH openstack compute nodes.

I connect same type of RH ceph storage (not the same version) to another pve cluster before for different project but it was running pve 6.x

The node we are talking about is running 15.2.14-pve1 (Octopus) according to the pveversion -v output. Could it be, that the version difference is just too large to be fully compatible?

Is there any way I can check some debug logs about this rbd connection ? If the ceph version is too old I may need to try different migration method like cloning VM using clonezilla. but that will add another layer of complexity and increased downtime while VM migration.

Is there a way to convert raw image on ceph to qcow2 ? if yes could give me example

Thanks

 

[aaron]

There is no other PVE nodes connected to this ceph cluster. only RH openstack compute nodes.

Ah okay, I was referring that you mentioned that you did this many times before. Would be interesting to know which versions the RH Ceph cluster had and the PVE node had in the situations where it worked.


RBD should be able to export images to raw format, at least it does so in the Ceph versions that I know with rbd export <image> <path to image file>.

You can then use the qm importdisk (man page) command to import that image to a PVE VM.

A few things I personally would try before falling back to the export / import way:

Check the infos of the image:

If I run rbd info <image> on a recent version, I see the following:

# rbd info vm-101-disk-0
rbd image 'vm-101-disk-0':
    size 3 GiB in 768 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: 80645a50f6c93
    block_name_prefix: rbd_data.80645a50f6c93
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features:
    flags:
    create_timestamp: Tue Feb  1 17:12:12 2022
    access_timestamp: Tue Feb  1 17:12:12 2022
    modify_timestamp: Tue Feb  1 17:12:12 2022

Is it possible, that some features are not available in your situation? Or the format might be different, and therefore maybe too old?

You could try to map the image manually to see if it works, and if it doesn't, you might get more infos from the error. rbd map <image>.


Right now I do suspect that the versions are way to far apart and suspect some incompatibilities since version 15 (ocotopus) is 7 releases after 0.95 (hammer)

 

[semira uthsala]

I used rbd export to nfs share finally. And managed to move all the vms