Unable to access ceph-based disks with krbd option enabled

[Waschbüsch]

Hi all,

I just upgraded my cluster to Proxmox VE 6.1 and wanted to give the updated krbd integration a spin.

So, I set my rbd storage to use krbd and then tried to migrate a vm to a different node. This is what I got:

2019-12-08 11:49:22 starting migration of VM 211 to node 'srv01' (192.168.1.1)
2019-12-08 11:49:22 starting VM 211 on remote node 'srv01'
2019-12-08 11:49:27 can't map rbd volume vm-211-disk-1: rbd: sysfs write failed
2019-12-08 11:49:27 ERROR: online migrate failure - command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=srv01' root@192.168.1.1 qm start 211 --skiplock --migratedfrom srv05 --migration_type secure --stateuri unix --machine pc-i440fx-2.11' failed: exit code 255
2019-12-08 11:49:27 aborting phase 2 - cleanup resources
2019-12-08 11:49:27 migrate_cancel
2019-12-08 11:49:28 ERROR: migration finished with problems (duration 00:00:06)
TASK ERROR: migration problems

The same happens if I add a new rbd storage (for the same cluster, just with the krbd switch turned on) and try to move the disk from rbd to krbd storage:

can't map rbd volume vm-211-disk-1: rbd: sysfs write failed (500)

Any and all operations work if krbd is not involved.

Are there any prerequisites to using krbd I need to be aware of?

 

[Waschbüsch]

Update:
The same is true when adding a new HDD to an existing VM or when creating an altogether new VM.
So, this is not limited to migration.

If (as the error message suggests it might be) this is to do with wrong permissions, where would I start looking?

ceph auth is set to optional:

     auth_client_required = none
     auth_cluster_required = none
     auth_service_required = none
     auth_supported = cephx

And I don't know what other permissions might be involved here?
Specifically: Which ceph and/ or system user is used and what permissions are needed?

 

[Waschbüsch]

OK. Resetting ceph.conf to the default auth stuff:

     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx

solved the problem, but I would still be very glad if someone could point out why that is?
AFAIK ceph auth does incur a (small) performance hit and seeing as this is a private, separate network, I really don't need the added security.

 

[Alwin]

Is there a keyring with the storage name under /etc/pve/priv/ceph?

 

[Waschbüsch]

Is there a keyring with the storage name under /etc/pve/priv/ceph?

Yes, there is.

 

[Alwin]

When you disable cephx, this keyring needs to be deleted. Our tooling will use it if it find one.

 

[Waschbüsch]

This results in new / different error. :-(

I tried the following:

- Setup a new storage for ceph rbd using the krbd flag named 'kdisks'
- removed

/etc/pve/priv/ceph/kdisks.keyring

- Tried to add a new disk to a VM using the new storage

/etc/pve/ceph.conf auth stuff reads like this:

     auth_client_required = none
     auth_cluster_required = none
     auth_service_required = none
     auth_supported = cephx

This resulted in the following error:

update VM 199: -scsi4 kdisks:32
TASK ERROR: error with cfs lock 'storage-kdisks': rbd error: rbd: listing images failed: (95) Operation not supported

 

[Waschbüsch]

For my part, I will now stick with the default config / enabled auth. Thank you for your help!

 

[Alwin]

auth_supported = cephx

If authentication is not needed, then remove this key. By default it is not set.

For my part, I will now stick with the default config / enabled auth. Thank you for your help!

You're welcome.

 

[roli8200]

Had the same problem with 6.2.
removed /etc/pve/priv/ceph/<pool-name>.keyring
Works now.