new krbd option on pve4 don`t work

[markusd]

Hi,
I tried to enable krbd in storage.cfg
But Vms didn`t start and I get this failures:

Oct 21 20:05:41 virt02 kernel: Key type ceph registered
Oct 21 20:05:41 virt02 kernel: libceph: loaded (mon/osd proto 15/24)
Oct 21 20:05:41 virt02 kernel: rbd: loaded (major 251)
Oct 21 20:05:41 virt02 kernel: libceph: client64567660 fsid 79feee35-196d-4f72-bf90-c34cd1d85cb5
Oct 21 20:05:41 virt02 kernel: libceph: mon2 192.168.0.5:6789 session established
Oct 21 20:05:41 virt02 kernel: libceph: read_partial_message ffff8813f733ea00 signature check failed
Oct 21 20:05:41 virt02 kernel: libceph: osd15 192.168.0.5:6812 bad crc

in storage.cfg I have:

rbd: rbd-images
monhost 192.168.0.15;192.168.0.16;192.168.0.5;192.168.0.20
pool rbd
username admin
content images
(at this moment without krbd)

I wonder, if this is related to my ceph.conf, where I've made some changes like:
auth client required = none
auth cluster required = none
auth service required = none
auth supported = cephx
cephx require signatures = false
cephx sign messages = false

Many thanks for your answers
Markus

 

[markusd]

Sorry,
is there really no one, who can give me a hint, where i have to look, to get krbd working for me?
Are there some modules, i have to load, or things in ceph.conf, i have to enable?
Many thanks.
Greetings
Markus

 

[markusd]

Re: new krbd option on pve4 don`t work- it works now, but....

Hi again...

I 've removed "auth supported = cephx" from ceph.conf.
After reboot i am able to start vms' with activated "krbd" in storage.cfg.
But now I can not add a new rbd-image to vm, backups don' t work and i can`t see the images I get always the error
"rbd error: rbd: couldn't connect to the cluster! (500)"
So i think i've lost something with authentification? Probably the keys.

My /etc/pve/ceph.conf:
auth client required = none
auth cluster required = none
auth service required = none
cephx require signatures = false
cephx sign messages = false
filestore xattr use omap = true
fsid = 79feee35-196d-4f72-bf90-c34cd1d85cb5
keyring = /etc/pve/priv/ceph.client.admin.keyring
#i've tested also with "keyring = /etc/pve/priv/$cluster.$name.keyring"
ms_dispatch_throttle_bytes = 0
ms_nocrc = true
osd journal size = 5120
osd pool default min size = 1

[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring

ls -l /etc/pve/priv/ceph/
rw------- 1 root www-data 63 May 16 2014 rbd-images.keyring
rw------- 1 root www-data 63 May 16 2014 rbd.keyring

ls -l /etc/pve/priv/
rw------- 1 root www-data 1679 Dec 10 2013 authkey.key
rw------ 1 root www-data 1971 Nov 1 21:32 authorized_keys
drwx------ 2 root www-data 0 May 16 2014 ceph
rw------- 1 root www-data 63 May 16 2014 ceph.client.admin.keyring
rw------- 1 root www-data 214 May 16 2014 ceph.mon.keyring
rw------- 1 root www-data 3536 Nov 1 21:32 known_hosts
drwx------ 2 root www-data 0 Dec 10 2013 lock
rw------- 1 root www-data 1679 Dec 10 2013 pve-root-ca.key
rw------- 1 root www-data 3 Jun 20 2014 pve-root-ca.srl

I would be glad, if you have an Idea for me!

Thanks Markus

 

[Q-wulf]

Re: new krbd option on pve4 don`t work- it works now, but....

just for comparison:


_________________________________________
[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
auth supported = cephx
cluster network = 10.0.0.0/16
filestore xattr use omap = true
fsid = 85db84d2-c915-4e6b-96c3-82343c38216e
keyring = /etc/pve/priv/$cluster.$name.keyring
osd journal size = 5120
osd pool default min size = 1
public network = 10.0.0.0/16
osd crush location hook = /home/datamile-crush-location-lookup.sh

[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.0]
host = De-Langen-A-202-Rw1-Rk1-Storagepod1
mon addr = 10.0.1.1:6789
mon osd allow primary affinity = true

_________________________________________

^^^That is the standard config of a ceph-server setup via Proxmox.
Red are the Parts that you setup manually during the ceph install/init process
Orange is setup by pveceph automatically.



Then if you set up your rdb pools as follows in Proxmox gui:

cat /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    maxfiles 0
    content images,iso,rootdir,vztmpl


rbd: rdb
    monhost 10.0.1.1
    username admin
    content images
    pool rbd
    krbd


rbd: hdd-r2
    monhost 10.0.1.1
    username admin
    content images
    pool HDD-Replication2
    krbd


rbd: hdd-r3
    monhost 10.0.1.1
    username admin
    content images
    pool HDD-Replication3
    krbd


rbd: ssd-r1
    monhost 10.0.1.1
    username admin
    content images
    pool SSD-Replication1
    krbd


rbd: ssd-r2
    monhost 10.0.1.1
    username admin
    content images
    pool SSD-Replication2
    krbd


rbd: ec_hdd-f1
    monhost 10.0.1.1
    username admin
    content images
    pool HDD-EC_Failure-1
    krbd


rbd: ec_hdd-f2
    monhost 10.0.1.1
    username admin
    content images
    pool HDD-EC_Failure-2
    krbd


rbd: ec_hdd-f3
    monhost 10.0.1.1
    username admin
    content images
    pool HDD-EC_Failure-3
    krbd


rbd: ec_cached_hdd-f1
    monhost 10.0.1.1
    username admin
    content images
    pool SSD-EC-Cache-HddFailure1
    krbd


rbd: ec_cached_hdd-f2
    monhost 10.0.1.1
    username admin
    content images
    pool SSD-EC-Cache-HddFailure2
    krbd


rbd: ec_cached_hdd-f3
    monhost 10.0.1.1
    username admin
    content images
    pool SSD-EC-Cache-HddFailure3
    krbd

then to get access to those pools, you'd have to add keys as follows:

mkdir /etc/pve/priv/ceph


[I]Select pool name after “rdb:” - e.g. SSD-R2 - that is your storageID
cp /etc/ceph/ceph.client.admin.keyring ceph/<storageID>.keyring[/I]


e.g. 
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/rdb.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/hdd-r2.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/hdd-r3.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ssd-r1.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ssd-r2.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ec_hdd-f1.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ec_hdd-f2.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ec_hdd-f3.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ec_cached_hdd-f1.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ec_cached_hdd-f2.keyring
cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/ec_cached_hdd-f3.keyring

and then it works.


just poking in the dark here, but your ceph.conf changes seem to have broken something on your server.
Try with the standard process - see if it works - then go back one step at a time to your "config".

 

[markusd]

Re: new krbd option on pve4 don`t work- it works now, but....

Thank you for your reply,
I will look after my cepx - related changes, i made last, when i 'am able to stop the vm's...
(The ceph-wiki says, "auth support=cephx / none" is not needed anymore, but maybe Proxmox / qemu does)
I will report.

Markus

 

[markusd]

Re: new krbd option on pve4 don`t work- it works now, but....

Hi,
I have tested...

For the Proxmox-Gui / backup the following seems to be necessary:
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
but then vms doesnt't start when krbd is enabled.
After removing "cephx sign messages = false"
it works like expected
"auth supported = cephx" isn't needed with newer ceph - versions

My initial goal to made all this changes was to reduce cpu usage, because we use old server
Now I wonder, if this is a reasonable compromise in every sense...
But it works, that's good

Thanks for looking

Markus