Ceph Authentication Error

[axion.joey]

Hey Everyone,

I'm hoping you can help me out. I can't for the life of me get proxmox to connect to a remote ceph cluster. I've followed all of the steps several times. But I keep getting this error: rbd error: rbd: couldn't connect to the cluster! (500). I've read several posts where this was caused by not copying over the keyring, but I've done that several times. Here's the info from my config.

Contents of /etc/pve/priv
drwx------ 2 root www-data 0 Apr 5 15:16 .
drwxr-xr-x 2 root www-data 0 Dec 31 1969 ..
-rw------- 1 root www-data 1.7K Apr 5 15:16 authkey.key
-rw------- 1 root www-data 1.6K Apr 6 14:27 authorized_keys
-rw------- 1 root www-data 2.6K Apr 5 17:01 known_hosts
drwx------ 2 root www-data 0 Apr 6 16:04 lock
-rw------- 1 root www-data 1.7K Apr 5 15:16 pve-root-ca.key
-rw------- 1 root www-data 3 Apr 5 17:01 pve-root-ca.srl
-rw------- 1 root www-data 63 Apr 6 17:32 rbd.keyring

md5sum of rbd.keyring
9edf14669c52f0b9a15787969c848f64 rbd.keyring

md5sum of /etc/ceph/ceph.client.admin.keyring
9edf14669c52f0b9a15787969c848f64 /etc/ceph/ceph.client.admin.keyring

Contents of /etc/pve/storage.cfg
dir: local
path /var/lib/vz
maxfiles 0
content rootdir,vztmpl,iso,images

rbd: rbd
monhost 10.3.16.20; 10.3.16.21; 10.3.16.22
pool rbd
content rootdir,images
krbd
username admin

Error message I see in /var/log/ceph-mon-hostname.log
2016-04-06 17:58:35.966697 7fdb831c4700 1 mon.proxmoxnj3@2(peon).auth v17 client did not provide supported auth type

Thanks in advance for your help.

 

[axion.joey]

Some more info:

pveversion -v
proxmox-ve: 4.1-26 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-1 (running version: 4.1-1/2f9650d4)
pve-kernel-4.2.6-1-pve: 4.2.6-26
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-41
pve-firmware: 1.1-7
libpve-common-perl: 4.0-41
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-17
pve-container: 1.0-32
pve-firewall: 2.0-14
pve-ha-manager: 1.0-14
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve1
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie

Ceph Version
ceph -v
ceph version 9.2.1

Contents of ceph.conf
cluster 7bd3c1b9-6d82-4601-b550-ec12fe318494
health HEALTH_OK
monmap e1: 3 mons at {proxmoxnj1=10.3.16.20:6789/0,proxmoxnj2=10.3.16.21:6789/0,proxmoxnj3=10.3.16.22:6789/0}
election epoch 14, quorum 0,1,2 proxmoxnj1,proxmoxnj2,proxmoxnj3
osdmap e48: 6 osds: 6 up, 6 in
flags sortbitwise
pgmap v92: 512 pgs, 2 pools, 0 bytes data, 0 objects
215 MB used, 5332 GB / 5332 GB avail
512 active+clean
root@proxmoxnj1:/etc/pve/priv# ceph -v
ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
root@proxmoxnj1:/etc/pve/priv# cat /etc/ceph/ceph.conf
[global]
fsid = 7bd3c1b9-6d82-4601-b550-ec12fe318494
mon_initial_members = proxmoxnj1, proxmoxnj2, proxmoxnj3
mon_host = 10.3.16.20,10.3.16.21,10.3.16.22
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd pool default size = 2
public network = 10.3.16.0/24
cluster network = 10.3.18.0/24
osd pool default pg num = 256
osd pool default pgp num = 256

[mon.proxmoxnj1]
host = proxmoxnj1
mon addr = 10.3.16.20

[mon.proxmoxnj2]
host = proxmoxnj2
mon addr = 10.3.16.21

[mon.proxmoxnj3]
host = proxmoxnj3
mon addr = 10.3.16.22

[osd]
osd recovery max active = 1
osd max backfills = 1
filestore max sync interval = 30
filestore min sync interval = 29
filestore flusher = false
filestore queue max ops = 10000

[osd.0]
host = cephnj1
public addr = 10.3.16.10
cluster addr = 10.3.18.10

[osd.1]
host = cephnj1
public addr = 10.3.16.10
cluster addr = 10.3.18.10

[osd.3]
host = cephnj1
public addr = 10.3.16.10
cluster addr = 10.3.18.10

[osd.2]
host = cephnj2
public addr = 10.3.16.11
cluster addr = 10.3.18.11

[osd.4]
host = cephnj2
public addr = 10.3.16.11
cluster addr = 10.3.18.11

[osd.5]
host = cephnj2
public addr = 10.3.16.11
cluster addr = 10.3.18.11

[client]
rbd cache = true
rbd cache writethrough until flush = true

 

[axion.joey]

My fault. It was a stupid mistake. I put the shared keys in /etc/pve/priv instead of /etc/pve/priv/ceph

 

[D.P.]

I have the shared keys in /etc/pve/priv/ceph but the error occurs.
What's wrong? Any ideas?

 

[axion.joey]

Can you post the output of ls -alh /etc/pve/priv/cep?

Also if you looking the ceph-mon log it may give you some clues.

 

[D.P.]

I manually restarted ceph on all nodes and it works.
I can connect to ceph but I can't upload content to it and can't create VM on it.