error 500 can't upload to storage type 'rbd'

[MogiePete]

I have 3 hosts configured to use a Ceph Shared Storage configured by following the Ceph Server guide on the Wiki.

I am able to see the storage on all three hosts but when I go to upload an ISO I get the error: error 500 can't upload to storage type 'rbd'

This is a test lab at work that we are using to learn more about KVM and Proxmox as a potential long term replacement for VMware and VSAN.

Everything works well except for the shared Ceph Storage. Any Help is greatly appreciated.

Here are my settings:

Proxmox: 4.0-48

storage.conf


rbd: shared
monhost 192.10.10.1 192.10.10.2 192.10.10.3
nodes proxmox-3,proxmox-2,proxmox-1
content images
username admin
pool rbd

ceph.conf



cat ceph.conf
[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
auth supported = cephx
cluster network = 192.10.10.0/24
filestore xattr use omap = true
fsid = dccc3deb-5cc9-444c-98cc-771adba51ea9
keyring = /etc/pve/priv/$cluster.$name.keyring
osd journal size = 5120
osd pool default min size = 1
public network = 192.10.10.0/24


[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring


[mon.1]
host = proxmox-2
mon addr = 192.10.10.2:6789


[mon.2]
host = proxmox-3
mon addr = 192.10.10.3:6789


[mon.0]
host = proxmox-1
mon addr = 192.10.10.1:6789



Health is listed as OK

Quorum Yes {0 1 2}

osdmap 9 osds 9 up, 9 in

pgmap v247: 512 pgs active+clean, 0 data 324MB used, 2.15 TB avail

 

[mensinck]

Hi MogiePete,

I am able to see the storage on all three hosts but when I go to upload an ISO I get the error: error 500 can't upload to storage type 'rbd'

This is expected by design..

rbd: shared
monhost 192.10.10.1 192.10.10.2 192.10.10.3
nodes proxmox-3,proxmox-2,proxmox-1
content images
username admin
pool rbd

As you can see here: ,,centent images", you can upload disk raw images to ceph storage. .iso are also not allowed as LCX containers.

You should could use NFS, GlusterFS or iscsi for shared .iso storage.

You should see the storage as

Enabled              Yes
Active                 Yes
Content              Disk Image
Type                  RBD
Shared               Yes
Size                   xxxx
Used                  yyyy
Avail                  zzzzz

on the storage tab inside a Node.

Try uploading a raw disk image to test availability.

Did You copy the ,,ceph.lient.admin.keyring" to /etc/pve/priv/ceph/my-ceph-pool.keyring ?

Regards Lukas

 

[MogiePete]

Lukas,

Thank you for the prompt reply. This makes sense to me now. I was looking at 'images' as ISO images. I added an NFS mount point and placed the ISOs and LXC Templates.

Everything else is working by design now that I understand the contents.

Thanks again.