we have a real cluster (3.4 supported version with kernel 3.10) and ubuntu 14.04 / ceph giant where this stuff works.
test "cluster" is single proxmox 3.4 (kernel 3.10) node connected to 3 node ceph cluster, all are vmware fusion virtual machines.
guests on them run fine with local storage, but dont run on ceph storage.
the proxmox node is using the proxmox no-subscription repository for its ceph client (ceph 0.80.0).
the test ceph cluster is running ubuntu 14.04 lts and ceph giant (0.87.2)
the test prox node can see the ceph cluster, its free space, make a new filesystem on it, even clone a file system onto it,
but doesnt seem to be able to mount it to kvm. "Could not open 'rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41': Operation not supported"
i copied the cephx key monhost: /etc/ceph/ceph.client.admin.keyring to /etc/pve/priv/ceph/rbd.keyring and tried both the web gui and pvesh
to add ceph storage. in both cases everything seemed fine as above, but same could not run kvm. running the vm from local storage works fine.
heres the full error
Use of uninitialized value in string ne at /usr/share/perl5/PVE/QemuServer.pm line 2907.
Use of uninitialized value in string eq at /usr/share/perl5/PVE/QemuServer.pm line 2909.
kvm: -drive file=rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41,192.168.113.42,192.168.113.43:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/rbd.keyring,if=none,id=drive-scsi1,aio=native,cache=none,detect-zeroes=on: could not open disk image rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41: Could not open 'rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41': Operation not supported
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=b2d545a6-3459-424b-8d93-7e5c722121d1' -name cephclone -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -m 128 -k en-us -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:11dc9937673f' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41,192.168.113.42,192.168.113.43:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/rbd.keyring,if=none,id=drive-scsi1,aio=native,cache=none,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -drive 'file=rbd:rbd/vm-101-disk-2:mon_host=192.168.113.41,192.168.113.42,192.168.113.43:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/rbd.keyring,if=none,id=drive-scsi0,aio=native,cache=none,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=101'' failed: exit code 1Could not open 'rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41': Operation not supported
test "cluster" is single proxmox 3.4 (kernel 3.10) node connected to 3 node ceph cluster, all are vmware fusion virtual machines.
guests on them run fine with local storage, but dont run on ceph storage.
the proxmox node is using the proxmox no-subscription repository for its ceph client (ceph 0.80.0).
the test ceph cluster is running ubuntu 14.04 lts and ceph giant (0.87.2)
the test prox node can see the ceph cluster, its free space, make a new filesystem on it, even clone a file system onto it,
but doesnt seem to be able to mount it to kvm. "Could not open 'rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41': Operation not supported"
i copied the cephx key monhost: /etc/ceph/ceph.client.admin.keyring to /etc/pve/priv/ceph/rbd.keyring and tried both the web gui and pvesh
to add ceph storage. in both cases everything seemed fine as above, but same could not run kvm. running the vm from local storage works fine.
heres the full error
Use of uninitialized value in string ne at /usr/share/perl5/PVE/QemuServer.pm line 2907.
Use of uninitialized value in string eq at /usr/share/perl5/PVE/QemuServer.pm line 2909.
kvm: -drive file=rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41,192.168.113.42,192.168.113.43:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/rbd.keyring,if=none,id=drive-scsi1,aio=native,cache=none,detect-zeroes=on: could not open disk image rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41: Could not open 'rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41': Operation not supported
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=b2d545a6-3459-424b-8d93-7e5c722121d1' -name cephclone -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -m 128 -k en-us -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:11dc9937673f' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41,192.168.113.42,192.168.113.43:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/rbd.keyring,if=none,id=drive-scsi1,aio=native,cache=none,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -drive 'file=rbd:rbd/vm-101-disk-2:mon_host=192.168.113.41,192.168.113.42,192.168.113.43:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/rbd.keyring,if=none,id=drive-scsi0,aio=native,cache=none,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=101'' failed: exit code 1Could not open 'rbd:rbd/vm-101-disk-1:mon_host=192.168.113.41': Operation not supported