Ceph hotplug problem in PVE 3.4 (Including cause and solution)

[casparsmit]

Hi,

I was testing the hotplug function of PVE 3.4 and found out it is not working with Ceph RBD storage.

When hotplugging a Hard Disk using Ceph i get the following error:


Parameter verification failed. (400)

virtio1: hotplug problem - adding drive failed: drive_add: string expected


Adding a Hard Disk using Ceph while the same VM is turned off works ok.

Also hotplugging using local directory (ZFS using writeback cache!) or local directory (EXT4) storage works.


Reading the following maillinglist thread (although somewhat different error message):

http://pve.proxmox.com/pipermail/pve-user/2015-February/008304.html

I'm was suspecting the same sub in the same file to be the problem: qemu_driveadd in /usr/share/perl5/PVE/QemuServer.pm
My guess was that while using Ceph the $drive variable is not set correctly using hotplug. (empty or malformed).

I reverted the patch (commit) and now it works! So while the commit fixes handling of spaces it breaks something else.

Kind regards,
Caspar

 

[tom]

works for me.

post your 'pveversion -v' and your VM config 'qm config VMID'

 

[casparsmit]

# pveversion -v
proxmox-ve-2.6.32: 3.3-147 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
pve-kernel-2.6.32-37-pve: 2.6.32-147
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.3-20
pve-firmware: 1.1-3
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-31
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-12
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

# qm config 103
bootdisk: virtio0
cores: 4
ide0: none,media=cdrom
ide2: none,media=cdrom
memory: 4096
name: test03
net0: virtio=DA:2CD:4A:61:07,bridge=vmbr0
ostype: win7
smbios1: uuid=8c686c7a-fe86-408b-abbd-7a9898503451
sockets: 1
vga: qxl
virtio0: my-ceph-pool:vm-103-disk-1,cache=writeback,size=100G

 

[tom]

# pveversion -v
...

looks all ok, I use the same and have no problems. are you sure that you hotplug a virtio disk?

 

[casparsmit]

looks all ok, I use the same and have no problems. are you sure that you hotplug a virtio disk?

Yes, absolutely sure!

With commit https://git.proxmox.com/?p=qemu-server.git;a=commitdiff;h=8ead5ec7dc342e991f2e8cef3e6b6afcba549250 it gives the error message.
After reverting the commit it works.

 

[casparsmit]

Tom,

I tried with a fresh PVE3.3 install and hotplug works fine.
I tried with a fresh PVE3.4 install and hotplug doesn't work.

What version of Ceph are you using? I'm using latest Firefly (0.80.8)

 

[spirit]

It's a regression bug.
We are currently discuss about it on pve-devel mailing.