Storage migration from Glusterfs

[FinnTux]

Hi,

I've been testing gluster as storage backend for my proxmox cluster. Everything looks good except vm images can't be moved from gluster to another storage.

Here is the error message:
create full clone of drive scsi0 (gl_ssd:10303/vm-10303-disk-1.qcow2)
transferred: 0 bytes remaining: 17179869184 bytes total: 17179869184 bytes progression: 0.00 %
qemu-img: block/gluster.c:1290: find_allocation: Assertion `offs >= start' failed.
TASK ERROR: storage migration failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f qcow2 -O raw gluster://172.16.0.161/gl_ssd/images/10303/vm-10303-disk-1.qcow2 zeroinit:/dev/zvol/rpool/data/vm-10303-disk-1' failed: got signal 6

Google gave this Red Hat bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=1451191

Gluster version (installed from gluster repo):
glusterfs-client 3.12.3-1
glusterfs-server 3.12.3-1

pveversion (running older kernel because of high io load on zfs storage):
proxmox-ve: not correctly installed (running kernel: 4.10.17-5-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.10.17-5-pve: 4.10.17-25
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.12.3-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9
openvswitch-switch: 2.7.0-2

Any idea how to fix this?

 

[Dark26]

Do you choose qcow2 when you migrate the machine ?

 

[FinnTux]

Migrating to local zfs storage, so choosing target format is not option. Source disk is qcow2.

 

[FinnTux]

Updated to the latest kernel
proxmox-ve: 5.1-30 (running kernel: 4.13.8-3-pve)

Now vm dies after starting storage migration.

create full clone of drive scsi0 (gl_ssd:10303/vm-10303-disk-1.qcow2)
drive mirror is starting for drive-scsi0
drive-scsi0: Cancelling block job
TASK ERROR: storage migration failed: mirroring error: VM 10303 qmp command 'query-block-jobs' failed - client closed connection

And here is the error when migrating to nfs (vm dies also):
create full clone of drive scsi0 (gl_ssd:10303/vm-10303-disk-1.qcow2)
Formatting '/mnt/pve/nfs_pve1_vmpool1/images/10303/vm-10303-disk-1.qcow2', fmt=qcow2 size=17179869184 encryption=off cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
drive mirror is starting for drive-scsi0
drive-scsi0: Cancelling block job
TASK ERROR: storage migration failed: mirroring error: VM 10303 not running

 

[FinnTux]

I noticed pvetest had new pve-qemu-kvm package, tested it and looks like it fixes the error above.

Good job devs and I hope glusterfs would get even more love from you. IMO It looks very promising on smaller clusters.

 

[drdownload]

Hi, where have you seen that the last pve-qemu-kvm (even in pvetest) addresses this isseu?

 

[FinnTux]

Sorry I don't quite understand the question, but the problem I mentioned was fixed in later version of pve-qemu-kvm. I don't know/remember the exact version numbers anymore.