GlusterFS & Proxmox Future & QCOW2 Issues

F

firegs

New Member
Apr 8, 2013
28
0
1
Hey gents,

I just want to know what the outlook is on more/better GlusterFS support in updated versions of Proxmox.

I've run into a few usability issues that basically make GlusterFS support almost unusable in Proxmox. For example, the simple creation of a QCOW2 image makes Proxmox timeout, and unable to continue. The reason is that the entire disk image (say 500GB image) is being written/created on the GlusterFS volume - which can take a really long time - and Proxmox's timeout is set incredibly short.

Why are Qcow2 images created on GlusterFS volumes 0% sparse and HUGE to start, but VMDK images are incredibly thin? Why does QCOW2 image creation take seconds on NFS/iSCSI but HOURS on GlusterFS?

Same with snapshot backups - if i want to restore a snapshot that contains a QCOW2 image, the restore fails/times out for the same reasons.

I can work around these issues, but my god is it making life miserable. Essentially, I have to restore the backup to an NFS share, then Move Disk to GlusterFS and change it from QCOW2 to VMDK, or else it fails.

I'm afraid to put this into production without the knowledge that better GlusterFS support will be coming.

Sam
 
Last edited:
Re: GlusterFS & Proxmox Future & Issues

Just a followup:

Using this command:

qemu-img create -f qcow2 gluster://server.domain.com:24007/testvol/images/100/vm-100-disk-1.qcow2 50G

Crates a 193KB disk image. 100% sparse, 100% thin. What the heck is making Proxmox's GUI make large disk images? I think if that issue could be changed/fixed/updated, almost all of my issues would go away.
 
Last edited:
Re: GlusterFS & Proxmox Future & Issues

firegs said:
Just a followup:

Using this command:

qemu-img create -f qcow2 gluster://server.domain.com:24007/testvol/a.img 50G

Crates a 193KB disk image. 100% sparse, 100% thin. What the heck is making Proxmox's GUI make large disk images?
Click to expand...

Maybe this is because proxmox create qcow2 with -o preallocation=metadata. (Which is good for performance with local disk).

Maybe this is bad with glusterfs ?

I found a bugreport here :
https://bugzilla.redhat.com/show_bug.cgi?id=895830
 
Last edited:
Re: GlusterFS & Proxmox Future & Issues

spirit said:
Maybe this is because proxmox create qcow2 with -o preallocation=metadata. (Which is good for performance with local disk).

Maybe this is bad with glusterfs ?

I found a bugreport here :
https://bugzilla.redhat.com/show_bug.cgi?id=895830
Click to expand...

Yep, thats exactly it. This renders QCOW2 with Proxmox basically unusable when creating or making qcow2 disk images with the Proxmox GUI.
 
Re: GlusterFS & Proxmox Future & Issues

It works as expected over NFS:
root@esx1:/mnt/pve/omnios_nfs/images/129# ls -hl vm-129-disk-1.qcow2
-rw-rw-rw-+ 1 root root 4,5G Jan 7 19:17 vm-129-disk-1.qcow2
root@esx1:/mnt/pve/omnios_nfs/images/129# du -hl vm-129-disk-1.qcow2
841M vm-129-disk-1.qcow2
 
Re: GlusterFS & Proxmox Future & Issues

Is NFS being shared over GlusterFS?

This is only using the FUSE mount.
 
Re: GlusterFS & Proxmox Future & Issues

No. Just mentioned it because (preallocation=metadata. (Which is good for performance with local disk). preallocation=metadata seems to only be a problem with glusterfs so it should be kept for all other solutions.
 
Re: GlusterFS & Proxmox Future & Issues

FYI - restore the backup to an NFS share, then Move Disk to GlusterFS and change it from QCOW2 to VMDK causes windows VM's to fail upon boot.

Attempting to then use qemu-img convert to convert the vmdk to qcow2.

qemu-img convert -f vmdk vm-103-disk-1.vmdk -O qcow2 vm-103-disk-1.qcow2

Will report back if the VM boots.

EDIT: NOPE. VM wont boot either way. Windows 7 VM goes into System Recovery. So currently, there is NO WORK AROUND.

Unless you're creating VM's from scratch with VMDK or RAW images, there is NO way to migrate qcow2 VM's to GlusterFS.

All of this could be easily solved if "preallocation='metadata'" was not there for GlusterFS volumes. Someone, please help. I'm dead in the water.

EDIT2: Last try - Move Disk w/ RAW conversion. If this doesnt work, I'm dead.

RAW WORKS! Wooo!! At least theres that.
 
Last edited:
Re: GlusterFS & Proxmox Future & Issues

Hi,

vdmk should be use only for compatibility, if you import from vmware. The format is not super fast in qemu, and maybe it can have bugs depend of the version of vmdk.

So, use raw, this is the fastest (but don't have snasphot features)
 
P reallocation gives problems on nfs too, backups process the entire image including 0's, so they take much longer than thin provisioned.

Sent from my MT27i using Tapatalk
 
Re: GlusterFS & Proxmox Future & Issues

I have done some research, maybe it's slow because we create the file through the fuse mount point.

It's possible to create the volume using the gluster qapi directly

qemu-img create gluster://$GLUSTER_HOST/$GLUSTER_VOLUME/images 5G

I'll do more tests
 
Re: GlusterFS & Proxmox Future & Issues

spirit said:
I have done some research, maybe it's slow because we create the file through the fuse mount point.

It's possible to create the volume using the gluster qapi directly

qemu-img create gluster://$GLUSTER_HOST/$GLUSTER_VOLUME/images 5G

I'll do more tests
Click to expand...

Yes yes, this works perfectly fine for a new VM. I've done it that way, too.

Problems still exist for restoring a VM from a backup. I'm looking to move all of my current VMs off a QNAP NAS device to a more hearty GlusterFS solution - and the only way to do that is to restore them from backups. Theres absolutely no way I'm going to remake all of our servers.

There are clearly workarounds, but yikes.

I wonder if its possible to use the gluster qapi during backup restore?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back