Could not read qcow2 header: Operation not permitted (500) on GlusterFS volume

[acidrop]

Hello,

I have a standalone proxmox node with with several VMs on ZFS volume and it is working fine.

Now, on a separate two node test cluster, I created 2 ZFS datasets. These 2 datasets will be replicated through GlusterFS on both nodes (2).Each dataset will store a separate VM which will be continously snapshoted through ZFS for backup.

I have configured zfs datasets with the following options:

compression=off
xattr=sa
sync=disabled

GlusterFS volume is configured with default settings.

Below is the ZFS configuration:

zpool status
  pool: zfspool1
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zfspool1    ONLINE       0     0     0
      sdb       ONLINE       0     0     0
      sdc       ONLINE       0     0     0
      sde       ONLINE       0     0     0
      sdf       ONLINE       0     0     0

errors: No known data errors

zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zfspool1             26.4G  12.7G  11.1G  /zfspool1
zfspool1/images      15.3G  12.7G    37K  /zfspool1/images
zfspool1/images/101  10.9G  12.7G  10.9G  /zfspool1/images/101
zfspool1/images/102  4.35G  12.7G   424K  /zfspool1/images/102

Gluster config is as follows:

Status of volume: glusterzfs
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick gluster1:/zfspool1                49155    Y    3057
Brick gluster2:/zfspool1                49155    Y    2949
NFS Server on localhost                    2049    Y    3064
Self-heal Daemon on localhost                N/A    Y    3068
NFS Server on gluster2                    2049    Y    3259
Self-heal Daemon on gluster2                N/A    Y    3264
 
There are no active volume tasks

Volume Name: glusterzfs
Type: Replicate
Volume ID: 976e38ae-bd13-4adf-929a-b63bbb6ec248
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/zfspool1
Brick2: gluster2:/zfspool1
Options Reconfigured:
cluster.server-quorum-ratio: 55%

Mount points:

df -h
Filesystem                Size  Used Avail Use% Mounted on
udev                       10M     0   10M   0% /dev
tmpfs                     299M  468K  298M   1% /run
/dev/mapper/pve-root      2.5G  2.0G  427M  83% /
tmpfs                     5.0M     0  5.0M   0% /run/lock
zfspool1                   24G   12G   13G  47% /zfspool1
zfspool1/images            13G  128K   13G   1% /zfspool1/images
zfspool1/images/101        24G   11G   13G  47% /zfspool1/images/101
zfspool1/images/102        13G  512K   13G   1% /zfspool1/images/102
tmpfs                     597M   50M  548M   9% /run/shm
/dev/mapper/pve-data      4.5G  138M  4.3G   4% /var/lib/vz
/dev/sda1                 495M   65M  406M  14% /boot
/dev/fuse                  30M   24K   30M   1% /etc/pve
172.21.3.3:/storage/nfs2  727G   28G  700G   4% /mnt/pve/nfs1
gluster:glusterzfs         24G   12G   13G  47% /mnt/pve/glusterzfs

Everything looks ok.I have also configured Gluster on Proxmox -> Datacenter -> Storage.
But when I try to create a VM on this storage, I get the following error:

unable to create image: qemu-img: gluster://gluster/glusterzfs/images/102/vm-102-disk-4.qcow2: Could not read qcow2 header: Operation not permitted (500)

PVE-Version:

proxmox-ve-2.6.32: 3.2-126 (running kernel: 2.6.32-29-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-29-pve: 2.6.32-126
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

If I dd or create anything via cli there is no problem.I can see that these files are replicated on both nodes.
However if I try to create a qcow2 disk with qemu-img I get the same error.
I tried the same scenario without ZFS but with hdd -> ext4+glusterfs combination and there is no problem there.
Must be something in ZFS+GlusterFS combination.

Anyone with such experience?

thanks

 

[liska_]

I experienced the same today. It is caused by split-brain of this file on gluster storage. Just try gluster volume heal YOUR_STORE info split-brain and you will probably see it in the output. Then you can use splitmount from this address http://blog.gluster.org/2014/02/glusterfs-split-brain-recovery-made-easy/ . Delete old/wrong replica and you can continue.

 

[acidrop]

thank you for the feedback.
I ended using zvol+drbd for replication.Seems more stable than gluster.Each vm resides on separate zvol + drbd resource.If I need to rollback I bring down the specific drbd resource first,do zfs rollback zvol and finally bring up drbd resource.The only problem is that I have to destroy the drbd resource on the other side,create md and initiate full sync which takes some time depending on vm size.