glusterfs 3.7.17-2 file descriptor errors

[ScottR]

Quick question for the glusterfs users out there. Recently we updated a dev cluster running 4.3 to the latest stable glusterfs. After the update it seems KVM using glusterfs is unable to perform any operations on the glusterfs cluster. Everything works fine on the fuse mount, but when KVM interacts directly, you end up with FD errors or migration errors/sig 11 exits.

I assume it's a problem with qemu 2.7 that was just released.

Could not read L1 table: Bad file descriptor

 

[tom]

pls also post your:

> pveversion -v

 

[ScottR]

Sure.

[root@dev-proxmox] /home/scott $ pveversion -v
proxmox-ve: 4.3-71 (running kernel: 4.4.21-1-pve)
pve-manager: 4.3-9 (running version: 4.3-9/f7c6f0cd)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-92
pve-firmware: 1.1-10
libpve-common-perl: 4.0-79
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-12
pve-qemu-kvm: 2.7.0-4
pve-container: 1.0-80
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.7.17-2
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80

[root@dev-proxmox] /home/scott $ dpkg --list | grep gluster
ii glusterfs-client 3.7.17-2 amd64 clustered file-system (client package)
ii glusterfs-common 3.7.17-2 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.7.17-2 amd64 clustered file-system (server package)

 

[Alessandro 123]

The latest gluster version is 3.8.x, please try that.

 

[ScottR]

The latest gluster version is 3.8.x, please try that.

Yes but stable is 3.7. I will try since it's a dev lab, but this wouldn't be a good option normally. In the past 3.8 was incompatible with the version of KVM that was used previously.

 

[ScottR]

Exactly the same result. Due to the dependencies, it's forcing an update to qemu 2.7 where the issue probably lies. Glusterfs works fine for the FUSE interface, just not the libgfapi.

kvm: -drive file=gluster://proxmox/dev_storage/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on: Could not read L1 table: Bad file descriptor.

 

[Alessandro 123]

Yes but stable is 3.7. I will try since it's a dev lab, but this wouldn't be a good option normally. In the past 3.8 was incompatible with the version of KVM that was used previously.

No, stable is 3.8

 

[ScottR]

No, stable is 3.8

Sorry you are correct, 3.8 is considered stable as well. That transition happened between the last review.

 

[ScottR]

I ran the experiment of pegging glusterfs at 3.7.15-1. As soon as you move the glusterfs-client package to anything greater (including 3.8), you start running into the locking issues. As long as you don't move this package, the following pve packages continue to work, including QEMU 2.7.0.

[scott@proxmox-dev-01] ~ $ pveversion -v
proxmox-ve: 4.3-71 (running kernel: 4.4.21-1-pve)
pve-manager: 4.3-9 (running version: 4.3-9/f7c6f0cd)
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-92
pve-firmware: 1.1-10
libpve-common-perl: 4.0-79
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-12
pve-qemu-kvm: 2.7.0-4
pve-container: 1.0-80
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.7.15-1
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
[scott@dev-proxmox-dev-01] ~ $ dpkg --list | grep glusterfs
ii glusterfs-client 3.7.15-1 amd64 clustered file-system (client package)
ii glusterfs-common 3.7.15-1 amd64 GlusterFS common libraries and translator modules

 

[proxtom]

Is someone aware of being something on the horizon to have proxmox and current Gluster-versions working again?
I ran into same issue(s) (on test servers, so nothing bad happened), but would like go with the 3.7 or 3.8 stable/latest as soon as possible without having to 'donwgrade' gluster.
Is this something proxmox can solve/mitigate or has this to be solved by qemu/gluster?

 

[ScottR]

I've done some more debugging here and it seems more likely to me that this is actually a proxmox issue. Today I've moved an external glusterfs server to the latest 3.8, while keeping the client libs at the 3.7.15-1 release. This works as expected. As soon as proxmox-ve, or glusterfs-client/common moves past this version it breaks. Part of why this may have been missed is that it appears **only with linked clones**. If you have a full clone of a VM, it starts and operates fine. As soon as you attempt to start a linked clone it's always the same error.

kvm: -drive file=gluster://localhost/vm_storage/images/102/vm-102-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on: Could not read L1 table: Input/output error

I'll sacrifice another support ticket to the proxmox gods for this one if nobody has any insight.

This is the last known working version that I can reliably clone linked images.

proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.3-1 (running version: 4.3-1/e7cdc165)
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-88
pve-firmware: 1.1-9
libpve-common-perl: 4.0-73
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-61
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.1-6
pve-container: 1.0-75
pve-firewall: 2.0-29
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.7.15-1
lxc-pve: 2.0.4-1
lxcfs: 2.0.3-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80

 

[ScottR]

To illustrate the same problem using the repo version of gluster with the latest 4.3, which should be supported/tested.

root@prox01:~# /usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=8880fe7e-a0ae-4a7e-8315-9db48e1d1b7a' -name linked -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/101.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 128 -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:b483b381a13f' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=gluster://localhost/vm_storage/images/101/vm-101-disk-1.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=6E:6F:4F:6A:6A:29,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'
kvm: -drive file=gluster://localhost/vm_storage/images/101/vm-101-disk-1.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=none,aio=native,detect-zeroes=on: Could not read L1 table: Input/output error

root@prox01:~# dpkg -l | grep glusterfs
ii glusterfs-client 3.5.2-2+deb8u2 amd64 clustered file-system (client package)
ii glusterfs-common 3.5.2-2+deb8u2 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.5.2-2+deb8u2 amd64 clustered file-system (server package)

root@prox01:~# pveversion -v
proxmox-ve: 4.3-71 (running kernel: 4.4.21-1-pve)
pve-manager: 4.3-10 (running version: 4.3-10/7230e60f)
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-47
qemu-server: 4.0-94
pve-firmware: 1.1-10
libpve-common-perl: 4.0-80
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-14
pve-qemu-kvm: 2.7.0-6
pve-container: 1.0-81
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80

 

[ScottR]

@proxtom - I was working on some repro images for support and found that proxmox released pve-qemu-kvm 2.7.0-8. After applying this latest update, linked clones started working again for me. I would suggest trying this while I work out the root cause with support. For now we're holding all systems at 4.2 or the last working 4.3-66.

 

[ScottR]

@proxtom, After working through this with support, we have a definite bug in the QEMU side of things. When on the latest 2.7 QEMU package, the images are created with a bad offset.

QEMU Defect: gluster partial reads refusal conflicts with qcow
https://bugs.launchpad.net/qemu/+bug/1644754

As a workaround, you can pin the QEMU version which allows you to upgrade to the latest proxmox.

cat > /etc/apt/preferences.d/pin-pve-qemu-kvm <<'_EOF'
Package: pve-qemu-kvm
Pin: version 2.6.2-2
Pin-Priority: 1001
_EOF

 

[dcsapak]

@proxtom, After working through this with support, we have a definite bug in the QEMU side of things. When on the latest 2.7 QEMU package, the images are created with a bad offset.

QEMU Defect: gluster partial reads refusal conflicts with qcow
https://bugs.launchpad.net/qemu/+bug/1644754

As a workaround, you can pin the QEMU version which allows you to upgrade to the latest proxmox.

cat > /etc/apt/preferences.d/pin-pve-qemu-kvm <<'_EOF'
Package: pve-qemu-kvm
Pin: version 2.6.2-2
Pin-Priority: 1001
_EOF

well this is only half right, the qemu bug exists, but on current versions of proxmox we workaround this by using the gluster backend instead of the file one (so the offset/filesize should be correct)
also the latest gluster version where most things *should* work is 3.8.7 ( i only tested a bit and could not see any glaring issues besides online full clone, see http://pve.proxmox.com/pipermail/pve-user/2016-December/167852.html ), but as i see it, gluster has some bugs from time to time and any upgrade should be well tested before updating your production machines

 

[ScottR]

Another take away, using libgfapi appears to be less common than just using CEPH. Our fallback option to this workaround was moving storage types away from GlusterFS entirely. As it stands, there's nothing in Proxmox as a product/build to stop you from running into this apart from running into this thread. In my mind, whoever is at fault, proxmox, gluster, or QEMU it's a risk if you are running on GlusterFS. The reality is you can't run on the latest packages without hitting the problem.

 

[ScottR]

Update for the folks who are watching. The latest proxmox seems to work with the latest 3.8 and 3.9 glustefs. I'm not sure what changed, no updates to the bug reference on proxmox/qemu, so it's a mystery how it broke and how it was fixed.