[SOLVED] [BUG] Proxmox's qemu and Debian 8 virtio drivers (as VM guest)

[RRJ]

Hi,

This thread is a result of my previous thread here http://forum.proxmox.com/threads/22142-debian-jessie-kvm-installation.
I crated new thread to get some attention, as there was not much activity on the previous one (devs attention, I mean of course )

So, I've been debugging this problem with GlusterFS devs and ended up with such result:

The problem is between qemu used in Proxmox and virtio drivers that come with D8 (atm both stable and testing versions are checked)
As soon as I choose not a virtio controller for VM with D8 (sata or scsi), installation goes well and I've got a totally usable system. If i choose virtio, I've got problems described in previous thread.

All other OSes (including D7, CentOS 7, Ubunti 14.04 LTS) do not have such problems with virtio drivers.
At the same time D8 runs perfectly on oVirt systems with virtio drivers.

So as for me it seems, like Proxmox should change qemu version...

Any ideas?

 

[tom]

I am not sure it the bug is in qemu, but test with 4.0beta (currently with qemu 2.3, quite soon with 2.4) - any difference?

 

[RRJ]

Hello Tom,

Thanks for your reply.
Do you mean Proxmox 4.0beta? I will try as soon, as I get some hardware.

 

[tom]

yes, see http://forum.proxmox.com/threads/22532-Proxmox-VE-4-0-beta1-released!

 

[RRJ]

meanwhile is there any chance to get updated kvm-qemu and qemu-server with current release? At least from pve-test repository? I've got one node I can restart pretty often, so can check it there (the most important, that node could boot with new changes )

 

[tom]

AFAIK there are no plans to update 3.x to a newer qemu version soon.

And we still have doubts that your issue is on the qemu side, as its ONLY seen on gluster. all other storages works, so it looks the problem is not on the qemu side.

 

[jampy]

I was able to reproduce the same problem.

Using native GlusterFS storage in Proxmox causes all kinds of disk corruption, whereas Gluster mounted via NFS works like a charm.

Test setup is a 3-node Proxmox 3.4 cluster.
Three physical machines: each is both a Proxmox node *and* a Gluster replica
The Gluster volume is replicated on all machines.

Network shouldn't matter as accessing Gluster via 127.0.0.1 produces similar results as when going across a 1Gbit link to one of the other nodes.

The same machines run a number of Debian 8.1 VMs via Gluster/NFS in a identical Gluster brick setup without any problems.


# pveversion -v
proxmox-ve-2.6.32: 3.4-157 (running kernel: 2.6.32-39-pve)
pve-manager: 3.4-6 (running version: 3.4-6/102d4547)
pve-kernel-2.6.32-39-pve: 2.6.32-157
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-18
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: not correctly installed
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: not correctly installed
glusterfs-client: 3.5.2-1

 

[RRJ]

Just to sum thing up:

D8 fails to install on glusterfs storage that is used by Proxmox qemu for VM storage.
Another affected Distros: Only D8 (D7, Ubuntu 14.04 LTS, Centos7, Centos6 tested and working)
D8 works if glusterfs is mounted via NFS, but this way libgfapi is not used !!
D8 works on ovirt as reported without problems (ovirt, as proxmox uses libgfapi out of the box with glusterfs, but here I did't ask how glusterfs is mounted, may be over nfs.. who knows)
Bug is reproducible very easy.

May be someone with a bit more fresh mind could say, where we should to look for the problem? proxmox's qemu, D8's virtio or glusterfs's libgfapi ?

 

[tom]

Just to sum thing up:

D8 fails to install on glusterfs storage that is used by Proxmox qemu for VM storage.
Another affected Distros: Only D8 (D7, Ubuntu 14.04 LTS, Centos7, Centos6 tested and working)
D8 works if glusterfs is mounted via NFS, but this way libgfapi is not used !!
D8 works on ovirt as reported without problems (ovirt, as proxmox uses libgfapi out of the box with glusterfs, but here I did't ask how glusterfs is mounted, may be over nfs.. who knows)
Bug is reproducible very easy.

May be someone with a bit more fresh mind could say, where we should to look for the problem? proxmox's qemu, D8's virtio or glusterfs's libgfapi ?

i am very "fresh minded" - did you already tested with 4.0 - like I asked last time? (to check newer qemu version)

 

[spirit]

seem that qemu gluster block driver don't have been updated since 1year. Not sure upgrade will help.

Maybe it could be a bug in libgfapi ?

 

[RRJ]

Guys, its an epic win.
Yesterday had another advice from GlusterFS devs, the problem is solved after I added:

performance.write-behind: off
performance.read-ahead: off

to the volume config.

I'll populate recommended config for volumes and qemu as soon as I test them enough, so someone could add them to proxmox's wiki

 

[RobFantini]

great you solved issue!

could you tell me which specific file you added those two lines?

 

[RRJ]

these are gluster volume VOLNAME set OPTION. not a file.

 

[spirit]

Guys, its an epic win.
Yesterday had another advice from GlusterFS devs, the problem is solved after I added:

performance.write-behind: off
performance.read-ahead: off

to the volume config.

I'll populate recommended config for volumes and qemu as soon as I test them enough, so someone could add them to proxmox's wiki

Do you have more infos about the relation between the virtio driver and theses options ?

 

[RRJ]

not yet. I'm on vacation atm. After I'm done with it, I've got some tests to do requested by glusterfs devs. We've got to clear it out, which option of these two does not work right and then they can see if they could do something to translators or it's virtio or even Debian8 bug (as i mentioned before, none of other OSes acts the same way as d8).

 

[RRJ]

OK,
as tests showed, the problem is only when I set performance.write-behind: on There are no problems with read-ahead on. Gluster devs said they will try to find out where exactly mistake is and fix, if it is possible to. They need to track the IO pattern during the installation process of D8 which causes this issue. But it seems like this won't be possible, as during installation I didn't see any kernel errors.. Just ended up with screwed FS.

 

[spirit]

OK,
as tests showed, the problem is only when I set performance.write-behind: on There are no problems with read-ahead on. Gluster devs said they will try to find out where exactly mistake is and fix, if it is possible to. They need to track the IO pattern during the installation process of D8 which causes this issue. But it seems like this won't be possible, as during installation I didn't see any kernel errors.. Just ended up with screwed FS.

Thanks for the report !

(Any link to your discuss with gluster devs ?)