Proxmox 3.4 Cluster with GlusterFS breaks VM images on node reboot

Roger Lehmann

Member
Apr 2, 2015
3
0
21
Hello, I'm having a serious problem with my Proxmox 3.4 Cluster.
It's using GlusterFS as storage for VM images and such.
Unfortunately, when I restart every node in the cluster sequentially one by one (with online migration of the running HA VM first of course) the qemu image of the HA VM gets corrupted and the VM itself has problems accessing it.

Code:
May 15 10:35:09 blog kernel: [339003.942602] end_request: I/O error, dev vda, sector 2048
May 15 10:35:09 blog kernel: [339003.942829] Buffer I/O error on device vda1, logical block 0
May 15 10:35:09 blog kernel: [339003.942929] lost page write due to I/O error on vda1
May 15 10:35:09 blog kernel: [339003.942952] end_request: I/O error, dev vda, sector 2072
May 15 10:35:09 blog kernel: [339003.943049] Buffer I/O error on device vda1, logical block 3
May 15 10:35:09 blog kernel: [339003.943146] lost page write due to I/O error on vda1
May 15 10:35:09 blog kernel: [339003.943153] end_request: I/O error, dev vda, sector 4196712
May 15 10:35:09 blog kernel: [339003.943251] Buffer I/O error on device vda1, logical block 524333
May 15 10:35:09 blog kernel: [339003.943350] lost page write due to I/O error on vda1
May 15 10:35:09 blog kernel: [339003.943363] end_request: I/O error, dev vda, sector 4197184

After the image is broken, it's impossible to migrate the VM or start it when it's down.

Code:
root@pve2 ~ # gluster volume heal pve-vol info
Gathering list of entries to be healed on volume pve-vol has been successful 

Brick pve1:/var/lib/glusterd/brick
Number of entries: 1
/images//200/vm-200-disk-1.qcow2

Brick pve2:/var/lib/glusterd/brick
Number of entries: 1
/images/200/vm-200-disk-1.qcow2

Brick pve3:/var/lib/glusterd/brick
Number of entries: 1
/images//200/vm-200-disk-1.qcow2

I'm using GlusterFS version 3.6.1. I know the Proxmox repo only has version 3.5.2 which has it's own problems.
Could anyone give a helping hand preventing this issue? Thanks!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!