Hello all,
I have been around and around trying to google my way out of issues that I have been having with ProxmoxVE and GlusterFS. I get the same io-error and KVM lockups that many people here have talked about before, but no fix has yet worked for me. I have a 4 node HA configuration using distributed-replicated volumes. At random intervals, KVM instances on the nodes will lock (as in, no further writes to the volume). The only remedy is to stop the VM and then start it again. All nodes are using GlusterFS version 3.7.6.
Here's pveversion -v:
storage.conf
VM 105.conf from /etc/pve/qemu-server
df -h from node 2
/etc/fstab from node 2
Notice from fstab that the "spool" volume isn't mounted. That was from some other tinkering in the past.
I've also include a redirected "ip a" from all nodes, gluster volume info, and glusterfs logs from the first node.
You will notice the common error of
At one point I had seen topology change errors showing on one of my switch logs at around the same time that the above error would display on the nodes. I'm unable to get to my switches currently to grab the logs, though. People have talked about changing the ping-timeout, which is something that I haven't tried yet.
I have tried setting:
On the gluster volume where the qcow2 disks live.
You'll likely notice that I've taken quite the sloppy approach to addressing the issue but, admittedly, I'm not an expert.
I've yet to find a definitive approach to handling the delicate balance between ProxmoxVE and GlusterFS, and I would really appreciate any guidance to narrow down and pinpoint the culprit behind this issue.
Thanks in advance!
I have been around and around trying to google my way out of issues that I have been having with ProxmoxVE and GlusterFS. I get the same io-error and KVM lockups that many people here have talked about before, but no fix has yet worked for me. I have a 4 node HA configuration using distributed-replicated volumes. At random intervals, KVM instances on the nodes will lock (as in, no further writes to the volume). The only remedy is to stop the VM and then start it again. All nodes are using GlusterFS version 3.7.6.
Here's pveversion -v:
Code:
proxmox-ve: 4.1-26 (running kernel: 4.2.6-1-pve)
pve-manager: 4.0-64 (running version: 4.0-64/fc76ac6c)
pve-kernel-4.2.6-1-pve: 4.2.6-26
pve-kernel-3.19.8-1-pve: 3.19.8-3
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-40
pve-firmware: 1.1-7
libpve-common-perl: 4.0-41
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-37
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-17
pve-container: 1.0-32
pve-firewall: 2.0-14
pve-ha-manager: 1.0-14
ksm-control-daemon: 1.2-1
glusterfs-client: 3.7.6-1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve1
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessiestorage.conf
Code:
dir: local
path /var/lib/vz
content images,iso,rootdir,vztmpl
maxfiles 0
glusterfs: data
volume data
path /mnt/data
server ppat-pdnhvs001
maxfiles 1
server2 ppat-pdnhvs002
content images,iso,backup,rootdir,vztmpl
dir: security
path /security
nodes ppat-pdnhvs004
content images
maxfiles 1VM 105.conf from /etc/pve/qemu-server
Code:
bootdisk: virtio0
cores: 4
hotplug: disk,network,usb
memory: 6144
name: vpat-pdnpdc002
net0: virtio=AE:48:8B:87:83:AC,bridge=vmbr0
net1: virtio=E6:2B:FA:A7:1E:63,bridge=vmbr10
net2: virtio=36:78:8C:C9:C9:1A,bridge=vmbr20
net3: virtio=32:D9:12:A7:B4:17,bridge=vmbr100
numa: 0
ostype: l26
parent: happyNewYear
smbios1: uuid=f3108ae0-64f3-4066-abfd-337970daccff
sockets: 1
vga: qxl
virtio0: data:105/vm-105-disk-1.qcow2,cache=writeback,size=52Gdf -h from node 2
Code:
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 3.2G 9.1M 3.2G 1% /run
/dev/dm-0 95G 2.8G 87G 4% /
tmpfs 7.9G 63M 7.8G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/mapper/pve-data 197G 5.3G 182G 3% /var/lib/vz
/dev/mapper/pve-gluster 5.2T 739G 4.5T 14% /gluster
ppat-pdnhvs002:/flexshare 11T 1.4T 9.0T 13% /mnt/flexshare
ppat-pdnhvs002:/homes 11T 1.4T 9.0T 13% /mnt/homes
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 100K 0 100K 0% /run/lxcfs/controllers
/dev/fuse 30M 56K 30M 1% /etc/pve
ppat-pdnhvs001:data 11T 1.4T 9.0T 13% /mnt/data/etc/fstab from node 2
Code:
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext4 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/mapper/pve-gluster /gluster xfs defaults,allocsize=4096,inode64,logbsize=256K,logbufs=8,noatime 1 2
ppat-pdnhvs002:/data /mnt/data glusterfs defaults,_netdev,acl,log-level=WARNING,log-file=/var/log/glusterfs/glusterfs.log,backup-volfile-servers=ppat-pdnhvs001:ppat-pdnhvs003:ppat-pdnhvs004 1 2
ppat-pdnhvs002:/flexshare /mnt/flexshare glusterfs defaults,_netdev,acl,log-level=WARNING,log-file=/var/log/glusterfs/glusterfs.log,backup-volfile-servers=ppat-pdnhvs001:ppat-pdnhvs003:ppat-pdnhvs004 1 2
ppat-pdnhvs002:/homes /mnt/homes glusterfs defaults,_netdev,acl,log-level=WARNING,log-file=/var/log/glusterfs/glusterfs.log,backup-volfile-servers=ppat-pdnhvs001:ppat-pdnhvs003:ppat-pdnhvs004 1 2
ppat-pdnhvs002:/spool /mnt/spool glusterfs defaults,_netdev,acl,log-level=WARNING,log-file=/var/log/glusterfs/glusterfs.log,backup-volfile-servers=ppat-pdnhvs001:ppat-pdnhvs003:ppat-pdnhvs004 1 2Notice from fstab that the "spool" volume isn't mounted. That was from some other tinkering in the past.
I've also include a redirected "ip a" from all nodes, gluster volume info, and glusterfs logs from the first node.
You will notice the common error of
Code:
All subvolumes are down. Going offline until atleast one of them comes back up.At one point I had seen topology change errors showing on one of my switch logs at around the same time that the above error would display on the nodes. I'm unable to get to my switches currently to grab the logs, though. People have talked about changing the ping-timeout, which is something that I haven't tried yet.
I have tried setting:
Code:
gluster volume set data cluster.self-heal-daemon offYou'll likely notice that I've taken quite the sloppy approach to addressing the issue but, admittedly, I'm not an expert.
I've yet to find a definitive approach to handling the delicate balance between ProxmoxVE and GlusterFS, and I would really appreciate any guidance to narrow down and pinpoint the culprit behind this issue.
Thanks in advance!
