GlusterFS storage cluster error when creating VM

[jan.straatman]

After configuring a GlusterFS storage with a arbiter and 2 storage nodes, I get the following error when creating a VM on our Proxmox 8.0.4 cluster:

Formatting 'gluster://89.47.9.17/clusterdata/images/101/vm-101-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=off compression_type=zlib size=34359738368 lazy_refcounts=off refcount_bits=16
[2023-10-16 23:12:51.553665 +0000] I [io-stats.c:3701:ios_sample_buf_size_configure] 0-clusterdata: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2023-10-16 23:12:51.661126 +0000] E [MSGID: 108006] [afr-common.c:6123:__afr_handle_child_down_event] 0-clusterdata-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2023-10-16 23:12:51.661215 +0000] E [MSGID: 108006] [afr-common.c:6123:__afr_handle_child_down_event] 0-clusterdata-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.
[2023-10-16 23:13:01.556623 +0000] I [io-stats.c:4033:fini] 0-clusterdata: io-stats translator unloaded
[2023-10-16 23:13:02.560883 +0000] I [io-stats.c:3701:ios_sample_buf_size_configure] 0-clusterdata: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2023-10-16 23:13:02.674999 +0000] E [MSGID: 108006] [afr-common.c:6123:__afr_handle_child_down_event] 0-clusterdata-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2023-10-16 23:13:02.675086 +0000] E [MSGID: 108006] [afr-common.c:6123:__afr_handle_child_down_event] 0-clusterdata-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.
[2023-10-16 23:13:12.563757 +0000] I [io-stats.c:4033:fini] 0-clusterdata: io-stats translator unloaded
virtio0: successfully created disk 'Clusterdata:101/vm-101-disk-0.qcow2,cache=writeback,iothread=1,size=32G'
TASK OK

When testing de storagecluster in SSH everythings seems fine, even when I directly mount the storage to a host it works without the errors.

Does anybody know what is causing this error and how I can resolve this?

 

[jan.straatman]

Update info:

The vm/disk is created fine and it is working, no perfomance issues or anything. I did notice that this error is quite old but still not fixed as of today? Just as a test we created a directory on all nodes, mounted them manually with mount -t glusterfs etc.... and then added a storage as a shared directory. When I use that to create a vm/disk we got no delay and no errors. It even seems a tiny bit faster when testing.

Can anyone explain to me what the difference is between attaching the glusterfs storage directly through proxmox vs manually with a mount and then adding a shared directory?

I would like to know what the difference is and maybe someone could tell me if this method will cause problems with Proxmox.

 

[Moayad]

Hello,

The vm/disk is created fine and it is working, no perfomance issues or anything. I did notice that this error is quite old but still not fixed as of today? Just as a test we created a directory on all nodes, mounted them manually with mount -t glusterfs etc.... and then added a storage as a shared directory. When I use that to create a vm/disk we got no delay and no errors. It even seems a tiny bit faster when testing.

Since you're not facing any performance issues and the VM/disk is created successfully, the error message might be more of a warning state that can occur under specific conditions. That being said, I would check the syslog/journalctl for any message around the time you tried to create the VM. Additionally, you can post the output of the following commands to get a better look:

gluster volume info clusterdata
gluster volume status
gluster volume heal clusterdata info

Can anyone explain to me what the difference is between attaching the glusterfs storage directly through proxmox vs manually with a mount and then adding a shared directory?

The Proxmox code for the GlusterFS plugin uses the `mount -t` command to mount the GlusterFS volume. Specifically, the command is constructed as follows in the `gluster cifs_mount` function [0]



[0] https://git.proxmox.com/?p=pve-stor...f56bf932c68e60a135c2;hb=refs/heads/master#l82

 

[jan.straatman]

So what is the difference? And does it have consequences if we use the manual method?

82 sub glusterfs_mount {
83 my ($server, $volume, $mountpoint) = @_;
84
85 my $source = "$server:$volume";
86
87 my $cmd = ['/bin/mount', '-t', 'glusterfs', $source, $mountpoint];
88
89 run_command($cmd, errmsg => "mount error");
90 }

Or

mount -t glusterfs manually on every host node

The first method gives errors and the second method works without errors...

 

[Dark26]

Sadly, you can't put container on the glusterfs storage if you mount with proxmox.