Throwing this solution up in case someone else runs into the issue. Scroll down to the bottom for the key Take Aways and dev recommendations.
Task Output when migrating a disk:
Task Output when creating a VM:
The error output indicates there was an issue establishing the cluster lock on storage or file. But as we'll see later this was a red herring. Not know that, I proceeded to spend several hours trying to understand and fix this supposed locking issue. Part way into this troubleshooting effort, the gluster volume status command began to hang and report no information. It would stop after hitting the 120 second timeout for Gluster CLI. The CLI log file (/var/log/glusterfs/cli.log) was not helpful either.
When I ran the migrate/create disk command, the cluster lock would appear in /etc/pve/priv/lock/. I could also see the directory was created at /mnt/pve/Dsp-Rz2-300GB/images/109. When the task failed, the directory (109) would be removed and the cluster lock cleared. I then noticed that the task would run for exactly 60 seconds before failing. This was key in understanding that the problem wasn't necessarily related to locking. A Staff Member posted in another thread that commands run with a cluster lock are limited to 60 seconds, because they don't want to be locking files indefinitely. In my case, the underlying command was pvesm alloc, and it was clearly hitting that 60 second limit. Running that command with even an extremely small disk size would generate the same errors as before.
Since I could see the cluster lock being created and removed, I began to think this might not be a locking issue. To get more information, I dug around in the source code and found that for Gluster volumes, this command is actually invoking qemu-img create. So I built a test command and ran it.
This makes it seem like a networking issue. But if that's the case, why would gluster peer status and gluster volume info show connected/active? Not sure, I decided to run the command against the second node in the cluster, and that actually worked!
Thinking this may just be a glitch in the gluster matrix, I got approval to shutdown all the VMs and restart all the nodes. But after all the nodes came up, none of the VMs could start. The Gluster volume was completely inaccessible. Now the qemu-img create command failed on every node. I ran pvesm status again, and Quorate was yes. All the gluster commands showed the same as before the restart:
Checking the gluster logs didn't give me a smoking gun, but there were clearly some communication issues:
/var/log/glusterfs/glusterd.log
/var/log/glusterfs/bricks/pve1-300GB-Rz2-proxmox.log
The PVE devs may want to consider adjusting the logic on Storage.pm, so that output from qemu-img create can be displayed within the PVE task output window. Getting the error "Failed to connect" would be a lot more helpful than getting the error "-locked". Starting on Line 923, it seems the function that allocates the disk (alloc_image) is embedded as a parameter within the function that creates the cluster lock (cluster_lock_storage). Not sure why it was done this way (maybe to enforce the 60 second timeout), but it would be nice if the Gluster plugin's alloc_image function could return an error message back to the PVE task output window, in the case when it's unable to return a $volname.
Environment
- 3x PVE 7.0-11 nodes clustered together
- Every node has a ZFS pool with a GlusterFS brick on it
- Glusterd version 9.2
- Gluster is configured in a 3 node Dispersed volume
- gluster volume create Dsp-Rz2-300GB disperse 3 redundancy 1 10.x.x.1:/pve1-300GB-Rz2/proxmox 10.x.x.2:/pve2-300GB-Rz2/proxmox 10.x.x.3:/pve3-300GB-Rz2/proxmox
The Problem
I was testing out some performance tuning setting on another volume in the Gluster pool. That pool was deleted and settings were reset. But shortly afterwards I noticed I was unable to migrate a disk to the Gluster volume, or create a new VM on the Gluster volume.Task Output when migrating a disk:
Task Output when creating a VM:
Troubleshooting
I ran pvesm status and Quorate was yes. I also ran these gluster commands, and everything showed active and connected. No issues.The error output indicates there was an issue establishing the cluster lock on storage or file. But as we'll see later this was a red herring. Not know that, I proceeded to spend several hours trying to understand and fix this supposed locking issue. Part way into this troubleshooting effort, the gluster volume status command began to hang and report no information. It would stop after hitting the 120 second timeout for Gluster CLI. The CLI log file (/var/log/glusterfs/cli.log) was not helpful either.
When I ran the migrate/create disk command, the cluster lock would appear in /etc/pve/priv/lock/. I could also see the directory was created at /mnt/pve/Dsp-Rz2-300GB/images/109. When the task failed, the directory (109) would be removed and the cluster lock cleared. I then noticed that the task would run for exactly 60 seconds before failing. This was key in understanding that the problem wasn't necessarily related to locking. A Staff Member posted in another thread that commands run with a cluster lock are limited to 60 seconds, because they don't want to be locking files indefinitely. In my case, the underlying command was pvesm alloc, and it was clearly hitting that 60 second limit. Running that command with even an extremely small disk size would generate the same errors as before.
Since I could see the cluster lock being created and removed, I began to think this might not be a locking issue. To get more information, I dug around in the source code and found that for Gluster volumes, this command is actually invoking qemu-img create. So I built a test command and ran it.
This makes it seem like a networking issue. But if that's the case, why would gluster peer status and gluster volume info show connected/active? Not sure, I decided to run the command against the second node in the cluster, and that actually worked!
Thinking this may just be a glitch in the gluster matrix, I got approval to shutdown all the VMs and restart all the nodes. But after all the nodes came up, none of the VMs could start. The Gluster volume was completely inaccessible. Now the qemu-img create command failed on every node. I ran pvesm status again, and Quorate was yes. All the gluster commands showed the same as before the restart:
Checking the gluster logs didn't give me a smoking gun, but there were clearly some communication issues:
/var/log/glusterfs/glusterd.log
/var/log/glusterfs/cli.log had nothing good
/var/log/glusterfs/bricks/pve1-300GB-Rz2-proxmox.log
/var/log/glusterfs/glustershd.log had one of these every 3 seconds
/var/log/glusterfs/mnt-pve-Dsp-Rz2-300GB.log
The Solution
After racking my brain for a while, I remembered that about 4 days ago I had changed the SAN NIC's MTU from 1500 to 6400, to try and boost performance. Changing the MTU back to 1500 and restarting the NICs brought Gluster back online instantly. Everything started working again. Praise God!Take Aways
If PVE is reporting a "lock" error, and the task exits after 60 seconds (120 seconds if it's unresponsive) then the problem might not be due to a cluster locking issue. It may be that the command is simply exceeding the 60 second timeout for "cluster lock" commands.The PVE devs may want to consider adjusting the logic on Storage.pm, so that output from qemu-img create can be displayed within the PVE task output window. Getting the error "Failed to connect" would be a lot more helpful than getting the error "-locked". Starting on Line 923, it seems the function that allocates the disk (alloc_image) is embedded as a parameter within the function that creates the cluster lock (cluster_lock_storage). Not sure why it was done this way (maybe to enforce the 60 second timeout), but it would be nice if the Gluster plugin's alloc_image function could return an error message back to the PVE task output window, in the case when it's unable to return a $volname.
