Hi everyone,
I have been using proxmox and gluster together for the last decade, I have a hyperconvergence setup based on them quite stable and fast. On my last cluster, though, I had a horrible issue a few weeks ago when upgrading proxmox with an apt upgrade that made VMs terribly slow, so I had had a bad time finding the culprit, as the upgrade, even all the upgrades were a minor version upgrade (Proxmox 7.1 to 7.2, GlusterFS from 9.3 to 9.6 and ZFS from 2.0.6 to 2.1.6), something made big damage to performance. After a lot of digging, and moving the VMs from gluster to ZFS and see performance was just a little better, I did found I had to disable zfs_prefect_disable, and performance came back! (I know, everybody suggests having it enabled, as I had, but if disabling it did the job, I can't do anything)
So, I thought it was a good time to upgrade gluster from 9.6 to 10.3, as I'd read it was performing from 15% to 300% faster depending on the case of use scenario, and as the 9.6 version gave me the creep, I did it, then moved all the VMs and my single LXC to it, and it seemed to work plain nice.
I need to say I don't use the gluster client to access glusterfs, but I have mounted gluster via fstab, add that directory as proxmox storage, just mount the same name path and make sure to check the "Shared" checkbox. I'm using that way since gluster 3.12 and don't remember if proxmox had gluster support from that day. I did the benchmarks when proxmox let to mount gluster via GUI through the gluster client, and performance was way better with this method.
So, after reinstalling everything, I placed all VMs and my LXC container back from ZFS to gluster, that was last Thursday. On Monday, the gluster mount point on my PVE02 node crashed, and all the VMs and the LXC container living in it, crashed with it. All of them had I/O errors and when I listed the directory it was shown with a "Transport endpoint is not connected" error:
I had to stop the VMs, stop the gluster daemon and unmount the directory, then start the daemon again, and mount back the directory, and start the VMs.
Fortunately, I had to recover from backup just one VM, the other VMs and the LXC just booted up fine.
I spoke with the gluster devs, and they ask for a core dump. I did not find any anywhere on the system, just a backtrace which wasn't very helpful for them, so I'd activated systemd-coredump to see if it would help in the case of another crash.
Today, exactly 4 days after the first crash, and 8 from the installation of gluster 10.3, the same issue happened again. Funny enough, systemd-coredump crashed so no coredumps were dump... (perfect systemd, clap, clap, clap)
This time, though, I noticed A LOT of I/O errors like this, just before the gluster mountpoint crash
As one of the gluster developers pointed, I need to know what this loop0 device were coming from, and try to disable if I could. After comparing the other proxmox nodes, there were no messages on dmesg about any loop device, and looking the messages from Monday, the same error messages were present just before the crash as well.
I noticed as well, the LXC today wasn't able to boot, and when I tried to boot this appeared on the dmesg log
And this on the proxmox log
Then I moved the disk from GlusterFS to ZFS and the LXC started just fine and the dmesg showed it was mounted properly
I won't move back the LXC container during some days, and I will migrate it to docker, but I was wondering if anyone can confirm this, if Proxmox devs are aware of this issue or can replicate it.
I have been using proxmox and gluster together for the last decade, I have a hyperconvergence setup based on them quite stable and fast. On my last cluster, though, I had a horrible issue a few weeks ago when upgrading proxmox with an apt upgrade that made VMs terribly slow, so I had had a bad time finding the culprit, as the upgrade, even all the upgrades were a minor version upgrade (Proxmox 7.1 to 7.2, GlusterFS from 9.3 to 9.6 and ZFS from 2.0.6 to 2.1.6), something made big damage to performance. After a lot of digging, and moving the VMs from gluster to ZFS and see performance was just a little better, I did found I had to disable zfs_prefect_disable, and performance came back! (I know, everybody suggests having it enabled, as I had, but if disabling it did the job, I can't do anything)
So, I thought it was a good time to upgrade gluster from 9.6 to 10.3, as I'd read it was performing from 15% to 300% faster depending on the case of use scenario, and as the 9.6 version gave me the creep, I did it, then moved all the VMs and my single LXC to it, and it seemed to work plain nice.
I need to say I don't use the gluster client to access glusterfs, but I have mounted gluster via fstab, add that directory as proxmox storage, just mount the same name path and make sure to check the "Shared" checkbox. I'm using that way since gluster 3.12 and don't remember if proxmox had gluster support from that day. I did the benchmarks when proxmox let to mount gluster via GUI through the gluster client, and performance was way better with this method.
So, after reinstalling everything, I placed all VMs and my LXC container back from ZFS to gluster, that was last Thursday. On Monday, the gluster mount point on my PVE02 node crashed, and all the VMs and the LXC container living in it, crashed with it. All of them had I/O errors and when I listed the directory it was shown with a "Transport endpoint is not connected" error:
d????????? ? ? ? ? ? vmdataI had to stop the VMs, stop the gluster daemon and unmount the directory, then start the daemon again, and mount back the directory, and start the VMs.
Fortunately, I had to recover from backup just one VM, the other VMs and the LXC just booted up fine.
I spoke with the gluster devs, and they ask for a core dump. I did not find any anywhere on the system, just a backtrace which wasn't very helpful for them, so I'd activated systemd-coredump to see if it would help in the case of another crash.
Today, exactly 4 days after the first crash, and 8 from the installation of gluster 10.3, the same issue happened again. Funny enough, systemd-coredump crashed so no coredumps were dump... (perfect systemd, clap, clap, clap)
Bash:
● systemd-coredump@0-3199871-0.service - Process Core Dump (PID 3199871/UID 0)
Loaded: loaded (/lib/systemd/system/systemd-coredump@.service; static)
Active: failed (Result: timeout) since Fri 2022-11-25 10:54:59 CET; 39min ago
TriggeredBy: ● systemd-coredump.socket
Docs: man:systemd-coredump(8)
Process: 3199873 ExecStart=/lib/systemd/systemd-coredump (code=killed, signal=TERM)
Main PID: 3199873 (code=killed, signal=TERM)
CPU: 15ms
Nov 25 10:49:59 pve02 systemd[1]: Started Process Core Dump (PID 3199871/UID 0).
Nov 25 10:54:59 pve02 systemd[1]: systemd-coredump@0-3199871-0.service: Service reached runtime time limit. Stopping.
Nov 25 10:54:59 pve02 systemd[1]: systemd-coredump@0-3199871-0.service: Failed with result 'timeout'.This time, though, I noticed A LOT of I/O errors like this, just before the gluster mountpoint crash
Bash:
[2022-11-25 10:52:12] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:12] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:12] EXT4-fs error (device loop0): kmmpd:179: comm kmmpd-loop0: Error writing to MMP block
[2022-11-25 10:52:12] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:12] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:12] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:18] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:18] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:18] loop: Write error at byte offset 4490452992, length 4096.
[2022-11-25 10:52:18] loop: Write error at byte offset 4490457088, length 4096.
[2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0, sector 8770416 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0, sector 8770424 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[2022-11-25 10:52:18] Aborting journal on device loop0-8.
[2022-11-25 10:52:18] loop: Write error at byte offset 4429185024, length 4096.
[2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0, sector 8650752 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0, sector 8650752 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[2022-11-25 10:52:18] Buffer I/O error on dev loop0, logical block 1081344, lost sync page write
[2022-11-25 10:52:18] JBD2: Error -5 detected when updating journal superblock for loop0-8.
[2022-11-25 10:52:23] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:23] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:23] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:28] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:28] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:28] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:33] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:33] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:33] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:38] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:38] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:38] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:43] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:43] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:43] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:48] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:48] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:48] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:53] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:53] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:53] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:52:59] loop: Write error at byte offset 37908480, length 4096.
[2022-11-25 10:52:59] blk_update_request: I/O error, dev loop0, sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[2022-11-25 10:52:59] Buffer I/O error on dev loop0, logical block 9255, lost sync page write
[2022-11-25 10:53:04] loop: Write error at byte offset 37908480, length 4096.As one of the gluster developers pointed, I need to know what this loop0 device were coming from, and try to disable if I could. After comparing the other proxmox nodes, there were no messages on dmesg about any loop device, and looking the messages from Monday, the same error messages were present just before the crash as well.
I noticed as well, the LXC today wasn't able to boot, and when I tried to boot this appeared on the dmesg log
Bash:
[2022-11-25 18:04:18] loop0: detected capacity change from 0 to 16777216
[2022-11-25 18:04:18] EXT4-fs (loop0): error loading journal
[2022-11-25 18:05:26] loop0: detected capacity change from 0 to 16777216
[2022-11-25 18:05:26] EXT4-fs (loop0): INFO: recovery required on readonly filesystem
[2022-11-25 18:05:26] EXT4-fs (loop0): write access unavailable, cannot proceed (try mounting with noload)And this on the proxmox log
Bash:
run_buffer: 321 Script exited with status 255
lxc_init: 847 Failed to run lxc.hook.pre-start for container "108"
__lxc_start: 2008 Failed to initialize container "108"
TASK ERROR: startup for container '108' failedThen I moved the disk from GlusterFS to ZFS and the LXC started just fine and the dmesg showed it was mounted properly
Bash:
[2022-11-25 18:24:06] loop0: detected capacity change from 0 to 16777216
[2022-11-25 18:24:06] EXT4-fs warning (device loop0): ext4_multi_mount_protect:326: MMP interval 42 higher than expected, please wait.
[2022-11-25 18:24:50] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: nonI won't move back the LXC container during some days, and I will migrate it to docker, but I was wondering if anyone can confirm this, if Proxmox devs are aware of this issue or can replicate it.