Snapshot breaks docker containers on debian VM?

T

thejp

Member
Dec 12, 2022
2
1
8
Hi,

I've searched around but couldn't quite find a solution.

So I have a node with two VMs only - Unraid and a Debian with docker and about 25 containers running.

I have noticed with trial and error that my Debian VM will running smoothly if I don't back it up, but when I place it in the backup job, that's configured with snapshot mode, a good portion of the containers will face issues and eventually stop working. Perfect example is Plex, which actually breaks and I then consistently get "Database corrupt".
In fact, one time this even led to having to run fsck to fix filesystem errors within the Debian VM.

Curiously enough, two other VMs in other machines also running docker, but over Ubuntu VMs and don't seem to have any issue (yet those have maybe 4 or 5 containers each).

So the question is, any ideias of other things I should look at/try? Or just accept the fact that I need to run a backup with stop mode just to make sure I have no issues?

Extra info: qemu agent running (regular shutdown and IP showing); backup is done via PBS; backup with snapshot does work, it is the underlying docker which causes issues, maybe because of the freeze operation.
 
Hey there!
Now that's curious - the next time it happens in the affected VM, can you look through your syslog, kernel log and systemd journal? I'm curious if there's anything in there regarding Docker.

Also, which filesystem is that VM using, and what's backing the filesystem of that VM? (For example, does it run ext4 and is backed by ZFS?)

Is the portion of containers that break completely random or are there any patterns (apart from the Plex container)?

thejp said:
Or just accept the fact that I need to run a backup with stop mode just to make sure I have no issues?
Click to expand...
This might just be the safest option if we can't identify what the exact problem is here. You could also try splitting that VM up into multiple smaller ones, since that seems to work for the other VMs. You could also test whether that problem really only exists on that Debian host - maybe you can replicate that setup on an Ubuntu VM (should be easy enough if you're using Docker Compose, right?) and see if the problem still persists.
 
Hi,

Both the VM and underlying Proxmox storage are plain old ext4.
Seems to me that some containers break more often that other - Plex is a very constant one... I first noticed when I was still messing around with the VM (I'm only about two months old using Proxmox... used to run everything without a hypervisor) and had a daily snapshot of my VMs at 1am, and would get the email with backup success, then at around 2am everyday I would get a message from Plex saying the database is corrupted. I would then fix the db, everything would work througout the day (since I had to restart Plex itself) but the other containers would need to be restarted when I found something not working... then at 1am that next day, rinse and repeat - this last week I took this VM out of the backup job and it's been very steady ever since.

Honestly, running it in stop mode is not going to be any sort of deal breaker, but for the sake of science I'm going to try a couple of other things, including switching the docker-compose over to a fresh ubuntu server VM.
Thanks :)
 
  • Like
Reactions: Max Carrara
Alright! Let me know how it goes. This is a rather interesting issue, so I'm curious what you'll find.
 
Hi, i have that same behavior on my kubernetes cluster and OpenEBS cSTOR storage engine & Longhorn storage engine. But in this case i disabled the backup on these VM (AlmaLinux 9) and when others VMs get backup done, then some PVC get corrupt and then must use fsck on that volumes. On Proxmox I'm using backup with snapshot over ZFS backend and saving on PBS. Is really strange behavior. Dmesg:

Code: 
[82698.901351] INFO: task OrientDB WAL Fl:296254 blocked for more than 122 seconds.
[82698.901985]       Not tainted 5.14.0-162.22.2.el9_1.x86_64 #1
[82698.902618] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[82698.903228] task:OrientDB WAL Fl state:D stack:    0 pid:296254 ppid:295441 flags:0x00004000
[82698.903790] Call Trace:
[82698.904367]  __schedule+0x248/0x590
[82698.904953]  ? out_of_line_wait_on_bit_lock+0xb0/0xb0
[82698.905510]  schedule+0x5a/0xc0
[82698.906118]  io_schedule+0x42/0x70
[82698.906658]  bit_wait_io+0xd/0x60
[82698.907212]  __wait_on_bit_lock+0x5b/0xb0
[82698.907739]  out_of_line_wait_on_bit_lock+0x92/0xb0
[82698.908274]  ? __ia32_sys_membarrier+0x20/0x20
[82698.908806]  ext4_update_super+0x3c0/0x410 [ext4]
[82698.909444]  ? wake_up_q+0x4a/0x90
[82698.910004]  ext4_commit_super+0x46/0x100 [ext4]
[82698.910559]  ext4_handle_error+0x1d9/0x1f0 [ext4]
[82698.911128]  __ext4_error+0x9c/0x120 [ext4]
[82698.911674]  ext4_journal_check_start+0x89/0xa0 [ext4]
[82698.912236]  __ext4_journal_start_sb+0x31/0x120 [ext4]
[82698.912771]  ext4_dirty_inode+0x35/0x80 [ext4]
[82698.913335]  __mark_inode_dirty+0x123/0x350
[82698.913883]  generic_update_time+0x6c/0xd0
[82698.914431]  file_update_time+0x127/0x140
[82698.915029]  ? generic_write_checks+0x61/0xc0
[82698.915550]  ext4_buffered_write_iter+0x50/0x110 [ext4]
[82698.916115]  new_sync_write+0x11c/0x1b0
[82698.916635]  vfs_write+0x1ef/0x280
[82698.917180]  __x64_sys_pwrite64+0x90/0xc0
[82698.917726]  do_syscall_64+0x59/0x90
[82698.918326]  ? __irq_exit_rcu+0x46/0xe0
[82698.918881]  ? sysvec_apic_timer_interrupt+0x3c/0x90
[82698.919442]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[82698.919989] RIP: 0033:0x7f6b85082487
[82698.920511] RSP: 002b:00007f6aebafb160 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
[82698.921069] RAX: ffffffffffffffda RBX: 00000000000003aa RCX: 00007f6b85082487
[82698.921608] RDX: 0000000000010000 RSI: 00007f6b4c001820 RDI: 00000000000003aa
[82698.922239] RBP: 00007f6b4c001820 R08: 0000000000000000 R09: 0000000757d46fc8
[82698.922836] R10: 0000000000030000 R11: 0000000000000293 R12: 0000000000010000
[82698.923425] R13: 0000000000030000 R14: 0000000000030000 R15: 00007f6b3c029800
[82724.777149] sd 7:0:0:0: [sde] tag#38 timing out command, waited 180s
[82724.780451] sd 7:0:0:0: [sde] tag#38 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=180s
[82724.781068] sd 7:0:0:0: [sde] tag#38 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[82724.781657] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82724.782329] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82724.782948] Buffer I/O error on dev sde, logical block 0, lost sync page write
[82724.783557] EXT4-fs (sde): I/O error while writing superblock
[82724.783584] EXT4-fs (sde): previous I/O error to superblock detected
[82724.783629] EXT4-fs (sde): previous I/O error to superblock detected
[82724.784217] EXT4-fs (sde): Remounting filesystem read-only
[82724.786191] EXT4-fs (sde): failed to convert unwritten extents to written extents -- potential data loss!  (inode 4458758, error -30)
[82724.786866] Buffer I/O error on device sde, logical block 17861959
[82724.787572] EXT4-fs (sde): failed to convert unwritten extents to written extents -- potential data loss!  (inode 4458753, error -30)
[82724.788275] Buffer I/O error on device sde, logical block 17861962
[82724.789007] Buffer I/O error on device sde, logical block 17861963
[82724.789988] EXT4-fs (sde): failed to convert unwritten extents to written extents -- potential data loss!  (inode 4456455, error -30)
[82724.790695] Buffer I/O error on device sde, logical block 17907658
[82724.791593] EXT4-fs (sde): failed to convert unwritten extents to written extents -- potential data loss!  (inode 4458762, error -30)
[82724.792327] Buffer I/O error on device sde, logical block 17866770
[82724.793087] Buffer I/O error on device sde, logical block 17866771
[82738.734358] sd 6:0:0:0: [sdd] tag#103 timing out command, waited 180s
[82738.735247] sd 6:0:0:0: [sdd] tag#103 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=180s
[82738.736021] sd 6:0:0:0: [sdd] tag#103 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[82738.736814] I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82738.737591] I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82738.738357] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[82738.739156] EXT4-fs (sdd): I/O error while writing superblock
[82738.739189] EXT4-fs (sdd): previous I/O error to superblock detected
[82738.739349] EXT4-fs (sdd): previous I/O error to superblock detected
[82738.739976] EXT4-fs (sdd): Remounting filesystem read-only
[82905.414351] sd 7:0:0:0: [sde] tag#89 timing out command, waited 180s
[82905.417982] sd 7:0:0:0: [sde] tag#89 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=180s
[82905.418951] sd 7:0:0:0: [sde] tag#89 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[82905.419975] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82905.420846] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82905.421626] Buffer I/O error on dev sde, logical block 0, lost sync page write
[82905.422517] EXT4-fs (sde): I/O error while writing superblock
[82919.386332] sd 6:0:0:0: [sdd] tag#42 timing out command, waited 180s
[82919.387239] sd 6:0:0:0: [sdd] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=180s
[82919.388113] sd 6:0:0:0: [sdd] tag#42 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[82919.388945] I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82919.389813] I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[82919.390676] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[82919.391558] EXT4-fs (sdd): I/O error while writing superblock
[83086.072000] sd 7:0:0:0: [sde] tag#28 timing out command, waited 180s
[83086.075441] sd 7:0:0:0: [sde] tag#28 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=180s
[83086.076401] sd 7:0:0:0: [sde] tag#28 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[83086.077263] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[83086.078164] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[83086.079019] Buffer I/O error on dev sde, logical block 0, lost sync page write
[83086.079896] EXT4-fs (sde): I/O error while writing superblock
[83086.079896] EXT4-fs (sde): I/O error while writing superblock
[83100.019596] sd 6:0:0:0: [sdd] tag#93 timing out command, waited 180s
[83100.020540] sd 6:0:0:0: [sdd] tag#93 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=180s
[83100.021509] sd 6:0:0:0: [sdd] tag#93 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[83100.022488] I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[83100.023392] I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
[83100.024183] Buffer I/O error on dev sdd, logical block 0, lost sync page write
[83100.025012] EXT4-fs (sdd): I/O error while writing superblock
[83100.025024] EXT4-fs (sdd): I/O error while writing superblock
[83100.025168] EXT4-fs (sdd): I/O error while writing superblock
[83100.025883] EXT4-fs (sdd): failed to convert unwritten extents to written extents -- potential data loss!  (inode 5505116, error -30)
[83100.028961] Buffer I/O error on device sdd, logical block 11573522
[84074.247686] EXT4-fs (sde): recovery complete
[84074.251125] EXT4-fs (sde): mounted filesystem with ordered data mode. Quota mode: none.
[84075.057172] sd 7:0:0:0: [sde] Synchronizing SCSI cache
[125216.888509] hrtimer: interrupt took 172921101 ns
[157327.585880] sd 6:0:0:0: [sdd] Synchronizing SCSI cache
 
My question is, why that VM fail on ext4 when the backup is not done on that VM, maybe the snapshot of ZFS is doing something to all the volumes?
 
In some research i think that i found the issue. The problem is (i think) with the Linked Clone and Base Disk image. These VMs are Linked Clone from a template, so these HDD have a base on hdd template, that are used on some others VM that have the snapshot backup enable. So i think that when other VM enter on backup state, the base HDD is freeze with snapshot and that have an impact on the Kubernetes VM. Can be?
 
I am observing the same thing on kubernetes (rke2) + longhorn as distributed storage. The weird thing is that the only container that always break is the elastic search instance. I have the backup starting at 1.30am and then I can see a spike in logs to 500k in just longhorn pods around 1h after the backups and then elastic search breaks at that point. Usually the log from longhorn complains that the volume is in ro state. Has anyone found a solution to this?
 
Same problems with Kubernetes (k3s) + longhorn on my side.
Proxmox stuck at "freeze guest" infinity and the corresponding VM becomes unresponsive.
 
Oof, blast from the past.

showiproute said:
Same problems with Kubernetes (k3s) + longhorn on my side.
Proxmox stuck at "freeze guest" infinity and the corresponding VM becomes unresponsive.
Click to expand...

Could you share some more details on your setup? How did you set up k3s and longhorn?
 
  • Like
Reactions: showiproute
Max Carrara said:
Oof, blast from the past.



Could you share some more details on your setup? How did you set up k3s and longhorn?
Click to expand...
Yes - thought I continue this thread instead of creating a new one.

I have four equal VMs which have following specs:

Code: 
agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=ide2;scsi0
cipassword: **********
ciuser: mathias
cores: 8
cpu: host
efidisk0: rpool:vm-128-disk-0,efitype=4m,format=raw,size=528K
ide2: rpool:vm-128-cloudinit,media=cdrom,size=4M
ipconfig0: ip=10.0.22.55/24,gw=10.0.22.254
machine: q35
memory: 16384
meta: creation-qemu=9.0.2,ctime=1728111009
name: k3s-worker-04
nameserver: 10.0.22.254
net0: virtio=BC:24:11:27:A0:8B,bridge=vlan22
numa: 0
onboot: 1
ostype: l26
scsi0: rpool:vm-128-disk-1,discard=on,format=raw,iothread=1,size=205312M,ssd=1
scsihw: virtio-scsi-single
searchdomain: hks.lan
smbios1: uuid=6df05d2d-8b2f-40e7-98f3-3a4bfe860f39
sockets: 1
sshkeys: ssh-rsa%20AAAAB3NzaC1yc2EAAAADAQABAAACAQC%2B and so on


The VM itself runs a current Ubuntu 24.04 OS with k3s installed according to SUSEs installation guide.
K3s is being fully managed via the Rancher UI (installed within the Kubernetes cluster) and uses Longhorn as the default filesystem for any PV.
Longhorn was installed and being updated via Rancher - I do not deal with any helm charts for that.

If I need to reboot PVE usually my worker nodes also take very long to shutdown (a couple of minutes).
I am not sure if this is being linked to this problem.
 
Hmm, interesting. I'm not sure what Longhorn does exactly under the hood, but anyhow, a couple questions that should hopefully aid in debugging:
  • How does Longhorn itself perform if you aren't running any backups or touching the VM otherwise?
  • Are there any logs that show up in the other VMs when any given VM is being frozen? (Syslogs, Kubernetes logs, etc.) Note that you can use the following commands on your host to issue commands to the guest agent to freeze / thaw and check the status:
    Bash: 
    qm guest cmd $VMID fsfreeze-freeze
qm guest cmd $VMID fsfreeze-thaw
qm guest cmd $VMID fsfreeze-status
  • What happens if you shut down Kubernetes + Longhorn in a given VM manually first and then freeze / thaw the VM? Does it still get stuck?
  • Is it possible for you to switch to a different CSI driver, e.g. NFS or something otherwise simple for the time being and see if the problem persists without Longhorn?
 
Max Carrara said:
Hmm, interesting. I'm not sure what Longhorn does exactly under the hood, but anyhow, a couple questions that should hopefully aid in debugging:
  • How does Longhorn itself perform if you aren't running any backups or touching the VM otherwise?
  • Are there any logs that show up in the other VMs when any given VM is being frozen? (Syslogs, Kubernetes logs, etc.) Note that you can use the following commands on your host to issue commands to the guest agent to freeze / thaw and check the status:
    Bash: 
    qm guest cmd $VMID fsfreeze-freeze
qm guest cmd $VMID fsfreeze-thaw
qm guest cmd $VMID fsfreeze-status
  • What happens if you shut down Kubernetes + Longhorn in a given VM manually first and then freeze / thaw the VM? Does it still get stuck?
  • Is it possible for you to switch to a different CSI driver, e.g. NFS or something otherwise simple for the time being and see if the problem persists without Longhorn?
Click to expand...

qm guest cmd $VMID fsfreeze-freeze exactly triggers my recognised behaviour.
From a Kubernetes point of view everything works as expected, I can also access the pods running on the VMs as the logic sits in the RAM but as soon as I would like to do some filesystem operations there are errors.

From a Longhorn point of view I get degraded warnings instantly as it tries to do write data.

The VM itself also becomes unresponsive: I can no longer log in via SSH and login via the NoVNC UI also does not work.

Exchanging the CSI driver to a different one would mean that I need to shift all my PVs and this would currently be out of scope.
 
Last edited:
Hmm, I see! Thanks for the update. That's very interesting.

My main guess is that Longhorn really does not play nice with filesystem freezes and always expects the disks it manages to respond within a certain time frame. I mean, quite obvious it doesn't; what I mean is that it expects disks to be responsive by design. After all it's a distributed storage system.

As a workaround, what you could try is set up an fsfreeze hook script in the QEMU guest agent config inside your VMs (see here) and temporarily evict all replicas from the node (the VM) when a freeze is issued. This is described more generally in the Longhorn docs here, but it seems that they've also posted an article on how to do that via the CLI not too long ago: https://longhorn.io/kb/how-to-evict-node-during-capi-node-rolling-replacement/

Of course, you'd also have to readmit the node back into Longhorn again after thawing. AFAICT, the script should run before the freeze and after the thaw, respectively.

Disclaimer: I have never done something like that myself (because I never needed to, admittedly), and it's been a while since I've messed with Kubernetes stuff, but that might just allow you to keep your k3s cluster healthy while backups are running. Of course, it will probably take a while until your replicas have been moved elsewhere, which means that your backups will most likely take longer in total, too. But, at least it should keep your k3s cluster / Longhorn happy and healthy. knocks on wood

Side note: If Longhorn really ends up being too painful for you or if the hook script proves to be infeasible, I can personally recommend setting up Ceph on your Proxmox VE cluster and using the Ceph CSI driver instead. I've set up a plain Kubernetes cluster with that driver ages ago and still got it sitting around somewhere, and the way it works is that you specify a dedicated RBD pool in Ceph for Kubernetes / k3s / etc. that the CSI driver then uses. That should give you most of the benefits of Longhorn while allowing you to manage Ceph via the PVE UI. Additionally, since there's no virtualization overhead between Ceph and the CSI driver, your storage's performance should also improve. You can of course also set quotas for the pool and use Ceph for VM storage as well.

If you've never dealt with Ceph before it can seem a little daunting at first, but it's honestly not too complicated once you've read through the docs a bit. The Ceph tooling we have in Proxmox VE should do most of the work for you. (Also, since you know your way around Kubernetes, I think you'd honestly have an easy time with it.)

One thing that just came to mind while writing this: I assume that Longhorn considers all nodes to be separate from one another in order to be able to guarantee redundancy. So, that means that you would have to ensure that every k3s VM runs on a different node of your Proxmox VE cluster. This might not seem like a big issue, but if (for example) you had to move most of your VMs to one node for some reason, and that node then suddenly failed, you might end up with data loss or inconsistencies. I am not 100% sure about this since I don't know how Longhorn handles IOPS and replicas, but it's one thing to keep in mind at least.

So, if you instead used Ceph in tandem with Proxmox VE, your infrastructure should be much more resilient as well, in adddition to the potential performance gains.

I know you said that switching to a different CSI driver is currently out of scope for you, but I wanted to leave this here regardless, because it might save you a lot of time and headaches in the long run. If your hardware is beefy enough, you could also try out all of this using nested virtualization.

I hope that all of this helps! Let me know if the fsfreeze hook worked for you.
 
  • Like
Reactions: showiproute
Might be possible - so far I haven't found any bug report on SUSEs side with regards to Longhorn + PVE + freezing.
For me it works the way that I kill the VM which hangs and restart it manual.

Migrating to Ceph would be out of scope as I only have two physical PVEs and running Ceph on VMs would bloat my small Kubernetes cluster.
 
You must log in or register to reply here.
Top Bottom
Back