[SOLVED] Workaround to use qemu-user-agent without guest-fsfreeze ?

[godzilla]

Hi all,

I think we're all aware of the the issue with the snapshot backup locking the guest FS with fsfreeze and not being able to thaw the FS, resulting in a completely locked VM. It's been discussed many times in and out of this forum.

I don't know if there is any progress in fixing the very root of the problem, however I'd like to know if Proxmox could implement a kind of workaround allowing us to keep the guest agent enabled but disabling the fsfreeze call on backups.

qemu-guest-agent has so many more useful features, it's often a lifesaver, and it's a pity you need to have it completely disabled because of a problem that at the moment seems not to have a solution.

Any feedback on this? Thank you

 

[mira]

Can you point at such issues?
Do you run the latest kernel versions? As far as I know the only issue we had recently was with `io_uring` in combination with DBs running in VMs.
But this should be fixed for a while now and only affected kernel 5.15 until 5.15.64 or so.

 

[godzilla]

Hi @mira, thanks for replying.

I'm currently running PVE 7.2-1 on kernel 5.13.19-6-pve, because of another issue I had, which impacted live migrate of VMs between different CPUs. I'm unsure if I can safely upgrade PVE to 7.3 and/or update the kernel to the latest version (I'd be happy to do, btw).

The problem is: sometimes, when qemu-guest-agent is active and running on the guest VM, the backup task executes the fs-freeze on the guest FS as expected, but the fs-thaw never happens.

I noticed it happens specifically with CentOS guests. No freeze hooks are involved.

Also I can't effectively reproduce the issue, for some odd reason it doesn't happen if I request the fsfreeze directly using qm, e.g.:

qm guest cmd <vmid> fsfreeze-freeze
qm guest cmd <vmid> fsfreeze-thaw

Any clue/suggestion?

Thank you!

 

[mira]

Do you use `io_uring` or a different Async I/O mode?
Do you still see the same issue with the latest kernel 5.15 or 5.19/6.1?

 

[godzilla]

Do you use `io_uring` or a different Async I/O mode?
Do you still see the same issue with the latest kernel 5.15 or 5.19/6.1?

Hi @mira

Yes, I use io_uring by default. Better using another I/O mode?

I can't test with different kernel at the moment, unless I'm sure the live migrate bug described in this thread is fixed. I'd like to do but I can't. Do you know anything about that?

Thank you!

 

[mira]

Both kernel 5.19 and kernel 6.1 should not be affected by the live migration issues. But you won't be able to live migrate from 5.13 to 5.19/6.1. A shutdown and start will be necessary.
If you have issues with fsfreeze, then yes, please, switch to `aio` or `threads` instead of `io_uring` until you can upgrade to a newer kernel.

Please also note that 5.13 hasn't received any fixes in a long time. It most likely is affected by multiple security vulnerabilities.

 

[godzilla]

Hi @mira ,

thanks a lot! We're planning to upgrade to Proxmox VE 7.3 soon, so we'll take the opportunity to upgrade the kernel as well.

I'm marking the thread as [SOLVED].

 

[godzilla]

Hi @mira ,

just to inform you that a VM with `threads` Async I/O and VirtIO SCSI Single controller just froze in the same way during the backup.

 

[mira]

Sorry that didn't help.

Was that still on kernel 5.13?

 

[godzilla]

Hi @mira ,

yes, the kernel is still version 5.13.19-6-pve

 

[godzilla]

Hi @mira ,

you said I will not be able to live migrate VMs from nodes running kernel 5.13 to nodes running kernel 5.19.

What if I update the kernel in steps, e.g. 5.13 -> 5.15 -> 5.19 ?

Thank you

 

[godzilla]

Also, what if I pause a VM and then resume it on the new node? it's always better than stop/start.

 

[mira]

Migration from 5.13 to 5.19 might work. I don't think anyone has tested it though.
But 5.13 to 5.15 and 5.15 to 5.19 won't work since 5.15 has incompatible changes.

 

[godzilla]

Hi @mira ,

I successfully migrated to Proxmox 7.3 & kernel 5.19.17-2-pve. Everything went smoothly, including live-migrations.

Unfortunately, though, the original issue still happens with CentOS 7.9 guests.

Any clue?

 

[mira]

I'm sorry to hear that. If newer host kernels don't help in any way, it may be an issue with the guest kernel itself or some software inside the guest.

There's a patch series adding the option to disable fsfreeze on backups: https://lists.proxmox.com/pipermail/pve-devel/2023-February/055656.html

You may want to follow the linked bugzilla entry [0] to get updates on the state of things.


[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4140

 

[godzilla]

Hi @mira , thanks for your feedback. I added myself to the CC list, hope it's going to be released soon.