Guest-agent fs-freeze command breaks the system on backup

[cglmicro]

Same problem here with multiple VM running WHM-CPANEL, some works, some won't.

 

[ruysilva]

Hello,

We have this issue too with some vm, on the past, the vzdump run fine but now don't. Not for all vms with cPanel and we can see a pattern here

 

[robertb]

Hey guys,

did anyone manage to find a cause for this?
Happens to me too since a few weeks now. Some virtual machines will at random switch to read only mode and present exact the same problem after rebooting. Seems to happen after taking a snapshot backup to pbs.
However, after running fsck manually it seems to be fine again.

 

[SOLTECSIS - Carles Munyoz]

Hi,
We have the same problem.
It seems that the "solution" is disable the QEMU Guest Agent option, as I suggested here:
https://forum.proxmox.com/threads/e...guest-fsfreeze-thaw-failed-got-timeout.68082/

 

[cglmicro]

Hi Soltecsis.

You are right, for the first time I'm able to do a backup of all my VMs, thanks.

 

[protonet]

Hi

I have the same problem with two VMs.

Centos + cyberpanel
cPanel + cloudlinux 7

Thanks

 

[jw6677]

Same issue here, cpanel on cloudlinux.

 

[ynorecex]

Same issue here, cPanel 96.09 on Cloudlinux 8.3
Proxmox 6.4-8
PBS 1.1-9

I had run a manual backup of the VPS on PBS and the VM froze, I needed to unlock by the host shell to be able to restart the VM.

Then I had to buy new underwear.

 

[jw6677]

Then I had to buy new underwear.

Yup. haha

Disabling qemu-agent does appear to work around this, but that's hardly a solution.

 

[Rawthink]

Bump. This is still an issue. I just tried a brand new install of Cloud Linux. Backups worked fine with the guest agent enabled... Then I installed cpanel. Now same problem. Totally freezes the VM and requires a stop and start...

Any chance we can find the cause and correct this?

 

[jmjosebest]

Same problem with:

CentOS 7 + cPanel
and
CloudLinux 7 + cPanel

"Solved" disabling Guest Agent in VM Options.

 

[Faris Raouf]

I don't like the idea of the guest agent causing problems -- or of some third party change or package causing problems with the guest agent, which then causes problems with backups.

Its a shame nobody has managed to narrow this down. I find it very worrying.

 

[Ben B]

Any updates on this one? I've got the same issues randomly with some CentOS 7 VMs running WHM/CPanel. These ones have the KernelCare stuff installed too...

 

[jmjosebest]

Hello, maybe this is the issue?
https://gitlab.com/qemu-project/qemu/-/issues/520

or there are different issues?

 

[Rawthink]

Hello, maybe this is the issue?
https://gitlab.com/qemu-project/qemu/-/issues/520

or there are different issues?

Yes. Just read this. Looks like the exact issue. I have restorted to running those guests without the agent. Just tested again a couple days ago and same problem.

 

[jmjosebest]

Inside CentOS 7, when we install the guest-agent there is a file: /etc/qemu-ga/fsfreeze-hook

#!/bin/sh

# This script is executed when a guest agent receives fsfreeze-freeze and
# fsfreeze-thaw command, if it is specified in --fsfreeze-hook (-F)
# option of qemu-ga or placed in default path (/etc/qemu/fsfreeze-hook).
# When the agent receives fsfreeze-freeze request, this script is issued with
# "freeze" argument before the filesystem is frozen. And for fsfreeze-thaw
# request, it is issued with "thaw" argument after filesystem is thawed.

LOGFILE=/var/log/qga-fsfreeze-hook.log
FSFREEZE_D=$(dirname -- "$0")/fsfreeze-hook.d

# Check whether file $1 is a backup or rpm-generated file and should be ignored
is_ignored_file() {
    case "$1" in
        *~ | *.bak | *.orig | *.rpmnew | *.rpmorig | *.rpmsave | *.sample)
            return 0 ;;
    esac
    return 1
}

# Iterate executables in directory "fsfreeze-hook.d" with the specified args
[ ! -d "$FSFREEZE_D" ] && exit 0
for file in "$FSFREEZE_D"/* ; do
    is_ignored_file "$file" && continue
    [ -x "$file" ] || continue
    printf "$(date): execute $file $@\n" >>$LOGFILE
    "$file" "$@" >>$LOGFILE 2>&1
    STATUS=$?
    printf "$(date): $file finished with status=$STATUS\n" >>$LOGFILE
done

exit 0

It's something here we can tune to fix the issue?

 

[InShield]

I have the same issue on an brandnew debian machine.

I disable the guest agent for now and do a test..

Disable the qemu agent is still working for me too. But not a nice option :-(

Can anybody help? Is this a kernel problem?

I use still this one:

Linux Proxmox 5.15.39-3-pve #2 SMP PVE 5.15.39-3 (Wed, 27 Jul 2022 13:45:39 +0200) x86_64 GNU/Linux

Now on this kernel the problem is gone on my vm:
Linux Proxmox 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64 GNU/Linux

 

[InShield]

My issue still persist´s :-( Has anyone a fix for this? Shutdown guest agent is no good option

 

[sahostking]

Strange we use Cloudlinux + Cpanel too and only on those servers we have the issue. Could it be cPanel related?

Our Directadmin servers that use Cloudlinux too dont have this issue it seems.

Well atleast nothing appears on our monitoring with respect to this.

Update: Read the post thoroughly and found its related to "/scripts/securetmp" of cpanel.

 

[hancok62]

Strange we use Cloudlinux + Cpanel too and only on those servers we have the issue. Could it be cPanel related?

Our Directadmin servers that use Cloudlinux too dont have this issue it seems.

Well atleast nothing appears on our monitoring with respect to this.

Update: Read the post thoroughly and found its related to "/scripts/securetmp" of cpanel.

in my case It also happened to me with cpanel + cloudlinux,

but some cpanel + cloudlinux work and another no, some crash in one day and some fail in random day's, don't care if i'm using pbs, nfs, local disk, crash some internally is the disventage of mantain very old kernel and try to patch

additionally this in some cases i remember that this occurs in machines without cloudlinux, but with centos7 + cpanel, which means that an old kernel was used too, but in newer ubuntu + cpanel for me is working well i don't no if this is failing in centos8 / alma linux based systems