Backup crashes VM - 'guest-fsfreeze-freeze' failed - got timeout

[RedVortex]

Similar issue here. I had the backups running fine for days, never had an issue. I need more information in Proxmox (IP address) so I decided to install and enable qemu-guest-agent. Since then I cannot backup the VM anymore, it fails every time.

Proxmox backup log

INFO: creating Proxmox Backup Server archive 'vm/114/2022-11-04T19:42:15Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption

---- IT HANGS HERE FOR A 1-2 MINS ---

INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 114 qmp command 'backup' failed - backup connect failed: command error: http request timed out
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 114 failed - VM 114 qmp command 'backup' failed - backup connect failed: command error: http request timed out
INFO: Failed at 2022-11-04 15:44:15

In the VM I get those 2 lines then nothing after

Nov  4 15:42:15 proxmoxbs qemu-ga: info: guest-ping called
Nov  4 15:42:15 proxmoxbs qemu-ga: info: guest-fsfreeze called

Has anybody found a solution ?

It works well without qemu-ga

INFO: creating Proxmox Backup Server archive 'vm/114/2022-11-04T19:56:25Z'
INFO: enabling encryption
INFO: started backup task 'e8dd49fc-48c0-4202-a92f-7027585d2ac0'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: created new
INFO:  14% (4.7 GiB of 32.0 GiB) in 3s, read: 1.6 GiB/s, write: 30.7 MiB/s
INFO:  22% (7.1 GiB of 32.0 GiB) in 6s, read: 812.0 MiB/s, write: 42.7 MiB/s
INFO:  31% (10.0 GiB of 32.0 GiB) in 9s, read: 977.3 MiB/s, write: 40.0 MiB/s
INFO:  38% (12.3 GiB of 32.0 GiB) in 12s, read: 800.0 MiB/s, write: 53.3 MiB/s
INFO:  40% (13.1 GiB of 32.0 GiB) in 15s, read: 262.7 MiB/s, write: 34.7 MiB/s
INFO:  55% (17.8 GiB of 32.0 GiB) in 18s, read: 1.6 GiB/s, write: 56.0 MiB/s
INFO:  62% (20.1 GiB of 32.0 GiB) in 21s, read: 804.0 MiB/s, write: 90.7 MiB/s
INFO:  76% (24.5 GiB of 32.0 GiB) in 24s, read: 1.5 GiB/s, write: 33.3 MiB/s
INFO:  88% (28.2 GiB of 32.0 GiB) in 27s, read: 1.2 GiB/s, write: 24.0 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 29s, read: 1.9 GiB/s, write: 0 B/s
INFO: backup is sparse: 28.14 GiB (87%) total zero data
INFO: backup was done incrementally, reused 30.81 GiB (96%)
INFO: transferred 32.00 GiB in 29 seconds (1.1 GiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 114 (00:00:30)
INFO: Backup finished at 2022-11-04 15:56:55
INFO: Backup job finished successfully

 

[EnigmA-X]

I had this very same issue and checked out the earlier linked gitlab issue.

One of the last comments (currently) is about the combination of Debian 11 + MariaDB + /tmp folder permissions.

Since I was running MariaDB (for Zabbix Server), I did the following:

• Stop most important services (systemctl stop <service>), in my case zabbix-server and mariadb
• Remove /tmp folder (rm -rf /tmp)
• Power off virtual machine (it really needs to get into the proxmox stopped state)
• Enable qemu guest agent in proxmox for this VM again (as I disabled this earlier)
• Power on machine (Reboot inside the vm is not sufficient)
• tmp folder recreated automatically

After this, I started a backup from the proxmox dashboard: VM --> Backup --> Backup now and everything worked smooth again.

Since it looks like this issue is related to Debian 11 and most likely MariaDB (this can't be a coincidence), I suspect maybe something is happening during the package install of mariadb with the /tmp permissions which in turn renders a serious issue for the qemu-ga.

Before diving into this, I would like to know if any of the earlier responders to this thread are also running mariadb.

 

[nodesolutions]

For us issue was vm running on zfs zvol with default async io (io_uring) and iothreads enabled.

Issue was resolved by migrating to virtio-scsi-single & async io threads.

https://bugzilla.kernel.org/show_bug.cgi?id=199727

 

[SGM]

Had the same issue on customer setup today. Turned out to be because of loop mounted FS. cPanel likes to loop mount /tmp (calls it secure tmp) and when fsfreeze is issued, it seems like qemu-ga freezes the underlying fs before the loop mounted one, so the later can't flush to disk and it goes nuts. In my case I just migrated to a regular /tmp volume and it's all good now.

 

[elemhsb]

I have got a lot of nfsd on ext4 kernel warnings and error traces while qemu-ga[26050]: info: guest-fsfreeze called.
As my solution a i have upgraded qemu-guest-agent on debian 11 bullseye qemu-guest-agent:amd64 from 1:5.2+dfsg-11+deb11u2 to 1:7.2+dfsg-1~bpo11+1. no reboot needed only restart solved it.

 

[slaptasBotas]

@elemhsb I have updated the qemu guest agent to version 7.2 and so far the manual Backup has worked, I will wait for tomorrow's backup.
Update: new day, new freeze....

 

[Malt6185]

I had this very same issue and checked out the earlier linked gitlab issue.

One of the last comments (currently) is about the combination of Debian 11 + MariaDB + /tmp folder permissions.

Since I was running MariaDB (for Zabbix Server), I did the following:

• Stop most important services (systemctl stop <service>), in my case zabbix-server and mariadb
• Remove /tmp folder (rm -rf /tmp)
• Power off virtual machine (it really needs to get into the proxmox stopped state)
• Enable qemu guest agent in proxmox for this VM again (as I disabled this earlier)
• Power on machine (Reboot inside the vm is not sufficient)
• tmp folder recreated automatically

After this, I started a backup from the proxmox dashboard: VM --> Backup --> Backup now and everything worked smooth again.

Since it looks like this issue is related to Debian 11 and most likely MariaDB (this can't be a coincidence), I suspect maybe something is happening during the package install of mariadb with the /tmp permissions which in turn renders a serious issue for the qemu-ga.

Before diving into this, I would like to know if any of the earlier responders to this thread are also running mariadb.

@EnigmA-X --

I've been having the same problems with a Zabbix VM. However, mine is running PostgreSQL and not MariaDB. I did do the /tmp folder removal as you suggested and then ran manual backup -- which worked fine (thanks for the info).

Will advise if I have issues again during automated backup run later this week.

EDIT Feb 20:
No issues this week with backup for Zabbix VM.