Qemu-guest-agent and snapshots

TwiX

Renowned Member
Feb 3, 2015
311
23
83
Hi,

On a relatively updated PVE node,
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.6-pve1
ceph-fuse: 15.2.6-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-6
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1
I do some PBS backups.

Backups are made with snapshot mode and qemu-guest-agent is installed on all VMs (Qemu Guest Agent is enabled indeed).
I noticed yesterday that during the backup, I have lot of iowaits on the VM.

Backup started @8:41 and ended @9:10



1623312615332.png

INFO: starting new backup job: vzdump 22117 --mode snapshot --node dc-prox-11 --remove 0 --storage PBS_Tuxis
INFO: Starting Backup of VM 22117 (qemu)
INFO: Backup started at 2021-06-10 08:41:14
INFO: status = running
INFO: VM Name: dc-zabbix-02
INFO: include disk 'scsi0' 'kvm_pool:vm-22117-disk-0' 82G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/22117/2021-06-10T06:41:14Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '7d90a700-91b9-4994-86fe-eb85b8f116aa'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (532.0 MiB of 82.0 GiB) in 3s, read: 177.3 MiB/s, write: 65.3 MiB/s
INFO: 1% (928.0 MiB of 82.0 GiB) in 8s, read: 79.2 MiB/s, write: 33.6 MiB/s
INFO: 2% (1.7 GiB of 82.0 GiB) in 26s, read: 44.2 MiB/s, write: 27.1 MiB/s
INFO: 3% (2.5 GiB of 82.0 GiB) in 47s, read: 39.2 MiB/s, write: 28.4 MiB/s
INFO: 4% (3.3 GiB of 82.0 GiB) in 1m 17s, read: 27.1 MiB/s, write: 20.4 MiB/s
INFO: 5% (4.1 GiB of 82.0 GiB) in 1m 54s, read: 22.8 MiB/s, write: 21.5 MiB/s
INFO: 6% (4.9 GiB of 82.0 GiB) in 2m 44s, read: 16.7 MiB/s, write: 16.1 MiB/s
INFO: 7% (5.7 GiB of 82.0 GiB) in 3m 32s, read: 17.5 MiB/s, write: 17.2 MiB/s
INFO: 8% (6.6 GiB of 82.0 GiB) in 4m 9s, read: 22.8 MiB/s, write: 18.7 MiB/s
INFO: 9% (7.4 GiB of 82.0 GiB) in 4m 54s, read: 18.8 MiB/s, write: 18.3 MiB/s
INFO: 10% (8.2 GiB of 82.0 GiB) in 5m 34s, read: 20.9 MiB/s, write: 20.6 MiB/s
INFO: 11% (9.0 GiB of 82.0 GiB) in 6m 8s, read: 24.5 MiB/s, write: 24.1 MiB/s
INFO: 12% (9.9 GiB of 82.0 GiB) in 6m 58s, read: 17.0 MiB/s, write: 16.4 MiB/s
INFO: 13% (10.7 GiB of 82.0 GiB) in 7m 37s, read: 21.8 MiB/s, write: 19.3 MiB/s
INFO: 14% (11.5 GiB of 82.0 GiB) in 8m 12s, read: 24.1 MiB/s, write: 20.6 MiB/s
INFO: 15% (12.3 GiB of 82.0 GiB) in 9m 14s, read: 13.2 MiB/s, write: 12.8 MiB/s
INFO: 16% (13.2 GiB of 82.0 GiB) in 10m 6s, read: 16.6 MiB/s, write: 16.3 MiB/s
INFO: 17% (14.0 GiB of 82.0 GiB) in 11m 1s, read: 15.1 MiB/s, write: 14.3 MiB/s
INFO: 18% (14.8 GiB of 82.0 GiB) in 11m 57s, read: 14.8 MiB/s, write: 13.2 MiB/s
INFO: 19% (15.6 GiB of 82.0 GiB) in 12m 44s, read: 17.7 MiB/s, write: 14.9 MiB/s
INFO: 20% (16.4 GiB of 82.0 GiB) in 13m 24s, read: 21.3 MiB/s, write: 20.9 MiB/s
INFO: 21% (17.2 GiB of 82.0 GiB) in 14m 11s, read: 17.8 MiB/s, write: 17.8 MiB/s
INFO: 22% (18.1 GiB of 82.0 GiB) in 15m 13s, read: 13.7 MiB/s, write: 13.7 MiB/s
INFO: 23% (18.9 GiB of 82.0 GiB) in 15m 53s, read: 20.9 MiB/s, write: 19.5 MiB/s
INFO: 24% (19.7 GiB of 82.0 GiB) in 16m 44s, read: 16.2 MiB/s, write: 14.9 MiB/s
INFO: 25% (20.5 GiB of 82.0 GiB) in 17m 37s, read: 16.4 MiB/s, write: 13.9 MiB/s
INFO: 26% (21.3 GiB of 82.0 GiB) in 18m 14s, read: 22.3 MiB/s, write: 14.7 MiB/s
INFO: 27% (22.1 GiB of 82.0 GiB) in 18m 38s, read: 34.2 MiB/s, write: 22.7 MiB/s
INFO: 28% (23.0 GiB of 82.0 GiB) in 19m 17s, read: 21.5 MiB/s, write: 18.5 MiB/s
INFO: 29% (23.8 GiB of 82.0 GiB) in 19m 58s, read: 21.0 MiB/s, write: 17.7 MiB/s
INFO: 30% (24.6 GiB of 82.0 GiB) in 20m 39s, read: 20.9 MiB/s, write: 12.3 MiB/s
INFO: 31% (25.4 GiB of 82.0 GiB) in 21m 19s, read: 20.7 MiB/s, write: 20.1 MiB/s
INFO: 32% (26.3 GiB of 82.0 GiB) in 21m 51s, read: 26.9 MiB/s, write: 22.5 MiB/s
INFO: 33% (27.1 GiB of 82.0 GiB) in 22m 20s, read: 28.4 MiB/s, write: 15.2 MiB/s
INFO: 34% (27.9 GiB of 82.0 GiB) in 22m 37s, read: 48.2 MiB/s, write: 29.2 MiB/s
INFO: 35% (28.7 GiB of 82.0 GiB) in 23m 5s, read: 31.3 MiB/s, write: 23.9 MiB/s
INFO: 36% (29.5 GiB of 82.0 GiB) in 23m 30s, read: 32.0 MiB/s, write: 28.3 MiB/s
INFO: 37% (30.3 GiB of 82.0 GiB) in 24m 0s, read: 28.0 MiB/s, write: 26.8 MiB/s
INFO: 38% (31.2 GiB of 82.0 GiB) in 24m 29s, read: 30.5 MiB/s, write: 26.6 MiB/s
INFO: 39% (32.0 GiB of 82.0 GiB) in 24m 50s, read: 40.6 MiB/s, write: 25.0 MiB/s
INFO: 40% (32.8 GiB of 82.0 GiB) in 25m 12s, read: 36.7 MiB/s, write: 30.5 MiB/s
INFO: 41% (33.8 GiB of 82.0 GiB) in 25m 29s, read: 60.9 MiB/s, write: 26.8 MiB/s
INFO: 42% (34.6 GiB of 82.0 GiB) in 25m 36s, read: 103.4 MiB/s, write: 39.4 MiB/s
INFO: 43% (35.3 GiB of 82.0 GiB) in 25m 49s, read: 56.3 MiB/s, write: 24.0 MiB/s
INFO: 44% (36.2 GiB of 82.0 GiB) in 26m 1s, read: 79.3 MiB/s, write: 28.7 MiB/s
INFO: 45% (37.0 GiB of 82.0 GiB) in 26m 8s, read: 112.6 MiB/s, write: 22.9 MiB/s
INFO: 46% (37.8 GiB of 82.0 GiB) in 26m 16s, read: 102.5 MiB/s, write: 40.0 MiB/s
INFO: 47% (38.7 GiB of 82.0 GiB) in 26m 32s, read: 58.5 MiB/s, write: 21.5 MiB/s
INFO: 48% (39.4 GiB of 82.0 GiB) in 26m 40s, read: 94.0 MiB/s, write: 23.5 MiB/s
INFO: 49% (40.2 GiB of 82.0 GiB) in 26m 46s, read: 131.3 MiB/s, write: 14.7 MiB/s
INFO: 50% (41.0 GiB of 82.0 GiB) in 26m 53s, read: 125.7 MiB/s, write: 32.0 MiB/s
INFO: 51% (41.8 GiB of 82.0 GiB) in 27m 7s, read: 58.0 MiB/s, write: 15.1 MiB/s
INFO: 52% (42.7 GiB of 82.0 GiB) in 27m 19s, read: 71.0 MiB/s, write: 10.3 MiB/s
INFO: 53% (43.5 GiB of 82.0 GiB) in 27m 24s, read: 162.4 MiB/s, write: 18.4 MiB/s
INFO: 54% (44.4 GiB of 82.0 GiB) in 27m 33s, read: 105.8 MiB/s, write: 20.9 MiB/s
INFO: 55% (45.1 GiB of 82.0 GiB) in 27m 38s, read: 152.0 MiB/s, write: 2.4 MiB/s
INFO: 56% (46.1 GiB of 82.0 GiB) in 27m 46s, read: 118.5 MiB/s, write: 43.5 MiB/s
INFO: 57% (46.8 GiB of 82.0 GiB) in 27m 50s, read: 181.0 MiB/s, write: 10.0 MiB/s
INFO: 58% (47.6 GiB of 82.0 GiB) in 28m 2s, read: 71.3 MiB/s, write: 20.7 MiB/s
INFO: 59% (48.7 GiB of 82.0 GiB) in 28m 9s, read: 158.3 MiB/s, write: 585.1 KiB/s
INFO: 68% (55.9 GiB of 82.0 GiB) in 28m 12s, read: 2.4 GiB/s, write: 1.3 MiB/s
INFO: 77% (63.2 GiB of 82.0 GiB) in 28m 15s, read: 2.4 GiB/s, write: 0 B/s
INFO: 86% (70.5 GiB of 82.0 GiB) in 28m 18s, read: 2.4 GiB/s, write: 0 B/s
INFO: 90% (74.6 GiB of 82.0 GiB) in 28m 21s, read: 1.4 GiB/s, write: 34.7 MiB/s
INFO: 91% (74.8 GiB of 82.0 GiB) in 28m 24s, read: 50.7 MiB/s, write: 50.7 MiB/s
INFO: 92% (75.5 GiB of 82.0 GiB) in 28m 34s, read: 80.0 MiB/s, write: 1.6 MiB/s
INFO: 93% (76.3 GiB of 82.0 GiB) in 28m 41s, read: 114.9 MiB/s, write: 0 B/s
INFO: 94% (77.1 GiB of 82.0 GiB) in 28m 48s, read: 112.0 MiB/s, write: 1.7 MiB/s
INFO: 95% (78.0 GiB of 82.0 GiB) in 28m 56s, read: 111.0 MiB/s, write: 0 B/s
INFO: 96% (78.7 GiB of 82.0 GiB) in 29m 1s, read: 156.8 MiB/s, write: 0 B/s
INFO: 100% (82.0 GiB of 82.0 GiB) in 29m 3s, read: 1.6 GiB/s, write: 0 B/s
INFO: backup is sparse: 34.86 GiB (42%) total zero data
INFO: backup was done incrementally, reused 50.25 GiB (61%)
INFO: transferred 82.00 GiB in 1743 seconds (48.2 MiB/s)
INFO: Finished Backup of VM 22117 (00:29:04)
INFO: Backup finished at 2021-06-10 09:10:18
Using encryption key from file descriptor
INFO: Backup job finished successfully
TASK OK

VM syslog (debian 10 with qemu-guest-agent v5.2 from backports) :
Code:
Jun 10 08:41:14 dc-zabbix-02 qemu-ga: info: guest-ping called
Jun 10 08:41:15 dc-zabbix-02 qemu-ga: info: guest-fsfreeze called

The fact is that during the backup VM is incredibly slow, mysql logs reports lots of slow queries, and some applications starts to have a wrong behavior, in that case VM 22117 is a zabbix server and sends wrong alerts emails.

Is it the right behavior for that kind of backups using qemu-fsfreeze ?
Do I have to deactivate qemu-guest-agent ?

Thanks in advance !
 
Hello
I have the same problem with a ceph storage.

I thought the backup is done via a snapshot. In this case, a write in the VM should not impact the backup and vice versa?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!