Backup scheduler stop every day the VM

V

virmix

Member
Sep 4, 2020
36
1
13
cpu.png

LOG BACKUP
Code: 
INFO: starting new backup job: vzdump 102 --compress zstd --mode snapshot --prune-backups 'keep-last=3' --mailnotification failure --quiet 1 --storage backup --node nodo1
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2022-01-22 06:00:01
INFO: status = running
INFO: VM Name: Win2022
INFO: include disk 'scsi0' 'local:102/vm-102-disk-1.qcow2' 700G
INFO: exclude disk 'virtio1' 'local:102/mail.qcow2' (backup=no)
INFO: include disk 'efidisk0' 'local:102/vm-102-disk-0.qcow2' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/backup/dump/vzdump-qemu-102-2022_01_22-06_00_01.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
ERROR: VM 102 qmp command 'guest-fsfreeze-freeze' failed - client closed connection
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 102 not running
ERROR: client closed connection
INFO: aborting backup job
ERROR: VM 102 not running
INFO: resuming VM again
ERROR: Backup of VM 102 failed - VM 102 not running
INFO: Failed at 2022-01-22 06:00:02
INFO: Backup job finished with errors
TASK ERROR: job errors



I think is the backup, I cannot found any log about this problem.


bk.png


I have a lot space in HD


space.png


info.png


INFO
Code: 
proxmox-ve: 7.1-1 (running kernel: 5.15.12-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-5.15: 7.1-8
pve-kernel-helper: 7.1-8
pve-kernel-5.13: 7.1-6
pve-kernel-5.4: 6.4-4
pve-kernel-5.15.12-1-pve: 5.15.12-3
pve-kernel-5.15.7-1-pve: 5.15.7-1
pve-kernel-5.13.19-3-pve: 5.13.19-7
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.3-1
proxmox-backup-file-restore: 2.1.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-4
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

SYSLOG
Jan 22 04:36:44 NODODOMAIN smartd[790]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 65 to 64 Jan 22 04:36:44 NODODOMAIN smartd[790]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 62 to 61 Jan 22 05:06:44 NODODOMAIN smartd[790]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 64 to 65 Jan 22 05:06:44 NODODOMAIN smartd[790]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 61 to 62 Jan 22 05:17:01 NODODOMAIN CRON[167750]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 05:17:01 NODODOMAIN CRON[167751]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 22 05:17:01 NODODOMAIN CRON[167750]: pam_unix(cron:session): session closed for user root Jan 22 06:00:01 NODODOMAIN pvescheduler[177755]: <root@pam> starting task UPID:nodo1:0002B65C:0044120D:61EB8F51:vzdump:102:root@pam: Jan 22 06:00:01 NODODOMAIN pvescheduler[177756]: INFO: starting new backup job: vzdump 102 --compress zstd --mode snapshot --prune-backups 'keep-last=3' --mailnotification failure --mailto mail@mail.com --quiet 1 --storage backup --node nodo1 Jan 22 06:00:01 NODODOMAIN pvescheduler[177756]: INFO: Starting Backup of VM 102 (qemu) [B]Jan 22 06:00:02 NODODOMAIN QEMU[1274]: KVM: entry failed, hardware error 0x80000021[/B] Jan 22 06:00:02 NODODOMAIN QEMU[1274]: If you're running a guest on an Intel machine without unrestricted mode Jan 22 06:00:02 NODODOMAIN QEMU[1274]: support, the failure can be most likely due to the guest entering an invalid Jan 22 06:00:02 NODODOMAIN QEMU[1274]: state for Intel VT. For example, the guest maybe running in big real mode Jan 22 06:00:02 NODODOMAIN QEMU[1274]: which is not supported on less recent Intel processors. Jan 22 06:00:02 NODODOMAIN kernel: set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. Jan 22 06:00:02 NODODOMAIN QEMU[1274]: EAX=00000000 EBX=95468fb0 ECX=95468fb0 EDX=00000000 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: ESI=0bb33040 EDI=9688bf50 EBP=95469000 ESP=9688bf50 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: ES =0000 00000000 ffffffff 00809300 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: CS =c200 7ffc2000 ffffffff 00809300 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: SS =0000 00000000 ffffffff 00809300 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: DS =0000 00000000 ffffffff 00809300 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: FS =0000 00000000 ffffffff 00809300 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: GS =0000 00000000 ffffffff 00809300 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: LDT=0000 00000000 000fffff 00000000 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: TR =0040 57faf000 00000067 00008b00 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: GDT= 57fb0fb0 00000057 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: IDT= 00000000 00000000 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: CR0=00050032 CR2=02a16060 CR3=268e3000 CR4=00000000 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: DR6=00000000ffff0ff0 DR7=0000000000000400 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: EFER=0000000000000000 Jan 22 06:00:02 NODODOMAIN QEMU[1274]: Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed. [B]Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: VM 102 qmp command failed - VM 102 qmp command 'guest-fsfreeze-freeze' failed - client closed connection[/B] Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: VM 102 qmp command failed - VM 102 not running Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: VM 102 qmp command failed - VM 102 not running Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: VM 102 qmp command failed - VM 102 not running Jan 22 06:00:02 NODODOMAIN kernel: fwbr102i0: port 2(tap102i0) entered disabled state Jan 22 06:00:02 NODODOMAIN kernel: fwbr102i0: port 2(tap102i0) entered disabled state Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: ERROR: Backup of VM 102 failed - VM 102 not running Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: INFO: Backup job finished with errors Jan 22 06:00:02 NODODOMAIN pvescheduler[177756]: job errors Jan 22 06:00:02 NODODOMAIN postfix/pickup[167828]: 99FB820131: uid=0 from=<root> Jan 22 06:00:02 NODODOMAIN postfix/cleanup[177796]: 99FB820131: message-id=<20220122050002.99FB820131@ded-une2798.domain.com> Jan 22 06:00:02 NODODOMAIN postfix/qmgr[1170]: 99FB820131: from=<root@domain.com>, size=4269, nrcpt=1 (queue active) Jan 22 06:00:02 NODODOMAIN systemd[1]: 102.scope: Succeeded. Jan 22 06:00:02 NODODOMAIN systemd[1]: 102.scope: Consumed 3h 59min 3.916s CPU time. Jan 22 06:00:03 NODODOMAIN qmeventd[177787]: Starting cleanup for 102 Jan 22 06:00:03 NODODOMAIN kernel: fwbr102i0: port 1(fwln102i0) entered disabled state Jan 22 06:00:03 NODODOMAIN kernel: vmbr0: port 2(fwpr102p0) entered disabled state Jan 22 06:00:03 NODODOMAIN kernel: device fwln102i0 left promiscuous mode Jan 22 06:00:03 NODODOMAIN kernel: fwbr102i0: port 1(fwln102i0) entered disabled state Jan 22 06:00:03 NODODOMAIN kernel: device fwpr102p0 left promiscuous mode Jan 22 06:00:03 NODODOMAIN kernel: vmbr0: port 2(fwpr102p0) entered disabled state Jan 22 06:00:03 NODODOMAIN qmeventd[177787]: Finished cleanup for 102 Jan 22 06:00:03 NODODOMAIN postfix/smtp[177798]: 99FB820131: to=<mail@mail.com>, relay=gmail-smtp-in.l.google.com[64.233.167.26]:25, delay=1.2, delays=0.02/0.01/0.26/0.95, dsn=2.0.0, status=sent (250 2.0.0 OK 1642827603 59si5517280wrk.581 - gsmtp) Jan 22 06:00:03 NODODOMAIN postfix/qmgr[1170]: 99FB820131: removed Jan 22 06:17:01 NODODOMAIN CRON[181498]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 06:17:01 NODODOMAIN CRON[181499]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 22 06:17:01 NODODOMAIN CRON[181498]: pam_unix(cron:session): session closed for user root Jan 22 06:25:01 NODODOMAIN CRON[183237]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 06:25:01 NODODOMAIN CRON[183238]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )) Jan 22 06:25:01 NODODOMAIN CRON[183237]: pam_unix(cron:session): session closed for user root Jan 22 06:34:41 NODODOMAIN systemd[1]: Starting Daily apt upgrade and clean activities... Jan 22 06:34:42 NODODOMAIN systemd[1]: apt-daily-upgrade.service: Succeeded. Jan 22 06:34:42 NODODOMAIN systemd[1]: Finished Daily apt upgrade and clean activities. Jan 22 06:36:44 NODODOMAIN smartd[790]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 65 to 66 Jan 22 06:36:44 NODODOMAIN smartd[790]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 62 to 63 Jan 22 07:17:01 NODODOMAIN CRON[194658]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 07:17:01 NODODOMAIN CRON[194659]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 22 07:17:01 NODODOMAIN CRON[194658]: pam_unix(cron:session): session closed for user root Jan 22 07:24:41 NODODOMAIN systemd[1]: Starting Daily apt download activities... Jan 22 07:24:42 NODODOMAIN systemd[1]: apt-daily.service: Succeeded. Jan 22 07:24:42 NODODOMAIN systemd[1]: Finished Daily apt download activities. Jan 22 08:17:01 NODODOMAIN CRON[207739]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 08:17:01 NODODOMAIN CRON[207740]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 22 08:17:01 NODODOMAIN CRON[207739]: pam_unix(cron:session): session closed for user root Jan 22 09:17:01 NODODOMAIN CRON[220773]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 09:17:01 NODODOMAIN CRON[220774]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 22 09:17:01 NODODOMAIN CRON[220773]: pam_unix(cron:session): session closed for user root Jan 22 10:17:01 NODODOMAIN CRON[233808]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 22 10:17:01 NODODOMAIN CRON[233809]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 22 10:17:01 NODODOMAIN CRON[233808]: pam_unix(cron:session): session closed for user root Jan 22 10:26:13 NODODOMAIN pvedaemon[1240]: <root@pam> successful auth for user 'root@pam' Jan 22 10:26:17 NODODOMAIN pvedaemon[1240]: <root@pam> starting task UPID:nodo1:00039936:005C72BB:61EBCDB9:qmstart:102:root@pam: Jan 22 10:26:17 NODODOMAIN pvedaemon[235830]: start VM 102: UPID:nodo1:00039936:005C72BB:61EBCDB9:qmstart:102:root@pam: Jan 22 10:26:17 NODODOMAIN pvedaemon[235830]: iothread is only valid with virtio disk or virtio-scsi-single controller, ignoring Jan 22 10:26:18 NODODOMAIN systemd[1]: Started 102.scope.

I run backup without scheduler task and work

rbk.png
 
Last edited:
Today all work fine, because I disabled the backup scheduler.
I dont know what happend, but problem is on backup module
 
virmix said:
Today all work fine, because I disabled the backup scheduler.
I dont know what happend, but problem is on backup module
Click to expand...
seems unlikely since the code that is executed is actually the same, regardless if you call it manually or via the schedule. (it is possible something interferes there though)

but, from the logs:

Code: 
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: KVM: entry failed, hardware error 0x800000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: If you're running a guest on an Intel machine without unrestricted mode
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: support, the failure can be most likely due to the guest entering an invalid
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: state for Intel VT. For example, the guest maybe running in big real mode
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: which is not supported on less recent Intel processors.
Jan 22 06:00:02 NODODOMAIN kernel: set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state.
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: EAX=00000000 EBX=95468fb0 ECX=95468fb0 EDX=00000000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: ESI=0bb33040 EDI=9688bf50 EBP=95469000 ESP=9688bf50
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: ES =0000 00000000 ffffffff 00809300
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: CS =c200 7ffc2000 ffffffff 00809300
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: SS =0000 00000000 ffffffff 00809300
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: DS =0000 00000000 ffffffff 00809300
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: FS =0000 00000000 ffffffff 00809300
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: GS =0000 00000000 ffffffff 00809300
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: LDT=0000 00000000 000fffff 00000000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: TR =0040 57faf000 00000067 00008b00
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: GDT=     57fb0fb0 00000057
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: IDT=     00000000 00000000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: CR0=00050032 CR2=02a16060 CR3=268e3000 CR4=00000000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: DR6=00000000ffff0ff0 DR7=0000000000000400
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: EFER=0000000000000000
Jan 22 06:00:02 NODODOMAIN QEMU[1274]: Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: Assertion `ret < cpu->num_ases && ret >= 0' failed.

it seems like this is a hardware/cpu error? did you maybe install a newer kernel since the last crash of the vm ?
 
dcsapak said:
parece poco probable ya que el código que se ejecuta es en realidad el mismo, independientemente de si lo llama manualmente o mediante el programa. (aunque es posible que algo interfiera allí)

pero, de los registros:

[código]
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: KVM: error de entrada, error de hardware 0x800000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: si está ejecutando un invitado en una máquina Intel sin modo sin restricciones
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: soporte, la falla probablemente se deba a que el invitado ingresó un
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: estado para Intel VT. Por ejemplo, el invitado puede ejecutarse en modo real grande
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: que no es compatible con los procesadores Intel menos recientes.
22 de enero 06:00:02 Núcleo NODODOMAIN: establecer kvm_intel.dump_invalid_vmcs=1 para volcar el estado interno de KVM.
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: EAX=00000000 EBX=95468fb0 ECX=95468fb0 EDX=00000000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: ESI=0bb33040 EDI=9688bf50 EBP=95469000 ESP=9688bf50
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: ES =0000 00000000 ffffffff 00809300
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: CS =c200 7ffc2000 ffffffff 00809300
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: SS =0000 00000000 ffffffff 00809300
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: DS =0000 00000000 ffffffff 00809300
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: FS =0000 00000000 ffffffff 00809300
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: GS =0000 00000000 ffffffff 00809300
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: LDT=0000 00000000 000fffff 00000000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: TR =0040 57faf000 00000067 00008b00
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: GDT= 57fb0fb0 00000057
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: IDT= 00000000 00000000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: CR0=00050032 CR2=02a16060 CR3=268e3000 CR4=00000000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: DR6=00000000ffff0ff0 DR7=0000000000000400
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: EFER=0000000000000000
22 de enero 06:00:02 NODODOMAIN QEMU[1274]: Code=kvm: ../hw/core/cpu-sysemu.c:77: cpu_asidx_from_attrs: la afirmación `ret < cpu->num_ases && ret >= 0' falló.
[/código]

parece que esto es un error de hardware/cpu? ¿Quizás instaló un kernel más nuevo desde el último bloqueo de la máquina virtual?
Click to expand...

Yes, I have kernel 5.15.12 (beta), and worked it fine before install. It is possible that the problem is kernel.

I rebooted Node and VM when was install kernel, but this happend after
 
Last edited:
I have a similar issue here: https://forum.proxmox.com/threads/b...ning-but-vm-never-starts-after-finish.104055/

when setting up scheduler to backup using snapshot mode it starts to backup freezes the VM Thaws the VM but my websites on that VM are down.. I have to manually stop and start the VM in order for it to work.

I have not tried manual backup snapshot mode, but will try it now and see if that starts the VM again without having to manually stop and start the VM.

just a note if I use scheduler with Stop mode. it works and after the backup it starts the VM itself and my sites go back online. Only the Snapshot mode in scheduler creates this issue for me.

Thanks and sorry not trying to hijack the OP's post.

Kind Regards,
Spiro
 
hey,

can you try applying the following diff to see if it fixes the issue:
Code: 
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 0071a06..bb78fac 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4012,7 +4012,7 @@ sub config_to_command {
     push @$machineFlags, 'accel=tcg';
     }
 
-    push @$machineFlags, 'smm=off' if should_disable_smm($conf, $vga);
+    push @$machineFlags, 'smm=off'; #if should_disable_smm($conf, $vga);
 
     my $machine_type_min = $machine_type;
     if ($add_pve_version) {

you can edit /usr/share/perl5/PVE/QemuServer.pm and then do a systemctl reload pvedaemon followed by stop/start of the affected VM.

also it would help us with identifying the issue if you could provide the following info:
* CPU model (lscpu)
* is this a nested setup? (are you running PVE on another hypervisor? if so, which?)
* which PVE kernel version is running? uname -a on the PVE host
 
oguz said:
hey,

can you try applying the following diff to see if it fixes the issue:
Code: 
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 0071a06..bb78fac 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4012,7 +4012,7 @@ sub config_to_command {
     push @$machineFlags, 'accel=tcg';
     }
 
-    push @$machineFlags, 'smm=off' if should_disable_smm($conf, $vga);
+    push @$machineFlags, 'smm=off'; #if should_disable_smm($conf, $vga);
 
     my $machine_type_min = $machine_type;
     if ($add_pve_version) {

you can edit /usr/share/perl5/PVE/QemuServer.pm and then do a systemctl reload pvedaemon followed by stop/start of the affected VM.

also it would help us with identifying the issue if you could provide the following info:
* CPU model (lscpu)
* is this a nested setup? (are you running PVE on another hypervisor? if so, which?)
* which PVE kernel version is running? uname -a on the PVE host
Click to expand...

oguz said:
Code:
Code: 
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 0071a06..bb78fac 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -4012,7 +4012,7 @@ sub config_to_command {
     push @$machineFlags, 'accel=tcg';
     }
 
-    push @$machineFlags, 'smm=off' if should_disable_smm($conf, $vga);
+    push @$machineFlags, 'smm=off'; #if should_disable_smm($conf, $vga);
 
     my $machine_type_min = $machine_type;
     if ($add_pve_version) {
Click to expand...
do I just add this code at the bottom or top or does this replace some code I need to find ?
oguz said:
lscpu
Click to expand...
Code: 
root@proxmox:~# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          56
On-line CPU(s) list:             0-55
Thread(s) per core:              2
Core(s) per socket:              14
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           79
Model name:                      Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Stepping:                        1
CPU MHz:                         3300.000
CPU max MHz:                     3300.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        4788.84
Virtualization:                  VT-x
L1d cache:                       896 KiB
L1i cache:                       896 KiB
L2 cache:                        7 MiB
L3 cache:                        70 MiB
NUMA node0 CPU(s):               0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52
                                 ,54
NUMA node1 CPU(s):               1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53
                                 ,55
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP condit
                                 ional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clf
                                 lush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm c
                                 onstant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid
                                  aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg
                                 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_
                                 timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb c
                                 at_l3 cdp_l3 invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpr
                                 iority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms inv
                                 pcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc
                                 cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
oguz said:
is this a nested setup? (are you running PVE on another hypervisor? if so, which?)
Click to expand...
I rented a server they installed Debian - Proxmox
i use Proxmox to make a VM
installed AlmaLinux 8.5 and cPanel copied over my websites from another server.
not sure what means nested setup ? or another hypervisor ?
oguz said:
which PVE kernel version is running? uname -a on the PVE host
Click to expand...
Linux proxmox 5.13.19-3-pve #1 SMP PVE 5.13.19-7 (Thu, 20 Jan 2022 16:37:56 +0100) x86_64 GNU/Linux

thanks for your help

Kind Regards
Spiro
 
Spirog said:
lscpu
Click to expand...
thanks!

Spirog said:
not sure what means nested setup ? or another hypervisor ?
Click to expand...
yes that's how i meant :) basically if your PVE is running on bare metal or not.

Spirog said:
do I just add this code at the bottom or top or does this replace some code I need to find ?
Click to expand...
you change that single line with the - to the one below with + (so you just comment out the if part and add a semicolon before the comment, the file is located in the path i've mentioned before)
 
oguz said:
- push @$machineFlags, 'smm=off' if should_disable_smm($conf, $vga); + push @$machineFlags, 'smm=off'; #if should_disable_smm($conf, $vga);
Click to expand...
so just this line with the -
Code: 
push @$machineFlags, 'smm=off' if should_disable_smm($conf, $vga);

and replace with the + line
Code: 
push @$machineFlags, 'smm=off'; #if should_disable_smm($conf, $vga);

ok will try now and try again and report back in a few mins thanks so much

Spiro
 
oguz said:
thanks!


yes that's how i meant :) basically if your PVE is running on bare metal or not.


you change that single line with the - to the one below with + (so you just comment out the if part and add a semicolon before the comment, the file is located in the path i've mentioned before)
Click to expand...
YOU ARE AWESOME!!!!!!!!!!!!!!!!!!!!

you found the Bug. and now its fixed..

Code: 
INFO: starting new backup job: vzdump 104 --mode snapshot --prune-backups 'keep-daily=1,keep-last=2,keep-weekly=1' --mailto **@gmail.com --node proxmox --storage backup_drive --all 0 --compress zstd --mailnotification always
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2022-02-02 06:39:56
INFO: status = running
INFO: VM Name: AlmaLinux
INFO: include disk 'scsi0' 'local-lvm:vm-104-disk-0' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/backup_drive/dump/vzdump-qemu-104-2022_02_02-06_39_56.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'ae0fb37b-0ac4-4db0-8a60-3a2dfc130a0b'
INFO: resuming VM again
INFO:   0% (2.2 GiB of 500.0 GiB) in 3s, read: 754.0 MiB/s, write: 79.0 MiB/s
INFO:   1% (5.0 GiB of 500.0 GiB) in 6s, read: 967.9 MiB/s, write: 14.5 MiB/s
INFO:   2% (10.0 GiB of 500.0 GiB) in 15s, read: 564.3 MiB/s, write: 97.3 MiB/s
INFO:   3% (16.8 GiB of 500.0 GiB) in 19s, read: 1.7 GiB/s, write: 31.0 KiB/s
INFO:   4% (24.6 GiB of 500.0 GiB) in 22s, read: 2.6 GiB/s, write: 0 B/s
INFO:   5% (29.4 GiB of 500.0 GiB) in 25s, read: 1.6 GiB/s, write: 0 B/s
INFO:   7% (37.0 GiB of 500.0 GiB) in 29s, read: 1.9 GiB/s, write: 417.0 KiB/s
INFO:   8% (44.0 GiB of 500.0 GiB) in 32s, read: 2.3 GiB/s, write: 2.3 MiB/s
INFO:   9% (48.7 GiB of 500.0 GiB) in 35s, read: 1.6 GiB/s, write: 0 B/s
INFO:  11% (56.3 GiB of 500.0 GiB) in 38s, read: 2.5 GiB/s, write: 0 B/s
INFO:  12% (61.7 GiB of 500.0 GiB) in 41s, read: 1.8 GiB/s, write: 0 B/s
INFO:  13% (66.3 GiB of 500.0 GiB) in 44s, read: 1.5 GiB/s, write: 21.3 KiB/s
INFO:  14% (73.3 GiB of 500.0 GiB) in 48s, read: 1.8 GiB/s, write: 9.0 KiB/s
INFO:  16% (81.1 GiB of 500.0 GiB) in 51s, read: 2.6 GiB/s, write: 0 B/s
INFO:  17% (86.6 GiB of 500.0 GiB) in 54s, read: 1.8 GiB/s, write: 0 B/s
INFO:  19% (95.2 GiB of 500.0 GiB) in 57s, read: 2.9 GiB/s, write: 0 B/s
INFO:  20% (101.0 GiB of 500.0 GiB) in 1m, read: 1.9 GiB/s, write: 0 B/s
INFO:  21% (109.7 GiB of 500.0 GiB) in 1m 3s, read: 2.9 GiB/s, write: 128.0 KiB/s
INFO:  22% (111.5 GiB of 500.0 GiB) in 1m 6s, read: 617.5 MiB/s, write: 77.3 MiB/s
INFO:  23% (115.2 GiB of 500.0 GiB) in 1m 13s, read: 547.9 MiB/s, write: 75.2 MiB/s
INFO:  24% (122.3 GiB of 500.0 GiB) in 1m 17s, read: 1.8 GiB/s, write: 0 B/s
INFO:  25% (128.0 GiB of 500.0 GiB) in 1m 20s, read: 1.9 GiB/s, write: 21.3 KiB/s
INFO:  27% (135.8 GiB of 500.0 GiB) in 1m 23s, read: 2.6 GiB/s, write: 0 B/s
INFO:  28% (140.7 GiB of 500.0 GiB) in 1m 26s, read: 1.6 GiB/s, write: 0 B/s
INFO:  29% (148.9 GiB of 500.0 GiB) in 1m 29s, read: 2.8 GiB/s, write: 2.3 MiB/s
INFO:  30% (154.0 GiB of 500.0 GiB) in 1m 32s, read: 1.7 GiB/s, write: 0 B/s
INFO:  32% (160.9 GiB of 500.0 GiB) in 1m 35s, read: 2.3 GiB/s, write: 0 B/s
INFO:  33% (165.7 GiB of 500.0 GiB) in 1m 38s, read: 1.6 GiB/s, write: 0 B/s
INFO:  34% (173.4 GiB of 500.0 GiB) in 1m 41s, read: 2.6 GiB/s, write: 0 B/s
INFO:  35% (179.2 GiB of 500.0 GiB) in 1m 44s, read: 1.9 GiB/s, write: 16.0 KiB/s
INFO:  37% (188.0 GiB of 500.0 GiB) in 1m 47s, read: 2.9 GiB/s, write: 0 B/s
INFO:  38% (194.1 GiB of 500.0 GiB) in 1m 50s, read: 2.0 GiB/s, write: 0 B/s
INFO:  40% (202.9 GiB of 500.0 GiB) in 1m 53s, read: 2.9 GiB/s, write: 21.3 KiB/s
INFO:  41% (208.1 GiB of 500.0 GiB) in 1m 56s, read: 1.8 GiB/s, write: 0 B/s
INFO:  43% (215.2 GiB of 500.0 GiB) in 1m 59s, read: 2.4 GiB/s, write: 85.3 KiB/s
INFO:  44% (220.3 GiB of 500.0 GiB) in 2m 8s, read: 574.8 MiB/s, write: 78.7 MiB/s
INFO:  45% (226.4 GiB of 500.0 GiB) in 2m 11s, read: 2.0 GiB/s, write: 0 B/s
INFO:  46% (234.4 GiB of 500.0 GiB) in 2m 15s, read: 2.0 GiB/s, write: 0 B/s
INFO:  48% (242.7 GiB of 500.0 GiB) in 2m 18s, read: 2.8 GiB/s, write: 0 B/s
INFO:  49% (247.3 GiB of 500.0 GiB) in 2m 21s, read: 1.5 GiB/s, write: 0 B/s
INFO:  50% (254.7 GiB of 500.0 GiB) in 2m 24s, read: 2.5 GiB/s, write: 600.0 KiB/s
INFO:  51% (259.9 GiB of 500.0 GiB) in 2m 27s, read: 1.7 GiB/s, write: 0 B/s
INFO:  53% (265.3 GiB of 500.0 GiB) in 2m 30s, read: 1.8 GiB/s, write: 106.7 KiB/s
INFO:  54% (272.3 GiB of 500.0 GiB) in 2m 34s, read: 1.8 GiB/s, write: 112.0 KiB/s
INFO:  55% (279.7 GiB of 500.0 GiB) in 2m 37s, read: 2.4 GiB/s, write: 373.3 KiB/s
INFO:  57% (286.5 GiB of 500.0 GiB) in 2m 40s, read: 2.3 GiB/s, write: 13.3 KiB/s
INFO:  58% (291.5 GiB of 500.0 GiB) in 2m 43s, read: 1.7 GiB/s, write: 0 B/s
INFO:  59% (298.9 GiB of 500.0 GiB) in 2m 46s, read: 2.5 GiB/s, write: 21.3 KiB/s
INFO:  60% (303.8 GiB of 500.0 GiB) in 2m 49s, read: 1.6 GiB/s, write: 0 B/s
INFO:  62% (313.0 GiB of 500.0 GiB) in 2m 53s, read: 2.3 GiB/s, write: 0 B/s
INFO:  64% (321.9 GiB of 500.0 GiB) in 2m 56s, read: 2.9 GiB/s, write: 0 B/s
INFO:  65% (325.8 GiB of 500.0 GiB) in 3m 5s, read: 439.0 MiB/s, write: 106.9 MiB/s
INFO:  66% (333.3 GiB of 500.0 GiB) in 3m 8s, read: 2.5 GiB/s, write: 0 B/s
INFO:  67% (339.1 GiB of 500.0 GiB) in 3m 11s, read: 2.0 GiB/s, write: 106.7 KiB/s
INFO:  68% (343.8 GiB of 500.0 GiB) in 3m 14s, read: 1.6 GiB/s, write: 0 B/s
INFO:  70% (350.3 GiB of 500.0 GiB) in 3m 17s, read: 2.2 GiB/s, write: 64.0 KiB/s
INFO:  71% (355.2 GiB of 500.0 GiB) in 3m 21s, read: 1.2 GiB/s, write: 95.0 KiB/s
INFO:  72% (362.4 GiB of 500.0 GiB) in 3m 24s, read: 2.4 GiB/s, write: 725.3 KiB/s
INFO:  73% (367.1 GiB of 500.0 GiB) in 3m 27s, read: 1.5 GiB/s, write: 0 B/s
INFO:  74% (375.0 GiB of 500.0 GiB) in 3m 30s, read: 2.6 GiB/s, write: 0 B/s
INFO:  75% (379.5 GiB of 500.0 GiB) in 3m 33s, read: 1.5 GiB/s, write: 104.0 KiB/s
INFO:  77% (386.4 GiB of 500.0 GiB) in 3m 37s, read: 1.7 GiB/s, write: 0 B/s
INFO:  78% (394.3 GiB of 500.0 GiB) in 3m 40s, read: 2.6 GiB/s, write: 22.7 KiB/s
INFO:  79% (399.1 GiB of 500.0 GiB) in 3m 43s, read: 1.6 GiB/s, write: 0 B/s
INFO:  81% (406.4 GiB of 500.0 GiB) in 3m 47s, read: 1.8 GiB/s, write: 0 B/s
INFO:  82% (413.7 GiB of 500.0 GiB) in 3m 50s, read: 2.4 GiB/s, write: 0 B/s
INFO:  83% (418.7 GiB of 500.0 GiB) in 3m 53s, read: 1.7 GiB/s, write: 0 B/s
INFO:  85% (426.4 GiB of 500.0 GiB) in 3m 56s, read: 2.6 GiB/s, write: 0 B/s
INFO:  86% (430.1 GiB of 500.0 GiB) in 3m 59s, read: 1.2 GiB/s, write: 12.3 MiB/s
INFO:  87% (435.9 GiB of 500.0 GiB) in 4m 19s, read: 301.6 MiB/s, write: 92.2 MiB/s
INFO:  88% (440.2 GiB of 500.0 GiB) in 4m 22s, read: 1.4 GiB/s, write: 21.3 KiB/s
INFO:  89% (446.8 GiB of 500.0 GiB) in 4m 26s, read: 1.6 GiB/s, write: 0 B/s
INFO:  90% (450.7 GiB of 500.0 GiB) in 4m 41s, read: 267.0 MiB/s, write: 81.4 MiB/s
INFO:  91% (457.2 GiB of 500.0 GiB) in 4m 45s, read: 1.6 GiB/s, write: 0 B/s
INFO:  92% (463.7 GiB of 500.0 GiB) in 4m 49s, read: 1.6 GiB/s, write: 32.0 KiB/s
INFO:  93% (465.2 GiB of 500.0 GiB) in 4m 52s, read: 495.4 MiB/s, write: 58.3 MiB/s
INFO:  94% (470.7 GiB of 500.0 GiB) in 5m 12s, read: 282.8 MiB/s, write: 100.9 MiB/s
INFO:  95% (477.0 GiB of 500.0 GiB) in 5m 16s, read: 1.6 GiB/s, write: 0 B/s
INFO:  96% (481.5 GiB of 500.0 GiB) in 5m 19s, read: 1.5 GiB/s, write: 0 B/s
INFO:  97% (485.5 GiB of 500.0 GiB) in 5m 36s, read: 236.9 MiB/s, write: 104.2 MiB/s
INFO:  98% (491.7 GiB of 500.0 GiB) in 5m 40s, read: 1.6 GiB/s, write: 0 B/s
INFO:  99% (495.9 GiB of 500.0 GiB) in 5m 43s, read: 1.4 GiB/s, write: 0 B/s
INFO: 100% (500.0 GiB of 500.0 GiB) in 5m 45s, read: 2.0 GiB/s, write: 0 B/s
INFO: backup is sparse: 489.58 GiB (97%) total zero data
INFO: transferred 500.00 GiB in 345 seconds (1.4 GiB/s)
INFO: archive file size: 3.44GB
INFO: prune older backups with retention: keep-daily=1, keep-last=2, keep-weekly=1
INFO: removing backup 'backup_drive:backup/vzdump-qemu-104-2022_01_31-23_15_07.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 104 (00:05:47)
INFO: Backup finished at 2022-02-02 06:45:43
INFO: Backup job finished successfully
TASK OK

Total time 6 m 12.3 s

I did not even notice any stop on my websites :) it was so super fast.

froze, thawed and started backup right away



@oguz is this going to get fixed next update? or will I have to redo the fix if any updates happen?


This is great news, I am so happy. I wish anyone else finds this will help them
until you can make a change and update for everyone

Thank you Thank You so much

Kind Regards
Spiro
 
Spirog said:
YOU ARE AWESOME!!!!!!!!!!!!!!!!!!!!

you found the Bug. and now its fixed..
Click to expand...
great!
Spirog said:
@oguz is this going to get fixed next update? or will I have to redo the fix if any updates happen?
Click to expand...
we'll see about a fix, this was more of a test (still need to find the root cause and do some testing for different setups, that's why i asked for CPU model and kernel version too) :)

your CPU is pretty old, so i was thinking that could be the issue here. [0]

most likely we will make an option to enable/disable smm on a VM basis, so that it can be tweaked when the hardware is too old/buggy ;)

in the meantime if there's an update on qemu-server package you'd have to reapply this fix.

[0]: https://ark.intel.com/content/www/u...n-processor-e52680-v4-35m-cache-2-40-ghz.html
 
  • Like
Reactions: Spirog
oguz said:
great!

we'll see about a fix, this was more of a test (still need to find the root cause and do some testing for different setups, that's why i asked for CPU model and kernel version too) :)

your CPU is pretty old, so i was thinking that could be the issue here. [0]

oguz said:
most likely we will make an option to enable/disable smm on a VM basis, so that it can be tweaked when the hardware is too old/buggy ;)

in the meantime if there's an update on qemu-server package you'd have to reapply this fix.
Click to expand...
Click to expand...

Last Question please?

- How will I know if there is an update - option ? or fix for my kind of server?

because if the system updates by itself my websites will be down without my knowledge from 3am until I wake up and check them?

( or will this only update if I do apt-get update - apt-get dist-upgrade etc.

thanks again
Spiro
 
Spirog said:
- How will I know if there is an update - option ? or fix for my kind of server?

because if the system updates by itself my websites will be down without my knowledge from 3am until I wake up and check them?
Click to expand...
system shouldn't upgrade by itself in the default configuration :)

for checking what's going to be upgraded, you can see it on the GUI, or just do apt update && apt list --upgradable to get a list ;)

upgrades won't happen unless you do it.
 
  • Like
Reactions: Spirog
@oguz

I spoke too soon. testing again from the actual VM

in GUI i slelct AlamLinux VM go to backup select backup now and its been stuck here for over 5 mins now?

before I did scheduled Backup and it worked ok?
now this one is stuck, I wonder why ?



Code: 
INFO: starting new backup job: vzdump 104 --compress zstd --remove 0 --mode snapshot --node proxmox --storage backup_drive
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2022-02-02 07:13:41
INFO: status = running
INFO: VM Name: AlmaLinux
INFO: include disk 'scsi0' 'local-lvm:vm-104-disk-0' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/backup_drive/dump/vzdump-qemu-104-2022_02_02-07_13_41.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
 
i Stopped it at 7m 22s it was frozen and websites were down ? :(

Code: 
INFO: starting new backup job: vzdump 104 --compress zstd --remove 0 --mode snapshot --node proxmox --storage backup_drive
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2022-02-02 07:13:41
INFO: status = running
INFO: VM Name: AlmaLinux
INFO: include disk 'scsi0' 'local-lvm:vm-104-disk-0' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/backup_drive/dump/vzdump-qemu-104-2022_02_02-07_13_41.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
closing with read buffer at /usr/share/perl5/IO/Multiplex.pm line 927.
ERROR: interrupted by signal
INFO: issuing guest-agent 'fs-thaw' command
 
seems like a different issue? is there any error or indication that the VM is crashing?

the fs-freeze just takes some time, and if you interrupt the backup that makes it only worse ;)
 
@oguz I stopped the backup and now it says AlmaLinux VM is Locked, all websites are down and it wont let me stop or start the VM,
it give error: VM is Locked (Backup_)
 
qm unlock VMID (change VMID to 104 for your VM)
 
oguz said:
qm unlock VMID (change VMID to 104 for your VM)
Click to expand...
ok I restart VM now... I did not see any errors, just when freeze it was 7 mins. and yesterday it was total 1 plus hours before it did finished
today was 6 mons with your fix on schedule backup

here is yesterdays log
Code: 
INFO: starting new backup job: vzdump 104 --node proxmox --storage backup_drive --remove 0 --compress zstd --mode snapshot
INFO: Starting Backup of VM 104 (qemu)
INFO: Backup started at 2022-01-31 23:15:07
INFO: status = running
INFO: VM Name: AlmaLinux
INFO: include disk 'scsi0' 'local-lvm:vm-104-disk-0' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/backup_drive/dump/vzdump-qemu-104-2022_01_31-23_15_07.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
ERROR: VM 104 qmp command 'guest-fsfreeze-freeze' failed - got timeout
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 104 qmp command 'guest-fsfreeze-thaw' failed - got timeout
INFO: started backup task '7c1bafde-a78a-4e9a-8b90-2b9c4d0fa40d'
INFO: resuming VM again
INFO:   2% (11.4 GiB of 500.0 GiB) in 3s, read: 3.8 GiB/s, write: 383.8 MiB/s
INFO:   3% (18.0 GiB of 500.0 GiB) in 6s, read: 2.2 GiB/s, write: 0 B/s
INFO:   4% (22.8 GiB of 500.0 GiB) in 9s, read: 1.6 GiB/s, write: 0 B/s
INFO:   7% (37.0 GiB of 500.0 GiB) in 12s, read: 4.7 GiB/s, write: 1.8 MiB/s
INFO:   8% (43.0 GiB of 500.0 GiB) in 15s, read: 2.0 GiB/s, write: 190.7 KiB/s
INFO:   9% (49.5 GiB of 500.0 GiB) in 18s, read: 2.2 GiB/s, write: 0 B/s
INFO:  11% (56.0 GiB of 500.0 GiB) in 21s, read: 2.2 GiB/s, write: 0 B/s
INFO:  12% (62.7 GiB of 500.0 GiB) in 24s, read: 2.2 GiB/s, write: 0 B/s
INFO:  16% (80.8 GiB of 500.0 GiB) in 27s, read: 6.0 GiB/s, write: 12.0 KiB/s
INFO:  19% (99.7 GiB of 500.0 GiB) in 30s, read: 6.3 GiB/s, write: 0 B/s
INFO:  22% (111.3 GiB of 500.0 GiB) in 33s, read: 3.9 GiB/s, write: 30.8 MiB/s
INFO:  23% (116.4 GiB of 500.0 GiB) in 41s, read: 651.2 MiB/s, write: 83.2 MiB/s
INFO:  24% (123.5 GiB of 500.0 GiB) in 44s, read: 2.4 GiB/s, write: 0 B/s
INFO:  26% (130.6 GiB of 500.0 GiB) in 47s, read: 2.4 GiB/s, write: 0 B/s
INFO:  27% (137.7 GiB of 500.0 GiB) in 50s, read: 2.4 GiB/s, write: 0 B/s
INFO:  28% (144.7 GiB of 500.0 GiB) in 53s, read: 2.3 GiB/s, write: 2.3 MiB/s
INFO:  30% (151.9 GiB of 500.0 GiB) in 56s, read: 2.4 GiB/s, write: 0 B/s
INFO:  32% (163.5 GiB of 500.0 GiB) in 59s, read: 3.8 GiB/s, write: 0 B/s
INFO:  36% (182.2 GiB of 500.0 GiB) in 1m 2s, read: 6.3 GiB/s, write: 16.0 KiB/s
INFO:  37% (189.7 GiB of 500.0 GiB) in 1m 5s, read: 2.5 GiB/s, write: 0 B/s
INFO:  39% (196.9 GiB of 500.0 GiB) in 1m 8s, read: 2.4 GiB/s, write: 0 B/s
INFO:  40% (204.1 GiB of 500.0 GiB) in 1m 11s, read: 2.4 GiB/s, write: 0 B/s
INFO:  42% (211.3 GiB of 500.0 GiB) in 1m 14s, read: 2.4 GiB/s, write: 0 B/s
INFO:  43% (217.5 GiB of 500.0 GiB) in 1m 17s, read: 2.1 GiB/s, write: 13.3 MiB/s
INFO:  44% (220.6 GiB of 500.0 GiB) in 1m 24s, read: 454.0 MiB/s, write: 93.3 MiB/s
INFO:  45% (227.7 GiB of 500.0 GiB) in 1m 27s, read: 2.4 GiB/s, write: 0 B/s
INFO:  46% (234.9 GiB of 500.0 GiB) in 1m 30s, read: 2.4 GiB/s, write: 0 B/s
INFO:  48% (242.1 GiB of 500.0 GiB) in 1m 33s, read: 2.4 GiB/s, write: 0 B/s
INFO:  49% (249.2 GiB of 500.0 GiB) in 1m 36s, read: 2.4 GiB/s, write: 0 B/s
INFO:  51% (256.4 GiB of 500.0 GiB) in 1m 39s, read: 2.4 GiB/s, write: 600.0 KiB/s
INFO:  52% (263.6 GiB of 500.0 GiB) in 1m 42s, read: 2.4 GiB/s, write: 0 B/s
INFO:  54% (270.8 GiB of 500.0 GiB) in 1m 45s, read: 2.4 GiB/s, write: 0 B/s
INFO:  55% (278.0 GiB of 500.0 GiB) in 1m 48s, read: 2.4 GiB/s, write: 0 B/s
INFO:  57% (285.2 GiB of 500.0 GiB) in 1m 51s, read: 2.4 GiB/s, write: 13.3 KiB/s
INFO:  58% (292.4 GiB of 500.0 GiB) in 1m 54s, read: 2.4 GiB/s, write: 0 B/s
INFO:  59% (299.8 GiB of 500.0 GiB) in 1m 57s, read: 2.5 GiB/s, write: 0 B/s
INFO:  61% (307.0 GiB of 500.0 GiB) in 2m, read: 2.4 GiB/s, write: 0 B/s
INFO:  62% (314.2 GiB of 500.0 GiB) in 2m 3s, read: 2.4 GiB/s, write: 0 B/s
INFO:  63% (319.0 GiB of 500.0 GiB) in 2m 6s, read: 1.6 GiB/s, write: 0 B/s
INFO:  64% (323.8 GiB of 500.0 GiB) in 2m 9s, read: 1.6 GiB/s, write: 23.5 MiB/s
INFO:  65% (326.9 GiB of 500.0 GiB) in 2m 16s, read: 450.9 MiB/s, write: 127.4 MiB/s
INFO:  66% (334.0 GiB of 500.0 GiB) in 2m 19s, read: 2.4 GiB/s, write: 0 B/s
INFO:  68% (341.1 GiB of 500.0 GiB) in 2m 22s, read: 2.4 GiB/s, write: 0 B/s
INFO:  69% (348.3 GiB of 500.0 GiB) in 2m 25s, read: 2.4 GiB/s, write: 0 B/s
INFO:  71% (355.4 GiB of 500.0 GiB) in 2m 28s, read: 2.4 GiB/s, write: 0 B/s
INFO:  72% (362.5 GiB of 500.0 GiB) in 2m 31s, read: 2.4 GiB/s, write: 665.3 KiB/s
INFO:  73% (369.5 GiB of 500.0 GiB) in 2m 34s, read: 2.4 GiB/s, write: 0 B/s
INFO:  75% (376.6 GiB of 500.0 GiB) in 2m 37s, read: 2.4 GiB/s, write: 0 B/s
INFO:  76% (383.7 GiB of 500.0 GiB) in 2m 40s, read: 2.4 GiB/s, write: 0 B/s
INFO:  78% (391.3 GiB of 500.0 GiB) in 2m 43s, read: 2.5 GiB/s, write: 22.7 KiB/s
INFO:  79% (399.0 GiB of 500.0 GiB) in 2m 46s, read: 2.6 GiB/s, write: 0 B/s
INFO:  81% (406.3 GiB of 500.0 GiB) in 2m 49s, read: 2.4 GiB/s, write: 0 B/s
INFO:  82% (413.5 GiB of 500.0 GiB) in 2m 52s, read: 2.4 GiB/s, write: 0 B/s
INFO:  84% (420.9 GiB of 500.0 GiB) in 2m 55s, read: 2.5 GiB/s, write: 0 B/s
INFO:  85% (428.5 GiB of 500.0 GiB) in 2m 58s, read: 2.5 GiB/s, write: 0 B/s
INFO:  86% (430.2 GiB of 500.0 GiB) in 3m 1s, read: 574.0 MiB/s, write: 57.0 MiB/s
INFO:  87% (435.8 GiB of 500.0 GiB) in 3m 19s, read: 320.7 MiB/s, write: 95.1 MiB/s
INFO:  88% (443.0 GiB of 500.0 GiB) in 3m 22s, read: 2.4 GiB/s, write: 0 B/s
INFO:  89% (447.6 GiB of 500.0 GiB) in 3m 25s, read: 1.5 GiB/s, write: 25.2 MiB/s
INFO:  90% (450.9 GiB of 500.0 GiB) in 3m 38s, read: 261.3 MiB/s, write: 87.4 MiB/s
INFO:  91% (459.0 GiB of 500.0 GiB) in 3m 41s, read: 2.7 GiB/s, write: 0 B/s
INFO:  93% (465.1 GiB of 500.0 GiB) in 3m 44s, read: 2.0 GiB/s, write: 17.5 MiB/s
INFO:  94% (471.0 GiB of 500.0 GiB) in 4m 8s, read: 255.0 MiB/s, write: 88.7 MiB/s
INFO:  95% (478.5 GiB of 500.0 GiB) in 4m 11s, read: 2.5 GiB/s, write: 0 B/s
INFO:  96% (482.6 GiB of 500.0 GiB) in 4m 14s, read: 1.4 GiB/s, write: 30.9 MiB/s
INFO:  97% (485.1 GiB of 500.0 GiB) in 4m 30s, read: 162.6 MiB/s, write: 103.8 MiB/s
INFO:  98% (493.3 GiB of 500.0 GiB) in 4m 33s, read: 2.7 GiB/s, write: 0 B/s
INFO: 100% (500.0 GiB of 500.0 GiB) in 4m 36s, read: 2.2 GiB/s, write: 0 B/s
INFO: backup is sparse: 489.64 GiB (97%) total zero data
INFO: transferred 500.00 GiB in 276 seconds (1.8 GiB/s)
INFO: archive file size: 3.47GB
INFO: Finished Backup of VM 104 (01:04:46)
INFO: Backup finished at 2022-02-01 00:19:53
INFO: Backup job finished successfully
TASK OK

you can see start and finish times over 1 hour for 3.79gb backup

also there is some errors

Code: 
]INFO: starting new backup job: vzdump 104 --node proxmox --storage backup_drive --remove 0 --compress zstd --mode snapshot

INFO: Starting Backup of VM 104 (qemu)

INFO: Backup started at 2022-01-31 23:15:07

INFO: status = running

INFO: VM Name: AlmaLinux

INFO: include disk 'scsi0' 'local-lvm:vm-104-disk-0' 500G

INFO: backup mode: snapshot

INFO: ionice priority: 7

INFO: creating vzdump archive '/mnt/backup_drive/dump/vzdump-qemu-104-2022_01_31-23_15_07.vma.zst'

INFO: issuing guest-agent 'fs-freeze' command

ERROR: VM 104 qmp command 'guest-fsfreeze-freeze' failed - got timeout

INFO: issuing guest-agent 'fs-thaw' command

ERROR: VM 104 qmp command 'guest-fsfreeze-thaw' failed - got timeout

INFO: started backup task '7c1bafde-a78a-4e9a-8b90-2b9c4d0fa40d'

INFO: resuming VM again

sorry and thanks
 
here is a pdf, i could not copy the error from the noVCN when I opened VCN screen from yesterday. not sure if that pertains to the freeze error ( stalling)
 

Attachments

  • QEMU (AlmaLinux) - noVNC.pdf
    63.6 KB · Views: 8
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back