Backup with STOP - seems VM is still running during backup?

[invisible999]

Hello everyone,

I configured backup schedule with STOP. My expectation is that VM is shutdown by the script, remains shutdown until backup is completed and after completion it is started.

However, looking at logs seems that VM is still running while during backup. Is this right? I separated important (from my point of view) entries from the log.

May 21 01:30:08 proxmox pvescheduler[225856]: <root@pam> starting task UPID:proxmox:00037241:0096AEE7:664C5B90:vzdump:100:root@pam:
May 21 01:30:08 proxmox pvescheduler[225857]: INFO: starting new backup job: vzdump 100 --quiet 1 --node proxmox --compress zstd --prune-backups 'keep-last=3' --storage local --mode stop --notes-template '{{guestname}}' --fleecing 0

May 21 01:30:08 proxmox pvescheduler[225857]: INFO: Starting Backup of VM 100 (qemu)

May 21 01:30:11 proxmox qm[225865]: <root@pam> starting task UPID:proxmox:0003724E:0096B021:664C5B93:qmshutdown:100:root@pam:

May 21 01:30:11 proxmox qm[225870]: shutdown VM 100: UPID:proxmox:0003724E:0096B021:664C5B93:qmshutdown:100:root@pam:

May 21 01:30:27 proxmox systemd-udevd[347]: /usr/lib/udev/rules.d/60-gasket-dkms.rules:1 Unknown group 'apex', ignoring
May 21 01:30:28 proxmox kernel: tap100i0: left allmulticast mode
May 21 01:30:28 proxmox kernel: vmbr0: port 2(tap100i0) entered disabled state
May 21 01:30:28 proxmox qmeventd[595]: read: Connection reset by peer
May 21 01:30:29 proxmox kernel: usb 1-3: reset full-speed USB device number 2 using xhci_hcd
May 21 01:30:29 proxmox kernel: cp210x 1-3:1.0: cp210x converter detected
May 21 01:30:29 proxmox kernel: usb 1-3: cp210x converter now attached to ttyUSB0
May 21 01:30:29 proxmox kernel: usb 1-9: reset full-speed USB device number 3 using xhci_hcd
May 21 01:30:29 proxmox kernel: Bluetooth: hci0: Firmware revision 0.1 build 19 week 44 2021
May 21 01:30:29 proxmox kernel: Bluetooth: hci0: Reading supported features failed (-16)
May 21 01:30:29 proxmox kernel: Bluetooth: hci0: Error reading debug features
May 21 01:30:29 proxmox kernel: Bluetooth: hci0: HCI LE Coded PHY feature bit is set, but its usage is not supported.
May 21 01:30:29 proxmox systemd[221183]: Reached target bluetooth.target - Bluetooth.
May 21 01:30:29 proxmox systemd[1]: Starting systemd-rfkill.service - Load/Save RF Kill Switch Status...
May 21 01:30:29 proxmox systemd[1]: Reached target bluetooth.target - Bluetooth Support.
May 21 01:30:29 proxmox systemd[1]: Started systemd-rfkill.service - Load/Save RF Kill Switch Status.
May 21 01:30:29 proxmox systemd[1]: 100.scope: Deactivated successfully.
May 21 01:30:29 proxmox systemd[1]: 100.scope: Consumed 4h 2min 17.738s CPU time.
May 21 01:30:29 proxmox qm[225865]: <root@pam> end task UPID:proxmox:0003724E:0096B021:664C5B93:qmshutdown:100:root@pam: OK
May 21 01:30:30 proxmox systemd[1]: Started 100.scope.
May 21 01:30:30 proxmox systemd-udevd[347]: /usr/lib/udev/rules.d/60-gasket-dkms.rules:1 Unknown group 'apex', ignoring
May 21 01:30:31 proxmox qmeventd[225943]: Starting cleanup for 100
May 21 01:30:31 proxmox qmeventd[225943]: trying to acquire lock...
May 21 01:30:33 proxmox kernel: tap100i0: entered promiscuous mode
May 21 01:30:33 proxmox kernel: vmbr0: port 2(tap100i0) entered blocking state
May 21 01:30:33 proxmox kernel: vmbr0: port 2(tap100i0) entered disabled state
May 21 01:30:33 proxmox kernel: tap100i0: entered allmulticast mode
May 21 01:30:33 proxmox kernel: vmbr0: port 2(tap100i0) entered blocking state
May 21 01:30:33 proxmox kernel: vmbr0: port 2(tap100i0) entered forwarding state
May 21 01:30:33 proxmox qmeventd[225943]:  OK

May 21 01:30:33 proxmox qmeventd[225943]: vm still running

May 21 01:30:33 proxmox kernel: cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0
May 21 01:30:33 proxmox kernel: cp210x 1-3:1.0: device disconnected
May 21 01:30:33 proxmox systemd-udevd[347]: /usr/lib/udev/rules.d/60-gasket-dkms.rules:1 Unknown group 'apex', ignoring
May 21 01:30:33 proxmox systemd[221183]: Stopped target bluetooth.target - Bluetooth.
May 21 01:30:33 proxmox systemd[1]: Stopped target bluetooth.target - Bluetooth Support.
May 21 01:30:38 proxmox systemd[1]: systemd-rfkill.service: Deactivated successfully.
May 21 01:30:44 proxmox kernel: usb 1-9: reset full-speed USB device number 3 using xhci_hcd
May 21 01:30:44 proxmox systemd-udevd[347]: /usr/lib/udev/rules.d/60-gasket-dkms.rules:1 Unknown group 'apex', ignoring
May 21 01:30:45 proxmox kernel: usb 1-3: reset full-speed USB device number 2 using xhci_hcd

May 21 01:34:17 proxmox pvescheduler[225857]: INFO: Finished Backup of VM 100 (00:04:09)

May 21 01:34:17 proxmox pvescheduler[225857]: INFO: Backup job finished successfully

 

[UdoB]

My expectation is that VM is shutdown by the script, remains shutdown until backup is completed

That is the magic of a KVM/QEMU snapshot

The VM is stopped for a short moment. The state of the whole VM is catched/frozen at that moment. Then the actual backup starts and in the very same moment the VM is allowed to start again and continue its work

Modifications in RAM or on a virtual disk are monitored and "catched". The mechanism ensures that the data from the moment of that early snapshot is written into the backup, not the now-changed data from the live RAM/storage.

Some things work better virtualized than "in the real world" ;-)

 

[fabian]

like Udo said, for VMs, the backup always happens from within qemu.

the difference between the modes is just how consistency is ensured:
- stop: highest form of consistency, the VM is shutdown, then started again in a paused state (just the Qemu process is started, but the VM is not executed yet) to start the backup. once the backup is running, the VM can be resumed with affecting consistency
- suspend: instead of shutting down and starting paused, the VM is suspended (execution is paused) to ensure consistency. again, the VM is only suspended to cover the start of the backup, after that point it is running again
- snapshot: just a freeze / thaw surrounding the backup start is executed to ensure disk consistency

the difference between suspend and snapshot is neglible, but snapshot is faster, so usually you'd either choose snapshot or stop.

for containers, there is more difference between the modes, and snapshot mode requires support from the storage layer.

 

[leesteken]

This feels as a continuation of this thread of OP: https://forum.proxmox.com/threads/h...hedule-option-with-proxmox.147423/post-666254

 

[UdoB]

This feels as a continuation

History repeats...

We have so many topics that come up again and again that keeping an overview of the current loops gets difficult.

 

[invisible999]

like Udo said, for VMs, the backup always happens from within qemu.

the difference between the modes is just how consistency is ensured:
- stop: highest form of consistency, the VM is shutdown, then started again in a paused state (just the Qemu process is started, but the VM is not executed yet) to start the backup. once the backup is running, the VM can be resumed with affecting consistency
- suspend: instead of shutting down and starting paused, the VM is suspended (execution is paused) to ensure consistency. again, the VM is only suspended to cover the start of the backup, after that point it is running again
- snapshot: just a freeze / thaw surrounding the backup start is executed to ensure disk consistency

the difference between suspend and snapshot is neglible, but snapshot is faster, so usually you'd either choose snapshot or stop.

for containers, there is more difference between the modes, and snapshot mode requires support from the storage layer.

Hey Fabian, thanks for explaining. Let me ask couple of questions to understand how this works - this is my first time dealing with Qemu/KVM, >15 years I was dealing with ESXi/vSphere.

What is 'VM is shutdown' in this context - what exactly does it mean? Does this mean that VM's OS is completely stopped and RAM is freed/flushed? Or some part of VM's OS continues running all this time? Is there concept 'copy on write' here at any time?

Thanks.

 

[fabian]

shutdown means shutdown as in completely stopped. then a new VM process is started, but before execution of the guest starts, the backup task is started, and only when that is set up, the guest starts executing (booting and then running). from that point on, the backed-up disks operate under copy-on-write semantics - that means any part of the disk that hasn't been backed up yet but is written to by the guest will first be copied to the backup target before the guest write goes through to the disk.

 

[invisible999]

Good explanation. Thanks!