Hi, I'd appreciate some helping figuring out why the backup on two of my VMs failed. I've attached a copy of the backup log. The two VMs that failed were not running and the two that succeeded were running. The backup job is configured to use Snapshot mode on all the VMs. I see that it tries to start the two VMs that failed before running the backup. Why is that? Is that a result of snapshot mode? I had expected any VM that wasn't running would easily backup. I also see the Cannot allocate memory errors and I'm not sure what that pertains to because I can manually start the VMs just fine and run all 4 simultaneously. Then I'm only using 8GB of 23.5GB total RAM and my backup destination also has 2.5TB free.
Interpreting failed backup log
- Thread starter MattE
- Start date
Are you using dynamic (balloon) memory for the VM configs, and/or combination with ZFS pool?
This might help:
https://forum.proxmox.com/threads/kvm-cannot-allocate-memory.13914/#post-118664
The solution in was make sure the sum total of all VM config memory (not actual used memory) did not exceed physical memory.
So if you have 4VMs each with dynamic (balloon) memory 4GB-8GB, that's 32GB total, which is over your 24GB physical. If you are using ZFS pool, expect that to take additional memory.
It was an older post, perhaps try dist-upgrade to the latest version of ProxMox, but make sure to full backup your VM's first (stopped), and also the data within those VM's.
This might help:
https://forum.proxmox.com/threads/kvm-cannot-allocate-memory.13914/#post-118664
The solution in was make sure the sum total of all VM config memory (not actual used memory) did not exceed physical memory.
So if you have 4VMs each with dynamic (balloon) memory 4GB-8GB, that's 32GB total, which is over your 24GB physical. If you are using ZFS pool, expect that to take additional memory.
It was an older post, perhaps try dist-upgrade to the latest version of ProxMox, but make sure to full backup your VM's first (stopped), and also the data within those VM's.
Are you backing up to a local or external hard drive? I would run smartctl (short offline test) on your backup drives to check for possible drive errors. If possible, can you try adding another hard drive locally, and see if it backs up fine?
https://www.thomas-krenn.com/en/wiki/SMART_tests_with_smartctl
https://www.thomas-krenn.com/en/wiki/SMART_tests_with_smartctl
That's good to hear. BTW this post might better explain the snapshot vs stop modes when doing full backups:
https://forum.proxmox.com/threads/backup-vs-snapshot.24332/#post-122684
also
https://wiki.qemu.org/Features/Snapshots
https://pve.proxmox.com/wiki/Backup_and_Restore
"Creating snapshots through the QEMU live snapshot commands allow for incremental guest image files to be created, with each image file containing differences from its parent backing file."
As I understand it, the first time snapshot is run it does a full backup, and then subsequent snapshots are the "differential" differences between the last full backup (parent backing file) and the current state of the VM. For example, I have 1 full backup (via first snapshot), and I do another snapshot before updating/adding software inside the VM (I'll label the new snapshot "current state of my VM before installing new updates"). Now after I updated software inside the VM, it turns out the update corrupted it, and now I want to go back to previous state before installing updates. So I choose the last snapshot and click restore, and is restored to the previous state when I took the snapshot.
Now if you have multiple snapshots, you can go back to any point in time when the snapshot was made. But beware this will be like a time machine, and restore it to that snapshot's state.
BTW I always recommend that snapshots/vm backups alone by themselves are not a good backup strategy. I recommend a multi-prong approach to backups
Backup data within the VM itself, backup the VM itself, and use 321 backup strategies for both:
https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
cheers!
https://forum.proxmox.com/threads/backup-vs-snapshot.24332/#post-122684
also
https://wiki.qemu.org/Features/Snapshots
https://pve.proxmox.com/wiki/Backup_and_Restore
"Creating snapshots through the QEMU live snapshot commands allow for incremental guest image files to be created, with each image file containing differences from its parent backing file."
As I understand it, the first time snapshot is run it does a full backup, and then subsequent snapshots are the "differential" differences between the last full backup (parent backing file) and the current state of the VM. For example, I have 1 full backup (via first snapshot), and I do another snapshot before updating/adding software inside the VM (I'll label the new snapshot "current state of my VM before installing new updates"). Now after I updated software inside the VM, it turns out the update corrupted it, and now I want to go back to previous state before installing updates. So I choose the last snapshot and click restore, and is restored to the previous state when I took the snapshot.
Now if you have multiple snapshots, you can go back to any point in time when the snapshot was made. But beware this will be like a time machine, and restore it to that snapshot's state.
BTW I always recommend that snapshots/vm backups alone by themselves are not a good backup strategy. I recommend a multi-prong approach to backups
Backup data within the VM itself, backup the VM itself, and use 321 backup strategies for both:
https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
cheers!
I think in this case, you are referring to the VM's snapshot feature, whereas in Backup and Restore, snapshot mode just means the VM backup is done live instead of stopping and restarting the VM.
