Proxmox VM automation

[EllyMae]

I have a few VMs where I would like to automate a few repetitive tasks, generally centered around taking backups. I want to issue commands to the VM that will shut down applications, perform the backups using proxmox-backup-client, and then start the applications. I would also like to schedule this process.

I realize that I can write scripts that do these things, and that I can schedule the scripts. But that's not what I want. I want to keep the coding non-existent, if possible. Declarative configurations I can handle, however.

I have researched some possible products, and have done some digging into Ansible. But these seem to be designed mainly for deploying systems and managing their operational states. I get the feeling that it would be a stretch for Ansible to handle my simple example. The VMs are already up and running, so I am not at all interested in that aspect. Operation status of individual processes is part of the exercise. But I need to monitor logs to verify successful shutdown and startup of these processes, and of the backup job itself.

I'm looking for something that is relatively easy to get going without a multitude of config files and scripts scattered all over the place. Maybe Ansible is the solution, but I have a feeling that this is not what it is designed for. Maybe you can convince me otherwise. Other than that, I would really appreciate your recommendations of other ways to do this. Thanks!

 

[Lukas Wagner]

I would say Ansible is the way to go if you don't want to write any bash scripts. Since all the tasks that you have described appear to be within the VM itself, you don't need the Ansible Plugin for Proxmox -- you can treat the VM in the same way as any other (physical) host.

However: Since you mentioned that you want to stop applications, take a backup and start the application again: Why don't you just take a backup of the whole VM? Because that is easily automated from the Proxmox VE GUI alone
If you want to use the 'snaphot' backup mode, you should consider installing and enabling the qemu-guest-agent. The guest agent freezes the filesystem before a backup in order to improve consistency.

 

[EllyMae]

However: Since you mentioned that you want to stop applications, take a backup and start the application again: Why don't you just take a backup of the whole VM? Because that is easily automated from the Proxmox VE GUI alone

I've tried to do it this way. But the usual way of doing backups in Proxmox has a couple of drawbacks in my case. These VMs have two separate disks: one for the system/home and the other is for (a large amount of) data. If I backed up using the usual methods, then when it comes time to restore the system drive, I would also have to unnecessarily restore the large data drive along with it. This is inconvenient, slow and incorrect. There have been others posting in here about this very issue, and Proxmox staff pushed back.

https://forum.proxmox.com/threads/separate-backups-by-disks.126594/
(There is another one that I submitted, but I'm unable to find it now.)

So, I am forced to find another solution. I am backing up the system drive using the GUI, and also excluding the data drive. Then I back up the data drive by the method I described in the beginning of this thread. This also has the advantage of getting a better quality backup by not having the applications running during the backup. Even with the fsfreeze, there is no guarantee that the applicaions are all completely between transactions at that moment. Shutting them down gives me this guarantee.

By the way, I do ensure that qemu-guest-agent is installed and running on all of my VMs. I may be wrong in my assumption that there is no request to the applications to finish and halt transaction updates. But, I have not come across any mention that this happens.

Thanks for your reply. I'll continue my learning of Ansible.

 

[LnxBil]

So, I am forced to find another solution.

For such a setup we use ZFS inside of the VM and are able to rollback to a specified snapshot we took in no time. Maybe this could also work for you?

 

[EllyMae]

Thanks for the suggestion.

Here is an old adage: Snapshots are no substitute for backups.

Here, in a Microsoft context, is why that is:
https://www.altaro.com/backup-dr/ch.../#Why_Checkpoints_Are_Not_Backup_Replacements
Everything there still applies to my case if you translate to a Proxmox context.

 

[LnxBil]

Here is an old adage: Snapshots are no substitute for backups.

Here, in a Microsoft context, is why that is:
https://www.altaro.com/backup-dr/ch.../#Why_Checkpoints_Are_Not_Backup_Replacements
Everything there still applies to my case if you translate to a Proxmox context.

Therefore you replicate the ZFS snapshots to another machine.

 

[EllyMae]

Therefore you replicate the ZFS snapshots to another machine.

These VMs don't have ZFS, and I'm not going to convert them. Did I say there is a large amount of data? But let's say that I did have a ZFS snapshot. How would I replicate it into Proxmox Backup Server. Did I say that I was using that?

 

[Dunuin]

These VMs don't have ZFS, and I'm not going to convert them. Did I say there is a large amount of data? But let's say that I did have a ZFS snapshot. How would I replicate it into Proxmox Backup Server. Did I say that I was using that?

You can incrementally replicate/sync/copy your ZFS datasets between pools or even hosts by piping "zfs send" into "zfs recv": https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html#gfwqb

Problem with the proxmox-backup-client is that you are limited to Debian VMs/LXCs...

 

[LnxBil]

These VMs have two separate disks: one for the system/home and the other is for (a large amount of) data. If I backed up using the usual methods, then when it comes time to restore the system drive, I would also have to unnecessarily restore the large data drive along with it. This is inconvenient, slow and incorrect.

My suggestion ZFS was with regard to this requirement. You don't need to involve a backup if you only want to rollback some stuff. You can still use PBS and do not have to change anything, yet you could of course use ZFS directly on PBS, many people - me included - do.