Multiple VMs on PVE - appear to backup 1-at-a-time

[bearhntr]

Is this true? I have 4 PVE servers, and want to stagger their backup start times. All of them have 1GB Ethernet connection.

I was running a test to backup 5 VMs on one server - 1 active VM the others are not running. It appears that it backs them up 1-at-a-time (i.e. 100, then 101, then 102, etc.)?

Is there no way to change this so that more than 1 at a time can backup? This is my home-lab, and even if backing up nightly for all the VMs, it will take 8-10 hours (based on how long it took just one VM to backup (1-100GB drive and 1-60GB drive).

 

[tcabernoch]

Look, I don't know if this behavior can be changed, but what you are seeing is 100% normal and _the_way_it_should_be_.
I guess it probably matters if you are using a PBS target or a NAS target.
But the same behavior comes out in both, for the same reason.

PBS backups work really strange to what you are probly familiar with. They sorta lock the VM's disk, and any new data that happens during the backup needs to write both locally and in the backup target ... so before the VM can move on to the next thing, it has to wait for that write to happen. It's complex, but particularly earlier in a PBS backup, it can cause freezing in the VM. So, you want your backup target to accept disk writes as fast as possible, and you wouldn't want to do more than one at a time because of this extreme sensitivity.

For a NAS target, I don't know if its as extremely sensitive, but when you start trying to dump multiple streams of data on a NAS, you start getting performance issues, failed backups, etc. So for this type as well, it runs one backup per PVE server at a time.

Now can you change this behavior? I don't know. I do not want to.

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=3347

 

[bearhntr]

Thanks for the information. I understand the reasoning behind it and the principle of 'shadow' copying, etc. I am guessing that I should not have jobs from 4 separate PVE servers scheduled at the same time, either?

Is there a way to set an 'order' of the way they backup from one PVE? I basically set it up under DataCenter >> Backup and used the defaults to do a Daily backup - with "keep-last=3" - I also want to set a 'monthly' for the 1st of each month with keeping 2 (maybe 3) of those as well. Can I do both?

 

[Bingo600]

Is there a way to set an 'order' of the way they backup from one PVE? I basically set it up under DataCenter >> Backup and used the defaults to do a Daily backup - with "keep-last=3" - I also want to set a 'monthly' for the 1st of each month with keeping 2 (maybe 3) of those as well. Can I do both?

I'm a newbie ...
But i think i read that you should set all backup strategies in "ONE ruleset".
I read that if you use several rulesets, then they will interfere with the wished retention periods. Lowest retention will "win".

Ie. day ruleset would purge week & month , and week ruleset would purge month.

 

[Falk R.]

You can set last backups, daily, weekly and so on up to yearly all in a purge rule.
If the backups take that long, they probably go to a NAS. Each PVE only does 1 VM at a time but if all hosts are in a job, all 4 hosts do it in parallel. This can also slow down a NAS, as it generates a lot of random traffic. This is not a problem with the PBS, as every I/O is random anyway, but you only make a full once and then incremental backups, from which synthetic fulls are generated.

 

[tcabernoch]

Whether you should run 4 backups at once is your call. Log into a VM that's being actively backed up. See what the performance is like. If its not good enough for you, then don't run 4 backups at once.

As to staggering the backups, that's easy. Create 4 backup jobs. Restrict each to 1 server. Stagger the schedule.

And then your daily weekly monthly question ... that's a retention setting. Go into PBS. Click on the datastore. Look over on the tabs on the right. Second tab. See the Pruning job there? Edit it. Set your intended retention.
(And also set your Garbage Collection. And then go over to Verify and set that job too.)

It occurs to me that you probably don't know how PBS really works ... in that it works completely differently than any other backup system you've used. It's a huge database of file parts. The 'backup' such as it is, is a directory to those file parts. If it already has a file part, it doesn't take a second copy. No data is transferred, just metadata.

You don't do things like daily weekly monthly. You don't do incremental and full. Everything is a full, but it leverages the dedupe capabilities of PBS to run like an incremental instead. In order to understand this, you need to look at the logs for a backup. Make a brand new VM. Put a bunch of stuff on it. Use it. Make a brand new backup job. Run the job. Watch the logs. Chances are its gonna do a whole lot of reading, but a quite small, frequently zero, amount of data transfer. PBS's dedupe is unreal.

 

[bearhntr]

@tcabernoch

Thanks for the explanation. Yes, I am used to Windows Backup methodology. I have an older Server 2012 server with a 12TB RAID5 setup with 2 HSP drives. Not really using it any more - and will find another use for it 'someday' - it has all my music and DVD/BR collection ripped to it.

The speed of the VMs is not critical to me - as I would be doing the backups at night (while I am asleep).

Guess I will have to play with the daily vs monthly mentality I had with my old backup method.

 

[tcabernoch]

Of course, the concept of daily weekly monthly is still absolutely useful.

Its just a retention concept with PBS. It's how many day/week/month do you want to be able to step back in time.

In a normal backup system with a full backup and incrementals stacked on top of that, a weekly, monthly, or yearly backup was necessarily a new full backup and could stand on its own without incremental files. So for a normal backup system, these concepts mean you will create a new file. In order to do so, you will either do a full backup (read every bit on the box) or alternately a synthetic backup (where existing files are collapsed on the NAS to make a new 'full'.)

 

[Darkk]

I run my backups on my three PVE servers at home and they all run in parallel several times a day without much of a noticeable difference in VM and container's performance. The backups actually only take a few mins as it's doing it incrementally. The only time I've seen backups take bit longer if the VMs or containers are off then it takes awhile. Same when moving the VMs/CTs to another PVE host. If the VMs/CTs stay on all the time and on the same host then the backups shouldn't take too long.

I'm using ZFS for the VMs/CTs.

I do make use of groups for the VMs/CTs meaning I have a crticial, non-crticial, testing and not in use groups. I've setup backup jobs based on those groups to run either several times a day, once a day or a week depending how important or frequent the changes are.

This is also how I've set up on our production servers at work running two clusters with 7 nodes each. Works very well for our needs.

 

[bearhntr]

I run my backups on my three PVE servers at home and they all run in parallel several times a day without much of a noticeable difference in VM and container's performance. The backups actually only take a few mins as it's doing it incrementally. The only time I've seen backups take bit longer if the VMs or containers are off then it takes awhile. Same when moving the VMs/CTs to another PVE host. If the VMs/CTs stay on all the time and on the same host then the backups shouldn't take too long.

I'm using ZFS for the VMs/CTs.

I do make use of groups for the VMs/CTs meaning I have a crticial, non-crticial, testing and not in use groups. I've setup backup jobs based on those groups to run either several times a day, once a day or a week depending how important or frequent the changes are.

This is also how I've set up on our production servers at work running two clusters with 7 nodes each. Works very well for our needs.

Perhaps it was the FIRST backup which took so long - I looked at the ones done last night and the night before. The longest one took 3m07s.

I have one server sending emails - still need to do the others:

 

[tcabernoch]

Dude. Look at the logs for one of those backups. See the dedupe in action. Go to your PBS and on the datastore Summary tab it gives you a dedupe ratio. You're gonna be impressed. If you aint getting 30:1, it just hasn't been running long enough yet.

 

[bearhntr]

Dude. Look at the logs for one of those backups. See the dedupe in action. Go to your PBS and on the datastore Summary tab it gives you a dedupe ratio. You're gonna be impressed. If you aint getting 30:1, it just hasn't been running long enough yet.

Thanks... I will have to let this run for a while. It seems that only the first instance of each VM. Looking a the times now after 3 iterations have run they appear to be quite quick.