Cool !!!, but will be better if PVE team developers add a column for bwlimit, then if you have a backup Server for several PVE hosts, or want to do a manual backup, the bwlimit can help you for performance reasons.
Option "max backups"
- Thread starter seedaumas
- Start date
Cool !!!, but will be better if PVE team developers add a column for bwlimit, then if you have a backup Server for several PVE hosts, or want to do a manual backup, the bwlimit can help you for performance reasons.
Claiming such nonsense will not help. It works perfectly for us, and that is why we have inplemented it this way.
Feel free to send patches if you want to improve thing:
http://pve.proxmox.com/wiki/Developer_Documentation
But there are serious flaws with the old approach
- confusing, because you have have multiple jobs with different max-backup settings on same storage.
- does not work at all with manual backup (when you don not have a job)
One idea that comes in mind is:
- keep max-backups setting for storage (upper limit)
- allow to specify max-backup in VM settings (must be lower that storage limit).
+1 for dietmar idea. Seems simply enough to understand and is useful too. Just another idea. I've done some tests that abuses some of our VM. I would like to revert to a state before the tests and made an extra backup. But I didn't find a way to "preserve" such an important backup or mark it "do not delete". I've copied the required backup away with shell access but maybe it would be a good idea to redefine the max backups like: max_backups = maximum number of backups at all, including manual planed_backups = number of backups done via backup schedule manual_backups = number of backups done via gui by operator And planed_backups + manual_backups
that's what i would expect from a backup solution;
if you have a dedicated shared backup storage you will send all backups from all nodes to that storage and the different backup jobs defines which vm's with which options;
what me confuses here is that each backup storage definition creates an extra nfs mount on every node what can easily result in large number of nfs mounts to the storage if you have several single pve hosts or clusters;
aside that you need to define consistent naming for each backup storage to identify (and they are all connected to the same storage), is that you have to lookup first which backup task covers a specific vm and then lookup on the storage to get the number for the backups what will be kept;
i'm not saying the new backup model is a bad solution, it's fine and working well - it just feels a little bit unhandy
yes indeed, the manual backup option is absolutely great feature
Last edited:
to define a max-backups on the storage side and override it with a lower value than the storage limit would be very cool, as it would significantly reduces the number of nfs mounts to the storage server and gives you the flexibility you had before;
would be very nice if you could implement it that way
for such important backups you want to 'preserve' i would recommend to keep them on an 'storage archive' and not on an regular backup storage as per definition backups are recurring and therefore overwritten - archives, per definition will be kept as long someone manually deletes them;
that means you have a separate storage mount for 'archives' where you do your 'important' manual backups with a max-backup value of 0 and it will stay there as long as you manually delete it;
though, for manual backups it would be very nice to have the option to enter a small description or custom file name with some variables (date) that you can remember for what reasons you did it and what you can expect when you roll back to that;
Last edited:
What is the benefit of asociating number of backups with storage definition ?
I made this question on post#5 of this thread, and never get an answer.
Actually it makes me have at least 3 storage definitions for each storage server. Makes no sense to me, as I suppose that system does periodicall check to each storage definition to know if it is available...
Could be a solution, but this allow one machine to have ONLY one max-backups, this is an unaceptable llimitation.
So I would prefer to have it on the backup job definition, as before.
This is the kind of backups I (can) use:
1. Daily (max-backups = 7)
2. Tuesday-Thursday-Saturay (max-backups :3)
3. Weekly (max-backups = 5)
And any machine can be on any (or more than one) backup definition.2. Tuesday-Thursday-Saturay (max-backups :3)
3. Weekly (max-backups = 5)
Having max-backups ONLY on backup definition, can make all my backup schedules work as expected.
May be there is another use of backups that will benefit from having max-backups on storage definition, but I can not imagine it...
Regards
According to Dietmar, the reason they moved this from backup job definition to storage definition is manual backups, so that if a user executes a manual backup to the same storage, the backups will be deleted accordingly. Although we don't use manual backups, this is understandable (although still unintuitive).
Sounds like a good solution.
The way our backups work is this:
- we have 2 different network storages (1 for weekly, 1 for daily backups, also for redundancy)
- we have 2 standard backup schedules (1 on every day but sunday, 1 on sunday)
- we retain 1 weekly backup for every VM by default
- we have a daily backup option for VM users
- users can also choose how many backups they want us to keep for them
So now for every user who is not satisfied with a single weekly and single daily backup kept, we have to create a new backup storage and a new backup job. Also the problem with having multiple backup jobs is that they cannot overlap with other backup jobs, so we have to plan them very carefully so each one executes under the time it has.
With Dietmar's proposed idea we could specify a larger number on the storage, and the user's desired number on the VM, and we would be able to use only a single backup job. It would also permit manual backups by the client.
You should keep two backups jobs (one per storage), each storage should have the larger number on max-backups, so you will end in controling only with VM's maxbackups.
This will end in that each machine will have the same number of backups for daily and weekly schedules (of course limited by this larger number)..
This is not intuitive, in your case :
- For daily backups, max-backups on VMs config should be lower than storage-definition
- For weekly backups, the priority would be in storage-definition
Of course it is better than actual, but I would prefer to have it on backup-schedule...
Regards
Yes, which would be much better for us than the current system. As you wrote, we would set the storages to higher numbers (effectively not using their max backups limitation), and we would be able to control this granularly for each VM.
If would be great to see this implemented.
I actually just came here to suggest EXACTLY this!
My backup storage filled up during the latest backup, partially due to the lack of flexibility in the retention settings. I have a few larger VMs that I really only need a couple of copies of, but a few smaller ones that I want daily backups going back a week or so.
The idea of configuring the same storage multiple times is not obvious - I only thought of it as I read through the Google results that got me here. I'd gone with a compromise - set the limit to 3 and see how it goes. Worked OK (aside from only having 3 days of backups for the daily ones), but has finally filled up as the larger ones got bigger.
Before the maxfiles was defined on the storage there were a number of people who made a manual backup resulting in all of their previous backups being deleted.
By attaching maxfiles to the storage itself, if you run a manual backup it still uses the maxfiles defined on the storage and does not result in all previous backups being accidently deleted.
Ensuring people do not accidently delete their backups seems like a great feature, not a flaw.
Seems like the use-case that others desire in this thread was overlooked when making this improvement.
dietmar suggestion sounds like a perfect solution:
I would also like to point out that you can override the storage setting for maxfiles by specifying maxfiles in the vzdump.conf.
This feature is very important to me, please make sure it is not removed.
I have hook script that attempts to calculate the number of backups that can fit onto my backup storage and dynamically update the vzdump.conf maxfiles setting.
This way I always have the most amount of backups of every VM that can fit onto my backup storage attached to each node.
I find this method better than hard coding some number on the storage and then running out of disk space when the size of VM backups grow too large or new VMs are added.
Last edited:
I think this is the 'perfect' solution for everybody. Could you please elaborate ?
Last edited:
It would be an excellent choice for setting max-storage backups if you make the calculation considering all PVE nodes for a shared storage of Backups
My setup is rather specific and my scripts are embarassingly poorly written so I will not share them until I can get them in a releasable format.
But basically I use the vzdump hooks and hook into job-start and job-end.
job-Start uses cryptsetup to mount the encrypted SATA Disk that we swap out like one would swap tape drives.
Once mounted it performs some calculations, how many VMs get backed up, how many VM backups are on the backup drive, how much free space is on the backup drive, what the average vm backup is.
Using that info it then attempts to calculate what maxfiles should be.
Then uses sed to change maxfiles value in /etc/vzdump.conf.
It has a failsafe that it will never set maxfiles below 1.
job-end unmounts the encrytpted volume
It is tricky to make this work right because vzdump first makes a backup, then removes old ones.
So you need enough free space for each backup to happen, then have its old ones purged.
When you have VMs that are large and small on same server getting this to work right is not so easy.
Some people do have different storage for each Proxmox server, not some common NFS where all backups are sent, I think dietmar suggestion to allow overriding the maxfiles on the storage is a solution that works for both use-cases.
When I see comments similar to yours I often wonder if people rely on that single online NFS(samba or whatever) server as their sole backup method.
Not saying you do, but anyone who relies on only online backups will be dissapointed when lightening strikes or hackers delete your VMs and the VM backups.
Don't get burnt (figuratively and literally by lightening) like I once did.
This public service announcement brought to you by Murphys Law
I assume you didn't read the whole thread - i already applied to Dietmars post 5 days ago that his suggestion would be the best solution ;-)
Not necessarily, i know several setups where the backups are send to a local NFS storage cluster which replicates it's data to a remote datacenter like we do - this is very common;
With a different storage for each Proxmox server you still have the problem you described when you only have that without archiving or a remote (offsite) backup;
my point on the max_backup setting per storage was that you end up with much more storage definitions as before - but this is covered way up in the thread and a possible solution was already suggested by Dietmar;