Prune setting per VM?

[lennart]

First off all congratulations on the beta release of Proxmox Backup Server.
After playing around for a little it's definitely a good addition to PVE.

One feature I'm missing is to have different prune settings per VM/CT on one single datastore.
Currently, we use 2 different datastores to handle different prune settings, but will there be an option to set the --keep-daily (for example) per VM/CT?

 

[fabian]

you can trigger the prune via the client instead of directly on the server and pass an arbitrary prune schedule per backup group (which equals guest type + ID for PVE backups). it's what PVE does as well via maxfiles => keep-last, with work in progress to bring the full flexibility of PBS pruning schedules to PVE itself.

e.g., if you backup to a PBS storage with maxfiles = 3, which currently has 3 backups stored for that guest, you'll see:

$ vzdump 9999 --storage pbs-noratest --mode suspend --remove
INFO: starting new backup job: vzdump 9999 --storage pbs-noratest --remove 1 --mode suspend
INFO: Starting Backup of VM 9999 (lxc)
INFO: Backup started at 2020-07-16 12:29:30
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: CT9999
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/test') in backup
INFO: starting first sync /proc/1439165/root// to /var/tmp/vzdumptmp1585246
INFO: Number of files: 24,394 (reg: 18,465, dir: 2,124, link: 3,774, special: 31)
INFO: Number of created files: 24,392 (reg: 18,465, dir: 2,122, link: 3,774, special: 31)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 18,446
INFO: Total file size: 796,665,253 bytes
INFO: Total transferred file size: 661,446,315 bytes
INFO: Literal data: 661,446,315 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 786,390
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 663,010,033
INFO: Total bytes received: 375,763
INFO: sent 663,010,033 bytes  received 375,763 bytes  102,059,353.23 bytes/sec
INFO: total size is 796,665,253  speedup is 1.20
INFO: first sync finished (6 seconds)
INFO: suspending guest
INFO: starting final sync /proc/1439165/root// to /var/tmp/vzdumptmp1585246
INFO: Number of files: 24,394 (reg: 18,465, dir: 2,124, link: 3,774, special: 31)
INFO: Number of created files: 0
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 0
INFO: Total file size: 796,665,253 bytes
INFO: Total transferred file size: 0 bytes
INFO: Literal data: 0 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 0
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 612,232
INFO: Total bytes received: 2,289
INFO: sent 612,232 bytes  received 2,289 bytes  1,229,042.00 bytes/sec
INFO: total size is 796,665,253  speedup is 1,296.40
INFO: final sync finished (0 seconds)
INFO: resuming guest
INFO: guest is online again after <1 seconds
INFO: creating Proxmox Backup Server archive 'ct/9999/2020-07-16T10:29:30Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp1585246/etc/vzdump/pct.conf root.pxar:/var/tmp/vzdumptmp1585246 --include-dev /var/tmp/vzdumptmp1585246/. --skip-lost-and-found --backup-type ct --backup-id 9999 --backup-time 1594895370 --repository test@pbs@127.0.0.1:backuptest
INFO: Starting backup: ct/9999/2020-07-16T10:29:30Z
INFO: Client name: nora
INFO: Starting protocol: 2020-07-16T12:29:36+02:00
INFO: Upload config file '/var/tmp/vzdumptmp1585246/etc/vzdump/pct.conf' to 'BackupRepository { user: Some("test@pbs"), host: Some("127.0.0.1"), store: "backuptest" }' as pct.conf.blob
INFO: Upload directory '/var/tmp/vzdumptmp1585246' to 'BackupRepository { user: Some("test@pbs"), host: Some("127.0.0.1"), store: "backuptest" }' as root.pxar.didx
INFO: root.pxar.didx: Uploaded 664407340 bytes as 190 chunks in 1 seconds (516 MB/s).
INFO: root.pxar.didx: Average chunk size was 3496880 bytes.
INFO: root.pxar.didx: Time per request: 6451 microseconds.
INFO: catalog.pcat1.didx: Uploaded 471653 bytes as 3 chunks in 1 seconds (0 MB/s).
INFO: catalog.pcat1.didx: Average chunk size was 157217 bytes.
INFO: catalog.pcat1.didx: Time per request: 410830 microseconds.
INFO: Upload index.json to 'BackupRepository { user: Some("test@pbs"), host: Some("127.0.0.1"), store: "backuptest" }'
INFO: Duration: PT1.256535526S
INFO: End Time: 2020-07-16T12:29:37+02:00
INFO: run: /usr/bin/proxmox-backup-client prune ct/9999 --quiet 1 --keep-last 3 --repository test@pbs@127.0.0.1:backuptest
INFO: ct/9999/2020-07-16T10:07:17Z Thu Jul 16 12:07:17 2020 remove
INFO: Finished Backup of VM 9999 (00:00:10)
INFO: Backup finished at 2020-07-16 12:29:40
INFO: Backup job finished successfully

INFO: run: /usr/bin/proxmox-backup-client prune ct/9999 --quiet 1 --keep-last 3 --repository test@pbs@127.0.0.1:backuptest
INFO: ct/9999/2020-07-16T10:07:17Z Thu Jul 16 12:07:17 2020 remove

shows the pruning in action, you can just disable pruning on the PVE and PBS side and run such a command yourself, either on the PBS, or the PVE, or some other host that has access to the PBS API.

bringing a bit more flexible/detailed pruning schedules to PBS might make sense as well. something like configure a default pruning schedule for the whole datastore, and then add override schedules per backup group if desired.

 

[Ramalama]

Hey Fabian, are the "Prune Settings per VM" or Group planned in near future?
For a Backup Server this feature is actually pretty important.

Cheers

 

[fabian]

not yet, you can always prune yourself using a script though if you want that (at least, prune more aggressively than the whole datastore).

 

[Ramalama]

not yet, you can always prune yourself using a script though if you want that (at least, prune more aggressively than the whole datastore).

What is with the Retention Settings Tab in the Backup Section of PVE-Datacenter?

I mean i can define there for every VM a Separate Backup Task with separate Retention Settings.
Isn't it the same as the Prune Settings in PBS?

Cheers

 

[fabian]

you can do that, yes. most people don't want to set up a job per guest though the PVE/client side prune settings are independent from those on the PBS/server side - if you configure both, the more aggressive one will determine the effective pruning settings, since both will be executed.

 

[Ramalama]

you can do that, yes. most people don't want to set up a job per guest though the PVE/client side prune settings are independent from those on the PBS/server side - if you configure both, the more aggressive one will determine the effective pruning settings, since both will be executed.

Awesome, thanks for the explanation, because that would have been my next question