vzdump backup retention with "delete oldest backups first" policy ?

[RolandK]

hello,
i'd wonder if there is an easy way to have some backup retention for vzdump full backups with some "use all available size of the backup datastore for backups and then start deleting the oldest backups first to free up necessary space for the new ones".

I know this is not possible with standard means in proxmox now, but maybe somebody has implemented it somehow by scripting !?

we have weekly vzdump full backups and i would favour deletion of old backups to free up space for new backups (and have more priority for new backups getting written) instead of re-tuning retention/deletion policy and have free space being wasted

opinions ?

 

[Hannes Laimer]

Hey,

with PBS you don't really know the amount of storage a backup will take up before making it. Under the assumption that the disk usage increases by about the same with every backup, you could let the datastore fill up until the storage usage is about as much as you'd like it. Then see how many backups you have in the datastore and set your retention up so when the next backup is taken the last one will be removed. So, lets say you kept ~the last 50 backups and have ~ 10% of your storage left, you can set the prune-job to keep the last 50 backups. Like that the storage usage should stay at about 90%. As I said this only works under the assumption that each new backup increases the storage usage by about the same amount, in other words, the amount of data that changes within a week in the VM is more or less the same every week.

 

[RolandK]

ok, thanks

 

[ivenae]

I have somthing like this:

/usr/local/bin/SmartStore.sh

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Import Config File
config=/usr/local/bin/config.cfg
source $config

# Check if Already Running
for pid in $(pidof -x SmartStore.sh); do
    if [ $pid != $$ ]; then
        echo "[$(date)] : SmartStore.sh : Process is already running with PID $pid"
        exit 1
    fi
done

# Get Zpool Status
zpool status
ZP=$(zpool status | grep degraded)
if [ -n "$ZP" ]; then
   echo ACHTUNG: ${ZP}
fi

# Get Filesystem Usage-Percentage
FS=$(df -h | grep -E '([7-9][0-9]|100)%')
echo ${FS}
if [ -n "$FS" ]; then
   echo ACHTUNG: ${FS}
fi

# Get Machine Count
pctList=$(pct list | wc -l)
if [ "$pctList" -ne "0" ]; then
   pctList=$(($pctList - 1))
fi

qmList=$(qm list | wc -l)
if [ "$qmList" -ne "0" ]; then
   qmList=$(($qmList - 1))
fi
NUMBER=$(($pctList + $qmList))

if [ -z "$mountPath" ]; then
   mountPath=$backupPath
fi

if [ -z "$mountPersistent" ]; then
   umount ${mountPath}
fi

mkdir -p ${mountPath}
folder=$(date +%A)

# Loop over UUIDs
for uuid in "${UUIDS[@]}"
do
   mount UUID=$uuid ${mountPath}
   mountpoint -q ${mountPath} || continue
   echo Sichere UUID $uuid
   echo "Backup ausgeführt am: $(date +"%Y-%m-%d-%H:%M")" on UUID $uuid >> "${logPath}"
   if [[ " ${BackupType[*]} " =~ " vzdump " ]]; then
      #Get Size of last Backups
      if [ "$(ls ${backupPath}/*.zst | wc -l)" -eq "0" ] || [ "${NUMBER}" -eq "0" ]; then
         backupCount='0'
         backupNeed='NA'
         backupPuffer='0'
      else
         backupSize=$(du -s ${backupPath}| cut -f 1)
         backupCount=$(($(ls ${backupPath}/*.zst | wc -l) / ${NUMBER}))
         backupNeed=$(($backupSize / $backupCount))
         doubleNeed=$(($backupNeed * 2))
         backupPuffer=$(($backupNeed * 11/10))
      fi
      backupAvail=$(df $mountPath | awk 'NR==2{print$4}')
      echo $backupCount Backup found
      echo $MAXFILE Backup in Backup-Retention-Policy
      echo $backupNeed Bytes estimated Backup Size
      echo $backupPuffer Bytes estimated Backup Size with Puffer
      echo $backupAvail Bytes estimated available Backup Space
      if [ ${backupPuffer} -ge ${backupAvail} ]; then
         if [ "${MAXFILE}" -eq "1" ]; then
            echo ACHTUNG: ZU WENIG SPEICHERPLATZ: Größere Backup Festplatte benötigt. Maxfile kann nicht weiter reduziert werden!
         else
            MAXFILE=$(($MAXFILE-1))
            sed -i -e "s/^MAXFILE.*/MAXFILE=$MAXFILE/" ${config}
            echo ACHTUNG: ZU WENIG SPEICHERPLATZ: Maxfile = ${MAXFILE}
         fi
      elif [ ${MAXFILE} == ${backupCount} ] && [ ${backupPuffer} -ge ${doubleNeed} ]; then
            MAXFILE=$(($MAXFILE+1))
            sed -i -e "s/^MAXFILE.*/MAXFILE=$MAXFILE/" ${config}
            echo ACHTUNG: Freier Speicher gefunden, Maxfile erhöht. Maxfile = ${MAXFILE}
      fi
      vzdump --compress zstd --prune-backups keep-last=$MAXFILE --all --mode snapshot --dumpdir $backupPath
   fi
   if [[ " ${BackupType[*]} " =~ " rsync " ]]; then
      #Get Last Backup Size / Maxfile Value
      backupSize=$(du -s ${backupPath}| cut -f 1)
      backupNeed=$(($backupSize / 6))
      backupPuffer=$(($backupNeed * 11/10))
      backupAvail=$(df $mountPath | awk 'NR==2{print$4}')
      echo $backupNeed Bytes estimated Backup Size
      echo $backupPuffer Bytes estimated Backup Size with Puffer
      echo $backupAvail Bytes estimated availlable Backup Space
      if [ ${backupPuffer} -ge ${backupAvail} ]; then
         echo ACHTUNG: ZU WENIG SPEICHERPLATZ
      fi
      if [ -d "${backupPath}/__latest" ]; then
         rsync -avh --delete --link-dest="${backupPath}/__latest" "$rsyncSource" "${backupPath}/$folder"
      else
         rsync -avh "$rsyncSource" "${backupPath}/$folder"
      fi
      rm -f "${backupPath}/__latest"
      ln -s "$folder" "${backupPath}/__latest"
   fi
   if [ -z "$mountPersistent" ]; then
      umount ${mountPath}
   fi
done

with a config like this

/usr/local/bin/config.cfg

UUIDS=(
2e522940-756e-449f-9035-0e0a779245c1
73e62b2d-1cc4-426c-9368-f0021fb5ab32
42a90b68-a9e5-403d-b8cf-370e6a0a46b3
)

BackupType=( vzdump )
rsyncSource=/share/
MAXFILE=8

backupPath=/media/SmartStore
logPath=/var/log/smartstore.log
filePath=/usr/local/bin/config.cfg

There is a Maxfile Parameter in the config which will be reduced if the estimated Backup size gets to large.
Only works if the size increase is small. If you double your backup size by copying large file to your server this will crash anyway, because it calculates the arithmetic mean of the last backups. But it helps me a lot.