Put VM name into backup name

[johjoh]

Hello,
it's possible to modify vzdump script for add to a backup name like:
vzdump-qemu-100-2016_05_20-15_10_00.vma.lzo
tha name of VM like:
vzdump-qemu-100-VMname-2016_05_20-15_10_00.vma.lzo

When a server crash or when purge old backup or when manage backup disk space it's difficult to understand what a backup refer to.
Can you implement in future release of Proxmox this feature?

Thank you

 

[ojaksch]

You can set a script in /etc/vzdump.conf for this, ie script: /usr/local/bin/vzdump-rename
Have a look to https://pve.proxmox.com/pve-docs/vzdump.1.html which describes everything you'll need.

edit: There's an example script at /usr/share/doc/pve-manager/examples/vzdump-hook-script.pl also.

 

[Zoltan Bakcsa]

I would like to have the VM name in the backup filename too. VM ID tells absolutely nothing about the backup. Moreover, using only VM ID in backups could lead to data loss.
Consider the following scenario:
I have a VM with VM ID 100, I create backups on daily basis and keep 2 versions. One day I decide to remove this VM and create a new one which happens to get the same ID as the previous one. Of course, I use different name. 2 days later when the manager asks me to recover the original VM I can only say sorry, because the backup service already deleted them. To avoid such a situation, manual configuration is needed.

Using a hook script is a nice and quick solution for the missing vm name issue. However, if I rename the backup file in the backup-end stage, the "max backup" will never be reached since VZDump module still searches for the original filename (without the name). This results in 2 things:

• Setting the max backups value on a storage does not make any difference (it would create infinite number of backups)
• backup rotating has to be duplicated in the hook script.

I don't mind writing my own hook script, but it would be very nice if could tell VZDump in advance what filename to use for the backup.

 

[fireon]

When a server crash or when purge old backup or when manage backup disk space it's difficult to understand what a backup refer to.

Why? A klick on "show config button" show you the vmname and all it's config details.

I would like to have the VM name in the backup filename too.

But yes, it would be nice too.

 

[LnxBil]

A few remarks on bigger installations:
* vm names are used multiple times depending on their network and usage (e.g. ns01, ldap, mail etc.)
* IDs are unique and should be grouped in pools concerning their logical environment (e.g. departments, subnets etc.) also in number groups, 1xxxx for this, 2xxxx for that.
* backups are normally archived and managed externally (not on a cluster)
* ID should not be reused unless the previous backups are taken care of after deleting a VM

 

[ojaksch]

[DEPRECATED - see below]

Um...edited the script to actual version, which includes a query for maxfiles: and deleted this post instead of editing. I'm a moron.

#!/bin/bash

cd "${DUMPDIR}"
MAXFILES=$(grep "maxfiles:" /etc/vzdump.conf | tr -d " " | awk -F":" '{print $2}')

case "${1}" in
   backup-end)
       VM_FNAM_OLD=$(basename ${TARFILE})
       VM_FNAM_NEW=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'-"$4"-"$5}')
       VM_FNAM_DUP=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'"}')
       mv "${VM_FNAM_OLD}" "${VM_FNAM_NEW}"
       NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.tar.* | wc -l)
       if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
           ls ${VM_FNAM_DUP}*.tar.* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
           echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
       fi
       ;;
   log-end)
       LOG_FNAM_OLD=$(basename ${LOGFILE})
       LOG_FNAM_NEW=$(basename ${LOGFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'-"$4"-"$5}')
       LOG_FNAM_DUP=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'"}')
       mv "${LOG_FNAM_OLD}" "${LOG_FNAM_NEW}"
       NUMFILES=$(ls -1 ${LOG_FNAM_DUP}*.log | wc -l)
       if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then ls ${LOG_FNAM_DUP}*.log | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm; fi
       ;;
esac

 

[Peter123]

We are saving the dumps in the new .zst-format.

Could someone tell us how we would have to adapt the script? It didn't help to replace tar for zst.

Thanks.

 

[ojaksch]

Nice to see that my script is still of use.
EDIT: But I don't see why the old code shouldn't work; PVE sends the full name of it's TAR file, therefore the code shouldn't care about the file extension.
Here is the actual version which is still working, also with zstd:
(I'm having problems with idents in the code preview. If in doubt please download, rename and use the attachment)

#!/bin/bash

# backup-start | backup-end | backup-abort | log-end | pre-stop | pre-restart | post-restart | job-start | job-end | job-abort

cd "${DUMPDIR}"
MAXFILES=$(grep "maxfiles:" /etc/vzdump.conf | tr -d " " | awk -F":" '{print $2}')

case "${1}" in
  backup-end)
VM_FNAM_OLD=$(basename ${TARFILE})
VM_FNAM_NEW=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'-"$4"-"$5}')
VM_FNAM_DUP=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'"}')
VM_TYPE=$(basename ${TARFILE} | awk -F "-" '{print $2}')
mv "${VM_FNAM_OLD}" "${VM_FNAM_NEW}"

case "${VM_TYPE}" in
lxc)
NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.tar* | wc -l)
if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
ls ${VM_FNAM_DUP}*.tar* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
fi
;;
qemu)
NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.vma* | wc -l)
if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
ls ${VM_FNAM_DUP}*.vma* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
fi
;;
esac
;;

log-end)
LOG_FNAM_OLD=$(basename ${LOGFILE})
LOG_FNAM_NEW=$(basename ${LOGFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'-"$4"-"$5}')
LOG_FNAM_DUP=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'"}')
mv "${LOG_FNAM_OLD}" "${LOG_FNAM_NEW}"
NUMFILES=$(ls -1 ${LOG_FNAM_DUP}*.log | wc -l)
if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then ls ${LOG_FNAM_DUP}*.log | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm; fi
;;

esac

 

[Peter123]

Great. It worked right away.

Thanks a lot.

 

[ojaksch]

Glad to hear. Still simple and stupid code, but continually improved and working since years

 

[Peter123]

Speaking about simple and stupid code

We use rclone to upload the VMs to different cloud storages, so we have now put the following at the end of your rename script:

#!/bin/bash

if [ "$1" = "backup-end" ]; then

rclone copy /path-to-dump  cloud-storage:path-for-dumps  --config /root/.config/rclone/rclone.conf

fi

In this way, we even get a note in the confirmation email whether the upload to the cloud worked.

 

[John Allison]

Fantastic little script! nice and neat. many thanks.

Quick question - do you need to restart proxmox in order for changes to take effect?
we have a few 5.4 hosts which just come up with an error after the script is applied, but the 6.2 machine seems fine.

 

[ojaksch]

No outstanding news to my renaming script, but a small addition to respect the new PBS (Proxmox Backp Server).

 

[Peter123]

Maybe just one small thing:

if you want to restore the renamed backup files from the GUI, Proxmox does not recognize the format and refuses to restore. So you have to manually rename the backup files each time before restoring.

According to Proxmox, the following is expected: "Currently, it has to start with vzdump- then any of qemu lxc or openvz, then a VMID number." https://forum.proxmox.com/threads/e...ormat-and-compression-type.70084/#post-324541

Could that be built into the script without much effort? Thank you again.

 

[Peter123]

Apparently Proxmox has made sure with a patch that the adapted format can now be used to restore directly.

Thank you, Proxmox.

 

[Peter123]

Strange. Suddenly the restore of a VM with a different name no longer worked. Happy too early.

 

[Peter123]

No outstanding news to my renaming script, but a small addition to respect the new PBS (Proxmox Backp Server).

One more small thing: after the backups have been renamed, Proxmox no longer automatically deletes them after the set number (Datacenter - Storage - "max backups").

Can that be changed in the script?

 

[Peter123]

Does anyone have any idea how to do that?

 

[andyn]

Seems that this script no longer works in PVE 7.

Backup window shows the following lines:

INFO: stopping kvm after backup task
INFO: archive file size: 10.86GB
INFO: basename: missing operand
INFO: Try 'basename --help' for more information.

I have determined that $target gives the full path to the backup file, but I can't get this to work either.
When I substitute $target for $(basename ${TARFILE}), I end up with a null value in the new name variable.
Example:

Replacing

case "${1}" in

    backup-end)
    VM_FNAM_OLD=$(basename ${TARFILE})
    VM_FNAM_NEW=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'-"$4"-"$5}')
    VM_FNAM_DUP=$(basename ${TARFILE} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'"}')
    VM_TYPE=$(basename ${TARFILE} | awk -F "-" '{print $2}')
    mv "${VM_FNAM_OLD}" "${VM_FNAM_NEW}"

with

case "${1}" in

    backup-end)
        VM_FNAM_OLD=$(${target})
        VM_FNAM_NEW=$(${target} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'-"$4"-"$5}')
        VM_FNAM_DUP=$(${target} | awk -F "-" '{print $1"-"$2"-"$3"-'${HOSTNAME}'"}')
        VM_TYPE=$(${target} | awk -F "-" '{print $2}')
        mv "${VM_FNAM_OLD}" "${VM_FNAM_NEW}"

results in the following in the backup progress window:

INFO: transferred 50.00 GiB in 55 seconds (930.9 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 2.44GB
INFO: mv: cannot stat '': No such file or directory
INFO: Finished Backup of VM 4684 (00:00:59)
INFO: Backup finished at 2021-08-10 20:36:54

 

[ojaksch]

I can remember that the PVE team changed something with the given parameters (and/or something else), but I don't use my own script anymore since the availability of PBS. But nevertheless I'll take a look at my script later that day.