Put VM name into backup name

[ojaksch]

@andyn A quick look at your script shows me that IMHO target should be TARGET ... Please try this out.

 

[ojaksch]

Here is the latest version of my script. Not tested with PVE6 but should work there too. I've taken into account the change from maxfiles to prune-backups in vzdump.conf, so please consider this. And yes, it would be more useful to respect prune-backups from VM/container's config, but this is a task for you guys out there as I'm too lazy for this and not using this script anymore, as mentioned

 

[andyn]

Thanks ojaksch !! works like a charm.



I am looking at PBS, but i haven't read enough of the docs yet to work out if I can use my NFS NAS as the backup storage back-end. It's the only thing I have with enough disk space to store everything. Currently the PVE hosts just back their VMs up to it.
With it being a lab, VMs are created and destroyed many times, and sometimes I will delete one VM running Debian then create a new one running Windows and it will end up with the same ID. Having names in the backups so that I can reinstate one is a great help.
Thank you again!

 

[ojaksch]

I am looking at PBS, but i haven't read enough of the docs yet to work out if I can use my NFS NAS as the backup storage back-end.

You are welcome. Glad to hear that it is working again and useful to you.
Using your NFS NAS with PBS should work if you setup the NFS mount as directory in and for PBS and NAS' OS isn't using any quirked NFS options.
I'm using PBS in two ways: On top of PVE for constant and quick backups of data and a dedicated PBS on top of an old decommissioned PC w/ a Core Duo (or somthing similar - don't remember and doesn't matter) for nightly backups of VMs and images of PCs like Clonezilla is doing...and I'm very satisfied with these soultion(s).

 

[andyn]

Thanks again! That's not totally clear in the documentation. Hardly surprising since we're only on V2. I'll take a look at it.

 

[ojaksch]

Nothing to thank for. IMHO it hasn't to be clear in the documentation...we're on Linux here at this the Linux way: There is always a solution - just think around it and you'll find it.

 

[andyn]

Yes, the Linux way is very interesting. Great for learning new things. I keep many notes!
Next question -
Inside the hook script, ${HOSTNAME} gets me the name of the VM being backed up
Outside the hook script, $HOSTNAME gets me the name of the PVE node.

I have worked this out so that I can now include the name of the PVE node in the dump file. New variable PVENODENAME added, and no, it's likely not the most efficent thing to create this variable every time the script runs, but it does work

#!/bin/bash

#     backup-start | backup-end | backup-abort | log-end | pre-stop | pre-restart | post-restart | job-start | job-end | job-abort

# If varable DUMPDIR is empty it might be a PBS backup
[ -z "${DUMPDIR}" ] && exit 0

cd "${DUMPDIR}"
MAXFILES=$(grep "prune-backups:" /etc/vzdump.conf | awk -F"=" '{print $2}')
case "${1}" in

    backup-end)
        PVENODENAME=$(hostname)
        VM_FNAM_OLD=$(basename ${TARGET})
        VM_FNAM_NEW=$(basename ${TARGET} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'-"$4"-"$5}')
        VM_FNAM_DUP=$(basename ${TARGET} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'"}')
        VM_TYPE=$(basename ${TARGET} | awk -F "-" '{print $2}')
        mv "${VM_FNAM_OLD}" "${VM_FNAM_NEW}"

        case "${VM_TYPE}" in
            lxc)
                NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.tar* | wc -l)
                if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
                    echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
                    ls ${VM_FNAM_DUP}*.tar* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
                fi
            ;;
            qemu)
                NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.vma* | wc -l)
                if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
                    echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
                    ls ${VM_FNAM_DUP}*.vma* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
                fi
            ;;
        esac
        ;;

    log-end)
        LOG_FNAM_OLD=$(basename ${LOGFILE})
        LOG_FNAM_NEW=$(basename ${LOGFILE} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'-"$4"-"$5}')
        LOG_FNAM_DUP=$(basename ${TARGET} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'"}')
        mv "${LOG_FNAM_OLD}" "${LOG_FNAM_NEW}"
        NUMFILES=$(ls -1 ${LOG_FNAM_DUP}*.log | wc -l)
        if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then ls ${LOG_FNAM_DUP}*.log | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm; fi
        ;;

esac

 

[ojaksch]

Next question -
Inside the hook script, ${HOSTNAME} gets me the name of the VM being backed up
Outside the hook script, $HOSTNAME gets me the name of the PVE node.

I have worked this out so that I can now include the name of the PVE node in the dump file. New variable PVENODENAME added, and no, it's likely not the most efficent thing to create this variable every time the script runs, but it does work

Well done. Your goal is (or should) to get it to work while learning, not to do this in the most efficient way at first try.
But...where is your question?

 

[andyn]

Ah, perhaps I worded that update badly.
It might have been better like this:

My next question was this:
Inside the hook script, ${HOSTNAME} gets me the name of the VM being backed up
Outside the hook script, $HOSTNAME gets me the name of the PVE node.
How do I make the hostname of the PVE node available inside the hook script?

I have worked the answer out for myself.
By adding a new variable (PVENODENAME in my example) you can include the PVE node name. This is likely not the most efficent way, but it does work.

I noticed that my first version doesn't correctly rename the log files, so below is a version that works for both the backups and the log files. This works for backups started from the GUI for an individual VM (select VM name/Backup/click Backup Now button), and also works when multiple backups are started from a backup job (Datacentre/Backup/Select the backup job/click Run Now button).
I shold point out that I only use VMs, and not containers, so I can't 100% confirm this works for containers, but I suspect it will because this is a simple extension of @ojaksch 's script.

#!/bin/bash

#     backup-start | backup-end | backup-abort | log-end | pre-stop | pre-restart | post-restart | job-start | job-end | job-abort

# If varable DUMPDIR is empty it might be a PBS backup
[ -z "${DUMPDIR}" ] && exit 0

cd "${DUMPDIR}"
MAXFILES=$(grep "prune-backups:" /etc/vzdump.conf | awk -F"=" '{print $2}')
case "${1}" in

backup-end)
        PVENODENAME=$(hostname)
        VM_FNAM_OLD=$(basename ${TARGET})
        VM_FNAM_NEW=$(basename ${TARGET} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'-"$4"-"$5}')
        VM_FNAM_DUP=$(basename ${TARGET} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'"}')
        VM_TYPE=$(basename ${TARGET} | awk -F "-" '{print $2}')
        mv "${VM_FNAM_OLD}" "${VM_FNAM_NEW}"

        case "${VM_TYPE}" in
            lxc)
                NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.tar* | wc -l)
                if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
                    echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
                    ls ${VM_FNAM_DUP}*.tar* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
                fi
            ;;
            qemu)
                NUMFILES=$(ls -1 ${VM_FNAM_DUP}*.vma* | wc -l)
                if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then
                    echo "-- maxfiles set to ${MAXFILES} but ${NUMFILES} present. Deleting the older ones."
                    ls ${VM_FNAM_DUP}*.vma* | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm
                fi
            ;;
esac
        ;;

log-end)
        PVENODENAME=$(hostname)
        LOG_FNAM_OLD=$(basename ${LOGFILE})
        LOG_FNAM_NEW=$(basename ${LOGFILE} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'-"$4"-"$5}')
        LOG_FNAM_DUP=$(basename ${TARGET} | awk -F "-" '{print $1"-"$2"-'${PVENODENAME}'-"$3"-'${HOSTNAME}'"}')
        mv "${LOG_FNAM_OLD}" "${LOG_FNAM_NEW}"
        NUMFILES=$(ls -1 ${LOG_FNAM_DUP}*.log | wc -l)
        if [[ $MAXFILES =~ ^-?[0-9]+$ ]] && [ ${NUMFILES} -gt ${MAXFILES} ]; then ls ${LOG_FNAM_DUP}*.log | head -n $((${NUMFILES}-${MAXFILES})) | xargs rm; fi
        ;;

esac

 

[popey]

Here is the latest version of my script. Not tested with PVE6 but should work there too. I've taken into account the change from maxfiles to prune-backups in vzdump.conf, so please consider this. And yes, it would be more useful to respect prune-backups from VM/container's config, but this is a task for you guys out there as I'm too lazy for this and not using this script anymore, as mentioned

how to use your script?

 

[andyn]

It's a hook script so can be referenced in /etc/vzdump.conf. Just save the script somewhere, set permissions on it so that it can be executed then edit the script: line in the vzdump.conf file to point to the script file you created. If you take a look at the top of this thread, I think it's discussed in more detail.

 

[ojaksch]

The link to the doc of vzdump in my post #2 has gone 404 which I've corrected now.

 

[user8923428934]

Hello,

is it possible to have multiple vzdump configs so I can actually have different retention policies for different storages? from what I understand, this applies to all of vzdump on that node. so when I set a retention for keep last 2 it will do that for all the vzdump operations on that node even though I want one vzdump storage to have a keep last of 4 and the other one of 2. further I assume the protected flag won't work as the backups get deleted via a regular rm command?

thanks

 

[fiona]

Hi,

Hello,

is it possible to have multiple vzdump configs

No.

so I can actually have different retention policies for different storages

Yes. Instead of configuring the retention in /etc/vzdump.conf, you can configure it directly on the backup storage in Datacenter > Storage > Edit > Backup Retention

from what I understand, this applies to all of vzdump on that node. so when I set a retention for keep last 2 it will do that for all the vzdump operations on that node even though I want one vzdump storage to have a keep last of 4 and the other one of 2.

The setting from the /etc/vzdump.conf will be used for all storages that don't configure an explicit retention themselves.

further I assume the protected flag won't work as the backups get deleted via a regular rm command?

Protected backups will not be considered for removal during pruning, that's the very purpose of it.

 

[RolandK]

regarding finding your appropriate backup files - there is .notes files together with the vzdump files, which contain the VM name, so you can simply grep trough *.notes when searching for a particular VM backup

 

[moxmox]

any chance this will be added into the offical build - I find the naming pretty bad when trying to find a particular backup you need to lookup the vm id.

 

[fiona]

Hi,

any chance this will be added into the offical build - I find the naming pretty bad when trying to find a particular backup you need to lookup the vm id.

the naming schema won't change to include the name in the near future, but you can set notes for your backups since a while now and there, you can include the name: https://pve.proxmox.com/pve-docs/chapter-vzdump.html#vzdump_notes

 

[TECNI.sistemas]

Hello,
it's possible to modify vzdump script for add to a backup name like:
vzdump-qemu-100-2016_05_20-15_10_00.vma.lzo
tha name of VM like:
vzdump-qemu-100-VMname-2016_05_20-15_10_00.vma.lzo

When a server crash or when purge old backup or when manage backup disk space it's difficult to understand what a backup refer to.
Can you implement in future release of Proxmox this feature?

Thank you

With sed it's easy, so I did it in my script

root@XD:~# HOSTNAME=VMname
root@XD:~# TARGET=/media/local/dump/vzdump-qemu-100-2016_05_20-15_10_00.vma.lzo
root@XD:~# NEWTARGET=`echo $TARGET|sed -r 's|(\-[0-9]{3}\-)|\1'$HOSTNAME'-|'`
root@XD:~# mv $TARGET $NEWTARGET
root@XD:~# TARGET=$NEWTARGET
root@XD:~# ls /media/local/dump/
vzdump-qemu-100-VMname-2016_05_20-15_10_00.vma.lzo
root@XD:~#