[solved] Help needed on a script for automating zfs snapshots to removable media

[Elleni]

I found a nice script that mostly does what we want. But I have to automate the snapshoting on a proxmox server further thus I 'd like to adapt the following script to our needs.

First thing is we need to do the backups on an encrypted pool, so adding a line with:
zfs load-key $BACKUPPOOL is easy but as we want to put the script in a cronjob I’d like to know how I can put the password in a variable or hardcode it into the script so there is no prompt asking for it.

Secondly - and that seems a bit more complicated, I’d like to know howto expand the script so it would be possible to also ssh to two remote proxmox servers to do the same as the script does locally, meaning taking snapshots of defined vm-disks and sending them through ssh back to the local server while keeping a number of $KEEPOLD of the snapshots.

Finally - in our setup we have no datasets for the vm-disks but rather created them directly on the pool.

But the script expects poolname\datasetname\vm-disk-xyz-disk-0 whereas we for now have poolname\vm-disk-xyz-disk-0 but if its easier I can create a dataset named disks and move the vm-disks there and modify the ProxMox vmconfigs instead of modifying the logic of the script if necessary.

I would be very glad/thankfull, if you guys could give me a hand on that one!

 

[Elleni]

Could I kindly ask for some hint to push me to the right direction on above request?

 

[Elleni]

We would like to create an encrypted pool with password -> prompt and then automate the zfs load-key step (no prompting for password but getting it from a file or hardcoded in a bash script). Is this possible? I tried creating a file and with the password in it. and then zfs load-key -L=/path/to/file/with/password and also zfs load-key "MyPassword" but that does not work.

 

[Dunuin]

We would like to create an encrypted pool with password -> prompt and then automate the zfs load-key step (no prompting for password but getting it from a file or hardcoded in a bash script). Is this possible? I tried creating a file and with the password in it. and then zfs load-key -L=/path/to/file/with/password and also zfs load-key "MyPassword" but that does not work.

I also searched for that. Looks like there is no argument to set a passphrase. But should be possible by piping stdin.
It would be way easier to just use keyfiles but I like to be able to unlock the pool using SSH with password.

 

[Elleni]

Hi Dunuin, thanks for your post. Being not skilled in scripting - could you give me an expample on how this would work with piping or linking me where I could read and learn about ?

 

[Dunuin]

Didn't tried it myself. But it is possible to pipe the content of a file to the stdin so it is handled as user input.
Something like: zfs load-key pool/dataset < /path/to/pwd.file

 

[Dunuin]

I made this script yesterday and looks like it is working.

If you add pairs of datasets and password files (just a file containing the password) to the arrays "DATASETS" and "PWDFILES" and run that script it will unlock them first, if not already unlocked and then it will run zfs mount -a to auto mount them.

But keep in mind that this would be useless if your password files aren't stored on an already unlocked storage. My proxmox is installed on an LUKS encrypted mdraid raid1 array with LVM thin which I unlock using SSH at boot. So my "/root/keys" folder is encrypted if not unlocked first using SSH.

#!/bin/bash

DATASETS=( "VMpool7/VLT" "VMpool8/VLT" )
PWDFILES=( "/root/keys/VMpool7_VLT.pwd" "/root/keys/VMpool8_VLT.pwd" )

unlockDataset () {
    local dataset=$1
    local pwdfile=$2
    # check if dataset exists
    type=$(zfs get -H -o value type ${dataset})
    if [ "$type" == "filesystem" ]; then
        # check if dataset isn't already unlocked
        keystatus=$(zfs get -H -o value keystatus ${dataset})
        if [ "$keystatus" == "unavailable" ]; then
            zfs load-key ${dataset} < ${pwdfile}
            # check if dataset is now unlocked
            keystatus=$(zfs get -H -o value keystatus ${dataset})
            if [ "$keystatus" != "available" ]; then
                echo "Error: Unlocking dataset '${dataset}' failed"
                return 1
            fi
        else
            echo "Info: Dataset already unlocked"
            return 0
        fi
    else
        echo "Error: No valid dataset found"
        return 1
    fi
}

unlockAll () {
    local noerror=0
    # check if number of datasets and pwdfiles are equal
    if [ ${#DATASETS[@]} -eq ${#PWDFILES[@]} ]; then
        # loop though each dataset pwdfile pair
        for (( i=0; i<${#DATASETS[@]}; i++ )); do
            unlockDataset "${DATASETS[$i]}" "${PWDFILES[$i]}"
            if [ $? -ne 0 ]; then
                noerror=1
                break
            fi
        done
    else
        echo "Error: Wrong number of datasets/pwdfiles"
        noerror=1
    fi
    # mount all datasets
    if [ $noerror -eq 0 ]; then
        zfs mount -a
    fi
    return $noerror
}

unlockAll

But I'm still not sure how to run that script automatically at boot at the right after ZFS is ready and before all other stuff like starting VMs is done.

Edit:
I used this systemd service "pve_post_start.service" to start my start script "start.sh" on boot:

[Unit]
Description=PVE Start Script
After=pve-guests.service

[Service]
Type=oneshot
ExecStart=/bin/bash /root/scripts/start.sh
User=root

[Install]
WantedBy=multi-user.target

It needs to be copied to "/etc/systemd/system" and enabled by systemctl enable pve_post_start.

The start.sh uses bash /root/scripts/unlock_zfs.sh to unlock and mount my passphrase encrypted ZFS datasets, mounts my SMB shares and starts all the VMs/LXCs in the right order (pct start 101 for LXCs, sleep 10 to wait a bit, qm start 103 to start VMs, ...)

 

[Elleni]

So for local snapshotting to a usb stick the script works as intended - including importing an encrypted zpool thanks to your tip, Dunuin. Now I have another requirenement.

I have to make the script working on a remote host meaning. The script has to run on a server1 where the removable media is attached, but the snapshots are to be taken on a server2 and sent back to server1 via ssh. Certificates are placed correctly so that ssh server2 from server1 is possible without password authentication. I tried to modify the script as follows:

Line 37 and 38:

zfs snapshot -r $NEWSNAP
zfs send -v $NEWSNAP | zfs recv -F "${BACKUPPOOL}/${DATASET}"

to

ssh server2 'zfs snapshot -r $NEWSNAP'
ssh server2 'zfs send -v $NEWSNAP | ssh server1 zfs recv -F "${BACKUPPOOL}/${DATASET}"'

But this does not work so far, and throws the following error:
The error it throws is missing snapshot argument.

I would be very glad for some additional hint.

 

[Dunuin]

So for local snapshotting to a usb stick the script works as intended - including importing an encrypted zpool thanks to your tip, Dunuin. Now I have another requirenement.

I have to make the script working on a remote host meaning. The script has to run on a server1 where the removable media is attached, but the snapshots are to be taken on a server2 and sent back to server1 via ssh. Certificates are placed correctly so that ssh server2 from server1 is possible without password authentication. I tried to modify the script as follows:

Line 37 and 38:

zfs snapshot -r $NEWSNAP
zfs send -v $NEWSNAP | zfs recv -F "${BACKUPPOOL}/${DATASET}"

to

ssh server2 'zfs snapshot -r $NEWSNAP'
ssh server2 'zfs send -v $NEWSNAP | ssh server1 zfs recv -F "${BACKUPPOOL}/${DATASET}"'

But this does not work so far, and throws the following error:
The error it throws is missing snapshot argument.

I would be very glad for some additional hint.

You are using the '-char. IF I remember right you need to use the "-char or variables won't be parsed.
Did you tried it like this?:

ssh server2 "zfs snapshot -r $NEWSNAP"
ssh server2 "zfs send -v $NEWSNAP | ssh server1 zfs recv -F \"${BACKUPPOOL}/${DATASET}\""

 

[Elleni]

Thank you very much Dunuin, I modified the script as you suggested and it works fine! I really appreciate your help very much!!

Now one last thing. We plan to alternate two removable medias daily, thus the destroying of snapshots the way it is implemented to just keep the number of KEEPOLD snapshots would not work, right? How would you implement it?

Is my thinking correct that if I add a separate KEEPOLD for the pool on the server that is bigger that should do it, right?

 

[Dunuin]

Thank you very much Dunuin, I modified the script as you suggested and it works fine! I really appreciate your help very much!!

Now one last thing. We plan to alternate two removable medias daily, thus the destroying of snapshots the way it is implemented to just keep the number of KEEPOLD snapshots would not work, right? How would you implement it?

Is my thinking correct that if I add a separate KEEPOLD for the pool on the server that is bigger that should do it, right?

Keep in mind that this script is sending snapshots incrementally. If a snapshot was deleted before it is sent to all backup drives you need to send the complete dataset again and not just the changes since the last backup.
I also modified that script once to backup my FreeNAS server because I missed some features. For example is it with that script not possible to backup recursive datasets. And if I remember right the way old snapshots were deleted was problematic.

 

[Elleni]

Dunuin - your help was very valuable and usefull for me so once again - a big thank you. Without it I would not have been able to implement this automatation of snapshotting which by the way is only one of the backup layers we have. Besides that we also have setup a proxmox backup server and as a third layer we do rsync of our valuable data to an offsite server, so I can greatly live with the limitations of the script.

One final question - until now I only use the script to backup vm disks but it came to my mind that in case of a need to reinstall the proxmox host(s) it would come in handy to also have a snapshot ready to restore the proxmox ve host configurations. Is my assumption correct that for this all I need is to add snapshots for rpool/ROOT/pve-1 of my proxmox servers? And would it be enough to only snapshot one of the proxmox nodes pve-1 (assuming that they are identical in a cluster) or would it be better to do this for every cluster node? Finally what would be the correct values for the MASTERPOOL and DATASET variables? MASTERPOOL=rpool/ROOT and Dataset=pve-1 ?

Edit to add that I wasn't yes successfull in modifying the script in order to to save the pve-1 snapshot with a different name on the same stick, so I can do snapshots from multiple pve-1 pools of all our servers. ex. node1-pve1, node2-pve2. Else it works fine with one pve1 snapshot, but will fail on the second. I still have to find out, where to put the prefix node1-, node2- in the script so creation and deletion of multiple pve1 snapshots from multiple servers can be sent to the same stick.

And is my assumption correct that for a restore of a configured proxmox pve node it would be enough to install proxmox, then boot in rescue mode and restore EXTBACKUP/ROOT/pve-1 to rpool/ROOT/pve-1 and everything including networking, vm configs would be there?

 

[edwin eefting]

Please have a look at ZFS autobackup.

It can do exactly what you need without extra steps i think: https://github.com/psy0rz/zfs_autobackup