How to migrate existing backups to PBS

proximoxi2

Active Member
Nov 1, 2019
15
0
41
48
We currently have two debian servers with a zfs pool to which we backups our containers and vm's (via NFS). We now want to migrate to PBS. Replace one of the servers to PBS, migrate the existing backups and then replace the other server.

But how do we migrate existing backups to PBS? I have not found any documentation about this.
 
Since it is on the roadmap since three and a half years I guess I should not expect that it gets implemented soon.
Restoring a few hundred backups does not scale very well, but I think we could extract the archives sequentially and use the proxmox backup client to backup them to pbs.
 
no, for container backups no such tool exists yet. you can always extract the backup (it's a tar file containing the rootfs and the container configs), and then back it up again using proxmox-backup-client like vzdump does (that allows matching the PBS snapshot timestamp to that of the old backup), or you could use pct restore and vzdump and lose the timestamps..
 
vma-to-pbs work well There were only two unexpected downsides. By default vma-to-pbs will use the current timestamp when importing the backup which makes importing the backup kind of useless when i have 10 backups from "now". There is `--backup-time <EPOCH>` , but that should be used in the examples imho. Also it would be nice if `vma-to-pbs --help` could specify what EPOCH is, but it seems to be a unix timestamp in seconds.
If you already have newer backups of the VM you get a
`Error: proxmox_backup_connect failed: command error: backup timestamp is older than last backup.`
But we could work around that by moving some files on the pbs server.

And here is a script you maybe also could add to the wiki because i guess migration from file based backups to pbs is the most common use case for this tool:
```bash
#!/bin/bash
for file in *.vma.zst; do
vmid=$(echo "$file" | grep -oP '(\d+)-\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}' | grep -oP '^\d+')
backup_time=$(date -d "$(echo "$file" | grep -oP '\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}' | tr '_' ' ')" +"%s")
zstd -d --stdout "$file" | \
PBS_FINGERPRINT='PBS-FINGERPRINT' vma-to-pbs \
--repository 'user@realm!token@server:port:datastore' \
--vmid "$vmid" \
--backup-time "$backup_time" \
--password-file path/to/pbs_password
done
```

And for imported VMs the comment is empty, but this information is there in the `qemu-server.conf` file. It would be nice if the name from there is used as comment.
 
In follow up to @proximoxi2's reply :
1. I fixed his script, it was not counting the date correctly
2. I added logging to the file
3. I found a way to record comments to backups, but I didn't add it, the reason will be below.

Bash:
#!/bin/bash
# Settings
PATH_TO_BACKUP_DUMPS="/mnt/old_backups/dump"
PBS_FINGERPRINT="aa:bb:cc......"
PBS_REPOSITORY="user@pam@192.168.1.1:my-backup-storage"
PBS_PASSWORD_FILE="/tmp/pbs-password"
LOG_FILE_PATH=/tmp/vma-to-pbs.log

# Prepre backup data
cd $PATH_TO_BACKUP_DUMPS
touch $LOG_FILE_PATH
for file in *.vma.zst; do
vmid=$(echo "$file" | grep -oP '(\d+)-\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2}' | grep -oP '^\d+')
backup_day="$(echo "$file" | grep -oP '\d{4}_\d{2}_\d{2}' | tr '_' '/')"
backup_time="$(echo "$file" | grep -oP '\-\d{2}_\d{2}_\d{2}' | tr '_' ':')"
backup_time="${backup_time:1}"
backup_full_time=$(date -d "$backup_day $backup_time" +"%s")


# Start backup
echo "Start backup: $file" | tee -a $LOG_FILE_PATH
zstd -d --stdout "$file" | \
vma-to-pbs \
--fingerprint "${PBS_FINGERPRINT}" \
--repository "${PBS_REPOSITORY}" \
--vmid "$vmid" \
--backup-time "$backup_full_time" \
--password-file ${PBS_PASSWORD_FILE} 2>&1 | tee -a $LOG_FILE_PATH >/dev/null
tail -n 1 /tmp/vma-to-pbs.log
echo -e "\n\n\n" | tee -a $LOG_FILE_PATH
done
echo "Log file path: $LOG_FILE_PATH"


And now about adding comments.
Comments can be added using proxmox API, here is a link to the right one:
https://pbs.proxmox.com/docs/api-viewer/index.html#/admin/datastore/{store}/notes

What's the problem:
I found instructions on how to use the API (https://pve.proxmox.com/wiki/Proxmox_VE_API), but following the instructions never worked for me, failed to authorize, but it works because - if you "steal" the request from the browser and edit it, everything works fine.

Now I will tell you how to "steal" this request:
1. open Datastore -> my-storage -> Content
2. select any backup -> click "Edit" in comment section -> type something and NOT click "ok".
3. type f12 key and open "Network" section
4. Click "Ok" to save comment
5. Find the query named "notes" in queries and click RMB -> Copy -> Copy as cURL
6. As a result you will get something like this

Bash:
curl 'https://192.168.1.1:8007/api2/extjs/admin/datastore/my-storage/notes' \
  -X 'PUT' \
  -H 'CSRFPreventionToken: djshfdhsgfhjdsgfhj' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' \
  -H 'Cookie: dlhgjdfkghjdfkhgjdfkh' \
  -H 'X-Requested-With: XMLHttpRequest' \
  --data-raw 'backup-type=vm&backup-id=114&backup-time=<timestamp>&notes=MyComment' \
  --compressed \
  --insecure

7. Now edit it as you want, replace the necessary id's and write how to read the comment file from the console into a variable and put this query after the import backup command.

Sources:
- https://pbs.proxmox.com/wiki/index.php/Import_VMA_Backups_into_Proxmox_Backup_Server
- https://forum.proxmox.com/threads/backup-server-api-access.133151/
- https://pve.proxmox.com/wiki/Proxmox_VE_API
- https://forum.proxmox.com/threads/query-if-a-backup-task-is-running.112019/#post-550198
- https://forum.proxmox.com/threads/pbs-client-authentication-with-api-token-doesnt-work.79457/
 
Last edited: