Backup not working anymore

Eric Delaet · Nov 2, 2020

Hello guys,

Just upgraded to 0.9.4-2.

My backups and restores are not working anymore. Reverting to a previous version doesn't seem to help.

2020-11-02 22:37:52 INFO: Starting Backup of VM 104 (qemu)
2020-11-02 22:37:52 INFO: status = running
2020-11-02 22:37:52 INFO: VM Name: jumphost
2020-11-02 22:37:52 INFO: include disk 'scsi0' 'stor01:vm-104-disk-0' 20G
2020-11-02 22:37:52 INFO: backup mode: snapshot
2020-11-02 22:37:52 INFO: ionice priority: 7
2020-11-02 22:37:52 INFO: creating Proxmox Backup Server archive 'vm/104/2020-11-02T21:37:52Z'
2020-11-02 22:37:52 INFO: issuing guest-agent 'fs-freeze' command
2020-11-02 22:37:52 INFO: enabling encryption
2020-11-02 22:37:52 INFO: issuing guest-agent 'fs-thaw' command
2020-11-02 22:37:52 ERROR: VM 104 qmp command 'backup' failed - proxmox_backup_new failed: failed to load decryption key from "/etc/pve/priv/storage/PBS.enc"
2020-11-02 22:37:52 INFO: aborting backup job
2020-11-02 22:37:52 ERROR: Backup of VM 104 failed - VM 104 qmp command 'backup' failed - proxmox_backup_new failed: failed to load decryption key from "/etc/pve/priv/storage/PBS.enc"

/etc/pve/priv/storage/PBS.enc is readable and not changed since my install on september. Permissions are root:www-data, 0600.

Also restoring old backups doesn't seem to work anymore in this version, when I try to restore something from the PBS store:

Error: failed to parse rfc3339 timestamp ("2020-09-22T11:39:02.821501271+02:00") - timestamp of unexpected length at line 1 column 60
TASK ERROR: command '/usr/bin/proxmox-backup-client restore '--crypt-mode=encrypt' '--keyfd=12' vm/104/2020-10-26T02:04:45Z index.json /var/tmp/vzdumptmp34851/index.json --repository root@pam@10.0.0.4:nfs-backup-store' failed: exit code 255

Any ideas ?

Thanks !

dcsapak · Nov 3, 2020

seems like this issue: https://forum.proxmox.com/threads/e...g-length-at-line-1-column-60-500.76572/page-2

solution is:

please remove the subsecond part of the timestamps in /etc/pve/priv/storage/STORAGEID.enc for that storage definition.

Klug · Nov 3, 2020

Same issue here since yesterday (this night backup did not happen).
What changed: yesterday I upgraded my PBS to 0.9-4 and my PVE nodes to 6.2-15.

Klug · Nov 3, 2020

dcsapak said:
seems like this issue: https://forum.proxmox.com/threads/e...g-length-at-line-1-column-60-500.76572/page-2

solution is:

It seems this fixed the issue, backup is working when launched from PVE GUI.

A new dirty-bitmap is created for each VM.
Is it because I rebooted the nodes (post-upgrade) or because I moved the VM (to reboot the nodes)?

dcsapak · Nov 3, 2020

Klug said:
A new dirty-bitmap is created for each VM.
Is it because I rebooted the nodes (post-upgrade) or because I moved the VM (to reboot the nodes)?

dirty bitmap support on migration is rather new (libproxmox-backup-qemu0 >= 0.7.1-1) (and i do not know right now if it is already available on the repos)

t.lamprecht · Nov 3, 2020

It is available on pve-no-subscription, it needs the following package versions installed:

Code:

libproxmox-backup-qemu0 0.7.1-1
pve-qemu-kvm 5.1.0-4
qemu-server 6.2-18

And the VM needs to me either migrated once to an updated node, or rebooted fully after installing those packages, so that the new qemu and librbary is loaded.

NOTE: this breaks "new" -> "old" migrations, but once you update also the "old" node it works again.

Eric Delaet · Nov 3, 2020

dcsapak said:
seems like this issue: https://forum.proxmox.com/threads/e...g-length-at-line-1-column-60-500.76572/page-2

solution is:

Thanks! I seem to have missed that post, sorry. But it seems to have fixed the issue, i can run a manual backup again and I succesfully restored a machine as well !

CgLU43d · Dec 30, 2020

I'm also having this issue
I'll continue troubleshooting but a bit concerned by the lack of mention of it on this forums

My versions
libproxmox-backup-qemu0/stable,now 1.0.2-1 amd64 [installed]
pve-qemu-kvm/stable,now 5.1.0-7 amd64 [installed]
qemu-server/stable,now 6.3-2 amd64 [installed]

Brand new PVE and brand new PBS on two dif boxes

Backup works beautiful without encryption but gives this error once encryption is added to the store

Code:

root@pvename:~# vzdump 110 --mode snapshot --node pvename --storage storagename
INFO: starting new backup job: vzdump 110 --node pvename --mode snapshot --storage storagename
INFO: Starting Backup of VM 110 (qemu)
INFO: Backup started at 2020-12-30 hh:mm:ss
INFO: status = running
INFO: VM Name: pass
INFO: include disk 'scsi0' 'zfspoolname:vm-110-disk-0' 15G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/110/2020-12-30T08:33:42Z'
INFO: enabling encryption
ERROR: VM 110 qmp command 'backup' failed - proxmox_backup_new failed: failed to load decryption key from "/etc/pve/priv/storage/storagename.enc"
INFO: aborting backup job
ERROR: Backup of VM 110 failed - VM 110 qmp command 'backup' failed - proxmox_backup_new failed: failed to load decryption key from "/etc/pve/priv/storage/storagename.enc"
INFO: Failed at 2020-12-30 hh:mm:ss
INFO: Backup job finished with errors
job errors

It's definitely not an issue with the timestamp thing
Time redacted for privacy but is in this format with no microseconds on the end
{"kdf":null,"created":"2020-12-30Thh:mm:ss+xx:xx","modified":"2020-12-30Thh:mm:ss+xx:xx","data":"CENSORED","fingerprint":"59:ed:09:01:63:64:8c:48:6c:9a:b5:cc:de:73:c5:ae:fd:4e:8a:34:c2:13:08:a9:b8:cb:eb:a7:b3:51:69:73"}

Starting to think this is a permissions issue but suppose the vzdump should be running as root and root can definitely access it so not sure at this stage

fabian · Dec 30, 2020

what does proxmox-backup-client key show /path/to/key say?

CgLU43d · Dec 30, 2020

fabian said:
what does proxmox-backup-client key show /path/to/key say?

Thank you! That totally fixed it - feel like such an idiot. It was very close to the original problem mentioned above

Error: failed to parse rfc3339 timestamp ("2020-12-30Thh:mm:ss+06:30") - value too large (30 > 23) at line 1 column 49

So most timezones have normal offset by the hour from UTC, but some timezones like mine have 30 minute increments too
This is not liked by the backup client
I was able to make it work by changing the created and modified timestamps to +00:00
And now it works perfect

I will include examples of working and non working for your debugging pleasure
Apparently there are also 45 minute incremental timezones somewhere

Broken

Code:

{"kdf":null,"created":"2020-12-30T00:00:00+06:30","modified":"2020-12-30T00:00:00+06:30","data":"7sqmWq9hTAtpVkZezh3im/vRE2Xzh6htgt67LnwF8FQ=","fingerprint":"36:0c:8b:5c:22:98:d3:34:47:0f:d3:60:32:79:8a:9b:3d:3d:37:5f:fb:74:65:82:d0:0d:10:63:62:ed:33:fd"}

Working

Code:

{"kdf":null,"created":"2020-12-30T00:00:00+00:00","modified":"2020-12-30T00:00:00+00:00","data":"7sqmWq9hTAtpVkZezh3im/vRE2Xzh6htgt67LnwF8FQ=","fingerprint":"36:0c:8b:5c:22:98:d3:34:47:0f:d3:60:32:79:8a:9b:3d:3d:37:5f:fb:74:65:82:d0:0d:10:63:62:ed:33:fd"}

I appreciate your prompt help a lot

fabian · Dec 30, 2020

thanks, that should be easy enough to fix (and sorry for the inconvenience!)

fabian · Dec 30, 2020

https://lists.proxmox.com/pipermail/pbs-devel/2020-December/001805.html

Search

Search

Backup not working anymore

Eric Delaet

Member

dcsapak

Proxmox Staff Member

Klug

Well-Known Member

Klug

Well-Known Member

dcsapak

Proxmox Staff Member

t.lamprecht

Proxmox Staff Member

Eric Delaet

Member

CgLU43d

New Member

fabian

Proxmox Staff Member

CgLU43d

New Member

fabian

Proxmox Staff Member

fabian

Proxmox Staff Member