Backup on S3 GLACIER : unable to rename (so fail)

[supervache]

Hi,
I'm trying to store my proxmox 7.2-4 VM's backup on Scaleway s3.
I followed this links :
- this blog post (translated)
- this documentation

Backups works fine, but are stored in S3 "STANDARD" class (more "expansive"). So I tried to change the s3fs parameters to use "glacier" like this :
[B]s3fs test-s3-glacier /mnt/pve-s3-scw -o passwd_file=/etc/passwd-s3fs -o url=https://s3.fr-par.scw.cloud/ -o storage_class=glacier[/B]

-o storage_class found here

But with this param, backup don't work as my logs explain :

INFO: starting new backup job: vzdump 236 --compress zstd --node acc-host-0006 --mode snapshot --remove 0 --storage s3-scw --notes-template '{{guestname}}'
INFO: Starting Backup of VM 236 (lxc)
INFO: Backup started at 2022-06-17 14:56:40
INFO: status = running
INFO: CT Name: ***
INFO: including mount point rootfs ('/') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: ***
INFO: including mount point rootfs ('/') in backup
INFO: starting first sync /proc/5676/root/ to /var/lib/vz/datastore-nvme/dump/vzdumptmp119014_236/
INFO: first sync finished - transferred 7.65G bytes in 24s
INFO: suspending guest
INFO: starting final sync /proc/5676/root/ to /var/lib/vz/datastore-nvme/dump/vzdumptmp119014_236/
INFO: final sync finished - transferred 0 bytes in 1s
INFO: resuming guest
INFO: guest is online again after 1 seconds
INFO: creating vzdump archive '/mnt/pve-s3-scw/dump/vzdump-lxc-236-2022_06_17-14_56_39.tar.zst'
INFO: Total bytes written: 7604183040 (7.1GiB, 267MiB/s)
ERROR: Backup of VM 236 failed - unable to rename '/mnt/pve-s3-scw/dump/vzdump-lxc-236-2022_06_17-14_56_39.tar.dat' to '/mnt/pve-s3-scw/dump/vzdump-lxc-236-2022_06_17-14_56_39.tar.zst'
INFO: Failed at 2022-06-17 14:57:49
INFO: Backup job finished with errors
TASK ERROR: job errors

My chmod seems good ?

root@acc-host-0006:/mnt/pve-s3-scw/dump# ls -lah
total 781M
drwxr-xr-x 1 root root    0 Jun 17 12:28 .
drwx------ 1 root root    0 Jan  1  1970 ..
-rw-r--r-- 1 root root   87 Jun 17 12:31 vzdump-lxc-236-2022_06_17-12_29_57.tar.zst.notes
-rw-r--r-- 1 root root 1.6K Jun 17 14:54 vzdump-lxc-236-2022_06_17-14_52_42.log
-rw-r--r-- 1 root root 1.6K Jun 17 14:57 vzdump-lxc-236-2022_06_17-14_56_39.log
-rw-r--r-- 1 root root 1.6K Jun 17 15:27 vzdump-lxc-236-2022_06_17-15_25_22.log
-rw-r--r-- 1 root root 781M Jun 17 15:26 vzdump-lxc-236-2022_06_17-15_25_22.tar.zst
-rw-r--r-- 1 root root   54 Jun 17 15:27 vzdump-lxc-236-2022_06_17-15_25_22.tar.zst.notes

Any idea why it works whitout class mode but not in glacier mode ? (Scaleway support explain me that the problem was not related to scaleway but proxmox.)

Thank you !

 

[sterzy]

Hi,

not entirely sure about this, so I'd be happily corrected, but:

ERROR: Backup of VM 236 failed - unable to rename '/mnt/pve-s3-scw/dump/vzdump-lxc-236-2022_06_17-14_56_39.tar.dat' to '/mnt/pve-s3-scw/dump/vzdump-lxc-236-2022_06_17-14_56_39.tar.zst'

Indicates that the backup archive could not be renamed (which is a step in the backup process). AFAICT S3 glacier doesn't really support rename operations [1]:

Updating an Archive in S3 Glacier​

After you upload an archive, you cannot update its content or its description. The only way you can update the archive content or its description is by deleting the archive and uploading another archive.

So I doubt you can use this storage type. Couldn't find this in Scaleway's own documentation, but I'd assume this also holds true there (or in any case, holds true for s3fs support for glacier type storage).

[1]: https://docs.aws.amazon.com/amazonglacier/latest/dev/working-with-archives.html

 

[supervache]

Thanks for your answer.
You probably right : I think like you that SCW has the same constraints with glacier as AWS.

I will try to mount s3fs in standard mode and then periodically run a cron job that will change the dump storage mode to glacier.
It's not ideal, but it could be the most efficient.

I could also store my backups locally, and do a cron job that copies to glacier and erases the local dump. It's more risky in my opinion: if the script crashes for some unknown reason, the local disk can fill up quickly.
In the previous case, if the script does not work, it is just the billing of the S3 service that will be higher, but without the risk of a service cut.

What do you think about this first solution ? Maybe you have a best idea ?

If it works, I'll share the script below for future readers.

 

[yetisbey]

Hi,
I would like to know if this worked for you because I’m thinking the same thing and it would be super for me to know before I invest time. Thank you in advance @supervache
Regards.

 

[supervache]

hi @yetisbey sorry for my late answer (back from vacation).
With S3 "Glacier" services, there is no capabilities to rename a file. So the backup workflow of proxmox 7.2-7 (an previous as I know) can't use directly this feature.

An option is to use your S3 provider's lifecycle option to turn your file to glacier after a certain time (E.G 24h, 3Days ...) In scaleway for my case, this feature is in beta.

All my best regards