Hello dear Community,
I am running a PBS 4.1 server (community repo) which stores backups from a PVE 9.1 HA cluster (also community repo) to an on-prem DeuxFleurs Garage S3 storage cluster. The PBS has a local cache made of an RAID5 array which has a usable size of about 130GB.
I experienced a weird behaviour in that small-ish (50-70GB VMs) got backed up perfectly, but all backups of large (1TB+ VMs) eventually failed. This was consistent.
All physical hosts are on the same site, physically connected over a fast network that is definitely not a bottleneck of any kind.
What this forum thread is NOT:
So, there are 2 tweaks to implement:
After this, don't forget to restart the services (or reboot if you're unsure) and the backups to S3 will work like a charm!
Enjoy!
Denis
I am running a PBS 4.1 server (community repo) which stores backups from a PVE 9.1 HA cluster (also community repo) to an on-prem DeuxFleurs Garage S3 storage cluster. The PBS has a local cache made of an RAID5 array which has a usable size of about 130GB.
I experienced a weird behaviour in that small-ish (50-70GB VMs) got backed up perfectly, but all backups of large (1TB+ VMs) eventually failed. This was consistent.
All physical hosts are on the same site, physically connected over a fast network that is definitely not a bottleneck of any kind.
What this forum thread is NOT:
- A complain of any sort. S3 support is still labeled as "tech-preview" and shall be handled as such
- A "pls-fix-asap" request to the dev team. Again, S3 support is tech preview
- An attempt to get free support since I'm too cheap to buy commercial support (okay, my infrastructure is a private, home deployment, but anyway...)
So, there are 2 tweaks to implement:
- On PBS, you need to enable the skip-if-none-match-header. From the CLI, enter:
proxmox-backup-manager s3 endpoint update garage-s3-ep --provider-quirks skip-if-none-match-header- On the Garage S3 gateway node (which accepts the client requests), you have to modify its nginx config file (
/etc/nginx/sites-available/my_s3_example.conf) to change its buffering proxy to a streaming proxy behaviour by modifying the following section:
Code:
location / {
proxy_pass http://my_S3_gateway_node;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_max_temp_file_size 0;
proxy_connect_timeout 600s;
proxy_send_timeout 3600s;
proxy_read_timeout 3600s;
proxy_http_version 1.1;
proxy_request_buffering off;
}After this, don't forget to restart the services (or reboot if you're unsure) and the backups to S3 will work like a charm!
Enjoy!
Denis