Proxmox Backup Server 4.0 BETA released!

Hi,
CRCinAU said:
I upgraded to PBS 4.0.8 today, mostly seems to be fine.
Click to expand...
allow me to ask why mostly? Did you ran into some issues? If so, please report them here or open an issue at bugzilla.proxmox.com, thanks!

CRCinAU said:
Is there any update on using LUKS encrypted disks as removable storage?
Click to expand...
No, this has not been implemented yet, the feature request is tracked as https://bugzilla.proxmox.com/show_bug.cgi?id=6473, please subscribe to get progress notifications on that.
 
Chris said:
allow me to ask why mostly? Did you ran into some issues? If so, please report them here or open an issue at bugzilla.proxmox.com, thanks!
Click to expand...

Nah - nothing to do with the update. Was more me switching from grub -> systemd-boot. My error :)

Chris said:
No, this has not been implemented yet, the feature request is tracked as https://bugzilla.proxmox.com/show_bug.cgi?id=6473, please subscribe to get progress notifications on that.
Click to expand...

Fair enough - I think I wrote that request :)

I might convert a couple of ones I've manually hacked in to non-encrypted to test out the auto-sync jobs etc and go from there until that feature becomes available.
 
Stubennerd said:
I successfully added Hetzner Object Storage in the "S3 Endpoints" configuration but add it as Datastore fails with "failed to access bucket (400)"
Click to expand...
Adding endpoints itself is currently not much checked, so being able to add one is not that telling.

I successfully tested the Hetzner S3 using
Endpoint: {{bucket}}.{{region}}.your-objectstorage.com
Region: nbg1 (for Nürnberg, just take the one for whatever you chose)
And do not select "Path Style"

If you still got problems please post your config (with access key censored!).
 
  • Like
Reactions: kavaa and Stubennerd
The update completed successfully without any issues. However, I’m now being logged out of the web UI every few seconds.

Each time I'm logged out, I see the following message in the browser console:

Code: 
__Host-PBSAuthCookie=; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; SameSite=Lax; HttpOnly; Path=/;

The server’s date and time zone are correctly configured.
Is this a known issue, or are there any recommended steps to resolve it?
 
Bent said:
The update completed successfully without any issues. However, I’m now being logged out of the web UI every few seconds.

Each time I'm logged out, I see the following message in the browser console:

Code: 
__Host-PBSAuthCookie=; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; SameSite=Lax; HttpOnly; Path=/;

The server’s date and time zone are correctly configured.
Is this a known issue, or are there any recommended steps to resolve it?
Click to expand...
We switched PBS 4 over to the more secure HttpOnly cookies for the login ticket, but we could iron out most issues on upgrade on internal testing, which doesn't mean that there isn't some edge case left-over.

Can you please try to clear the browser cache and its local storage for the PBS instance? That can normally be done either fine-grained through the lock icon (firefox) or settings icon (chromium) on the left side of the browser's url bar, alternatively by opening the developer tools/console of your browser when having the PBS UI loaded. Then force-reload the web UI.

While that should not be required, it might still be good to check, especially force-reloading the UI might clear some leftover frontend code from your browsers caches.
 
Any idea if we can speed up the transfer out to Wasabi? Currently pulling from the local datastore to the wasabi datastore and the average MB/s is around 5 or so. We do have a 258 Mbps upload to the WAN which calculates out to something like 32 MB/s so it doesn't need to go this slow, I have no throttling enabled or anything of the sort configured.

1754374727209.png
 
Last edited:
complexplaster27 said:
Any idea if we can speed up the transfer out to Wasabi? Currently pulling from the local datastore to the wasabi datastore and the average MB/s is around 5 or so. We do have a 258 Mbps upload to the WAN which calculates out to something like 32 MB/s so it doesn't need to go this slow, I have no throttling enabled or anything of the sort configured.

View attachment 88867
Click to expand...
How much upload speed do you expect to get from wasabi? Note that this not only depends on your local upload speed but also the rate wasabi allows you to upload with. Maybe you can check by uploading a chunk folder to a bucket by some external tooling and compare the speeds.

And yes, there is still a lot of room for improvement. Currently there is a 1 to 1 mapping for chunks, so each chunk upload requires an individual api request and no multipart upload is used. There are plans however to improve this by more intelligently handling the chunks and pack them into larger portions for upload.
 
  • Like
Reactions: complexplaster27
t.lamprecht said:
We switched PBS 4 over to the more secure HttpOnly cookies for the login ticket, but we could iron out most issues on upgrade on internal testing, which doesn't mean that there isn't some edge case left-over.

Can you please try to clear the browser cache and its local storage for the PBS instance? That can normally be done either fine-grained through the lock icon (firefox) or settings icon (chromium) on the left side of the browser's url bar, alternatively by opening the developer tools/console of your browser when having the PBS UI loaded. Then force-reload the web UI.

While that should not be required, it might still be good to check, especially force-reloading the UI might clear some leftover frontend code from your browsers caches.
Click to expand...
Thanks for the reply. I tried force reloading the page "forget this page", clearing browsing data and used a completely clean browser (Chromium) for testing. All with the same outcome unfortunately.

For what it's worth I found these errors in systemd logs, does this help getting to the root of the issue or do you need more information?

Bash: 
root@pbs ~ # journalctl -b0 --no-pager --priority err
[...]]
Aug 03 19:42:56 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:49016 user=root@pam msg=authentication error - SUCCESS (0)
Aug 03 19:43:04 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:48990 user=root@pam msg=authentication error - SUCCESS (0)
Aug 03 19:43:07 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:48990] permission check failed.
Aug 03 20:01:38 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:36910 user=root@pam msg=authentication error - SUCCESS (0)
Aug 03 20:01:41 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:36910] permission check failed.
Aug 03 20:01:45 <redacted> proxmox-backup-proxy[773]: authentication failure; rhost=[::ffff:<redacted>]:36910 msg=no authentication credentials provided.
Aug 03 20:01:48 <redacted> proxmox-backup-proxy[773]: GET /api2/extjs/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:36910] authentication failed - no authentication credentials provided.
Aug 03 20:01:48 <redacted> proxmox-backup-proxy[773]: GET /favicon.ico: 404 Not Found: [client [::ffff:<redacted>]:36910] no such file: 'favicon.ico'
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37486] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37558] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37546] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37562] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37572] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:19:13 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:58340 user=root@pam msg=authentication error - SUCCESS (0)
Aug 04 08:19:16 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:58340] permission check failed.
Aug 05 08:11:17 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:39930 user=root@pam msg=authentication error - SUCCESS (0)
Aug 05 08:11:20 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:39930] permission check failed.
 
Bent said:
Thanks for the reply. I tried force reloading the page "forget this page", clearing browsing data and used a completely clean browser (Chromium) for testing. All with the same outcome unfortunately.

For what it's worth I found these errors in systemd logs, does this help getting to the root of the issue or do you need more information?

Bash: 
root@pbs ~ # journalctl -b0 --no-pager --priority err
[...]]
Aug 03 19:42:56 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:49016 user=root@pam msg=authentication error - SUCCESS (0)
Aug 03 19:43:04 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:48990 user=root@pam msg=authentication error - SUCCESS (0)
Aug 03 19:43:07 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:48990] permission check failed.
Aug 03 20:01:38 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:36910 user=root@pam msg=authentication error - SUCCESS (0)
Aug 03 20:01:41 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:36910] permission check failed.
Aug 03 20:01:45 <redacted> proxmox-backup-proxy[773]: authentication failure; rhost=[::ffff:<redacted>]:36910 msg=no authentication credentials provided.
Aug 03 20:01:48 <redacted> proxmox-backup-proxy[773]: GET /api2/extjs/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:36910] authentication failed - no authentication credentials provided.
Aug 03 20:01:48 <redacted> proxmox-backup-proxy[773]: GET /favicon.ico: 404 Not Found: [client [::ffff:<redacted>]:36910] no such file: 'favicon.ico'
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37486] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37558] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37546] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37562] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:02:26 <redacted> proxmox-backup-proxy[773]: [[::ffff:<redacted>]:37572] failed to check for TLS handshake: couldn't peek into incoming TCP stream
Aug 04 08:19:13 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:58340 user=root@pam msg=authentication error - SUCCESS (0)
Aug 04 08:19:16 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:58340] permission check failed.
Aug 05 08:11:17 <redacted> proxmox-backup-api[719]: authentication failure; rhost=[::ffff:<redacted>]:39930 user=root@pam msg=authentication error - SUCCESS (0)
Aug 05 08:11:20 <redacted> proxmox-backup-api[719]: POST /api2/json/access/ticket: 401 Unauthorized: [client [::ffff:<redacted>]:39930] permission check failed.
Click to expand...
Hi,
what version of proxmox-backup-server is currently running? It seems that 4.0.7-1 pulled in some fixes not present in 4.0.6-1, especially in the newer version the `Expire` is not set for HttpOnly cookies anymore, but it does seem to be present in your cookie as shown above.
 
Bent said:
For what it's worth I found these errors in systemd logs, does this help getting to the root of the issue or do you need more information?
Click to expand...
Thank you, can you try to log in via ssh and update to the latest version right now? Installing from the ISO, it seems that PAM has some problem logging in. Especially this part:

Code: 
authentication failure; rhost=[::ffff:10.10.10.1]:42160 user=root@pam msg=authentication error - SUCCESS (0)

Seems suspicious. Basically this can only happen if PAM returns an error code that isn't Success but that makes no sense because the line itself states that it was successful. Something here seems to have gotten mixed up. However, it is fixed when I update everything to current state of the test repository.

The issue mentioned by Chris could also be at fault, but would not explain that message in the logs.
 
  • Like
Reactions: Chris
sterzy said:
Thank you, can you try to log in via ssh and update to the latest version right now? Installing from the ISO, it seems that PAM has some problem logging in. Especially this part:

Code: 
authentication failure; rhost=[::ffff:10.10.10.1]:42160 user=root@pam msg=authentication error - SUCCESS (0)

Seems suspicious. Basically this can only happen if PAM returns an error code that isn't Success but that makes no sense because the line itself states that it was successful. Something here seems to have gotten mixed up. However, it is fixed when I update everything to current state of the test repository.

The issue mentioned by Chris could also be at fault, but would not explain that message in the logs.
Click to expand...
Updating from 4.0.8-1 -> 4.0.9-1 did not resolve the issue unfortunately. Just saw @sterzy latest post regarding the patch. Any 'easy' way to test this immediately on our system without waiting for the next release / any build instructions?
 
Bent said:
Updating from 4.0.8-1 -> 4.0.9-1 did not resolve the issue unfortunately. Just saw @sterzy latest post regarding the patch. Any 'easy' way to test this immediately on our system without waiting for the next release / any build instructions?
Click to expand...
No, but the error message being incorrect is only cosmetic (as the wrong error state was logged), the underlying checks are correct. Please post the output of proxmox-backup-manager version --verbose --output-format json-pretty, best as file attachment
 
Last edited:
  • Like
Reactions: sterzy
Chris said:
No, but the error message being incorrect is only cosmetic (as the wrong error state was logged), the underlying checks are correct. Please post the output of proxmox-backup-manager version --verbose --output-format json-pretty, best as file attachment
Click to expand...
Here you go, let me know if there is anything else!
 

Attachments

Chris said:
Status update on this one, I was able to reproduce and identify the issue. The Backblaze B2 implementation of the S3 api seems to not support the If-None-Match http header, therefore the requests failing with above error. Patches to fix this have been implemented [0], with these I was able to operate just fine on a Backblaze B2 bucket, so this will be fixed in an upcoming version.

Nevertheless, may I ask you to also report this issue to Backblaze, asking for supporting the If-None-Match header in put object requests as well. The more interest the see, the more likely this will be implemented and improve S3 api compatibility.

[0] https://lore.proxmox.com/pbs-devel/20250728100154.556438-1-c.ebner@proxmox.com/T/
Click to expand...
LOL just came here for this exact issue :)

Installed Beta 1 and using B2 got the same deal so guess I have to wait for a new Beta Build for the fix?
 
The fix from Chris is already included in Proxmox Backup Server 4.0.10-1, which was uploaded to the pbs-test beta repo a few hours ago. Check the advanced options in the edit window that opens for creating or editing and S3 Endpoint.
 
  • Like
Reactions: Johannes S and ccolotti
You must log in or register to reply here.
Top Bottom