OpenID 401 with Azure AD

And today this issue is back, i took a quick look into pveproxy/access.log. I can see there is a callback which starts with ?code=... and a long string which is about 1066 bytes long, followed by some more stuff like state=, session_state= and so on. Maybe the issue is that code= is over 1024 characters? or the whole string itself is to large with around 1220 bytes.
 
I've sent a patch increasing the limit to 2048 for now. This should fix the issues with Azure for the time being.
 
  • Like
Reactions: talos
What repository is this patch on? I need it in the non enterprise repo to apply to my servers since it's impacting some of my users
 
The patch has been applied and will be part of the next libpve-access-control release (7.2-4 or higher).
If you want to manually change it until then, you can do so by editing /usr/share/perl5/PVE/API2/OpenId.pm and changing the limit for the `code` parameter in the `login` API call accordingly.

Code: 
__PACKAGE__->register_method ({
    name => 'login',
    path => 'login',
    method => 'POST',
    protected => 1,
    description => " Verify OpenID authorization code and create a ticket.",
    parameters => {
    additionalProperties => 0,
    properties => {
        'state' => {
        description => "OpenId state.",
        type => 'string',
        maxLength => 1024,
            },
 
mira said:
The patch has been applied and will be part of the next libpve-access-control release (7.2-4 or higher).
If you want to manually change it until then, you can do so by editing /usr/share/perl5/PVE/API2/OpenId.pm and changing the limit for the `code` parameter in the `login` API call accordingly.

Code: 
__PACKAGE__->register_method ({
    name => 'login',
    path => 'login',
    method => 'POST',
    protected => 1,
    description => " Verify OpenID authorization code and create a ticket.",
    parameters => {
    additionalProperties => 0,
    properties => {
        'state' => {
        description => "OpenId state.",
        type => 'string',
        maxLength => 1024,
            },
Click to expand...
So change the 1024 to 2048?
And what service do you need to restart to get the new value active?
And can this been done on an running environment?
 
Restart pvedaemon and pveproxy with the following command:
systemctl restart pvedaemon.service pveproxy.service

The patch will set the new limit to 4096, so choose whichever you prefer and is enough for your use case.

If you don't have any errors in the code, it should not affect the running environment. Check the status of pvedaemon and pveproxy after the restart with systemctl status pvedaemon.service pveproxy.service
 
Last edited:
mira said:
This might be a similar issue as in https://bugzilla.proxmox.com/show_bug.cgi?id=3916
The issue in the bug tracker is with lemonldap-ng.

Could you try downgrading libpve-rs-perl to version 0.5.0?
Click to expand...

Just to be clear, has this package been moved to production version yet, because it still doesn't work?

We've got Linux [server name would be here] 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64

pvedaemon[1489500]: openid authentication failure; rhost=::ffff:[ip address would be here] msg=Failed to contact token endpoint: Failed to parse server response

AAD app has proper permissions for email and profile, everything else is in default settings..

....

After reading your suggestion of updating the OpenId.pm script I have found out the problem is on this line

Perl: 
my $info = $openid->verify_authorization_code($param->{code}, $private_auth_state);

I tried changing the max allowed size to 4096 but it had not effect
 
Last edited:
Package libpve-access-control starting from version 7.2-4 already contain the fix.
 
mira said:
Package libpve-access-control starting from version 7.2-4 already contain the fix.
Click to expand...
just FYI- it still does not work. I am on 7.3-6 (updated as of today). same exact error "Failed to contact token endpoint: Request failed"
 
@mira i can reproduce the issue on an an patched proxmox installation 7.4.3 with the message: openid authentication failure; rhost=::ffff:xxx.xxx.xxx.xxx msg=Failed to contact token endpoint: Request failed
 
We are on 7.4-3 here, no issues with Azure AD or Keycloak (two different installations). If it never worked, check your client id and key again.
 
I'll have to see if we can improve the logging and error messages, especially the generic `Failed to contact token endpoint: Request failed`.
 
I verified that the client id and secrect are correct.
We use Proxmox Backup Server with Azure AD and have no such problems. Is there any difference in the used libraries or something like that?
 
Unfortunately it seems the flaw is that its not able to parse the endpoint URL correctly using the .well-known url when the error is "Failed to contact token endpoint". i wish there was a direct way to enter the endpoint url.
 
Same error but with different oauth provider (Synology SSO). We definitely need more verbose logging. I have a service that successfully works with the same settings.

Syslog:
Bash: 
openid authentication failure; rhost=::ffff:192.168.2.4 msg=Failed to contact token endpoint: Request failed

I tried to compare requests from nginx between pve and oauth server, but this is clearly not enough to understand the problem.

Another service Portainer (working):
Bash: 
[02/Apr/2023:19:46:16 +0000] - 200 200 - POST https sso.internal.local"/webman/sso/SSOAccessToken.cgi" [Client 192.168.1.101] [Length 119] [Gzip -] [Sent-to 192.168.2.200] "Go-http-client/2.0" "-"
[02/Apr/2023:19:46:17 +0000] - 200 200 - GET https sso.internal.local"/webman/sso/SSOUserInfo.cgi" [Client 192.168.1.101] [Length 56] [Gzip -] [Sent-to 192.168.2.200] "Go-http-client/2.0" "-"
Proxmox 7.4-3 (not working):
Bash: 
[02/Apr/2023:19:46:22 +0000] - 200 200 - GET https sso.internal.local "/webman/sso/openid-jwks.json" [Client 192.168.1.101] [Length 435] [Gzip -] [Sent-to 192.168.2.200] "ureq/2.4.0" "-"
[02/Apr/2023:19:46:22 +0000] - 400 400 - POST https sso.internal.local "/webman/sso/SSOAccessToken.cgi" [Client 192.168.1.101] [Length 46] [Gzip -] [Sent-to 192.168.2.200] "ureq/2.4.0" "-"

There are no errors from the oauth provider side.

401.jpg
 
Last edited:
Some kind of debug option would be great. I also recreated the whole config for the oidc authentication to azure and it won't change the behaviour. From my logs in the azuread everything seems fine, so the problem seems to be in the handling of the response.
 
ondrejvalenta said:
Just to be clear, has this package been moved to production version yet, because it still doesn't work?

We've got Linux [server name would be here] 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64

pvedaemon[1489500]: openid authentication failure; rhost=::ffff:[ip address would be here] msg=Failed to contact token endpoint: Failed to parse server response

AAD app has proper permissions for email and profile, everything else is in default settings..

....

After reading your suggestion of updating the OpenId.pm script I have found out the problem is on this line

Perl: 
my $info = $openid->verify_authorization_code($param->{code}, $private_auth_state);

I tried changing the max allowed size to 4096 but it had not effect
Click to expand...
Updated to the latest version of everything, whole server restarted, AAD TenantId, ClientId and Client Secret checked and recreated, nothing worked..

I found this guide https://www.reddit.com/r/Proxmox/comments/pqxu2o/proxmox_oidc_authentication_azure_ad/ and there they use standard
https://login.microsoftonline.com/[tenant id]/v2.0 instead my original https://sts.windows.net/[tenant id] that I have found in some other guide.


This is the working configuration.

1680565783298.png

These are app permissions, nothing extra here.
1680565959558.png
 
Last edited:
@ondrejvalenta this is interesting. I used the the login.microsoftonline.com endpoint all the time.
But after changing the url to sts.windows.net....(that didn't worked) and back to the original url the login works now. Maybe it was not correctly saved. I have no real explanation for this . I only copy pasted the url so that it should not be changed....
 
  • Like
Reactions: ondrejvalenta
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back