OpenID 401 with Azure AD

[talos]

Hi,

i get that "code: value may only be 1024 characters long." sometimes, it appears and disappears from time to time.

 

[talos]

And today this issue is back, i took a quick look into pveproxy/access.log. I can see there is a callback which starts with ?code=... and a long string which is about 1066 bytes long, followed by some more stuff like state=, session_state= and so on. Maybe the issue is that code= is over 1024 characters? or the whole string itself is to large with around 1220 bytes.

 

[mira]

I've sent a patch increasing the limit to 2048 for now. This should fix the issues with Azure for the time being.

 

[daros]

I've sent a patch increasing the limit to 2048 for now. This should fix the issues with Azure for the time being.

When can we expect the patch to be live?

 

[bunderuso]

What repository is this patch on? I need it in the non enterprise repo to apply to my servers since it's impacting some of my users

 

[mira]

The patch has been applied and will be part of the next libpve-access-control release (7.2-4 or higher).
If you want to manually change it until then, you can do so by editing /usr/share/perl5/PVE/API2/OpenId.pm and changing the limit for the `code` parameter in the `login` API call accordingly.

__PACKAGE__->register_method ({
    name => 'login',
    path => 'login',
    method => 'POST',
    protected => 1,
    description => " Verify OpenID authorization code and create a ticket.",
    parameters => {
    additionalProperties => 0,
    properties => {
        'state' => {
        description => "OpenId state.",
        type => 'string',
        maxLength => 1024,
            },

 

[daros]

The patch has been applied and will be part of the next libpve-access-control release (7.2-4 or higher).
If you want to manually change it until then, you can do so by editing /usr/share/perl5/PVE/API2/OpenId.pm and changing the limit for the `code` parameter in the `login` API call accordingly.

__PACKAGE__->register_method ({
    name => 'login',
    path => 'login',
    method => 'POST',
    protected => 1,
    description => " Verify OpenID authorization code and create a ticket.",
    parameters => {
    additionalProperties => 0,
    properties => {
        'state' => {
        description => "OpenId state.",
        type => 'string',
        maxLength => 1024,
            },

So change the 1024 to 2048?
And what service do you need to restart to get the new value active?
And can this been done on an running environment?

 

[mira]

Restart pvedaemon and pveproxy with the following command:
systemctl restart pvedaemon.service pveproxy.service

The patch will set the new limit to 4096, so choose whichever you prefer and is enough for your use case.

If you don't have any errors in the code, it should not affect the running environment. Check the status of pvedaemon and pveproxy after the restart with systemctl status pvedaemon.service pveproxy.service

 

[ondrejvalenta]

This might be a similar issue as in https://bugzilla.proxmox.com/show_bug.cgi?id=3916
The issue in the bug tracker is with lemonldap-ng.

Could you try downgrading libpve-rs-perl to version 0.5.0?

Just to be clear, has this package been moved to production version yet, because it still doesn't work?

We've got Linux [server name would be here] 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64

pvedaemon[1489500]: openid authentication failure; rhost=::ffff:[ip address would be here] msg=Failed to contact token endpoint: Failed to parse server response

AAD app has proper permissions for email and profile, everything else is in default settings..

....

After reading your suggestion of updating the OpenId.pm script I have found out the problem is on this line

my $info = $openid->verify_authorization_code($param->{code}, $private_auth_state);

I tried changing the max allowed size to 4096 but it had not effect

 

[mira]

Package libpve-access-control starting from version 7.2-4 already contain the fix.

 

[saggy2]

Package libpve-access-control starting from version 7.2-4 already contain the fix.

just FYI- it still does not work. I am on 7.3-6 (updated as of today). same exact error "Failed to contact token endpoint: Request failed"

 

[bk-lg]

@mira i can reproduce the issue on an an patched proxmox installation 7.4.3 with the message: openid authentication failure; rhost=::ffff:xxx.xxx.xxx.xxx msg=Failed to contact token endpoint: Request failed

 

[talos]

We are on 7.4-3 here, no issues with Azure AD or Keycloak (two different installations). If it never worked, check your client id and key again.

 

[mira]

I'll have to see if we can improve the logging and error messages, especially the generic `Failed to contact token endpoint: Request failed`.

 

[bk-lg]

I verified that the client id and secrect are correct.
We use Proxmox Backup Server with Azure AD and have no such problems. Is there any difference in the used libraries or something like that?

 

[saggy2]

Unfortunately it seems the flaw is that its not able to parse the endpoint URL correctly using the .well-known url when the error is "Failed to contact token endpoint". i wish there was a direct way to enter the endpoint url.

 

[Dexogen]

Same error but with different oauth provider (Synology SSO). We definitely need more verbose logging. I have a service that successfully works with the same settings.

Syslog:

openid authentication failure; rhost=::ffff:192.168.2.4 msg=Failed to contact token endpoint: Request failed

I tried to compare requests from nginx between pve and oauth server, but this is clearly not enough to understand the problem.

Another service Portainer (working):

[02/Apr/2023:19:46:16 +0000] - 200 200 - POST https sso.internal.local"/webman/sso/SSOAccessToken.cgi" [Client 192.168.1.101] [Length 119] [Gzip -] [Sent-to 192.168.2.200] "Go-http-client/2.0" "-"
[02/Apr/2023:19:46:17 +0000] - 200 200 - GET https sso.internal.local"/webman/sso/SSOUserInfo.cgi" [Client 192.168.1.101] [Length 56] [Gzip -] [Sent-to 192.168.2.200] "Go-http-client/2.0" "-"

Proxmox 7.4-3 (not working):

[02/Apr/2023:19:46:22 +0000] - 200 200 - GET https sso.internal.local "/webman/sso/openid-jwks.json" [Client 192.168.1.101] [Length 435] [Gzip -] [Sent-to 192.168.2.200] "ureq/2.4.0" "-"
[02/Apr/2023:19:46:22 +0000] - 400 400 - POST https sso.internal.local "/webman/sso/SSOAccessToken.cgi" [Client 192.168.1.101] [Length 46] [Gzip -] [Sent-to 192.168.2.200] "ureq/2.4.0" "-"

There are no errors from the oauth provider side.

 

[bk-lg]

Some kind of debug option would be great. I also recreated the whole config for the oidc authentication to azure and it won't change the behaviour. From my logs in the azuread everything seems fine, so the problem seems to be in the handling of the response.

 

[ondrejvalenta]

Just to be clear, has this package been moved to production version yet, because it still doesn't work?

We've got Linux [server name would be here] 5.15.64-1-pve #1 SMP PVE 5.15.64-1 (Thu, 13 Oct 2022 10:30:34 +0200) x86_64

pvedaemon[1489500]: openid authentication failure; rhost=::ffff:[ip address would be here] msg=Failed to contact token endpoint: Failed to parse server response

AAD app has proper permissions for email and profile, everything else is in default settings..

....

After reading your suggestion of updating the OpenId.pm script I have found out the problem is on this line

my $info = $openid->verify_authorization_code($param->{code}, $private_auth_state);

I tried changing the max allowed size to 4096 but it had not effect

Updated to the latest version of everything, whole server restarted, AAD TenantId, ClientId and Client Secret checked and recreated, nothing worked..

I found this guide https://www.reddit.com/r/Proxmox/comments/pqxu2o/proxmox_oidc_authentication_azure_ad/ and there they use standard
https://login.microsoftonline.com/[tenant id]/v2.0 instead my original https://sts.windows.net/[tenant id] that I have found in some other guide.


This is the working configuration.



These are app permissions, nothing extra here.

 

[bk-lg]

@ondrejvalenta this is interesting. I used the the login.microsoftonline.com endpoint all the time.
But after changing the url to sts.windows.net....(that didn't worked) and back to the original url the login works now. Maybe it was not correctly saved. I have no real explanation for this . I only copy pasted the url so that it should not be changed....