LDAP authentication broke recently - possibly related to libpve-access-control update?

[victorhooi]

I have a Proxmox cluster, that uses LDAP to authenticate against a G Suite domain.

Previously, users were able to login successfully through their LDAP credentials.

However, recently I updated some packages (e.g. libpve-access-control) in order to try with the new LDAP sync feature (discussion thread).

Along the way, I've suddenly noticed that LDAP authentication seems to have broken. When users try to login via LDAP, they are getting:

Login failed. Please try again

Could this be related to the updated packages, or is it something else?

Here is my current /etc/pve/domains.cfg file:

# cat /etc/pve/domains.cfg
pam: pam
        comment Linux PAM standard authentication

ldap: example.io
        base_dn dc=example,dc=io
        bind_dn uid=victorhooi,ou=Users,dc=example,dc=io
        server1 ldap.google.com
        user_attr uid
        cert /etc/pve/priv/Google_2022_05_22_3494.crt
        certkey /etc/pve/priv/Google_2022_05_22_3494.key
        port 636
        secure 1
        verify 1

pve: pve
        comment Proxmox VE authentication server

In the audit logs for G Suite SecureLDAP, I see an error about:

"LDAP bind with "" failed with NO_SUCH_OBJECT.":

(The username we're trying to sign in with here is "testvendor")



I assume the LDAP bind returning "NO_SUCH_OBJECT" might be the issue?
In /var/log/syslog, I see this:

May  8 15:18:48 example-kvm01 pvedaemon[1714641]: authentication failure; rhost=127.0.0.1 user=testvendor@example.io msg=Invalid username at /usr/share/perl5/PVE/LDAP.pm line 83.

 

[dcsapak]

what does the ldapsearch query

ldapsearch -H ldap://YOURLDAPHOST -D YOURBIND_DN -b YOURBASE_DN '(uid=testvendor)' dn

return?

 

[victorhooi]

Hi,

It returns the following:

$  LDAPTLS_REQCERT=allow LDAPTLS_CERT=Google_2022_05_22_3494.crt LDAPTLS_KEY=Google_2022_05_22_3494.key ldapsearch -H ldaps://ldap.google.com:636 -b dc=example,dc=io '(uid=testvendor)' dn
SASL/EXTERNAL authentication started
SASL username: st=California,c=US,ou=GSuite,cn=LDAP Client,l=Mountain View,o=Google Inc.
SASL SSF: 0
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=io> with scope subtree
# filter: (uid=testvendor)
# requesting: dn
#

# testvendor, Users, example.io
dn: uid=testvendor,ou=Users,dc=example,dc=io

# search result
search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

Also - here is a screenshot from the LDAP audit log from last week logins were working correctly:



The main difference I can see is that LDAP seemingly isn't passing through the user to the bind/search?

I've checked, and the /etc/pve/domains.cfg and /etc/pve/users.cfg files are the same. (I did make a change to add "bind_dn", but I've commented that out now, and the error still occurs).

Thanks,
Victor

 

[dcsapak]

mhmm for some reason our code does not get the dn...

can you post your pveversion -v please

 

[victorhooi]

Of course!

Here is the output from pveversion -v:

# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-11 (running version: 6.1-11/f2f18736)
pve-kernel-helper: 6.1-9
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.9-pve1
ceph-fuse: 14.2.9-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.2
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-1
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-6
pve-cluster: 6.1-8
pve-container: 3.1-4
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.1-1
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-20
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

EDIT: In the above output, libpve-access-control is 6.0-6, as I did a manual downgrade from 6.0-7. However, it still didn't work (unless it needs a reboot after?) and I've just updated it back.

(I sent you the credentials in a separate email as well, if that would help at all).

 

[dcsapak]

ok i did check around (with the mail you sent me)

the following thing changed:

when connecting and no 'bind_dn' is configured, we now do an anonymous bind
it seems that the google ldap server authenticates with the clientcert and key, but the
anonymous bind afterwards 'logs out' and we are no longer be able to query the dn of the user...
(there is a bugfix necessary where we die if we did not get a dn back...)

as workaround you can configure a bind_dn with password that has read access.. this should work normally
(it seems the password of my account is not valid... maybe i changed it last time and forgot...)

 

[victorhooi]

Dominik - I just sent you a reset password - let me know if you didn't get it.

For the Proxmox server - I do have bind_dn set now in /etc/pve/domains.cfg:

pam: pam
        comment Linux PAM standard authentication

ldap: example.io
        base_dn dc=example,dc=io
        bind_dn uid=victorhooi,ou=Users,dc=example,dc=io
        server1 ldap.google.com
        user_attr uid
        cert /etc/pve/priv/Google_2022_05_22_3494.crt
        certkey /etc/pve/priv/Google_2022_05_22_3494.key
        port 636
        secure 1
        verify 1

pve: pve
        comment Proxmox VE authentication server

I then put the password for victorhooi@example.io in:

/etc/pve/priv/ldap/example.io.pw

In the LDAP audit console, after I try to login with testvendor I see:

https://i.imgur.com/5YGT3m5.png

In /var/log/syslog, I still see:

May  8 03:48:39 examplemtv-vm01 pvedaemon[2571709]: authentication failure; rhost=127.0.0.1 user=testvendor@example.io msg=Invalid username at /usr/share/perl5/PVE/LDAP.pm line 83.

 

[dcsapak]

weird, i get a different error (Failure authenticating with password) but i guess this is because google does not allow to bind to the ldap with password at all? ( even an ldapsearch with -D and -W gets denied, at least for my account)

anyway a fix is already applied on master for access-control, no need for a bind_dn and password anymore when using a client cert/key

edit: i got it to work (my account was not completly set up), so i can confirm it works with our latest patch, or alternatively the bind_dn and password set correctly

 

[victorhooi]

Thanks Dominik for checking!

I do have bind_dn and the password set. I am pretty sure they are set correctly, as I can run the below command:

# pveum realm sync anguslab.io --dry-run --full --purge --scope both

and it returns a list of users in the domain.

These are the corresponding entries that the SecureLDAP audit log shows when I run pveum realm sync:


However, when I try to login with the user "testvendor" again - the login fails, and the SecureLDAP audit record shows:



From the "User" column - it does seem to be doing an authenticated LDAP lookup right?

Is there perhaps a way I can easily apply the patch right now, to confirm it fixes the issue?

 

[victorhooi]

I can confirm that the patch fixes the issue for me.

I was able to apply the patch manually, by doing a git checkout on https://git.proxmox.com/?p=pve-access-control.git;a=summary, then manually copying the file over to /usr/share/perl5/PVE/Auth/LDAP.pm

However, I also had to remove the bind_dn argument from /etc/domains.cfg - it would not work without removing this line.