Proxmox LDAP Sync - Hard Limit on number of users?

[victorhooi]

Hi,

Does anybody know if there's a hard limit on the number of users that the LDAP sync in Proxmox can do? Or is the below a bug?

I have a Proxmox 6.4 cluster, and I'm trying to an LDAP sync as follows:

pveum realm sync "anguslab.io"  --full --purge --scope both

However, after running through the list of users, it then gives an error message:

Warning: unable to close filehandle GEN3 properly: File too large at /usr/share/perl5/PVE/Tools.pm line 248.
syncing users and groups failed: unable to write '/etc/pve/user.cfg.tmp.238030' - File too large

I checked, and there isn't a file /etc/pve/user.cfg.tmp/238030, so I assume it got cleaned up.

I'm not super familiar with Perl file handling - is this a language limit, or something in our code?

Thanks,
Victor

 

[dcsapak]

we have a 512kbyte file limit on the pmxcfs, so assuming ~80 bytes per user (minus a bit overhead) it's about 6500 users..

 

[victorhooi]

Hmm, what happens beyond that point?

Or is there any way to increase this limit in pmxcfs?

(I think we probably have around 5000-ish users - it's possible the overhead, or metadata per user that's boosting it up - but we are close).

 

[victorhooi]

Are there any workarounds we can use, to get a larger user.cfg file?

 

[dcsapak]

Are there any workarounds we can use, to get a larger user.cfg file?

no and i would not recommend letting the file get that big... do you really have 5000 users that need to login to your pve ?

 

[victorhooi]

It's probably around 4000-4500 - it's used as an internal lab for experimenting with VMs.

It's a four-node cluster, each node has around 2TB of memory, and 128 cores, so we can get some decent utilisation.

Obviously it won't be concurrent usage from all the users at once, but a portion of them at each time, and all of them need the ability to login. (They're also geographically dispersed, so timezone wise, it would be different cohorts at different times normally).

Can you think of any way to make Proxmox work for this use case?

 

[victorhooi]

So I found an earlier patch, where we increased the limit from 128Kb to 512Kb, back in 2016:

https://lists.proxmox.com/pipermail/pve-devel/2016-September/022699.html
https://lists.proxmox.com/pipermail/pve-devel/2020-August/044639.html

Is it viable at all to increase this limit again? (e.g. to 1Mb):

https://git.proxmox.com/?p=pve-clus...a4138317529840d56d912628ec11f7965;hb=HEAD#l31

Or are there other limitations around doing something like that?

 

[dcsapak]

i mean simply upping the limit could work (for now) but maybe later you run into that limit again...

i don't know what your users need, but maybe an external api client with it's own user auth system would make sense?
so that it relays api calls to only some 'actual' user accounts on pve?

another alternative would be ofc to have multiple clusters for different groups of people...

 

[victorhooi]

Aha, this is maybe another of those "640K ought to be enough for anybody" moments . (Possibly an apocryphal quote).

For our specific use case, I'm very confident that boosting it to 1MB would resolve our issue. (Assuming it scales linearly).

We were around 3000 users before, we added another 1000 or so, and that should be our upper limit for the foreseeable future.

These users are store in a Google Workspace domain, which we then sync to Proxmox.

All the users share the same VMs/templates/configurations, so it makes sense to have a single cluster. (That, and we only physically have one cluster with 4 nodes). You can basically think of it like follow-the-sun support teams, only a subset of them will be on at any point in time, but they all need ability to logon in their timezone.

I was looking with interest at some efforts to get SAMLv2 or OpenID Connect support in Proxmox - but not sure if that would hit the same issue as well? (Since it seems the way Proxmox is setup, you need the users defined in both the local user.cfg file, as well as the external IdP).

How hard would it be increase the limit to 1MB? Is it just a case of editing "#define MEMDB_MAX_FILE_SIZE" in memdb.h, or is there more to it? Would you need to edit Tools.pm as well, for the reader? (Sorry if this is an obvious question).

(I'm assuming making the limit a runtime configurable option would be too tricky).

 

[dcsapak]

I was looking with interest at some efforts to get SAMLv2 or OpenID Connect support in Proxmox - but not sure if that would hit the same issue as well? (Since it seems the way Proxmox is setup, you need the users defined in both the local user.cfg file, as well as the external IdP).

there is works on openid connect realm, but yes you're right, the users would have to be in pve as well since the acls always references the users defined in the user.cfg

How hard would it be increase the limit to 1MB? Is it just a case of editing "#define MEMDB_MAX_FILE_SIZE" in memdb.h, or is there more to it? Would you need to edit Tools.pm as well, for the reader? (Sorry if this is an obvious question).

my guess is that increasing this variable + the one occurrance in pve-common: src/PVE/Tools.pm in the sub 'safe_read_from' should be enough

but there may be other places that use that limit

could you maybe open a feature request for increasing the limit, so that we can discuss and track that there?

 

[victorhooi]

Of course - I've raised https://bugzilla.proxmox.com/show_bug.cgi?id=3492 to track this.

 

[iwik]

With corporate Active Directory (trying to replace vmware) it is easy possible to hit this 1MB limit, when PVE tries to sync all users in AD..now what?

 

[cheiss]

With corporate Active Directory (trying to replace vmware) it is easy possible to hit this 1MB limit, when PVE tries to sync all users in AD..now what?

Hi,

do you really need to sync all users? Does everyone need to log into Proxmox VE?
If not, please consider simply creating a separate group for these users and only syncing only that one group via a filter.
See also e.g. this thread: https://forum.proxmox.com/threads/a..._max_file_size-restriction-too-narrow.142806/

This is standard & best pratice anyway, for a variety of reason - starting with e.g. security alone.

 

[iwik]

Hi,

do you really need to sync all users? Does everyone need to log into Proxmox VE?
If not, please consider simply creating a separate group for these users and only syncing only that one group via a filter.
See also e.g. this thread: https://forum.proxmox.com/threads/a..._max_file_size-restriction-too-narrow.142806/

This is standard & best pratice anyway, for a variety of reason - starting with e.g. security alone.

Thanks for that link, at least partial success. We have multiple teams, each have own security group. That would mean, when I want to use this existing security groups for provisioning access to various resources in PVE I need to add them into new group, which I will use as filter in user query?

Question number 2 - nested groups membership is ignored? https://forum.proxmox.com/threads/ldap-sync-with-nested-groups.80749/#post-624699

 

[cheiss]

That would mean, when I want to use this existing security groups for provisioning access to various resources in PVE I need to add them into new group, which I will use as filter in user query?

If it's already an existing LDAP group, you can use it directly in the filter like you would with any other LDAP tool. You can also test these with the ldapsearch tool.

To filter by multiple groups (i.e. to sync multiple groups) from LDAP, you should be able use a filter of this sort:

(|(memberOf=CN=group1,OU=users,DC=example,DC=com)(memberOf=CN=group2,OU=users,DC=example,DC=com))

Question number 2 - nested groups membership is ignored? https://forum.proxmox.com/threads/ldap-sync-with-nested-groups.80749/#post-624699

I'll look into it at times and try to reproduce it. Seems this just unfortunately got "swallowed" in the depths of the forum.

 

[iwik]

If it's already an existing LDAP group, you can use it directly in the filter like you would with any other LDAP tool. You can also test these with the ldapsearch tool.

To filter by multiple groups (i.e. to sync multiple groups) from LDAP, you should be able use a filter of this sort:

(|(memberOf=CN=group1,OU=users,DC=example,DC=com)(memberOf=CN=group2,OU=users,DC=example,DC=com))

I'll look into it at times and try to reproduce it. Seems this just unfortunately got "swallowed" in the depths of the forum.

OK, but I think it is better to manage it using AD, it is more comfortable way.
Anyway we used to have vcenter where anybody can login and using groups permission on vcenter objects each department/team have own set of resources which they can use. Limiting users in ldap query makes sometimes sense, but sometimes not. Like in the original topic from victorhooi.