Active Directory Sync issues: MEMDB_MAX_FILE_SIZE restriction too narrow.

[fatllama]

Please pardon any lack of knowledge, I am a young sysadmin still learning PVE.

I am in an environment where we have ~40,000 users in Active Directory. This proves to be an issue when syncing through BIND, and I'm assuming LDAP would result in the same issue since it's a restriction that Proxmox has in place on config file sizes. The 1MiB limit placed on individual config file sizes allows for (I believe) less than 10,000 users in the user.cfg file.

Trying to change this value manually has been tedious and in order for us to move forward with Proxmox in our environment, we need a clean solution to this problem. We wouldn't need more than 100 users authenticating into the nodes, but we have a vast infrastructure that is tied in with our current AD environment, and the creation and management of a separate Active Directory instance would just add more complexity to this environment.

From what I understand, this restriction is in place to prevent excessive memory usage and allow for easier synchronization between hosts. I believe we have the resources to allow synchronization of slightly larger config files between hosts. Of course, not everyone would have these capabilities so it would be unreasonable to expect a change to be made that prevents others from being able to use Proxmox on their hardware. However, if there were some way to make this setting easier to change for an individual user, then this problem would be much less of a hassle.

Below is the error and references to others having this issue:

Warning: unable to close filehandle GEN17 properly: File too large at /usr/share/perl5/PVE/Tools.pm line 254.
TASK ERROR: syncing users and groups failed: unable to write '/etc/pve/user.cfg.tmp.1630' - File too large

https://forum.proxmox.com/threads/memdb_max_file_size-on-pmxcfs-is-too-small-1-mib.136392/
https://forum.proxmox.com/threads/proxmox-ldap-sync-hard-limit-on-number-of-users.91320/
Sublinks from the above thread:
https://bugzilla.proxmox.com/show_bug.cgi?id=3492
https://git.proxmox.com/?p=pve-cluster.git;a=commit;h=a8df0863b5851dacb4f76ae6364ac1a02fbd56db

 

[pille99]

what for a strange way of getting ADS User auth.
you dont need to sync 10.000 user if you just need 100. just set the realm correct. (i checked the available option and it should not be a problem)

you can use LDAP as well. will work. i havent checked it myself but from the option which are available it should be possible to filter for group which contains proxmox users (should be an own group).

usually, how its done - the programm should authenticate in real time against any source.

 

[cheiss]

We wouldn't need more than 100 users authenticating into the nodes, but we have a vast infrastructure that is tied in with our current AD environment, and the creation and management of a separate Active Directory instance would just add more complexity to this environment.

Normally, in such cases you simply have e.g. a AD Group, where all users that actually need to authenticate for Proxmox VE are members.
That would be the clean solution, as well as having the benefit of added security in that only users that should be able to login to Proxmox VE can do it.

Following this, you can apply either a user or group filter under the Sync options of an Active Directory realm, such that only the needed users are actually synced. You don't need a separate Active Directory instance or such for this, simply an appropriate user/group filter.

 

[fatllama]

what for a strange way of getting ADS User auth.
you dont need to sync 10.000 user if you just need 100. just set the realm correct. (i checked the available option and it should not be a problem)

I was wondering if this was possible. I couldn't find anything on it but I admittedly didn't look as much as I should have.

you can use LDAP as well. will work. i havent checked it myself but from the option which are available it should be possible to filter for group which contains proxmox users (should be an own group).

I will try this out today. Thank you for the info!

usually, how its done - the programm should authenticate in real time against any source.

This seems way more reasonable to me than syncing the entire directory over. I guess that's where the Lightweight part comes from!

 

[fatllama]

Normally, in such cases you simply have e.g. a AD Group, where all users that actually need to authenticate for Proxmox VE are members.
That would be the clean solution, as well as having the benefit of added security in that only users that should be able to login to Proxmox VE can do it.

I was not aware this was possible, that simplifies things quite a bit. I thought the permissions/scope would need to be set after syncing the entire directory over.

Following this, you can apply either a user or group filter under the Sync options of an Active Directory realm, such that only the needed users are actually synced. You don't need a separate Active Directory instance or such for this, simply an appropriate user/group filter.

That simplifies this process a ton. Thank you! I will be trying this today.

 

[fatllama]

Got it! I was able to filter down the import using User filters and Group filters. I started with the later details in this article:

https://forum.proxmox.com/threads/h...cluster-to-an-active-directory-domain.100395/

Basing it mostly off of this query format:

(&(objectclass=user)(samaccountname=*)(MemberOf=CN=GROUPNAME,OU=GROUP_OU,DC=DOMAIN,DC=TLD))

For anyone who stumbles across this with a somewhat complicated AD structure: When filtering for users that are in a specific group, the value for MemberOf= should be the distinguishedName of the group that you are trying to pull. That took me quite a while to pin down since this was my first time making LDAP queries by hand.

Thanks cheiss and pille99 !

 

[pille99]

you are welcome.

 

[cheiss]

Got it! I was able to filter down the import using User filters and Group filters.

Great to hear!

Please just mark the thread as SOLVED, by editing the first port - there should be a dropdown for that near the title field. This helps others with the same problem to find it more easily in the future!