MEMDB_MAX_FILE_SIZE on pmxcfs is too small (1 MiB)

[rozaq]

Hello everyone,
here I get that the value of the MEMDB_MAX_FILE_SIZE variable in memdb.h is too small, the impact is that when the /etc/pve/user.cfg file exceeds 1MB it cannot create any more users. Are there any suggestions to improve this?

I tried editing the max value of the variable, but it was very difficult, there were lots of errors in the proxmox service.

root@Research-Proxmox02:~# systemctl | grep pve | grep failed
● pve-firewall.service loaded failed failed Proxmox VE firewall
● pve-guests.service loaded failed failed PVE guests
● pve-ha-crm.service loaded failed failed PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service loaded failed failed PVE Local HA Resource Manager Daemon
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service loaded failed failed PVE Status Daemon
root@Research-Proxmox02:~#

 

[hd--]

Hello
The value of the variable and also the maxsize of the pmxfs are due to the fact that the entire pmxfs is stored in memory and is also synchronized in real time on all computers in the cluster.
I don't think that manually changing the value is a good option (You can run into a lot of problems, not least synchronization problems, which can be difficult to debug)
Hence, the question how many users do you require and is there the possibility to require less ?

[1] https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)

 

[fabian]

1MB seems very excessive.. could you describe your setup?

 

[rozaq]

Hello
The value of the variable and also the maxsize of the pmxfs are due to the fact that the entire pmxfs is stored in memory and is also synchronized in real time on all computers in the cluster.
I don't think that manually changing the value is a good option (You can run into a lot of problems, not least synchronization problems, which can be difficult to debug)
Hence, the question how many users do you require and is there the possibility to require less ?

[1] https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)

maybe just in case the maximum number of users is 6000 and does not include ACLs in the user.cfg file, and maybe the size of the user.cfg file will continue to grow.

 

[rozaq]

1MB seems very excessive.. could you describe your setup?

I'm using 20 hosts here and using Ceph as storage, do you have any suggestions?

 

[fabian]

no, I mean, what are you putting into user.cfg that you run into this limit? it's a plain text file, it only contains the user/permission related settings..

 

[rozaq]

no, I mean, what are you putting into user.cfg that you run into this limit? it's a plain text file, it only contains the user/permission related settings..

sorry, I misunderstood, as you said the user.cfg file contains users, permissions, etc.

I put into user.cfg is user, token, ACL. And it will always increase the file size.

 

[LnxBil]

sorry, I misunderstood, as you said the user.cfg file contains users, permissions, etc.

I put into user.cfg is user, token, ACL. And it will always increase the file size.

How many lines are in there?

 

[fabian]

exactly, that is my question - what is your setup like that you need so many (or, alternatively, so big, if that is the case ) entries in user.cfg?

 

[rozaq]

hello, I have made improvements to this case, there are 2 options:
1. You can hardcode edits on pve-cluster.
2. You can clean up resources on user.cfg that are not used.

 

[esi_y]

1MB seems very excessive.. could you describe your setup?

I understand this sort of back-at-you question, but holistically, why not let user set it (hence also the pmxcfs limit as well) to whatever they want? Clearly there's enough space in RAM nowadays, so the only concern might be the syncing over network. I would rather wonder if files can't by synced as "diffs" rather than full refresh to minimize the traffic.

 

[fabian]

because then people would start to abuse /etc/pve as a shared storage like thing, which it is not. it's meant to sync small files that need to be available and consistent in a cluster. we have bumped the limits where sensible in the past, but we can only do so while keeping performance and failure conditions in mind.