Proxmox ulimit hell : how to really increase open files ?

[quantum]

I have already read posts related to ulimit on this forum, but don't understand anything, so please do NOT post me as a reply, either links to the forum, either links to proxmox mostly oudated wiki on this subject.

My case is pretty simple. I have a fresh Hetzner dedicated server. And I want to ensure that my heavy loaded production-grade VMs (like MariaDB) will not struggle because of too low "open files limit".

So, I setup Proxmox with the official and latest ISO, to get the best config defaults :

root@physical:~# pveversion
> pve-manager/6.3-2/22f57405 (running kernel: 5.4.73-1-pve)

Then I checked file-max on the physical host. Don't know how fs-max is related to ulimit, but the defaults sound good (something unlimited) :
cat /proc/sys/fs/file-max
> 9223372036854775807

Now i check ulimit hard limits as root on the physical host, I have something like that :
root@physical:~# ulimit -aH
open files (-n) 1048576

Now i do another check INSIDE the MariaDB container :
root@container:~# prlimit -p 1
NOFILE max number of open files 524288 524288 files

The file-max, the ulimit, and the prlimit command print differents values.. But, at this point, Proxmox Staff seems to have done a good job, because the defaults are indeed relevant for a 2020 (now 2021) high performance dedicated server.

My problem is about the ulimit soft limit, both inside a container, and on the physical PVE host :
root@physical:~# ulimit -aS
open files (-n) 1024
root@container:~# ulimit -aS
open files (-n) 1024

As you can see, my soft limit is 1024. So i'm afraid that my containers are actually limited to 1024 open files, which is too low, for my needs.

So my questions are :
- is ulimit soft "open files" matters inside a container ?
- how to apply an "open files limit" properly to a given container, and how to check the value ?
- why Proxmox Staff apply such a very low soft ulimit "open files" per default, and do anything else right ?

Please give a clear list of relevant commands to the Proxmox community, on how to increase "open files limit" on both the physical host, and containers !

 

[quantum]

It finally works !

On physical host do :

A. add to /etc/sysctl.conf :

fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
vm.max_map_count = 262144

B. add to /etc/security/limits.conf

*       soft    nofile  1048576 unset
*       hard    nofile  1048576 unset
root    soft    nofile  1048576 unset
root    hard    nofile  1048576 unset
*       soft    memlock 1048576 unset
*       hard    memlock 1048576 unset

C. foreach container X do :

add to /etc/pve/lxc/X.conf
lxc.prlimit.nofile: 30000

Inside each container do :

D. pct enter X (with X = ID of the container)

E. add to /etc/security/limits.conf

*       soft    nofile  1048576 unset
*       hard    nofile  1048576 unset
root    soft    nofile  1048576 unset
root    hard    nofile  1048576 unset
*       soft    memlock 1048576 unset
*       hard    memlock 1048576 unset

F. reboot the whole PVE physical host

Then when I do 'ulimit -aS' or 'ulimit -aH' I have larger value that 1024 ! (both physical host or containers !!)

At the APPLICATION level, inside a container :

G. You must ensure that the processes are running with appropriate ulimit. I will show you an example with nginx/debian LXC container.

root@nginx-debian-container:~# ps -aux | grep nginx

root 215 0.0 0.0 10604 868 ? Ss 16:24 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 218 0.0 0.1 11324 3584 ? S 16:24 0:00 nginx: worker process
root 379 0.0 0.0 6072 816 ? S+ 16:25 0:00 grep nginx

root@nginx-debian-container:~# grep '^Max open files' /proc/218/limits

Limit Soft Limit Hard Limit Units
Max open files 1024 30000 files

As you can see here, the ulimit soft "open files limit", of the relevant nginx process (worker processes) is still at 1024 !

Keep in mind that a given software can increase the soft limit, but not the hard limit. However, a soft limit IS a limit. So if your soft limit is 1024, your nginx can't allocate more than 1024 connections (one socket = one file), that's why it is very important to go further, and also increase the soft limit. Then ensure that the processes of your app work with the appropriate ulimit "open files".

As said before, software can set the soft limit itself, so it is likely that your software provide a configuration entry to do that. With nginx you need to have a "nginx.conf" something like that :

user  nginx;
worker_processes  1;
worker_rlimit_nofile 28672;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  28672;
}

http {
  ...
}

Relevant options are worker_rlimit_nofile and worker_connections. It then works as expected :

root@nginx:~# ps -aux | grep nginx
root       445  0.0  0.0  10604   868 ?        Ss   18:16   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx      446  0.0  0.7  22816 15108 ?        S    18:16   0:00 nginx: worker process
root       448  0.0  0.0   6072   820 ?        S+   18:17   0:00 grep nginx
root@nginx:~# cat  /proc/446/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            28672                28672                files

Soft limit is now OK !

H. if it doesn't work, with certain services you may also have to add the ulimit at the service

root@myservice-debian-container:~# echo "ulimit -n 28672" >> /etc/default/a_service

I. if it doesn't work, with systemd you may also have to add the ulimit in :

/lib/systemd/system/MYSERVICE.service

in the [Service] section, add "LimitNOFILE=30000"

Also try to reboot your Container, in order to ensure the ulimit persist after reboot

 

[quantum]

As a side note, I continue my research, and found that

prlimit -p 1

(inside a given container) is the only reliable way to print the current "open files" hard limit set. So after applying instructions in my previous post, ensure to run such command to verify.

Indeed, as explained before, when you check

ulimit -aS or ulimit -aH

after entering in the container with

pct enter CONTAINER_ID

(or after doing SSH into the container), you may have different result. So prlimit is the reliable way.

Remember that even if the hard limit is correctly set, you must also configure your application to set the appropriate "files limits". For nginx, it is the

worker_rlimit_nofile 28672;

and just found that php-fpm pool have also a

rlimit_files = 28672

in order to do that (assuming that you set

lxc.prlimit.nofile: 30000

(at the container conf level, as explained before, to take a bit of margin)

 

[kotakomputer]

This thread help me so much.

In my case (tested on Promox 6.4), I don't need to change any value on Node. Only change this inside LXC:

1. Enter to VPS using Proxmox - Select VPS - Console
2. # nano /etc/security/limits.conf .... enter this value:
   

   *       soft    nofile  1048576 unset
   *       hard    nofile  1048576 unset
   root    soft    nofile  1048576 unset
   root    hard    nofile  1048576 unset
   *       soft    memlock 1048576 unset
   *       hard    memlock 1048576 unset

3. # exit ---> No need to restart LXC, just type exit
4. Login again
5. # ulimit -n

NB: do not do this using "pct enter vps-id", you must login using ssh or Proxmox-Console

 

[quantum]

Hi,

You're welcome to the Proxmox ulimit hell

It might be fine like that, but I advise you to double check at the application level (running inside the container), that the ulimit have indeed really increased (both soft and hard).

root@nginx:~# ps -aux | grep nginx
root       445  0.0  0.0  10604   868 ?        Ss   18:16   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx      446  0.0  0.7  22816 15108 ?        S    18:16   0:00 nginx: worker process
root       448  0.0  0.0   6072   820 ?        S+   18:17   0:00 grep nginx
root@nginx:~# cat  /proc/446/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            28672                28672                files

As said before, in the example above, I ensure that nginx has softlimit of 28672 inside its container, by first getting the ID of the parent nginx process (or any process related to nginx), then query /proc/PROC_ID/limits to ensure that it is really the case

Have a good day

 

[gmbeniamin]

This thread help me so much.

In my case (tested on Promox 6.4), I don't need to change any value on Node. Only change this inside LXC:

1. Enter to VPS using Proxmox - Select VPS - Console
2. # nano /etc/security/limits.conf .... enter this value:
   

   *       soft    nofile  1048576 unset
   *       hard    nofile  1048576 unset
   root    soft    nofile  1048576 unset
   root    hard    nofile  1048576 unset
   *       soft    memlock 1048576 unset
   *       hard    memlock 1048576 unset

3. # exit ---> No need to restart LXC, just type exit
4. Login again
5. # ulimit -n

NB: do not do this using "pct enter vps-id", you must login using ssh or Proxmox-Console

This actually works, at least for now. I followed these instructions for Proxmox 6 and all the limits are exactly where they should be. Will test more in order to be sure this is a permanent solution.

 

[gmbeniamin]

With version 6.5-13 this does not work anymore. The Soft Files limit is still 1024 and we have no clue how to change this. A few months ago everything worked perfectly but now we cannot make it work at all.

 

[Andrey Zentavr]

With version 6.5-13 this does not work anymore. The Soft Files limit is still 1024 and we have no clue how to change this. A few months ago everything worked perfectly but now we cannot make it work at all.

Please try my solution (steps 3...6) from this reply.
You can check the limits with this script:

#!/usr/bin/env bash

for PID in $(ps aux | grep /usr/bin/kvm | grep -v grep | awk '{ print $2 }'); do
  SOFT_LIMIT=$(cat /proc/${PID}/limits 2>/dev/null | grep "Max open files" | awk '{ print $4 }')
  HARD_LIMIT=$(cat /proc/${PID}/limits 2>/dev/null | grep "Max open files" | awk '{ print $5 }')
  echo "PID ${PID} opened files: $(ls -1 /proc/${PID}/fd 2>/dev/null | wc -l)/${SOFT_LIMIT}/${HARD_LIMIT}"
done

 

[Andrey Zentavr]

You can also change limits on already running process as well:

#!/usr/bin/env bash

for PID in $(ps aux | grep /usr/bin/kvm | grep -v grep | awk '{ print $2 }'); do
  SOFT_LIMIT="1048576"
  HARD_LIMIT="2097152"
  echo "Changing the limits for PID ${PID}"
  prlimit --nofile=${SOFT_LIMIT}:${HARD_LIMIT} --pid ${PID}
done

 

[kotakomputer]

With version 6.5-13 this does not work anymore. The Soft Files limit is still 1024 and we have no clue how to change this. A few months ago everything worked perfectly but now we cannot make it work at all.

For version >= 6.5.x the simple approach is edit VMID.conf then enter this line:

lxc.prlimit.nofile: 150000

Then .. restart the CT.

Above code set open file limit to 150.000 files.

 

[melonion]

A few mentions of VMs, but settings all target CTs - does this affect VMs as well? Or are they completely separate in such things?

 

[ajtatum]

Hi, I wanted to follow up on this thread as I'm trying to install Graylog which requires MongoDB which has a requirement that the number of open files must be greater than 64000. I need to make this change to a VM and I'm running PVE 7.4-3.

 

[LnxBil]

Hi, I wanted to follow up on this thread as I'm trying to install Graylog which requires MongoDB which has a requirement that the number of open files must be greater than 64000. I need to make this change to a VM and I'm running PVE 7.4-3.

If you have a VM and NOT a container, please refer to your guest os handbook, the problem is then NOT related to PVE.