"Too Many open files" on Proxmox API + GUI (with KVM)

[Ch@rlus]

Hey guys,

Since we updated our PVE cluster to PVE6, we're having quite a lot of API + GUI errors with this message :

"RPCEnvironment init request failed: Unable to load access control list: Too many open files"

Theses nodes only host KVM VMs, and most of the threads I've found on the subject deal with LXC.

Here's what ulimit is looking like :

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031334
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1031334
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Should we simply consider increasing the "open files" value ? Any tips on how to do it ? (If I remember correctly, "ulimit -p" only affect the current shell, and doesn't affect daemons or background processes)

Thanks !

 

[oguz]

hi,

if you add the nofile option into /etc/security/limits.conf it should be permanent. see here[0]

also the lsof command can help you find which process is using all the file descriptors

[0]: https://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/

 

[Ch@rlus]

Thanks !

I added the following at the end of /etc/security/limits.conf

root soft  nofile 9000
root hard  nofile 65000
*  soft  nofile 9000
*  hard  nofile 65000

Now I do get 9000 when running "ulimit -n".

I'll keep you posted if the problem re-appears, and I'll try to use lsof to find what is responsible for that issue.

 

[chrigiboy]

Hello, we have the same issue, we got every few minutes this error on the GUI:
RPCEnvironment init request failed: Unable to load access control list: Too many open files (500)

root@px10:~# ulimit -n
400000
root@px10:~# ulimit -Hn
499999

Any suggestions?

 

[oguz]

RPCEnvironment init request failed: Unable to load access control list: Too many open files (500)

root@px10:~# ulimit -n
400000
root@px10:~# ulimit -Hn
499999

Any suggestions?

lsof | awk '{ print $1; $2; }' | uniq -c | sort -rn | head

 

[chrigiboy]

Hello,
response from main server, where i was connected to.

proxmox1:~# lsof | awk '{ print $1; $2; }' | uniq -c | sort -rn | head
15792 kvm
12840 kvm
12540 kvm
10246 kvm
9200 kvm
8268 kvm
7992 kvm
7942 kvm
7276 kvm
6603 kvm


Response from node, which i had opened, when the errormessage appear:

proxmox6:~# lsof | awk '{ print $1; $2; }' | uniq -c | sort -rn | head
16524 kvm
14640 kvm
13514 kvm
13014 kvm
12243 kvm
7488 kvm
7280 kvm
5902 kvm
3780 kvm
3604 kvm

 

[oguz]

hmm looks like your kvm processes are using a lot of files?

can you check the lsof output for those vm?

could you also post your pveversion -v

 

[chrigiboy]

I have restarted now all server. It looks better now. Let me check them.