"to many open files" for LXC

[jbates58]

Hi All,

I have the following setup at home

Proxmox Server
    VMs
        Truenas VM
        Other VMs (not important to thread)
        
    LXCs
        InfluxDB
        Wazuh Server
        Wazuh Dashboards
        Wazux index 1
        Wazux index 2
        Wazux index 3
        Other LXCs (not important to thread)

I have a dataset on truenas that i share to Proxmox using SMB and mount it in FStab like below

# Mount CIFS share on demand with rwx permissions for use in LXCs (manually added)
//truenas.localdomain/pve_mount /mnt/pve/truenas_ssd/ cifs _netdev,x-systemd.automount,noatime,uid=100000,gid=110000,dir_mode=0770,file_mode=0770,username=username,password=password 0 0

In this dataset, i have the following structure



And i have config like below mapping these folders into relevant containers

mp0: /mnt/pve/truenas_ssd/network_logs/<directory as needed>,mp=/mnt/<directory as needed>

Recently, i deployed the Wazuh platform, and then about 2 days later influxDB broke, and so did Wazuh. some investigations led me to the error when attempting to look in the mounted directories

to many open files

So, i looked into this a little bit, and then rebooted Influx, and shutdown the Wazuh containers, as Influx is more important to me than Wazuh currently.

I would like some assistance on how i can diagnose this issue, and remediate it? I am also not sure what the actual issue is. But I have rebuilt my PVE from the ground up, as originally, i had all the shares mapped into the LXCs directly using "privileged" containers and using NFS for a direct mount. I decided to fix that security issue and move to SMB as it allowed authentication vs NFS. I plan to run up some other databases and things in future like postgress / MariaDB and who knows what else. So dont want to run into these issues for a production home server (so to speak)

Many thanks for your time.

 

[jbates58]

just a friendly bump to see if anyone has any suggestions on where i can go to look into this issue.

Cheers.

Jason

 

[Johannes S]

lsof will give you a list of open files

 

[waltar]

ulimit -a will show actual settings.
Look to /etc/security/limits.conf and create some own limits in eg. like
echo '* hard nofile 16384' >/etc/security/limits.d/90-pve-config.conf
Do a reboot of the pve node after to take affect on all processes.
For a vm do changes inside the vm, for lxc (or even the host itself) on host.

 

[Grasmachien]

@jbates58 Did you manage to figure it out in the end? I'm hitting this issue too.