Unable to create new inotify object - Proxmox 5.1.35

[itvietnam]

Hi,

This morning we detected problem on our cluster running Proxmox 5.1.35

We try to migrate all VM and CT to new node, then try start services with ha-manager and see following errors:

root@node02:~# ha-manager set ct:119 --state started
Unable to create new inotify object: Too many open files at /usr/share/perl5/PVE/INotify.pm line 390.

Then i try list all CT with pct command:

root@node02:~# pct list
Unable to create new inotify object: Too many open files at /usr/share/perl5/PVE/INotify.pm line 390.
root@node02:~#

And we can not list all VM too:

root@node02:~# qm list
Unable to create new inotify object: Too many open files at /usr/share/perl5/PVE/INotify.pm line 390.
root@node02:~#

May i know how to fix this issue?

 

[itvietnam]

I read this topic, and try to set inotify value to higher but no luck:

 

[fabian]

you probably also need to bump your max_user_watches , see https://bugzilla.proxmox.com/show_bug.cgi?id=1042

 

[itvietnam]

Hi Fabian,

I already try it but no luck yesterday.



I red your instruction in this thread: https://forum.proxmox.com/threads/f...-watch-too-many-open-files.28700/#post-145947

 

[fabian]

please double check that you sufficiently bumped both values on all your nodes - your screenshots are from different nodes

 

[itvietnam]

Hi Fabian,

I can confirm this is impacted on LXC node and parameter adjusted on node2 and node1 (LXC node).

I try to separate LXC and KVM to separate nodes: some host KVM VPS and some host LXC VPS.

• KVM nodes: not affected by this bug.
• LXC nodes: yes, only LXC VPS caused this problem.

Firstly, we detected problem on node1 -->caused this node reboot --> all VPS migrated to node2 and node3.

After 4 hours, node2 error and i can not type any cluster command like this comment: https://forum.proxmox.com/threads/u...tify-object-proxmox-5-1-35.40881/#post-197091

I decided reboot node2 and move KVM VPS to node3 and LXC VPS to node1.

• Node3: running up to now and no reboot
• Node1: return errors -->i try to adjust inotify parameters but no luck. Later i have to force this node reboot.

Temporary solution: i narrow down and detected some LXC VPS just activated on that day and stop them. This LXC node is running up to date.

May we have some tip for further debug? we can start back 3 VPS caused this problem and monitor again.

 

[fabian]

note that those limits are not persistent across reboots unless you set them in /etc/sysctl.d/ or /etc/sysctl.conf . the only way you get those messages is if you are hitting the limits - you either need to increase them (further), or kill what/whoever is using up so much inotify resources in your containers. note that these resources are shared between containers and host, but not between VMs and host.

 

[itvietnam]

what/whoever is using up so much inotify resources

Hi Fabian,

May i know how to find out top container consume this resources ?

Thanks,

 

[fabian]

find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null

will show which PIDs have open inotify FDs, then you can simply count and map to containers via cgroups.

 

[itvietnam]

find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null

will show which PIDs have open inotify FDs, then you can simply count and map to containers via cgroups.

Thanks for your clearly TIP. I see this command on internet but not detail as your explaination.

 

[VitoHoang]

There is result of command

lsof |awk '{print $1" - " $2}'|uniq -c| sort|sort -n

The result format is

Numbers of row - Command - PID

And many of pve-ha-lr.

pmxcfs open too much files. Why is it? Thanks!