PVEProxy error

[itvietnam]

Hello,

We receive 1 cluster reboot random (HP Blade server) and our WHMCS connect to this cluster is really slow. We check pveproxy status and it showing this output:

 pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-06-30 11:48:08 +07; 24h ago
  Process: 20107 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
  Process: 3411 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
 Main PID: 3449 (pveproxy)
    Tasks: 4 (limit: 9830)
   Memory: 209.4M
      CPU: 4h 8min 25.624s
   CGroup: /system.slice/pveproxy.service
           ââ 3449 pveproxy
           ââ55506 pveproxy worker
           ââ60144 pveproxy worker
           ââ62747 pveproxy worker

Jul 01 12:13:58 hv101 pveproxy[62746]: ipcc_send_rec[1] failed: Too many open files
Jul 01 12:13:58 hv101 pveproxy[62746]: ipcc_send_rec[2] failed: Too many open files
Jul 01 12:13:58 hv101 pveproxy[62746]: ipcc_send_rec[3] failed: Too many open files
Jul 01 12:13:58 hv101 pveproxy[3449]: worker 16408 finished
Jul 01 12:13:58 hv101 pveproxy[3449]: starting 1 worker(s)
Jul 01 12:13:58 hv101 pveproxy[3449]: worker 62747 started
Jul 01 12:14:01 hv101 pveproxy[62746]: got inotify poll request in wrong process - disabling inotify
Jul 01 12:14:02 hv101 pveproxy[62746]: worker exit
Jul 01 12:14:02 hv101 pveproxy[62747]: Clearing outdated entries from certificate cache
Jul 01 12:14:52 hv101 pveproxy[60144]: internal error at /usr/share/perl5/PVE/RESTHandler.pm line 349.

May i know how to fix this error?

Thanks,

 

[wolfgang]

As the error message saying, you have too many files open.
This can happen if you use Containers in this case you should increase the open files limit in the limits.conf .

 

[itvietnam]

This is strange. We did not use any LXC on this cluster. We have 2 incidents last weekend. Proxmox random reboot on Saturday and Sunday after upgrade to Proxmox latest version. Is this the root cause of issue?

proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
pve-manager: 5.2-3 (running version: 5.2-3/785ba980)
pve-kernel-4.15: 5.2-3
pve-kernel-4.13: 5.1-45
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.13.16-3-pve: 4.13.16-49
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-34
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
openvswitch-switch: 2.7.0-2
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

 

[wolfgang]

If you have no LXC than I would investigate what process has so many files open.

 

[itvietnam]

Hi,

Do you have any guide/command to investigate this cause?

 

[wolfgang]

You can use this

#!/bin/bash

cd /proc
for pid in [0-9]*
do
   echo "PID: $pid \tFD: $(ls /proc/$pid/fd/ | wc -l)"
done