PVEProxy error

itvietnam

Renowned Member
Aug 11, 2015
132
4
83
Hello,

We receive 1 cluster reboot random (HP Blade server) and our WHMCS connect to this cluster is really slow. We check pveproxy status and it showing this output:

Code:
 pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-06-30 11:48:08 +07; 24h ago
  Process: 20107 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
  Process: 3411 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
 Main PID: 3449 (pveproxy)
    Tasks: 4 (limit: 9830)
   Memory: 209.4M
      CPU: 4h 8min 25.624s
   CGroup: /system.slice/pveproxy.service
           ââ 3449 pveproxy
           ââ55506 pveproxy worker
           ââ60144 pveproxy worker
           ââ62747 pveproxy worker

Jul 01 12:13:58 hv101 pveproxy[62746]: ipcc_send_rec[1] failed: Too many open files
Jul 01 12:13:58 hv101 pveproxy[62746]: ipcc_send_rec[2] failed: Too many open files
Jul 01 12:13:58 hv101 pveproxy[62746]: ipcc_send_rec[3] failed: Too many open files
Jul 01 12:13:58 hv101 pveproxy[3449]: worker 16408 finished
Jul 01 12:13:58 hv101 pveproxy[3449]: starting 1 worker(s)
Jul 01 12:13:58 hv101 pveproxy[3449]: worker 62747 started
Jul 01 12:14:01 hv101 pveproxy[62746]: got inotify poll request in wrong process - disabling inotify
Jul 01 12:14:02 hv101 pveproxy[62746]: worker exit
Jul 01 12:14:02 hv101 pveproxy[62747]: Clearing outdated entries from certificate cache
Jul 01 12:14:52 hv101 pveproxy[60144]: internal error at /usr/share/perl5/PVE/RESTHandler.pm line 349.

May i know how to fix this error?

Thanks,
 
As the error message saying, you have too many files open.
This can happen if you use Containers in this case you should increase the open files limit in the limits.conf .
 
This is strange. We did not use any LXC on this cluster. We have 2 incidents last weekend. Proxmox random reboot on Saturday and Sunday after upgrade to Proxmox latest version. Is this the root cause of issue?

Code:
proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
pve-manager: 5.2-3 (running version: 5.2-3/785ba980)
pve-kernel-4.15: 5.2-3
pve-kernel-4.13: 5.1-45
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.13.16-3-pve: 4.13.16-49
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-34
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
openvswitch-switch: 2.7.0-2
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9
 
If you have no LXC than I would investigate what process has so many files open.
 
You can use this
Code:
#!/bin/bash

cd /proc
for pid in [0-9]*
do
   echo "PID: $pid \tFD: $(ls /proc/$pid/fd/ | wc -l)"
done
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!