Fresh Proxmox VE 6.1 install on a three node cluster. Started seeing this after building several VMs (no containers):
I added a few lines before and after for context.
For some reason, I'm only seeing this on the second and third nodes (vmh02 and vmh03). I'm currently connected and managing the cluster via vmh01.
This has prevented me from editing memory on one of the affected nodes but otherwise everything seems to be working as expected (I tried to edit memory again and it worked the second time).
I see a lot of posts about inotify and containers but I don't think that applies here. This appears to be a new issue. It looks like pveproxy runs as www-data, so I added this to /etc/security/limits.conf:
www-data - nofile 65536
Then I ran:
systemctl restart pveproxy
I'm not sure if this is the correct way to fix this but we'll see. Let me know if there is something else I should be doing to fix this.
Code:
Mar 29 16:03:00 daemon.info vmh03 systemd[1]: Starting Proxmox VE replication runner...
Mar 29 16:03:00 daemon.info vmh03 systemd[1]: pvesr.service: Succeeded.
Mar 29 16:03:00 daemon.info vmh03 systemd[1]: Started Proxmox VE replication runner.
Mar 29 16:03:03 daemon.err vmh03 pveproxy[13557]: failed to accept connection: Too many open files
Mar 29 16:03:03 daemon.warning vmh03 pveproxy[11732]: ipcc_send_rec[1] failed: Too many open files
Mar 29 16:03:03 daemon.warning vmh03 pveproxy[11732]: ipcc_send_rec[2] failed: Too many open files
Mar 29 16:03:03 daemon.warning vmh03 pveproxy[11732]: ipcc_send_rec[3] failed: Too many open files
Mar 29 16:03:03 daemon.info vmh03 pveproxy[29619]: worker 13557 finished
Mar 29 16:03:03 daemon.info vmh03 pveproxy[29619]: starting 1 worker(s)
Mar 29 16:03:03 daemon.info vmh03 pveproxy[29619]: worker 11733 startedI added a few lines before and after for context.
For some reason, I'm only seeing this on the second and third nodes (vmh02 and vmh03). I'm currently connected and managing the cluster via vmh01.
This has prevented me from editing memory on one of the affected nodes but otherwise everything seems to be working as expected (I tried to edit memory again and it worked the second time).
I see a lot of posts about inotify and containers but I don't think that applies here. This appears to be a new issue. It looks like pveproxy runs as www-data, so I added this to /etc/security/limits.conf:
www-data - nofile 65536
Then I ran:
systemctl restart pveproxy
I'm not sure if this is the correct way to fix this but we'll see. Let me know if there is something else I should be doing to fix this.
Code:
# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-22
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1