REST API connection error

jan.svoboda

Member
Jun 11, 2020
32
1
13
Hello,

I have some issues with REST API and I'm unable to find out what is the problem.

I have Python 3 wrapper over proxmoxer (version 1.1.1) Python library with HTTPS backend. During script execution, after several minutes the script fails on following exceptions like there were some network issues but I cannot se any problem causing this in logs on the Proxmox node. All I could read from the traceback is only that the Proxmox node terminated HTTP session for some reason.

Code:
root@devel1:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 8.0-2
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

Traceback (vm_management_http.py is my wrapper over proxmoxer):
Code:
Traceback (most recent call last):
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 53, in <module>
    ip_address = proxmox.get_machine_ip(vm_name)
  File "/home/svoboda/src/proxmox-mgmt/vm_management_http.py", line 522, in get_machine_ip
    vm_handle = self._get_vm_handle_by_vm_name(vm_name)
  File "/home/svoboda/src/proxmox-mgmt/vm_management_http.py", line 1955, in _get_vm_handle_by_vm_name
    nodes = self.get_proxmox_nodes()
  File "/home/svoboda/src/proxmox-mgmt/vm_management_http.py", line 54, in get_proxmox_nodes
    nodes = self._proxmox.nodes.get()
  File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
    return self(args)._request("GET", params=params)
  File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
    resp = self._store["session"].request(method, url, data=data or None, params=params)
  File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
    return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I found out that the same thing happens when I manually stop or restart daemon pveproxy the script execution. Unfortunately I didn't find any logs that would cause it when pveproxy runs fine.

Any idea what could be wrong?

Thank you for advises.
 
pve-cluster contains errors only right after its start:
root@devel1:/var/log# journalctl -u pve-cluster
-- Logs begin at Mon 2020-07-27 10:14:49 CEST, end at Wed 2020-07-29 13:33:38 CEST. --
Jul 27 10:15:02 devel1 systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 27 10:15:02 devel1 pmxcfs[1800]: [quorum] crit: quorum_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [quorum] crit: can't initialize service
Jul 27 10:15:02 devel1 pmxcfs[1800]: [confdb] crit: cmap_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [confdb] crit: can't initialize service
Jul 27 10:15:02 devel1 pmxcfs[1800]: [dcdb] crit: cpg_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [dcdb] crit: can't initialize service
Jul 27 10:15:02 devel1 pmxcfs[1800]: [status] crit: cpg_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [status] crit: can't initialize service

After that it doesn't contain any error. There are a lot of logs like the following:

Code:
root@devel1:/var/log# journalctl -u pve-cluster
Jul 29 13:29:29 devel1 pmxcfs[1800]: [status] notice: received log


pve-daemon doesn't contain any error related to the REST API.

Are there any other daemons related to the REST API?
 
I set debugging of HTTP to print HTTP headers and run the script by following code:
Python:
import http.client as http_client
http_client.HTTPConnection.debuglevel = 1

I got following output:

send: b'GET /api2/json/nodes/devel1/tasks/UPID:devel1:00007E2F:011D7FC0:5F2167D4:qmstart:225:root@pam:/status HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F2167D1::SdTzNXBPQNwMoG76mg2LpDlQHn8U3Yc4nrf6DFCMTQ5PlNpL337ws7JSIDcIUscQUpI2g7UgrM0b7OmDagWbxlJeGrQfnV/9nu7mAQ2+aIBRz/2cwPKWKiWAyXo224Sc3sDOQ1USuzPA14XwQvuHsAxkkeKglv+geuQIQecIzv4HzI0jIaaT9fZgu22wUzmtCyrcKJBVOnKfsEpP+rf9M+EmUgEAqOmJOOhD+k5h2SqwENeno5tJKpZdaUw6eiIGKvUqA3G89JxWWCKAN9R/ZtkKC/BYH2rCeOlT5cDPxde0z8DdgrMXg23wlMCkVFP/FulVtOSsOS/BgxppcZHeqw==\r\n\r\n'
reply: ''
Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./test.py", line 49, in <module>
proxmox.poweron_machine(vm_name=vm_name)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 113, in poweron_machine
while self._is_task_in_progress(upid):
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 2083, in _is_task_in_progress
task_status = self._get_task_status(upid)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 1932, in _get_task_status
return self._proxmox.nodes(node).tasks(upid).status.get()['status']
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
return self(args)._request("GET", params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
resp = self._store["session"].request(method, url, data=data or None, params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
 
another script run, this time it wasn't terminated just after its start
send: b'POST /api2/json/nodes/devel1/qemu/225/status/start HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F216A77::G/TtypyL000aQpkNc4gJkxus7Gvfnu3a33wT9ZTAg2LE9ayBF16bA+7Tx+KvCCmDjMpc5ROak3XbOZX9eUDRGg9QTvN/de84Ph5Nd58vtYg/jZ4CFKdLHWulNGopfxtT6ujkUhqUHSz2MBcapN1RDCs9/sV7pG26vsnaJuyuJan5scQUcTBDjoc5YqV4LFl9aLkJyKYYOggRdtpQ78ap2ysha34xDmOJq3nGHOZZrhiTu+KD2ffcr1HhKwU7EWyTqCWPph+yS8kWNvygsFUyKzVCebf0Gp5NI6KTZI+yFNCgN1bqrqujHwCUqhsCrhz5z4Lgse0DEE6YEoipQq1bRQ==\r\nContent-Length: 0\r\nCSRFPreventionToken: 5F216A77:GTCXtHnEuF3p1AVSbsU3Ka0dgu6T3t7t8CJAMg1f8VU\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0
header: Connection: close
header: Connection: Keep-Alive
header: Date: Wed, 29 Jul 2020 12:25:29 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Content-Length: 71
header: Content-Type: application/json;charset=UTF-8
header: Expires: Wed, 29 Jul 2020 12:25:29 GMT
send: b'GET /api2/json/nodes/devel1/tasks/UPID:devel1:00008B0A:011EA169:5F216AB9:qmstart:225:root@pam:/status HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F216A77::G/TtypyL000aQpkNc4gJkxus7Gvfnu3a33wT9ZTAg2LE9ayBF16bA+7Tx+KvCCmDjMpc5ROak3XbOZX9eUDRGg9QTvN/de84Ph5Nd58vtYg/jZ4CFKdLHWulNGopfxtT6ujkUhqUHSz2MBcapN1RDCs9/sV7pG26vsnaJuyuJan5scQUcTBDjoc5YqV4LFl9aLkJyKYYOggRdtpQ78ap2ysha34xDmOJq3nGHOZZrhiTu+KD2ffcr1HhKwU7EWyTqCWPph+yS8kWNvygsFUyKzVCebf0Gp5NI6KTZI+yFNCgN1bqrqujHwCUqhsCrhz5z4Lgse0DEE6YEoipQq1bRQ==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0
header: Connection: Keep-Alive
header: Connection: Keep-Alive
header: Date: Wed, 29 Jul 2020 12:25:30 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Content-Length: 214
header: Content-Type: application/json;charset=UTF-8
header: Expires: Wed, 29 Jul 2020 12:25:30 GMT
send: b'GET /api2/json/nodes/devel1/tasks/UPID:devel1:00008B0A:011EA169:5F216AB9:qmstart:225:root@pam:/status HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F216A77::G/TtypyL000aQpkNc4gJkxus7Gvfnu3a33wT9ZTAg2LE9ayBF16bA+7Tx+KvCCmDjMpc5ROak3XbOZX9eUDRGg9QTvN/de84Ph5Nd58vtYg/jZ4CFKdLHWulNGopfxtT6ujkUhqUHSz2MBcapN1RDCs9/sV7pG26vsnaJuyuJan5scQUcTBDjoc5YqV4LFl9aLkJyKYYOggRdtpQ78ap2ysha34xDmOJq3nGHOZZrhiTu+KD2ffcr1HhKwU7EWyTqCWPph+yS8kWNvygsFUyKzVCebf0Gp5NI6KTZI+yFNCgN1bqrqujHwCUqhsCrhz5z4Lgse0DEE6YEoipQq1bRQ==\r\n\r\n'
reply: ''
Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./test.py", line 49, in <module>
proxmox.poweron_machine(vm_name=vm_name)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 113, in poweron_machine
while self._is_task_in_progress(upid):
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 2083, in _is_task_in_progress
task_status = self._get_task_status(upid)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 1932, in _get_task_status
return self._proxmox.nodes(node).tasks(upid).status.get()['status']
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
return self(args)._request("GET", params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
resp = self._store["session"].request(method, url, data=data or None, params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
 
do you see anything on the PVE side in the logs? can you attempt starting 'pveproxy' in debug/foreground mode and then repeat and watch the output?
 
Sadly, nothing that would give me some clue. I run pveproxy in debug mode by command systemctl stop pveproxy && pveproxy start --debug 1 There are no errors or warnings in the log, just accepting and closing connections like following piece of the log. This is a snippet a while before the script was interrupted:
Code:
root@devel1:~# systemctl stop pveproxy && pveproxy start --debug 1
31642: ACCEPT FH48 CONN13
close connection AnyEvent::Handle=HASH(0x55a034e28ea8)
31642: CLOSE FH48 CONN12
close connection AnyEvent::Handle=HASH(0x55a034ea9a68)
31640: CLOSE FH132 CONN6
31640: ACCEPT FH89 CONN7
close connection AnyEvent::Handle=HASH(0x55a034eac8c0)
31640: CLOSE FH89 CONN6
31640: ACCEPT FH89 CONN7
31640: ACCEPT FH118 CONN8
31641: ACCEPT FH14 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e9f280)
31641: CLOSE FH14 CONN7
31641: ACCEPT FH14 CONN8
close connection AnyEvent::Handle=HASH(0x55a034dc0c80)
31642: CLOSE FH32 CONN11
31641: ACCEPT FH19 CONN9
close connection AnyEvent::Handle=HASH(0x55a034d8e408)
31641: CLOSE FH19 CONN8
31640: ACCEPT FH123 CONN9
31640: ACCEPT FH132 CONN10
close connection AnyEvent::Handle=HASH(0x55a034e974d8)
31641: CLOSE FH14 CONN7
31641: ACCEPT FH14 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e19ee8)
31642: CLOSE FH35 CONN10
close connection AnyEvent::Handle=HASH(0x55a034fc7160)
31640: CLOSE FH132 CONN9
close connection AnyEvent::Handle=HASH(0x55a034f55f40)
31642: CLOSE FH40 CONN9
31640: ACCEPT FH132 CONN10
31640: ACCEPT FH135 CONN11
31641: ACCEPT FH21 CONN9
close connection AnyEvent::Handle=HASH(0x55a034f2bfe8)
31641: CLOSE FH21 CONN8
close connection AnyEvent::Handle=HASH(0x55a034d60c60)
31641: CLOSE FH14 CONN7
close connection AnyEvent::Handle=HASH(0x55a034dc3058)
31642: CLOSE FH31 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e289c8)
31642: CLOSE FH17 CONN7
31642: ACCEPT FH17 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e04b90)
31642: CLOSE FH17 CONN7
31642: ACCEPT FH17 CONN8

In /var/log/pveproxy/access.log there are no error messages either.
Code:
192.168.7.10 - root@pam [30/07/2020:13:15:21 +0200] "POST /api2/json/nodes/devel1/qemu/228/status/start HTTP/1.1" 200 71
192.168.7.10 - root@pam [30/07/2020:13:15:21 +0200] "GET /api2/json/nodes/devel1/tasks/UPID:devel1:00007CDE:019C0B46:5F22ABC8:qmstart:228:root@pam:/status HTTP/1.1" 200 214
192.168.7.10 - root@pam [30/07/2020:13:15:22 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4136
192.168.7.10 - root@pam [30/07/2020:13:15:25 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4167
192.168.7.10 - root@pam [30/07/2020:13:15:28 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4167
192.168.7.10 - root@pam [30/07/2020:13:15:35 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4236
192.168.7.10 - root@pam [30/07/2020:13:15:38 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4236
192.168.7.10 - root@pam [30/07/2020:13:15:42 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4234
192.168.7.10 - root@pam [30/07/2020:13:15:45 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4277
192.168.7.10 - root@pam [30/07/2020:13:15:48 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4277
192.168.7.10 - root@pam [30/07/2020:13:15:52 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4299
192.168.7.10 - root@pam [30/07/2020:13:15:55 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4276
192.168.7.10 - root@pam [30/07/2020:13:15:59 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4276
192.168.7.10 - root@pam [30/07/2020:13:16:02 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4264
192.168.7.10 - root@pam [30/07/2020:13:16:05 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4282
192.168.7.10 - root@pam [30/07/2020:13:16:09 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4282
192.168.7.10 - root@pam [30/07/2020:13:16:12 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4243
192.168.7.10 - root@pam [30/07/2020:13:16:15 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4246
192.168.7.10 - root@pam [30/07/2020:13:16:19 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4246
192.168.7.10 - root@pam [30/07/2020:13:16:22 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4252
192.168.7.10 - root@pam [30/07/2020:13:16:25 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4256
192.168.7.10 - root@pam [30/07/2020:13:16:28 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4256
192.168.7.10 - root@pam [30/07/2020:13:16:32 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4300
192.168.7.10 - root@pam [30/07/2020:13:16:35 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4315
192.168.7.10 - root@pam [30/07/2020:13:16:38 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4315
192.168.7.10 - root@pam [30/07/2020:13:16:41 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4258
192.168.7.10 - root@pam [30/07/2020:13:16:45 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4254
192.168.7.10 - root@pam [30/07/2020:13:16:48 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4254
192.168.7.10 - root@pam [30/07/2020:13:16:51 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4293
192.168.7.10 - root@pam [30/07/2020:13:16:54 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4288
192.168.7.10 - root@pam [30/07/2020:13:16:58 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4288
192.168.7.10 - root@pam [30/07/2020:13:17:01 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4306
192.168.7.10 - root@pam [30/07/2020:13:17:04 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4266
 
I have enabled debug printing of HTTP headers by Python 3 library requests and the last HTTP request failed with code HTTP 596 Connection timed out
Code:
send: b'POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCSRFPreventionToken: 5F22BB73:VkYwrd9a8GTUOa21bznirwQTSaAwndcXjQAuPr+1sTc\r\nCookie: PVEAuthCookie=PVE:svoboda@ldap:5F22BB73::rTvZyUk6ewTAyNRaqk1C3k5nUOEfobuoJRr93QZ7+x9o6c+SsCt/M+XduCv+owZz4AFxKaMTAnOo8yiJ6OX6NQRyxdD72Y7HEykrJ4RsL5B8QX7CRi/JpxOUJ4nRxBWCP2WvCKfIGgqhzGFVh2cGGlJ8XyvVOWVlsI3vIEYJzdkZAfdLDiVi+RcU+QpyI07MHqhHXHI18kBFGEhjbti1ienaxosJvfTUcEdWP3XZAdZxIGoDTJrQe3d/l2gslZ+8Wrygcj8Lb71oWIkG2sGz/5AnWqDYqwmeWIB2x1itBxERr/8jkFYPv87yF7eB03k+JWWJYH0R8nphX2Uo5k4juA==\r\nContent-Length: 0\r\n\r\n'
reply: 'HTTP/1.1 596 Connection timed out\r\n'
header: Cache-Control: max-age=0
header: Connection: close
header: Date: Thu, 30 Jul 2020 12:34:48 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Expires: Thu, 30 Jul 2020 12:34:48 GMT
===================================================
{'Cache-Control': 'max-age=0', 'Connection': 'close', 'Date': 'Thu, 30 Jul 2020 12:34:48 GMT', 'Pragma': 'no-cache', 'Server': 'pve-api-daemon/3.0', 'Expires': 'Thu, 30 Jul 2020 12:34:48 GMT'}
 
we don't set that error ourselves AFAICT..
 
is this a cluster?
 
Those HTTP 596 errors are in /var/log/pveproxy/access.log
Code:
root@devel1:~# grep 'HTTP/1.1" 596' /var/log/pveproxy/access.log
192.168.7.10 - root@pam [30/07/2020:12:28:37 +0200] "GET /api2/json/nodes/testcl/qemu/225/status/current HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:14:24:48 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
192.168.7.12 - secka@ldap [30/07/2020:14:24:48 +0200] "GET /api2/json/nodes/devel1/qemu/287/status/current HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:14:34:48 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
192.168.3.210 - root@pam [30/07/2020:15:04:08 +0200] "POST /api2/json/nodes/devel2/qemu/104/agent/shutdown HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:15:23:38 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
192.168.70.76 - dolinsky@ldap [30/07/2020:15:23:39 +0200] "GET /api2/json/nodes/devel1/qemu/252/status/current HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:15:44:54 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
 
hmm. it could be proxying either to another node or to pvedaemon that times out - in that case we return the HTTP status code from the proxying request.. you can verify whether that is the case by adding a print in /usr/share/perl5/PVE/APIServer/AnyEvent.pm around line 650 - running apt install --reinstall libpve-http-server-perl will revert to the stock version again. is the system under high load or very underpowered?
 
Looks like the problem is back. I created a simple Python script using requests library to be able to print raw HTTP headers.

Python:
#!/usr/bin/env python3

import requests
import json
import sys

# ticket and token obtained by: curl -k -d "username=svoboda@ldap&password=mypassword" https://devel1.proxmox.example.com:8006/api2/json/access/ticket
TICKET = 'PVE:svoboda@ldap:5F314237::LpDSrR2jPw6STdOLj0bkzdah7yRqXhAi/pVnbU6vkvmuNHagEikF8+jUUqMSMxbtu7FWTwzRcRoQ6q8UqSutZsUiUbpSbsAn4l8Lv/9VaMyRMh4+dRK/eKckUKGJdPOqz2I3GoPeThzGcjCBTpuNzCH4lldnHuWNjE1il3db5fdhFl/w9ZqjxLNjAqlYISxpu8K3SKNhILzhmtY6ofr+cRR1M/Z02iszvpqAU9Lloot33pj4AG9G8DvzmJn5CtfyGFUBCaT4cPtknfOaOgiwz+0pLFU6lZm2bCTkIJI5Ls5QI/EYPpJpZ3VQqoxDzlJD/iMSJgK4DddG33h2/PBsrA=='

TOKEN = '5F314237:23feZZDRcawyvm8hIsLF0j61sgqU6zX5/iCmzXuRVI0'

cookies = {'PVEAuthCookie': TICKET}
headers = {'CSRFPreventionToken': TOKEN}

def shutdown_vm():
  response = requests.post('https://devel1.proxmox.example.com:8006/api2/json/nodes/devel1/qemu/245/agent/shutdown', cookies=cookies, headers=headers, verify=False)

  print('===================================================')
  print(response.headers)
  print('===================================================')

  if response.status_code != 200:
    print(f'ERROR: failed to shutdown VM with error code: {response.status_code}', file=sys.stderr)
    sys.exit(4)

Code:
send: b'POST /api2/json/nodes/devel1/qemu/245/agent/shutdown HTTP/1.1\r\nHost: devel1.proxmox.flowmon.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCSRFPreventionToken: 5F314237:23feZZDRcawyvm8hIsLF0j61sgqU6zX5/iCmzXuRVI0\r\nCookie: PVEAuthCookie=PVE:svoboda@ldap:5F314237::LpDSrR2jPw6STdOLj0bkzdah7yRqXhAi/pVnbU6vkvmuNHagEikF8+jUUqMSMxbtu7FWTwzRcRoQ6q8UqSutZsUiUbpSbsAn4l8Lv/9VaMyRMh4+dRK/eKckUKGJdPOqz2I3GoPeThzGcjCBTpuNzCH4lldnHuWNjE1il3db5fdhFl/w9ZqjxLNjAqlYISxpu8K3SKNhILzhmtY6ofr+cRR1M/Z02iszvpqAU9Lloot33pj4AG9G8DvzmJn5CtfyGFUBCaT4cPtknfOaOgiwz+0pLFU6lZm2bCTkIJI5Ls5QI/EYPpJpZ3VQqoxDzlJD/iMSJgK4DddG33h2/PBsrA==\r\nContent-Length: 0\r\n\r\n'
reply: 'HTTP/1.1 596 Connection timed out\r\n'
header: Cache-Control: max-age=0
header: Connection: close
header: Date: Mon, 10 Aug 2020 13:36:26 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Expires: Mon, 10 Aug 2020 13:36:26 GMT
===================================================
{'Cache-Control': 'max-age=0', 'Connection': 'close', 'Date': 'Mon, 10 Aug 2020 13:36:26 GMT', 'Pragma': 'no-cache', 'Server': 'pve-api-daemon/3.0', 'Expires': 'Mon, 10 Aug 2020 13:36:26 GMT'}
===================================================
ERROR: failed to shutdown VM with error code: 596

I have added debug print to file /usr/share/perl5/PVE/APIServer/AnyEvent.pm as you advised (string "************** HERE1 ***************\n") but it did not print to logs :
Code:
eval {
        my $code = delete $hdr->{Status};
        my $msg = delete $hdr->{Reason};
        delete $hdr->{URL};
        delete $hdr->{HTTPVersion};
        my $header = HTTP::Headers->new(%$hdr);
        if (my $location = $header->header('Location')) {
      $location =~ s|^http://localhost:85||;
      $header->header(Location => $location);
      print "************** HERE1 ***************\n";
        }
        my $resp = HTTP::Response->new($code, $msg, $header, $body);
        # Note: disable compression, because body is already compressed
        $self->response($reqstate, $resp, undef, 1);
    };
 
could you put the print outside of the if?

e.g., like this
Code:
eval {
        my $code = delete $hdr->{Status};
        my $msg = delete $hdr->{Reason};
        delete $hdr->{URL};
        delete $hdr->{HTTPVersion};
        my $header = HTTP::Headers->new(%$hdr);
        if (my $location = $header->header('Location')) {
      $location =~ s|^http://localhost:85||;
      $header->header(Location => $location);
        }
       warn "$code - $msg\n";
        my $resp = HTTP::Response->new($code, $msg, $header, $body);
        # Note: disable compression, because body is already compressed
        $self->response($reqstate, $resp, undef, 1);
    };
 
This is interesting. There are 595 - no route to host errors.
Code:
Aug 17 10:11:31 testcl systemd[1]: Started PVE API Proxy Server.
Aug 17 10:18:21 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:27 testcl pveproxy[2163]: 200 - OK
Aug 17 10:18:32 testcl pveproxy[2163]: 200 - OK
Aug 17 10:18:35 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:39 testcl pveproxy[2162]: 595 - No route to host
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2163]: 200 - OK
Aug 17 10:18:42 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:45 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:48 testcl pveproxy[2164]: 595 - No route to host
Aug 17 10:18:48 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:48 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:49 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:49 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:49 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:51 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:54 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:57 testcl pveproxy[2163]: 595 - No route to host
 
After adding the debug print warn "$code - $msg\n"; to /usr/share/perl5/PVE/APIServer/AnyEvent.pm, the script was interrupted and no errors appeared in the pveproxy log, just 200 - OK at the time when the script was interrupted.
 
Meanwhile I found another interesting logs
Code:
Aug 17 08:25:50 devel1 pveproxy[24109]: Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 256.
<omitted>
Aug 17 08:27:31 devel1 pveproxy[24448]: 2020-08-17 08:27:31.158150 +0200 error AnyEvent::Util: Runtime error in AnyEvent::guard callback: Can't call method "_put_session" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent/Handle.pm line 2259 during global destruction.


Sometimes there are logs Too many open files from pveproxy though I changed max open files
Code:
root@devel1:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031416
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 9000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1031416
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
as proposed here: https://forum.proxmox.com/threads/too-many-open-files-on-proxmox-api-gui-with-kvm.66986
Code:
Aug 14 09:22:26 devel1 pveproxy[31510]: starting 1 worker(s)
Aug 14 09:22:26 devel1 pveproxy[31510]: worker 4746 started
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:27 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:27 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:27 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:28 devel1 pveproxy[4746]: Clearing outdated entries from certificate cache
Aug 14 09:22:29 devel1 pveproxy[4745]: got inotify poll request in wrong process - disabling inotify
Aug 14 09:22:31 devel1 pveproxy[4745]: worker exit
Aug 14 09:27:23 devel1 pveproxy[31510]: worker 4647 finished

and another
Code:
Aug 14 09:22:03 devel1 pveproxy[31510]: starting 1 worker(s)
Aug 14 09:22:03 devel1 pveproxy[31510]: worker 4647 started
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/testcl/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel1/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel2/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel4/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel3/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel5/pve-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/testcl/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel1/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel2/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel4/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel3/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel5/pve-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/testcl/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel1/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel2/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel4/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel3/pveproxy-ssl.pem' - Too many open files
<ommited>
Aug 14 09:22:05 devel1 pveproxy[4152]: failed to accept connection: Too many open files
Aug 14 09:22:05 devel1 pveproxy[31510]: worker 4152 finished