REST API connection error

J

jan.svoboda

Member
Jun 11, 2020
32
1
13
Hello,

I have some issues with REST API and I'm unable to find out what is the problem.

I have Python 3 wrapper over proxmoxer (version 1.1.1) Python library with HTTPS backend. During script execution, after several minutes the script fails on following exceptions like there were some network issues but I cannot se any problem causing this in logs on the Proxmox node. All I could read from the traceback is only that the Proxmox node terminated HTTP session for some reason.

Code: 
root@devel1:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 8.0-2
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

Traceback (vm_management_http.py is my wrapper over proxmoxer):
Code: 
Traceback (most recent call last):
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 53, in <module>
    ip_address = proxmox.get_machine_ip(vm_name)
  File "/home/svoboda/src/proxmox-mgmt/vm_management_http.py", line 522, in get_machine_ip
    vm_handle = self._get_vm_handle_by_vm_name(vm_name)
  File "/home/svoboda/src/proxmox-mgmt/vm_management_http.py", line 1955, in _get_vm_handle_by_vm_name
    nodes = self.get_proxmox_nodes()
  File "/home/svoboda/src/proxmox-mgmt/vm_management_http.py", line 54, in get_proxmox_nodes
    nodes = self._proxmox.nodes.get()
  File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
    return self(args)._request("GET", params=params)
  File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
    resp = self._store["session"].request(method, url, data=data or None, params=params)
  File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
    return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I found out that the same thing happens when I manually stop or restart daemon pveproxy the script execution. Unfortunately I didn't find any logs that would cause it when pveproxy runs fine.

Any idea what could be wrong?

Thank you for advises.
 
pve-cluster contains errors only right after its start:
root@devel1:/var/log# journalctl -u pve-cluster
-- Logs begin at Mon 2020-07-27 10:14:49 CEST, end at Wed 2020-07-29 13:33:38 CEST. --
Jul 27 10:15:02 devel1 systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 27 10:15:02 devel1 pmxcfs[1800]: [quorum] crit: quorum_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [quorum] crit: can't initialize service
Jul 27 10:15:02 devel1 pmxcfs[1800]: [confdb] crit: cmap_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [confdb] crit: can't initialize service
Jul 27 10:15:02 devel1 pmxcfs[1800]: [dcdb] crit: cpg_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [dcdb] crit: can't initialize service
Jul 27 10:15:02 devel1 pmxcfs[1800]: [status] crit: cpg_initialize failed: 2
Jul 27 10:15:02 devel1 pmxcfs[1800]: [status] crit: can't initialize service

After that it doesn't contain any error. There are a lot of logs like the following:

Code: 
root@devel1:/var/log# journalctl -u pve-cluster
Jul 29 13:29:29 devel1 pmxcfs[1800]: [status] notice: received log


pve-daemon doesn't contain any error related to the REST API.

Are there any other daemons related to the REST API?
 
I set debugging of HTTP to print HTTP headers and run the script by following code:
Python: 
import http.client as http_client
http_client.HTTPConnection.debuglevel = 1

I got following output:

send: b'GET /api2/json/nodes/devel1/tasks/UPID:devel1:00007E2F:011D7FC0:5F2167D4:qmstart:225:root@pam:/status HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F2167D1::SdTzNXBPQNwMoG76mg2LpDlQHn8U3Yc4nrf6DFCMTQ5PlNpL337ws7JSIDcIUscQUpI2g7UgrM0b7OmDagWbxlJeGrQfnV/9nu7mAQ2+aIBRz/2cwPKWKiWAyXo224Sc3sDOQ1USuzPA14XwQvuHsAxkkeKglv+geuQIQecIzv4HzI0jIaaT9fZgu22wUzmtCyrcKJBVOnKfsEpP+rf9M+EmUgEAqOmJOOhD+k5h2SqwENeno5tJKpZdaUw6eiIGKvUqA3G89JxWWCKAN9R/ZtkKC/BYH2rCeOlT5cDPxde0z8DdgrMXg23wlMCkVFP/FulVtOSsOS/BgxppcZHeqw==\r\n\r\n'
reply: ''
Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./test.py", line 49, in <module>
proxmox.poweron_machine(vm_name=vm_name)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 113, in poweron_machine
while self._is_task_in_progress(upid):
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 2083, in _is_task_in_progress
task_status = self._get_task_status(upid)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 1932, in _get_task_status
return self._proxmox.nodes(node).tasks(upid).status.get()['status']
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
return self(args)._request("GET", params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
resp = self._store["session"].request(method, url, data=data or None, params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
 
another script run, this time it wasn't terminated just after its start
send: b'POST /api2/json/nodes/devel1/qemu/225/status/start HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F216A77::G/TtypyL000aQpkNc4gJkxus7Gvfnu3a33wT9ZTAg2LE9ayBF16bA+7Tx+KvCCmDjMpc5ROak3XbOZX9eUDRGg9QTvN/de84Ph5Nd58vtYg/jZ4CFKdLHWulNGopfxtT6ujkUhqUHSz2MBcapN1RDCs9/sV7pG26vsnaJuyuJan5scQUcTBDjoc5YqV4LFl9aLkJyKYYOggRdtpQ78ap2ysha34xDmOJq3nGHOZZrhiTu+KD2ffcr1HhKwU7EWyTqCWPph+yS8kWNvygsFUyKzVCebf0Gp5NI6KTZI+yFNCgN1bqrqujHwCUqhsCrhz5z4Lgse0DEE6YEoipQq1bRQ==\r\nContent-Length: 0\r\nCSRFPreventionToken: 5F216A77:GTCXtHnEuF3p1AVSbsU3Ka0dgu6T3t7t8CJAMg1f8VU\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0
header: Connection: close
header: Connection: Keep-Alive
header: Date: Wed, 29 Jul 2020 12:25:29 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Content-Length: 71
header: Content-Type: application/json;charset=UTF-8
header: Expires: Wed, 29 Jul 2020 12:25:29 GMT
send: b'GET /api2/json/nodes/devel1/tasks/UPID:devel1:00008B0A:011EA169:5F216AB9:qmstart:225:root@pam:/status HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F216A77::G/TtypyL000aQpkNc4gJkxus7Gvfnu3a33wT9ZTAg2LE9ayBF16bA+7Tx+KvCCmDjMpc5ROak3XbOZX9eUDRGg9QTvN/de84Ph5Nd58vtYg/jZ4CFKdLHWulNGopfxtT6ujkUhqUHSz2MBcapN1RDCs9/sV7pG26vsnaJuyuJan5scQUcTBDjoc5YqV4LFl9aLkJyKYYOggRdtpQ78ap2ysha34xDmOJq3nGHOZZrhiTu+KD2ffcr1HhKwU7EWyTqCWPph+yS8kWNvygsFUyKzVCebf0Gp5NI6KTZI+yFNCgN1bqrqujHwCUqhsCrhz5z4Lgse0DEE6YEoipQq1bRQ==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: max-age=0
header: Connection: Keep-Alive
header: Connection: Keep-Alive
header: Date: Wed, 29 Jul 2020 12:25:30 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Content-Length: 214
header: Content-Type: application/json;charset=UTF-8
header: Expires: Wed, 29 Jul 2020 12:25:30 GMT
send: b'GET /api2/json/nodes/devel1/tasks/UPID:devel1:00008B0A:011EA169:5F216AB9:qmstart:225:root@pam:/status HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\naccept: application/json, application/x-javascript, text/javascript, text/x-javascript, text/x-json\r\nConnection: keep-alive\r\nCookie: PVEAuthCookie=PVE:root@pam:5F216A77::G/TtypyL000aQpkNc4gJkxus7Gvfnu3a33wT9ZTAg2LE9ayBF16bA+7Tx+KvCCmDjMpc5ROak3XbOZX9eUDRGg9QTvN/de84Ph5Nd58vtYg/jZ4CFKdLHWulNGopfxtT6ujkUhqUHSz2MBcapN1RDCs9/sV7pG26vsnaJuyuJan5scQUcTBDjoc5YqV4LFl9aLkJyKYYOggRdtpQ78ap2ysha34xDmOJq3nGHOZZrhiTu+KD2ffcr1HhKwU7EWyTqCWPph+yS8kWNvygsFUyKzVCebf0Gp5NI6KTZI+yFNCgN1bqrqujHwCUqhsCrhz5z4Lgse0DEE6YEoipQq1bRQ==\r\n\r\n'
reply: ''
Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/svoboda/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.8/http/client.py", line 1332, in getresponse
response.begin()
File "/usr/lib64/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.8/http/client.py", line 272, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./test.py", line 49, in <module>
proxmox.poweron_machine(vm_name=vm_name)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 113, in poweron_machine
while self._is_task_in_progress(upid):
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 2083, in _is_task_in_progress
task_status = self._get_task_status(upid)
File "/home/svoboda/src/proxmox-mgmt/vm_management/vm_management_http.py", line 1932, in _get_task_status
return self._proxmox.nodes(node).tasks(upid).status.get()['status']
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
return self(args)._request("GET", params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
resp = self._store["session"].request(method, url, data=data or None, params=params)
File "/home/svoboda/.local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/svoboda/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
 
do you see anything on the PVE side in the logs? can you attempt starting 'pveproxy' in debug/foreground mode and then repeat and watch the output?
 
Sadly, nothing that would give me some clue. I run pveproxy in debug mode by command systemctl stop pveproxy && pveproxy start --debug 1 There are no errors or warnings in the log, just accepting and closing connections like following piece of the log. This is a snippet a while before the script was interrupted:
Code: 
root@devel1:~# systemctl stop pveproxy && pveproxy start --debug 1
31642: ACCEPT FH48 CONN13
close connection AnyEvent::Handle=HASH(0x55a034e28ea8)
31642: CLOSE FH48 CONN12
close connection AnyEvent::Handle=HASH(0x55a034ea9a68)
31640: CLOSE FH132 CONN6
31640: ACCEPT FH89 CONN7
close connection AnyEvent::Handle=HASH(0x55a034eac8c0)
31640: CLOSE FH89 CONN6
31640: ACCEPT FH89 CONN7
31640: ACCEPT FH118 CONN8
31641: ACCEPT FH14 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e9f280)
31641: CLOSE FH14 CONN7
31641: ACCEPT FH14 CONN8
close connection AnyEvent::Handle=HASH(0x55a034dc0c80)
31642: CLOSE FH32 CONN11
31641: ACCEPT FH19 CONN9
close connection AnyEvent::Handle=HASH(0x55a034d8e408)
31641: CLOSE FH19 CONN8
31640: ACCEPT FH123 CONN9
31640: ACCEPT FH132 CONN10
close connection AnyEvent::Handle=HASH(0x55a034e974d8)
31641: CLOSE FH14 CONN7
31641: ACCEPT FH14 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e19ee8)
31642: CLOSE FH35 CONN10
close connection AnyEvent::Handle=HASH(0x55a034fc7160)
31640: CLOSE FH132 CONN9
close connection AnyEvent::Handle=HASH(0x55a034f55f40)
31642: CLOSE FH40 CONN9
31640: ACCEPT FH132 CONN10
31640: ACCEPT FH135 CONN11
31641: ACCEPT FH21 CONN9
close connection AnyEvent::Handle=HASH(0x55a034f2bfe8)
31641: CLOSE FH21 CONN8
close connection AnyEvent::Handle=HASH(0x55a034d60c60)
31641: CLOSE FH14 CONN7
close connection AnyEvent::Handle=HASH(0x55a034dc3058)
31642: CLOSE FH31 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e289c8)
31642: CLOSE FH17 CONN7
31642: ACCEPT FH17 CONN8
close connection AnyEvent::Handle=HASH(0x55a034e04b90)
31642: CLOSE FH17 CONN7
31642: ACCEPT FH17 CONN8

In /var/log/pveproxy/access.log there are no error messages either.
Code: 
192.168.7.10 - root@pam [30/07/2020:13:15:21 +0200] "POST /api2/json/nodes/devel1/qemu/228/status/start HTTP/1.1" 200 71
192.168.7.10 - root@pam [30/07/2020:13:15:21 +0200] "GET /api2/json/nodes/devel1/tasks/UPID:devel1:00007CDE:019C0B46:5F22ABC8:qmstart:228:root@pam:/status HTTP/1.1" 200 214
192.168.7.10 - root@pam [30/07/2020:13:15:22 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4136
192.168.7.10 - root@pam [30/07/2020:13:15:25 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4167
192.168.7.10 - root@pam [30/07/2020:13:15:28 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4167
192.168.7.10 - root@pam [30/07/2020:13:15:35 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4236
192.168.7.10 - root@pam [30/07/2020:13:15:38 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4236
192.168.7.10 - root@pam [30/07/2020:13:15:42 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4234
192.168.7.10 - root@pam [30/07/2020:13:15:45 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4277
192.168.7.10 - root@pam [30/07/2020:13:15:48 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4277
192.168.7.10 - root@pam [30/07/2020:13:15:52 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4299
192.168.7.10 - root@pam [30/07/2020:13:15:55 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4276
192.168.7.10 - root@pam [30/07/2020:13:15:59 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4276
192.168.7.10 - root@pam [30/07/2020:13:16:02 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4264
192.168.7.10 - root@pam [30/07/2020:13:16:05 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4282
192.168.7.10 - root@pam [30/07/2020:13:16:09 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4282
192.168.7.10 - root@pam [30/07/2020:13:16:12 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4243
192.168.7.10 - root@pam [30/07/2020:13:16:15 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4246
192.168.7.10 - root@pam [30/07/2020:13:16:19 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4246
192.168.7.10 - root@pam [30/07/2020:13:16:22 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4252
192.168.7.10 - root@pam [30/07/2020:13:16:25 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4256
192.168.7.10 - root@pam [30/07/2020:13:16:28 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4256
192.168.7.10 - root@pam [30/07/2020:13:16:32 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4300
192.168.7.10 - root@pam [30/07/2020:13:16:35 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4315
192.168.7.10 - root@pam [30/07/2020:13:16:38 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4315
192.168.7.10 - root@pam [30/07/2020:13:16:41 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4258
192.168.7.10 - root@pam [30/07/2020:13:16:45 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4254
192.168.7.10 - root@pam [30/07/2020:13:16:48 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4254
192.168.7.10 - root@pam [30/07/2020:13:16:51 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4293
192.168.7.10 - root@pam [30/07/2020:13:16:54 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4288
192.168.7.10 - root@pam [30/07/2020:13:16:58 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4288
192.168.7.10 - root@pam [30/07/2020:13:17:01 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4306
192.168.7.10 - root@pam [30/07/2020:13:17:04 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 4266
 
I have enabled debug printing of HTTP headers by Python 3 library requests and the last HTTP request failed with code HTTP 596 Connection timed out
Code: 
send: b'POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1\r\nHost: devel1.proxmox.somedomain.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCSRFPreventionToken: 5F22BB73:VkYwrd9a8GTUOa21bznirwQTSaAwndcXjQAuPr+1sTc\r\nCookie: PVEAuthCookie=PVE:svoboda@ldap:5F22BB73::rTvZyUk6ewTAyNRaqk1C3k5nUOEfobuoJRr93QZ7+x9o6c+SsCt/M+XduCv+owZz4AFxKaMTAnOo8yiJ6OX6NQRyxdD72Y7HEykrJ4RsL5B8QX7CRi/JpxOUJ4nRxBWCP2WvCKfIGgqhzGFVh2cGGlJ8XyvVOWVlsI3vIEYJzdkZAfdLDiVi+RcU+QpyI07MHqhHXHI18kBFGEhjbti1ienaxosJvfTUcEdWP3XZAdZxIGoDTJrQe3d/l2gslZ+8Wrygcj8Lb71oWIkG2sGz/5AnWqDYqwmeWIB2x1itBxERr/8jkFYPv87yF7eB03k+JWWJYH0R8nphX2Uo5k4juA==\r\nContent-Length: 0\r\n\r\n'
reply: 'HTTP/1.1 596 Connection timed out\r\n'
header: Cache-Control: max-age=0
header: Connection: close
header: Date: Thu, 30 Jul 2020 12:34:48 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Expires: Thu, 30 Jul 2020 12:34:48 GMT
===================================================
{'Cache-Control': 'max-age=0', 'Connection': 'close', 'Date': 'Thu, 30 Jul 2020 12:34:48 GMT', 'Pragma': 'no-cache', 'Server': 'pve-api-daemon/3.0', 'Expires': 'Thu, 30 Jul 2020 12:34:48 GMT'}
 
we don't set that error ourselves AFAICT..
 
is this a cluster?
 
Those HTTP 596 errors are in /var/log/pveproxy/access.log
Code: 
root@devel1:~# grep 'HTTP/1.1" 596' /var/log/pveproxy/access.log
192.168.7.10 - root@pam [30/07/2020:12:28:37 +0200] "GET /api2/json/nodes/testcl/qemu/225/status/current HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:14:24:48 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
192.168.7.12 - secka@ldap [30/07/2020:14:24:48 +0200] "GET /api2/json/nodes/devel1/qemu/287/status/current HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:14:34:48 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
192.168.3.210 - root@pam [30/07/2020:15:04:08 +0200] "POST /api2/json/nodes/devel2/qemu/104/agent/shutdown HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:15:23:38 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
192.168.70.76 - dolinsky@ldap [30/07/2020:15:23:39 +0200] "GET /api2/json/nodes/devel1/qemu/252/status/current HTTP/1.1" 596 -
192.168.7.10 - svoboda@ldap [30/07/2020:15:44:54 +0200] "POST /api2/json/nodes/devel1/qemu/225/agent/shutdown HTTP/1.1" 596 -
 
hmm. it could be proxying either to another node or to pvedaemon that times out - in that case we return the HTTP status code from the proxying request.. you can verify whether that is the case by adding a print in /usr/share/perl5/PVE/APIServer/AnyEvent.pm around line 650 - running apt install --reinstall libpve-http-server-perl will revert to the stock version again. is the system under high load or very underpowered?
 
Looks like the problem is back. I created a simple Python script using requests library to be able to print raw HTTP headers.

Python: 
#!/usr/bin/env python3

import requests
import json
import sys

# ticket and token obtained by: curl -k -d "username=svoboda@ldap&password=mypassword" https://devel1.proxmox.example.com:8006/api2/json/access/ticket
TICKET = 'PVE:svoboda@ldap:5F314237::LpDSrR2jPw6STdOLj0bkzdah7yRqXhAi/pVnbU6vkvmuNHagEikF8+jUUqMSMxbtu7FWTwzRcRoQ6q8UqSutZsUiUbpSbsAn4l8Lv/9VaMyRMh4+dRK/eKckUKGJdPOqz2I3GoPeThzGcjCBTpuNzCH4lldnHuWNjE1il3db5fdhFl/w9ZqjxLNjAqlYISxpu8K3SKNhILzhmtY6ofr+cRR1M/Z02iszvpqAU9Lloot33pj4AG9G8DvzmJn5CtfyGFUBCaT4cPtknfOaOgiwz+0pLFU6lZm2bCTkIJI5Ls5QI/EYPpJpZ3VQqoxDzlJD/iMSJgK4DddG33h2/PBsrA=='

TOKEN = '5F314237:23feZZDRcawyvm8hIsLF0j61sgqU6zX5/iCmzXuRVI0'

cookies = {'PVEAuthCookie': TICKET}
headers = {'CSRFPreventionToken': TOKEN}

def shutdown_vm():
  response = requests.post('https://devel1.proxmox.example.com:8006/api2/json/nodes/devel1/qemu/245/agent/shutdown', cookies=cookies, headers=headers, verify=False)

  print('===================================================')
  print(response.headers)
  print('===================================================')

  if response.status_code != 200:
    print(f'ERROR: failed to shutdown VM with error code: {response.status_code}', file=sys.stderr)
    sys.exit(4)

Code: 
send: b'POST /api2/json/nodes/devel1/qemu/245/agent/shutdown HTTP/1.1\r\nHost: devel1.proxmox.flowmon.com:8006\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCSRFPreventionToken: 5F314237:23feZZDRcawyvm8hIsLF0j61sgqU6zX5/iCmzXuRVI0\r\nCookie: PVEAuthCookie=PVE:svoboda@ldap:5F314237::LpDSrR2jPw6STdOLj0bkzdah7yRqXhAi/pVnbU6vkvmuNHagEikF8+jUUqMSMxbtu7FWTwzRcRoQ6q8UqSutZsUiUbpSbsAn4l8Lv/9VaMyRMh4+dRK/eKckUKGJdPOqz2I3GoPeThzGcjCBTpuNzCH4lldnHuWNjE1il3db5fdhFl/w9ZqjxLNjAqlYISxpu8K3SKNhILzhmtY6ofr+cRR1M/Z02iszvpqAU9Lloot33pj4AG9G8DvzmJn5CtfyGFUBCaT4cPtknfOaOgiwz+0pLFU6lZm2bCTkIJI5Ls5QI/EYPpJpZ3VQqoxDzlJD/iMSJgK4DddG33h2/PBsrA==\r\nContent-Length: 0\r\n\r\n'
reply: 'HTTP/1.1 596 Connection timed out\r\n'
header: Cache-Control: max-age=0
header: Connection: close
header: Date: Mon, 10 Aug 2020 13:36:26 GMT
header: Pragma: no-cache
header: Server: pve-api-daemon/3.0
header: Expires: Mon, 10 Aug 2020 13:36:26 GMT
===================================================
{'Cache-Control': 'max-age=0', 'Connection': 'close', 'Date': 'Mon, 10 Aug 2020 13:36:26 GMT', 'Pragma': 'no-cache', 'Server': 'pve-api-daemon/3.0', 'Expires': 'Mon, 10 Aug 2020 13:36:26 GMT'}
===================================================
ERROR: failed to shutdown VM with error code: 596

I have added debug print to file /usr/share/perl5/PVE/APIServer/AnyEvent.pm as you advised (string "************** HERE1 ***************\n") but it did not print to logs :
Code: 
eval {
        my $code = delete $hdr->{Status};
        my $msg = delete $hdr->{Reason};
        delete $hdr->{URL};
        delete $hdr->{HTTPVersion};
        my $header = HTTP::Headers->new(%$hdr);
        if (my $location = $header->header('Location')) {
      $location =~ s|^http://localhost:85||;
      $header->header(Location => $location);
      print "************** HERE1 ***************\n";
        }
        my $resp = HTTP::Response->new($code, $msg, $header, $body);
        # Note: disable compression, because body is already compressed
        $self->response($reqstate, $resp, undef, 1);
    };
 
could you put the print outside of the if?

e.g., like this
Code: 
eval {
        my $code = delete $hdr->{Status};
        my $msg = delete $hdr->{Reason};
        delete $hdr->{URL};
        delete $hdr->{HTTPVersion};
        my $header = HTTP::Headers->new(%$hdr);
        if (my $location = $header->header('Location')) {
      $location =~ s|^http://localhost:85||;
      $header->header(Location => $location);
        }
       warn "$code - $msg\n";
        my $resp = HTTP::Response->new($code, $msg, $header, $body);
        # Note: disable compression, because body is already compressed
        $self->response($reqstate, $resp, undef, 1);
    };
 
This is interesting. There are 595 - no route to host errors.
Code: 
Aug 17 10:11:31 testcl systemd[1]: Started PVE API Proxy Server.
Aug 17 10:18:21 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:22 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:27 testcl pveproxy[2163]: 200 - OK
Aug 17 10:18:32 testcl pveproxy[2163]: 200 - OK
Aug 17 10:18:35 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:39 testcl pveproxy[2162]: 595 - No route to host
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2162]: 200 - OK
Aug 17 10:18:39 testcl pveproxy[2163]: 200 - OK
Aug 17 10:18:42 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:45 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:48 testcl pveproxy[2164]: 595 - No route to host
Aug 17 10:18:48 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:48 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:49 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:49 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:49 testcl pveproxy[2164]: 200 - OK
Aug 17 10:18:51 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:54 testcl pveproxy[2163]: 595 - No route to host
Aug 17 10:18:57 testcl pveproxy[2163]: 595 - No route to host
 
After adding the debug print warn "$code - $msg\n"; to /usr/share/perl5/PVE/APIServer/AnyEvent.pm, the script was interrupted and no errors appeared in the pveproxy log, just 200 - OK at the time when the script was interrupted.
 
Meanwhile I found another interesting logs
Code: 
Aug 17 08:25:50 devel1 pveproxy[24109]: Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 256.
<omitted>
Aug 17 08:27:31 devel1 pveproxy[24448]: 2020-08-17 08:27:31.158150 +0200 error AnyEvent::Util: Runtime error in AnyEvent::guard callback: Can't call method "_put_session" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent/Handle.pm line 2259 during global destruction.


Sometimes there are logs Too many open files from pveproxy though I changed max open files
Code: 
root@devel1:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031416
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 9000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1031416
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
as proposed here: https://forum.proxmox.com/threads/too-many-open-files-on-proxmox-api-gui-with-kvm.66986
Code: 
Aug 14 09:22:26 devel1 pveproxy[31510]: starting 1 worker(s)
Aug 14 09:22:26 devel1 pveproxy[31510]: worker 4746 started
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:26 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:27 devel1 pveproxy[4745]: ipcc_send_rec[1] failed: Too many open files
Aug 14 09:22:27 devel1 pveproxy[4745]: ipcc_send_rec[2] failed: Too many open files
Aug 14 09:22:27 devel1 pveproxy[4745]: ipcc_send_rec[3] failed: Too many open files
Aug 14 09:22:28 devel1 pveproxy[4746]: Clearing outdated entries from certificate cache
Aug 14 09:22:29 devel1 pveproxy[4745]: got inotify poll request in wrong process - disabling inotify
Aug 14 09:22:31 devel1 pveproxy[4745]: worker exit
Aug 14 09:27:23 devel1 pveproxy[31510]: worker 4647 finished

and another
Code: 
Aug 14 09:22:03 devel1 pveproxy[31510]: starting 1 worker(s)
Aug 14 09:22:03 devel1 pveproxy[31510]: worker 4647 started
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/testcl/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel1/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel2/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel4/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel3/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel5/pve-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/testcl/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel1/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel2/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel4/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel3/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel5/pve-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/testcl/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel1/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel2/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel4/pveproxy-ssl.pem' - Too many open files
Aug 14 09:22:04 devel1 pveproxy[4152]: 4152: unable to read '/etc/pve/nodes/devel3/pveproxy-ssl.pem' - Too many open files
<ommited>
Aug 14 09:22:05 devel1 pveproxy[4152]: failed to accept connection: Too many open files
Aug 14 09:22:05 devel1 pveproxy[31510]: worker 4152 finished
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back