Intermittent network loss from VM -> Node

[aph3rson]

Hello,

I'm running Ansible on a VM running on my single PVE node. Ansible is currently being used to call the Proxmox API on the node itself, using the community.general.proxmox_kvm module, which uses Proxmoxer under the hood (which, in turn, uses Requests to call the Proxmox API).

For the most part, things work well, except intermittent errors when calling the Proxmox API, like so (192.168.2.1 is my PVE node):

The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1303, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1159, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 448, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 337, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='192.168.2.1', port=8006): Read timed out. (read timeout=5)
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/tmp/ansible_community.general.proxmox_kvm_payload_k_zx1e0k/ansible_community.general.proxmox_kvm_payload.zip/ansible_collections/community/general/plugins/module_utils/proxmox.py", line 101, in _connect
    return ProxmoxAPI(api_host, verify_ssl=validate_certs, **auth_args)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/proxmoxer/core.py", line 210, in __init__
    self._backend = importlib.import_module(f".backends.{backend}", "proxmoxer").Backend(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/proxmoxer/backends/https.py", line 302, in __init__
    self.auth = ProxmoxHTTPAuth(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/proxmoxer/backends/https.py", line 59, in __init__
    self._get_new_tokens(********=********, otp=otp)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/proxmoxer/backends/https.py", line 70, in _get_new_tokens
    response_data = requests.post(
  File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='192.168.2.1', port=8006): Read timed out. (read timeout=5)

This is an intermittent exception - calls to the API will work sometimes, and fail other times. The exception implies that the TLS socket itself isn't receiving packets.

I haven't noticed other issues with Proxmox's UI from my device on the same network, and `journalctl` shows nothing out-of-the-ordinary on both the node and the VM. Rebooting both has not helped.

What may be causing the issue here?

 

[aph3rson]

Looking more into this, I grabbed some PCAPs from both the node and the guest. I can see where the request is sent off, and the TCP ACK back from the node. No TLS data is sent in the meantime, and the node disconnects five seconds later.

Looking at pveproxy logs, I can see a proxy detected vanished client connection warning - it seems PVE knows that the client disconnected. However, it doesn't tell me why this request is taking so long to respond. Based on what I can tell, this appears to be happening at authentication time within Proxmoxer - the task this Ansible play should be doing (starting a VM) isn't getting started, and the exception suggests this is in Proxmoxer's auth layer. I'm using PAM auth here, nothing special.

I've attached some selected pieces of the PCAPs from both node and VM, in case that helps. I could really use some help here.