ERR_CONTENT_LENGTH_MISMATCH

[archerne]

Hi all. I am stuck, I restarted my PVE server and now I no longer have a GUI. When I go to the site I get:

    GET https://<IPv4>:8006/pve2/ext6/charts.js?ver=7.0.0 net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
    <IPv4>/:33 GET https://<IPv4>:8006/pve2/js/pvemanagerlib.js?ver=7.1-8 net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
    <IPv4>/:18 GET https://<IPv4>:8006/pve2/ext6/ext-all.js?ver=7.0.0 net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
    proxmoxlib.js?ver=3.4-4:2 Uncaught ReferenceError: Ext is not defined
        at proxmoxlib.js?ver=3.4-4:2
    (anonymous) @ proxmoxlib.js?ver=3.4-4:2
    locale-en.js?ver=7.0.0:1 Uncaught ReferenceError: Ext is not defined
        at locale-en.js?ver=7.0.0:1
    (anonymous) @ locale-en.js?ver=7.0.0:1
    (index):38 Uncaught ReferenceError: Ext is not defined
        at (index):38
    (anonymous) @ (index):38

I have googled and tried many things and none have them brought the GUI back. Anyone have suggestions?I tried:

• Browser hard reload
• systemctl restart pveproxy.service
• apt install --reinstall libjs-extjs
• apt install --reinstall proxmox-widget-toolkit
• apt install --reinstall pve-manager

Output of pveversion:

    proxmox-ve: 7.1-1 (running kernel: 5.15.7-1-pve)
    pve-manager: 7.1-8 (running version: 7.1-8/5b267f33)
    pve-kernel-5.15: 7.1-7
    pve-kernel-helper: 7.1-6
    pve-kernel-5.13: 7.1-5
    pve-kernel-5.11: 7.0-10
    pve-kernel-5.15.7-1-pve: 5.15.7-1
    pve-kernel-5.13.19-2-pve: 5.13.19-4
    pve-kernel-5.11.22-7-pve: 5.11.22-12
    pve-kernel-5.11.22-4-pve: 5.11.22-9
    ceph-fuse: 15.2.14-pve1
    corosync: 3.1.5-pve2
    criu: 3.15-1+pve-1
    glusterfs-client: 9.2-1
    ifupdown2: 3.1.0-1+pmx3
    ksm-control-daemon: 1.4-1
    libjs-extjs: 7.0.0-1
    libknet1: 1.22-pve2
    libproxmox-acme-perl: 1.4.0
    libproxmox-backup-qemu0: 1.2.0-1
    libpve-access-control: 7.1-5
    libpve-apiclient-perl: 3.2-1
    libpve-common-perl: 7.0-14
    libpve-guest-common-perl: 4.0-3
    libpve-http-server-perl: 4.0-4
    libpve-storage-perl: 7.0-15
    libspice-server1: 0.14.3-2.1
    lvm2: 2.03.11-2.1
    lxc-pve: 4.0.11-1
    lxcfs: 4.0.11-pve1
    novnc-pve: 1.3.0-1
    proxmox-backup-client: 2.1.2-1
    proxmox-backup-file-restore: 2.1.2-1
    proxmox-mini-journalreader: 1.3-1
    proxmox-widget-toolkit: 3.4-4
    pve-cluster: 7.1-3
    pve-container: 4.1-3
    pve-docs: 7.1-2
    pve-edk2-firmware: 3.20210831-2
    pve-firewall: 4.2-5
    pve-firmware: 3.3-4
    pve-ha-manager: 3.3-1
    pve-i18n: 2.6-2
    pve-qemu-kvm: 6.1.0-3
    pve-xtermjs: 4.12.0-1
    qemu-server: 7.1-4
    smartmontools: 7.2-1
    spiceterm: 3.2-2
    swtpm: 0.7.0~rc1+2
    vncterm: 1.7-1
    zfsutils-linux: 2.1.1-pve3

 

[Moayad]

Hi,

You can ssh or ping the PVE right? are all pve services up and running?

Can you post the output of the following command:

curl -k https://IP:8006 | grep title

 

[archerne]

Hi,

You can ssh or ping the PVE right? are all pve services up and running?

Can you post the output of the following command:

curl -k https://IP:8006 | grep title

curl -k https://IP:8006 | grep title
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2217  100  2217    0     0  16669      0 --:--:-- --:--:-- --:--:-- 16795
    <title>pve - Proxmox Virtual Environment</title>

 

[t.lamprecht]

net::ERR_CONTENT_LENGTH_MISMATCH

This is a weird error to get from Proxmox VE's API, do you have a (reverse) proxy running in-between?

what's the response if you manually load the JS file, e.g.:
curl -vk https://<IPv4>:8006/pve2/js/pvemanagerlib.js?ver=7.1-8 >/dev/null

Does the file exists and is it readable by pveproxy?
stat /usr/share/pve-manager/js/pvemanagerlib.js

 

[archerne]

This is a weird error to get from Proxmox VE's API, do you have a (reverse) proxy running in-between?

what's the response if you manually load the JS file, e.g.:
curl -vk https://<IPv4>:8006/pve2/js/pvemanagerlib.js?ver=7.1-8 >/dev/null

Does the file exists and is it readable by pveproxy?
stat /usr/share/pve-manager/js/pvemanagerlib.js

No reverse proxy, i am trying to connect to it via VPN as I am not at the location that the server is. It was working great until I restarted it the other day, and then the GUI didn't come back and is throwing these errors in the dev console.

curl -vk https://IP:8006/pve2/js/pvemanagerlib.js?ver=7.1-8 >/dev/null
*   Trying IP:8006...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to IP (IP) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1235 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=pve.yournamehere.com
*  start date: Nov 13 03:18:49 2021 GMT
*  expire date: Nov 13 03:18:49 2023 GMT
*  issuer: Cert Data
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
} [5 bytes data]
> GET /pve2/js/pvemanagerlib.js?ver=7.1-8 HTTP/1.1
> Host: IP:8006
> User-Agent: curl/7.74.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [217 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [217 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Date: Tue, 11 Jan 2022 17:48:53 GMT
< Server: pve-api-daemon/3.0
< Content-Length: 1119994
< Content-Type: application/javascript
< Last-Modified: Tue, 07 Dec 2021 17:55:42 GMT
<
{ [16167 bytes data]
100 1093k  100 1093k    0     0  18.4M      0 --:--:-- --:--:-- --:--:-- 18.4M
* Connection #0 to host IP left intact

stat /usr/share/pve-manager/js/pvemanagerlib.js
  File: /usr/share/pve-manager/js/pvemanagerlib.js
  Size: 1119994         Blocks: 2192       IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 5898686     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-01-10 21:37:19.994925475 -0600
Modify: 2021-12-07 11:55:42.000000000 -0600
Change: 2022-01-09 20:32:24.193472627 -0600
 Birth: 2022-01-09 20:32:23.817463043 -0600

 

[cerox]

@archerne I'm facing the same problem as you with my Proxmox server.
Did you find a solution??

 

[Joris L.]

@archerne same issue here, this is beyond weird to me at this time

 

[Alexey Pavlyuts]

I am not sure if it relevant to the issue, but I had experience absolutely the same symptoms with my Proxmox web interface.

My conditions seems specific: the servers are very far from me, in network mean 330-350ms RTT, and I use OpenVPN based tuneling to reach the interface. There no extra proxies or any other kind of intermediate, just VPN access into internal network.

I fond that the interface just can't load biggest Javascript libraries, the connection may be active for 15-18 sec and the close with this error. And also I feel all other tools slow and unstable, even SSH goes worse.

Same time, /var/log/pveproxy/access.log reports that the file was sent in a full with code 200.

Fortunately, I have VM with desktop and browser on the same farm and I can try from there - and everything works fine.

So, my conclusiton is that big RTT and, probably, narrow channel and packet drops may cause this, and it is not completely pveproxy fault.

Researching with Chrome inspect mode on (network tab) I found that if I set local cache off, each reload of the interface is inpredictable: it may success or fail, depending of success to load the biggest .js files (in my case):

• /pve2/ext6/ext-all.js?ver=7.0.0 - 684kB to load
• /pve2/js/pvemanagerlib.js?ver=7.4-16 - 268kB to load

All the interface fails goes from unability to load these files, in a case of success it may take op to 40s to load.

Then, I had taken the file ext-all.js alone and started to refresh it once and once to see how it behave. And true, the download was unstable. Sometimes it takes 40 to load successfully. Sometimes it fails after 7 secs (!!!).

I was trying to setup wireshark and check the packet sequence on failure but it seems like my connectivity become better to the moment and I ther eno more failures for dosens of try, so I gived up on that, but I seen a strange behaviour like SSL re-handshaking in the middle of the transfer, it sems like SSL session failed and new one was negotiated.

My conclusion is simple: pveproxy behaves strange if the IP connectivity is poor (big RTT) and sometimes it may lead to the situation when browser will not receive longest files. It may be a problem on either TCP, SSL or applicaton level. If the browser nearby, it loads for about 500ms. It probably cold be reproducable in lab by setting delay to 350ms between pveproxy host and browser.

I dare to blame pveproxy on probably wrong behavior just becase has other web apps at the same conditons and even behind of nginx proxy, and neither of that demonstrate same behaviour. Other app loads 200-300kb js scripts within 3s with great stability under the same network condition (located side-to-side) and I beleive nginx has much better connection/SSL handling.

BTW, it could be a solution to put nginx as a proxy, so there will be a very short leg between 'nginx as client' and pveproxy and this leg should work well. In other hand, nginx has proven performans under any condition and able to ise http2 for even better.

Please, let me know if it helps and feel free to ask.

UPD: probably, pveproxy reacts to packet lost (drop) not to clear delay. Becase my RTT is always over 300ms, and it works fine most of time, bt sometimes it start to behave like described. Packed drop is the most probable reason as it requires retransmission and then creates even more delay and may trigger some protocol timeouts which was not tuned to the besto for this scenarion.

 

[t.lamprecht]

My conclusion is simple: pveproxy behaves strange if the IP connectivity is poor (big RTT) and sometimes it may lead to the situation when browser will not receive longest files. It may be a problem on either TCP, SSL or applicaton level. If the browser nearby, it loads for about 500ms. It probably cold be reproducable in lab by setting delay to 350ms between pveproxy host and browser.

FWIW, I occasionally travel by train and use Proxmox VE's interface between, and while the internet access is pretty great nowadays, it drops to barely useable when passing through some regions (cough Deutsches Eck transport link cough) with very spiky latencies and package drops, and I also spent some time explicitly doing some site load tests there, i.e., to see if there are some low-hanging fruits where we can improve on, but in general, besides needing quite a bit of time to fully load it worked OK, with cache enabled or disabled. I also worked remote from my hometown, and there I had a mediocre ADSL link and OpenVPN in between just a few years ago, having also 300+ ms latency; worked ok there too, never had any issue loading the web interface of a Proxmox VE server from our headquarters.

And now that I think more about it, we also had other devs doing explicit high-latency and package loss tests, while those were mostly for improving the backup performance from PBS on such high-latency tests, it always included loading the web interface too. Also with the servers hosting our CDN, those are spread over the whole world and using the PVE web UI works fine on any of them, even if on the other side of the globe, so I doubt that it's a general thing...

BTW, it could be a solution to put nginx as a proxy, so there will be a very short leg between 'nginx as client' and pveproxy and this leg should work well.

Yeah, would be great if you could try that, if still doesn't work check your network, if it works flawlessly then it might be indeed a component used by pveproxy, maybe anyevent based HTTP server, possibly (rather likely) also limited to some more specific characteristics of your setup than just that slightly higher latency, because as mentioned above, our devs and admins here use PVE in such higher latency setups relatively often ourselves without issues..

In other hand, nginx has proven performans under any condition and able to ise http2 for even better.

http2 won't help you if you have high latency or package loss as it is still affected by head of line blocking.
http3 (QUIC) would improve on that, but got only stabilized relatively recently, but we plan to at least evaluate it in the mid/long term, albeit our motivation is mostly for PBS and its backup protocol (where we already use http2 currently), as there the benefits of http2/3 are relevant, HTTP 1.1 is far good enough by a wide margin for simple files and API calls like most of what the PVE web UI does.

UPD: probably, pveproxy reacts to packet lost (drop) not to clear delay. Becase my RTT is always over 300ms, and it works fine most of time, bt sometimes it start to behave like described. Packed drop is the most probable reason as it requires retransmission and then creates even more delay and may trigger some protocol timeouts which was not tuned to the besto for this scenarion.

pveproxy uses HTTP 1.1 which uses TCP which can cope with (quite) some package loss, and as mentioned above that was also part of our testing in the past and at least I are exposed to such environments in my personal real-life use case every few months due to travel.

Nonetheless, many thanks for your detailed write-up and that you spent some time investigating this.
And we'd naturally appreciate if you got any pointer for what environment defect besides some package loss and slightly higher latency you observe when this triggers, so that we can try to reproduce it and make Proxmox VE's API more resilient, if it's fixable at that level, that is.