HTTP 501 when using web reverse-proxy (Cloudflare Argo Tunnel) to start VM - but stopping VM works?

victorhooi

Well-Known Member
Apr 3, 2018
250
20
58
38
Hi,

I'm using Cloudflare Argo Tunnel to provide a reverse web-proxy in front of the Proxmox UI. (We are using this to provide integration with our SSO).

I've setup the tunnel, and I can load the Proxmo UI, but certain actions seem to fail - for example, starting VMs returns a HTTP 501:

gN4hT7R.png


This is the HTTP response when using the tunnel:

5NT2GaR.png

And in a working situation (actually using ZeroTier here):
1dDUO9h.png

Strangely enough - stopping a VM works.

Any ideas what's going on?

Is this some bug/glitch in the web UI, or something else?

Thanks,
Victor
 
does the proxy forward the header for the csrf prevention token?
 
Hmm, I checked a working and non-working request - the csrfpreventiontoken is present in both.

Failed request:

wNB75Td.png


Code:
:authority: syd1.example.dev
:method: POST
:path: /api2/extjs/nodes/syd1/qemu/101/status/start
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-GB,en-US;q=0.9,en;q=0.8
content-length: 0
cookie: __cfduid=d615b6082599a758d09528756d5ca1f0c1553110670; CF_Authorization=eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ2YTU0ZmMzNTBkYzA3OTFmODA0OWY0MmIwMGY3NDY2MGVlOThjNjg3ZTRiOWEyYTBlZDQ0NGM4YzRjNWNlYTUiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOlsiMDM5Mjk2ZjIwODA2M2U4OTk1MDgwMmNiNzA3ODVmZGY4YmMwODMyMmU4NDBjZDIzNWU5Y2ViNzI4MDQ5ZmVkZiJdLCJlbWFpbCI6InZpY3Rvcmhvb2lAYW5ndXNsYWIuaW8iLCJleHAiOjE1NTMxOTk1MDIsImlhdCI6MTU1MzExMzEwNCwiaXNzIjoiaHR0cHM6Ly9hbmd1c2xhYi5jbG91ZGZsYXJlYWNjZXNzLmNvbSIsIm5vbmNlIjoiYTViMjU1OWI5MmI1NzdkMjA2MGZhMTMwMzkxOGIxOWU1MDY5NjVlNzI0YmY5MTViYzNlMTNlMzcxYzVjZWE1NiIsInN1YiI6IjNjNzljNmQ5LTk0N2MtNDViYy1iZTBjLTc5NWQ3OWM0Y2UxYSJ9.MK8aPYD4B5UBcV8NSAGKwJXSkljiqyRzJPEOS_fTBaZUKKvwPKGzXeHOFyz3_oXpPjfnE2_GO5wGMLAOSsN5pjbVR_6gOZk_D5u-Js6U8rEL2wGLg8zmomhOM4lkThV2Xv2SCc0J6d17Byd3Uk4eZiMowp9ECOKVHhmpeIKVySYOZNirapuOkJWD_Rzimm2G_rnvmFQpj2yegdPHXDEpgExBFrYjox6wvrbw2x-R_wiM3sO-ovR4u0jtaWhmxTJg1VafSdqLVTcbShwtXLjo_bQEF8SnE_41zcshugV1xFADx2cgCdrdbEq8eerqjuBTRY_vDmem_FlrFsxWr6gZNg; PVEAuthCookie=PVE%3Aroot@pam%3A5C93E417%3A%3APoMCqft0qtqyhWnX211FgQkb7HWROiRsKfPMcfAgKN882u1eP4LAaIYXe/yJXkcjinphWT9bqd6j5DPHijosLVUOcGWXVa3gIoYWNxHYS7ksYwR0VGjwSWSa4mwTzVGr1drb3m1V4ZjaVqjpIFHJpf/l6nlu/I8rWKemJR9AOsBsANp3UVJt+c14RDOUtd8FQAFgqYIWUBGwic4r3mOmFEJT4S/A/rFw5LwaVnA5RPgiNxUqh5XIKTiUlTRPciymTKpF1i/t8OV8+7QX0vhIwigV7EsAm303fG3hg8L0hhhDSncQa1e1rxsoeRraTKubEOBzT3Vl7BmcMB0eE40GKA%3D%3D
csrfpreventiontoken: 5C93E417:2ZEqsOQEBCBTJGLGr/DdB7k1t74
origin: https://syd1.example.dev
referer: https://syd1.example.dev/
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36
x-requested-with: XMLHttpRequest

Working request:

AbQHiAe.png


Code:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive
Content-Length: 0
Cookie: PVEAuthCookie=PVE%3Aroot@pam%3A5C93E673%3A%3AItWW+ViucXEG9Df9LoRrqoYoiASymmdT6QgqTM2/UE6lMAYTw+/lYkKWIER3s1cVh7r10gN7JZFky4P2Hswi/sx5KNFpCuOPAiZbAnTNnbEMstGbv6Yjo/NOSEIws7LTBrTAqX3KkC2Dn7svUpbg1c52aWZLuS6p/hlaMleMISRGHVtSV/JDqKPYbbwY9PyZE8NpOh+JRWY9utErNXrySFfSQVEHlXmIkooHzQz4mdpuFote7tz5MtL0SSM9lfBevR4p3gyl//qtTQDeXxzWAXL6mg5oOtixlGtwGwrGINkyNkbGf0az4JlEzsi3zSIUuCSxyMR6hSnoDMKwgbAcXw%3D%3D
CSRFPreventionToken: 5C93E673:QqiwUUr7zCMJYNguXtjUFTzOtYc
Host: 10.144.125.63:8006
Origin: https://10.144.125.63:8006
Referer: https://10.144.125.63:8006/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36
X-Requested-With: XMLHttpRequest

Does that tell us anything?

What could be causing Proxmox to think the server is offline for this type of requests?
 
Last edited:
can you post the complete response of a failed api call?
 
Hi,

The response is empty, here is the response headers:

Code:
cache-control: max-age=0
cf-ray: 4bc2ad40d8155203-MEL
content-length: 0
date: Sat, 23 Mar 2019 19:05:25 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expires: Sat, 23 Mar 2019 19:05:25 GMT
pragma: no-cache
server: cloudflare
status: 501

I've also attached a complete HAR file as well, and made a gist:

https://gist.github.com/victorhooi/0b5c55fc799aa09fc92c576f85c66774

Thanks,
Victor
 

Attachments

  • cloudflare_failed_start_request.har.txt.zip
    313.8 KB · Views: 1
i would try to get the original answer from the node, because it seems that the proxy filters the error response somehow
 
Hmm - do you know of any way to get the original answer from the node?

Or is this something I need to get from the proxy itself?
 
I checked /var/log/pveproxy/access.log but it didn't reveal much. Is there a way to reveal more verbose information on the HTTP server side?
 
Is there a way to reveal more verbose information on the HTTP server side?
not really (you can start pveproxy/pvedaemon in debug mode, but i don't think that its useful for this case)

Hmm - do you know of any way to get the original answer from the node?

Or is this something I need to get from the proxy itself?
yes, you probably have to get that from the proxy, or you could do a tcpdump locally on the node (the connection between pveproxy and pvedaemon is over http, so that should be easily readable)
 
Wait - tcpdump locally will show the traffic between pveproxy and pvedaemon? =)

How does that work? What command variant can I use there?
 
OK, I captured traffic on the Proxmox node as follow:

Code:
root@syd1:~# tcpdump -i lo -v -w proxmox_cloudflare_issue.pcap 'port 85'
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
^C142 packets captured
296 packets received by filter
0 packets dropped by kernel

I then logged in to the Proxmox node via CloudFlare Access, and stopped a VM, then attempted to start it. Stopping it appears to be successful, but starting it gave me the "Connection error - server offline?" message as before.

I've attached both the PCAP file from tcpdump, as well as a HAR capture from Chrome browser.

This is very odd - I see the POST request where I stop the VM instance:

ghbAw0l.png


There is a second HTTP 500 which says the machine isn't running:

c6yWzjH.png


However, there is no subsequent POST for when I try to start the machine?

I'm very confused now - as from the HAR, it does seem like something comes back from Cloudflare (although the response is empty - and Cloudlfare says that exactly as it came to them).

I then tried performing the same start VM action without the proxy, and going straight to the machine (via ZeroTier). You can see here that a POST is seen, and Proxmox works as expected, starting the machine:

xkko1fu.png


Is there any chance this could be a bug in the ExtJS web app itself?
 

Attachments

  • cloudflare_http_501_on_start_vm.har.zip
    79.1 KB · Views: 1
  • proxmox_cloudflare_issue.pcap.zip
    14.9 KB · Views: 1
Last edited:
2 ideas,

first the proxy does not relay the start post call and returns an empty 500
or it reaches pve, but the pveproxy does not relay the api call to the pvedaemon, this can be verified if you check /var/log/pveproxy/access.log and see if there is a corresponding start api call and its error code
additionally every start api call should generate a task in the task log, even if it fails (with the expection if something from the api is not right, but that would return an error like "parameter verification failed" for example)
 
OK, I did a tail -f on /var/lob/pveproxy/access/log, as I was hitting the "Start" button in the UI - I do see a 501 in the access log:
Code:
127.0.0.1 - - [29/03/2019:05:32:26 +1100] "POST /api2/extjs/nodes/syd1/qemu/102/status/start HTTP/1.1" 501 -
127.0.0.1 - - [29/03/2019:05:32:27 +1100] "GET /pve2/ext6/theme-crisp/resources/images/shared/icon-error.png HTTP/1.1" 200 18494
127.0.0.1 - root@pam [29/03/2019:05:32:27 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 732
127.0.0.1 - root@pam [29/03/2019:05:32:27 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:29 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:30 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2755
127.0.0.1 - root@pam [29/03/2019:05:32:30 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 725
127.0.0.1 - root@pam [29/03/2019:05:32:31 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:32 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:33 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 720
127.0.0.1 - root@pam [29/03/2019:05:32:33 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2740
127.0.0.1 - root@pam [29/03/2019:05:32:33 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:35 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:35 +1100] "GET /api2/json/nodes/syd1/qemu/102/agent/network-get-interfaces HTTP/1.1" 500 13
127.0.0.1 - root@pam [29/03/2019:05:32:36 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 716
127.0.0.1 - root@pam [29/03/2019:05:32:36 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2716
127.0.0.1 - root@pam [29/03/2019:05:32:37 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - - [29/03/2019:05:32:38 +1100] "POST /api2/extjs/nodes/syd1/qemu/102/status/start HTTP/1.1" 501 -
127.0.0.1 - root@pam [29/03/2019:05:32:38 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:39 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 751
127.0.0.1 - root@pam [29/03/2019:05:32:40 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:40 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2724
127.0.0.1 - root@pam [29/03/2019:05:32:42 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:43 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 749
127.0.0.1 - root@pam [29/03/2019:05:32:43 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:44 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2741
127.0.0.1 - - [29/03/2019:05:32:45 +1100] "POST /api2/extjs/nodes/syd1/qemu/102/status/start HTTP/1.1" 501 -
127.0.0.1 - root@pam [29/03/2019:05:32:45 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:46 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 733
127.0.0.1 - root@pam [29/03/2019:05:32:46 +1100] "GET /api2/json/nodes/syd1/qemu/102/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 200 2382
127.0.0.1 - root@pam [29/03/2019:05:32:46 +1100] "GET /api2/json/nodes/syd1/qemu/102/agent/network-get-interfaces HTTP/1.1" 500 13
127.0.0.1 - root@pam [29/03/2019:05:32:46 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:47 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2770
127.0.0.1 - root@pam [29/03/2019:05:32:48 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:49 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
127.0.0.1 - root@pam [29/03/2019:05:32:49 +1100] "GET /api2/json/cluster/resources HTTP/1.1" 200 730
127.0.0.1 - root@pam [29/03/2019:05:32:50 +1100] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2772
127.0.0.1 - root@pam [29/03/2019:05:32:51 +1100] "GET /api2/json/nodes/syd1/qemu/102/status/current HTTP/1.1" 200 277
In Chrome, you can see all the HTTP 501's as well for the POST calls to start:
yweo78W.png


In the Proxmox UI Task list, I do not see any failed calls though - the last action you see is the "VM 102 - Stop", when I stopped the VM before trying to start it several times:

HHGrFeF.png


Does the above mean it reached the pveproxy from the client, but didn't go further than that? The tcpdump I took is traffic between pveproxy and pvedaemon - and it wasn't present there. What does the fact it's missing from the UI show?

Also - is there any workaround to starting VMs via this proxy - or would all of those rely on the same exact API call itself working, and getting passed from pveproxy to pveademon to Proxmox?
 
mhmm... that would mean that the pveproxy throws some error on those api calls before handing it to the pvedaemon (thus no task log)
it would be very interesting to capture such an http request that goes from the proxy to the pveproxy, i am not sure if this can be done easily
on the pve side (tls etc.)... is there a possibility that you can do that on the proxy side?
 
The thing is - the issue only seems to occur when going via Cloudflare's Argo Tunnel. Yet the issue seems to be within Proxmox itself.

Are you thinking of using mitmproxy as a reverse proxy instead of Cloudflare?
 
Are you thinking of using mitmproxy as a reverse proxy instead of Cloudflare?
no i meant between cloudflare and pve, to see what exactly cloudflare sends to pve and what comes back to cloudflare
 
I ran cloudlfared with debug logging - I see this:

Code:
DEBU[0101] 501 chunked transfer encoding not supported   CF-RAY=4bf3856bd8795d96-BNE

Does that mean something? Is it possible pveproxy is choking on some "chunked transfer encoding" thing?

Full output around that request is:
Code:
DEBU[0101] POST https://localhost:8006/api2/extjs/nodes/syd1/qemu/102/status/start HTTP/1.1  CF-RAY=4bf3856bd8795d96-BNE
DEBU[0101] Request Headers map[Cookie:[__cfduid=d2880c87e3e3b1c1b8e4593210a79e7ba1552572954; CF_Authorization=eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ2YTU0ZmMzNTBkYzA3OTFmODA0OWY0MmIwMGY3NDY2MGVlOThjNjg3ZTRiOWEyYTBlZDQ0NGM4YzRjNWNlYTUiLCJ0eXAiOiJK
V1QifQ.eyJhdWQiOlsiYjEzYzlhMmM1YjhiM2M5NDc2NzhhOGQ5YmZkYjdmYzllZmQyMWI5MDI5MDkxNTA4Y2EwYTMzZmQyYTU4ZmE0NSJdLCJlbWFpbCI6InZpY3Rvcmhvb2lAYW5ndXNsYWIuaW8iLCJleHAiOjE1NTM5MTY5ODUsImlhdCI6MTU1MzgzMDU4NywiaXNzIjoiaHR0cHM6Ly9hbmd1c2xhYi5jbG91ZGZ
sYXJlYWNjZXNzLmNvbSIsIm5vbmNlIjoiYzBkNzIyMzk2MjI5ZmQ2YmQxOWNhMTQ5NzNhZjBhNGNjNDFkM2ZjN2QyZmNkMGU2ODc1YWJiYWIxNTM4OGRjNyIsInN1YiI6IjNjNzljNmQ5LTk0N2MtNDViYy1iZTBjLTc5NWQ3OWM0Y2UxYSJ9.fY1iQHS-mb3pu237fAAgwhFU6lfeEE3Rn4HhRE1URa-_I8hTBSQwIzcz
BJtrHZQYyLnaedYRvHUm-rSip55hTV73MByHd-WRcX5UQMR0CTeaCIxbEcvrdjdaZ7N4mg7xEc8a9OXaQxVYqAY3aQV72RTZaOunQ71gXUoALMHYufhw7bw61yfytyn_pTn5WyJK2_asq9KGoYqwW2meC60WBNoCb15ByGPGzlqdj-fpRNvFVFRy3BZlZPK7qI5qPfUZ8S5Pl1oKbPfxrQjgi7IGAKZtT-TLWKEgxRf-wm
QMqHrHw1xs9YZaEQ3OljIL6pHDpMXzgUCVarIE6rxQNzJSJQ; PVEAuthCookie=PVE%3Aroot@pam%3A5C9E5406%3A%3AnMO/9wIgUzno4z0FdHsR+stdGIcUQ9kllQI6noVDrhgykeqQnffTxY0qo0ccNJQD4bfxbXsWMC1jCe8zZNaa9ciO0viguBTYMLNtM/WgbNTzZQdJBP/5tJ86//dTUWUqMw8OX2/ktS2OZi8
TWJGt2I8SV+QzDk2j9ldkXUAy9YxctucPdCbE6hUJx2Qmz5zKzSgI5K8U0cFVanw4t/gLh5D21YLkaAOgkB9S3gdxRSpxd2jyWx7E9SGd1S18ZMyzNqc8FdSa2Rkg7pUXeKQzx2LJLqT2Xgw5Z00nXQkvteFfDdNZ+hMKKc30GSdfdzB5AACrqWXtVI/Me1Eb765zIA%3D%3D] User-Agent:[Mozilla/5.0 (Macint
osh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36] Csrfpreventiontoken:[5C9E5406:BPEeEhYkU4c+5CnTgQF6JR/ecMM] Cf-Access-Jwt-Assertion:[eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ2YTU0ZmMzNTBkYzA3OTF
mODA0OWY0MmIwMGY3NDY2MGVlOThjNjg3ZTRiOWEyYTBlZDQ0NGM4YzRjNWNlYTUiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOlsiYjEzYzlhMmM1YjhiM2M5NDc2NzhhOGQ5YmZkYjdmYzllZmQyMWI5MDI5MDkxNTA4Y2EwYTMzZmQyYTU4ZmE0NSJdLCJlbWFpbCI6InZpY3Rvcmhvb2lAYW5ndXNsYWIuaW8iLCJleHAiOj
E1NTM5MTY5ODUsImlhdCI6MTU1MzgzMDU4NywiaXNzIjoiaHR0cHM6Ly9hbmd1c2xhYi5jbG91ZGZsYXJlYWNjZXNzLmNvbSIsIm5vbmNlIjoiYzBkNzIyMzk2MjI5ZmQ2YmQxOWNhMTQ5NzNhZjBhNGNjNDFkM2ZjN2QyZmNkMGU2ODc1YWJiYWIxNTM4OGRjNyIsInN1YiI6IjNjNzljNmQ5LTk0N2MtNDViYy1iZTBj
LTc5NWQ3OWM0Y2UxYSJ9.fY1iQHS-mb3pu237fAAgwhFU6lfeEE3Rn4HhRE1URa-_I8hTBSQwIzczBJtrHZQYyLnaedYRvHUm-rSip55hTV73MByHd-WRcX5UQMR0CTeaCIxbEcvrdjdaZ7N4mg7xEc8a9OXaQxVYqAY3aQV72RTZaOunQ71gXUoALMHYufhw7bw61yfytyn_pTn5WyJK2_asq9KGoYqwW2meC60WBNoCb
15ByGPGzlqdj-fpRNvFVFRy3BZlZPK7qI5qPfUZ8S5Pl1oKbPfxrQjgi7IGAKZtT-TLWKEgxRf-wmQMqHrHw1xs9YZaEQ3OljIL6pHDpMXzgUCVarIE6rxQNzJSJQ] Cdn-Loop:[cloudflare] Accept-Encoding:[gzip] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8] X-Forwarded-For:[115.
70.36.138] Origin:[https://syd1.example.com] X-Requested-With:[XMLHttpRequest] Cf-Warp-Tag-Id:[8787216b53ebccb14b1a62ef01a1b52ea3bdfacdf8c72457d35c7e61af430640] Cf-Visitor:[{"scheme":"https"}] Cf-Ray:[4bf3856bd8795d96-BNE] Cf-Connecting-I
p:[115.70.36.138] Cf-Access-Authenticated-User-Email:[victorhooi@example.com] X-Forwarded-Proto:[https] Referer:[https://syd1.example.com/] Accept:[*/*] Cf-Ipcountry:[AU]]  CF-RAY=4bf3856bd8795d96-BNE
DEBU[0101] Request content length 0                      CF-RAY=4bf3856bd8795d96-BNE
DEBU[0101] 501 chunked transfer encoding not supported   CF-RAY=4bf3856bd8795d96-BNE
DEBU[0101] Response Headers map[Date:[Fri, 29 Mar 2019 17:21:36 GMT] Pragma:[no-cache] Server:[pve-api-daemon/3.0] Expires:[Fri, 29 Mar 2019 17:21:36 GMT] Cache-Control:[max-age=0]]  CF-RAY=4bf3856bd8795d96-BNE
DEBU[0101] Response content length unknown               CF-RAY=4bf3856bd8795d96-BNE
DEBU[0101] GET https://localhost:8006/pve2/ext6/theme-crisp/resources/images/shared/icon-error.png HTTP/1.1  CF-RAY=4bf3856c789b5d96-SYD
DEBU[0101] Request Headers map[Cf-Warp-Tag-Id:[8787216b53ebccb14b1a62ef01a1b52ea3bdfacdf8c72457d35c7e61af430640] Cf-Connecting-Ip:[115.70.36.138] Cf-Ray:[4bf3856c789b5d96-SYD] Cf-Visitor:[{"scheme":"https"}] Cf-Access-Jwt-Assertion:[eyJhb
GciOiJSUzI1NiIsImtpZCI6IjQ2YTU0ZmMzNTBkYzA3OTFmODA0OWY0MmIwMGY3NDY2MGVlOThjNjg3ZTRiOWEyYTBlZDQ0NGM4YzRjNWNlYTUiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOlsiYjEzYzlhMmM1YjhiM2M5NDc2NzhhOGQ5YmZkYjdmYzllZmQyMWI5MDI5MDkxNTA4Y2EwYTMzZmQyYTU4ZmE0NSJdLCJlbWFp
bCI6InZpY3Rvcmhvb2lAYW5ndXNsYWIuaW8iLCJleHAiOjE1NTM5MTY5ODUsImlhdCI6MTU1MzgzMDU4NywiaXNzIjoiaHR0cHM6Ly9hbmd1c2xhYi5jbG91ZGZsYXJlYWNjZXNzLmNvbSIsIm5vbmNlIjoiYzBkNzIyMzk2MjI5ZmQ2YmQxOWNhMTQ5NzNhZjBhNGNjNDFkM2ZjN2QyZmNkMGU2ODc1YWJiYWIxNTM4OG
RjNyIsInN1YiI6IjNjNzljNmQ5LTk0N2MtNDViYy1iZTBjLTc5NWQ3OWM0Y2UxYSJ9.fY1iQHS-mb3pu237fAAgwhFU6lfeEE3Rn4HhRE1URa-_I8hTBSQwIzczBJtrHZQYyLnaedYRvHUm-rSip55hTV73MByHd-WRcX5UQMR0CTeaCIxbEcvrdjdaZ7N4mg7xEc8a9OXaQxVYqAY3aQV72RTZaOunQ71gXUoALMHYufh
w7bw61yfytyn_pTn5WyJK2_asq9KGoYqwW2meC60WBNoCb15ByGPGzlqdj-fpRNvFVFRy3BZlZPK7qI5qPfUZ8S5Pl1oKbPfxrQjgi7IGAKZtT-TLWKEgxRf-wmQMqHrHw1xs9YZaEQ3OljIL6pHDpMXzgUCVarIE6rxQNzJSJQ] X-Forwarded-Proto:[https] Referer:[https://syd1.example.com/pve2/
ext6/theme-crisp/resources/theme-crisp-all_1.css] If-Modified-Since:[Wed, 25 Jan 2017 17:32:08 GMT] Accept-Encoding:[gzip] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683
.86 Safari/537.36] X-Forwarded-For:[115.70.36.138] Cf-Access-Authenticated-User-Email:[victorhooi@example.com] Cf-Ipcountry:[AU] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8] Accept:[image/webp,image/apng,image/*,*/*;q=0.8] Cookie:[__cfdui
d=d2880c87e3e3b1c1b8e4593210a79e7ba1552572954; CF_Authorization=eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ2YTU0ZmMzNTBkYzA3OTFmODA0OWY0MmIwMGY3NDY2MGVlOThjNjg3ZTRiOWEyYTBlZDQ0NGM4YzRjNWNlYTUiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOlsiYjEzYzlhMmM1YjhiM2M5NDc2Nzh
hOGQ5YmZkYjdmYzllZmQyMWI5MDI5MDkxNTA4Y2EwYTMzZmQyYTU4ZmE0NSJdLCJlbWFpbCI6InZpY3Rvcmhvb2lAYW5ndXNsYWIuaW8iLCJleHAiOjE1NTM5MTY5ODUsImlhdCI6MTU1MzgzMDU4NywiaXNzIjoiaHR0cHM6Ly9hbmd1c2xhYi5jbG91ZGZsYXJlYWNjZXNzLmNvbSIsIm5vbmNlIjoiYzBkNzIyMzk2M
jI5ZmQ2YmQxOWNhMTQ5NzNhZjBhNGNjNDFkM2ZjN2QyZmNkMGU2ODc1YWJiYWIxNTM4OGRjNyIsInN1YiI6IjNjNzljNmQ5LTk0N2MtNDViYy1iZTBjLTc5NWQ3OWM0Y2UxYSJ9.fY1iQHS-mb3pu237fAAgwhFU6lfeEE3Rn4HhRE1URa-_I8hTBSQwIzczBJtrHZQYyLnaedYRvHUm-rSip55hTV73MByHd-WRcX5UQM
R0CTeaCIxbEcvrdjdaZ7N4mg7xEc8a9OXaQxVYqAY3aQV72RTZaOunQ71gXUoALMHYufhw7bw61yfytyn_pTn5WyJK2_asq9KGoYqwW2meC60WBNoCb15ByGPGzlqdj-fpRNvFVFRy3BZlZPK7qI5qPfUZ8S5Pl1oKbPfxrQjgi7IGAKZtT-TLWKEgxRf-wmQMqHrHw1xs9YZaEQ3OljIL6pHDpMXzgUCVarIE6rxQNzJS
JQ; PVEAuthCookie=PVE%3Aroot@pam%3A5C9E5406%3A%3AnMO/9wIgUzno4z0FdHsR+stdGIcUQ9kllQI6noVDrhgykeqQnffTxY0qo0ccNJQD4bfxbXsWMC1jCe8zZNaa9ciO0viguBTYMLNtM/WgbNTzZQdJBP/5tJ86//dTUWUqMw8OX2/ktS2OZi8TWJGt2I8SV+QzDk2j9ldkXUAy9YxctucPdCbE6hUJx2Qmz
5zKzSgI5K8U0cFVanw4t/gLh5D21YLkaAOgkB9S3gdxRSpxd2jyWx7E9SGd1S18ZMyzNqc8FdSa2Rkg7pUXeKQzx2LJLqT2Xgw5Z00nXQkvteFfDdNZ+hMKKc30GSdfdzB5AACrqWXtVI/Me1Eb765zIA%3D%3D] Cdn-Loop:[cloudflare]]  CF-RAY=4bf3856c789b5d96-SYD
 
Last edited:
yes, our http daemons cannot handle chunked transfer encoding, but why does the cloudflare proxy do this?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!