Proxmox login returns HTTP 401 with valid password

victorhooi

Active Member
Apr 3, 2018
250
20
38
37
I've setup a new Proxmox 6.0 cluster with three nodes. Version info is here:
Code:
root@example-vm01:/var/log/pveproxy# pveversion
pve-manager/6.0-5/f8a710d7 (running kernel: 5.0.18-1-pve)
I'm using Cloudflared as a proxy to provide SSO in front of Proxmox. This was previously working on a separate cluster running Proxmox 5.4. However, on this cluster, when I try to login to Proxmox, I get an error message:
p7lFUSx.png

Code:
Connection error 401: Unauthorized
I also tail-ed /var/log/pveproxy/access.log, and I see this:
Code:
127.0.0.1 - - [15/08/2019:01:19:42 -0700] "POST /api2/extjs/access/ticket HTTP/1.1" 200 690
127.0.0.1 - - [15/08/2019:01:19:42 -0700] "GET /api2/extjs/version?_dc=1565857182452 HTTP/1.1" 401 -
127.0.0.1 - - [15/08/2019:01:19:42 -0700] "GET /api2/extjs/nodes/localhost/subscription?_dc=1565857182453 HTTP/1.1" 401 -
If I check Chrome Dev tools, I see the following two HTTP calls that return 401:

fOKz90Y.png

Code:
http://mtv1.example.com/api2/extjs/version?_dc=1565806584792
and
HbyJtGi.png

Code:
http://mtv1.example.com/api2/extjs/nodes/localhost/subscription?_dc=1565806584793

Any idea what's going on here, or why we're suddenly getting this HTTP 401?

Thanks,
Victor
 
Last edited:
Also - this is the cloudflared (HTTPS proxy) logs at the same time:

Code:
{"CF-RAY":"5069cb9deae8cec8-LAX","level":"debug","msg":"POST https://localhost:8006/api2/extjs/access/ticket HTTP/1.1","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9deae8cec8-LAX","level":"debug","msg":"Request Headers map[Accept:[*/*] Accept-Encoding:[gzip] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8] Cdn-Loop:[cloudflare] Cf-Cloudflared-Proxy-Tunnel-Hostname:[yy20xkixzs1773bzlpdtu3b8s5utvu821kyyan16yqkky062q8sg.cftunnel.com] Cf-Connecting-Ip:[172.85.43.201] Cf-Ipcountry:[US] Cf-Ray:[5069cb9deae8cec8-LAX] Cf-Trace-Id:[5988385e694e202:5988385e694e202:0:1:5d5517ad:544c964820d5725121827d4ab4a75233b7dcbd82d54ce6b4516654fc4755630a] Cf-Visitor:[{\"scheme\":\"http\"}] Cf-Warp-Tag-Id:[c2cec706-069f-4de4-8444-9182d9862203] Content-Type:[application/x-www-form-urlencoded; charset=UTF-8] Cookie:[__cfduid=da53e32d25be03d8282930a737dff619b1563309006] Csrfpreventiontoken:[null] Origin:[http://mtv1.example.com] Referer:[http://mtv1.example.com/] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3882.0 Safari/537.36] X-Forwarded-For:[172.85.43.201] X-Forwarded-Proto:[http] X-Requested-With:[XMLHttpRequest]]","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9deae8cec8-LAX","level":"debug","msg":"Request content length 41","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9deae8cec8-LAX","level":"debug","msg":"200 OK","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9deae8cec8-LAX","level":"debug","msg":"Response Headers map[Cache-Control:[max-age=0] Content-Encoding:[gzip] Content-Length:[684] Content-Type:[application/json;charset=UTF-8] Date:[Thu, 15 Aug 2019 08:28:29 GMT] Expires:[Thu, 15 Aug 2019 08:28:29 GMT] Pragma:[no-cache] Server:[pve-api-daemon/3.0]]","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9deae8cec8-LAX","level":"debug","msg":"Response content length 684","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edc41cec8-LAX","level":"debug","msg":"GET https://localhost:8006/api2/extjs/version?_dc=1565857709886 HTTP/1.1","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edc41cec8-LAX","level":"debug","msg":"Request Headers map[Accept:[*/*] Accept-Encoding:[gzip] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8] Cdn-Loop:[cloudflare] Cf-Cloudflared-Proxy-Tunnel-Hostname:[yy20xkixzs1773bzlpdtu3b8s5utvu821kyyan16yqkky062q8sg.cftunnel.com] Cf-Connecting-Ip:[172.85.43.201] Cf-Ipcountry:[US] Cf-Ray:[5069cb9edc41cec8-LAX] Cf-Trace-Id:[7f59cbfea14b5d65:7f59cbfea14b5d65:0:1:5d5517ad:39ba53c4493c69cd49b6fc1b658ff22a31e30f394cd61ef8cd2f1cbd7611407d] Cf-Visitor:[{\"scheme\":\"http\"}] Cf-Warp-Tag-Id:[c2cec706-069f-4de4-8444-9182d9862203] Cookie:[__cfduid=da53e32d25be03d8282930a737dff619b1563309006] Csrfpreventiontoken:[5D5517AD:oJ3eg5UDpypgZXMqVmhkwEh9IvtASVdf3qKmNlaB5RQ] Referer:[http://mtv1.example.com/] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3882.0 Safari/537.36] X-Forwarded-For:[172.85.43.201] X-Forwarded-Proto:[http] X-Requested-With:[XMLHttpRequest]]","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edc41cec8-LAX","level":"debug","msg":"Request content length 0","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edf83cf34-LAX","level":"debug","msg":"GET https://localhost:8006/api2/extjs/nodes/localhost/subscription?_dc=1565857709886 HTTP/1.1","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edf83cf34-LAX","level":"debug","msg":"Request Headers map[Accept:[*/*] Accept-Encoding:[gzip] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8] Cdn-Loop:[cloudflare] Cf-Cloudflared-Proxy-Tunnel-Hostname:[yy20xkixzs1773bzlpdtu3b8s5utvu821kyyan16yqkky062q8sg.cftunnel.com] Cf-Connecting-Ip:[172.85.43.201] Cf-Ipcountry:[US] Cf-Ray:[5069cb9edf83cf34-LAX] Cf-Trace-Id:[66726c8f9132a101:66726c8f9132a101:0:1:5d5517ad:9b009726e9c581f0292408d86f3c54f819cd8d99124aea0725364b8586c5b6d7] Cf-Visitor:[{\"scheme\":\"http\"}] Cf-Warp-Tag-Id:[c2cec706-069f-4de4-8444-9182d9862203] Cookie:[__cfduid=da53e32d25be03d8282930a737dff619b1563309006] Csrfpreventiontoken:[5D5517AD:oJ3eg5UDpypgZXMqVmhkwEh9IvtASVdf3qKmNlaB5RQ] Referer:[http://mtv1.example.com/] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3882.0 Safari/537.36] X-Forwarded-For:[172.85.43.201] X-Forwarded-Proto:[http] X-Requested-With:[XMLHttpRequest]]","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edf83cf34-LAX","level":"debug","msg":"Request content length 0","time":"2019-08-15T01:28:29-07:00"}
{"CF-RAY":"5069cb9edc41cec8-LAX","level":"debug","msg":"401 No ticket","time":"2019-08-15T01:28:32-07:00"}
{"CF-RAY":"5069cb9edc41cec8-LAX","level":"debug","msg":"Response Headers map[Cache-Control:[max-age=0] Date:[Thu, 15 Aug 2019 08:28:29 GMT] Expires:[Thu, 15 Aug 2019 08:28:29 GMT] Pragma:[no-cache] Server:[pve-api-daemon/3.0]]","time":"2019-08-15T01:28:32-07:00"}
{"CF-RAY":"5069cb9edc41cec8-LAX","level":"debug","msg":"Response content length unknown","time":"2019-08-15T01:28:32-07:00"}
{"CF-RAY":"5069cb9edf83cf34-LAX","level":"debug","msg":"401 No ticket","time":"2019-08-15T01:28:32-07:00"}
{"CF-RAY":"5069cb9edf83cf34-LAX","level":"debug","msg":"Response Headers map[Cache-Control:[max-age=0] Date:[Thu, 15 Aug 2019 08:28:29 GMT] Expires:[Thu, 15 Aug 2019 08:28:29 GMT] Pragma:[no-cache] Server:[pve-api-daemon/3.0]]","time":"2019-08-15T01:28:32-07:00"}
{"CF-RAY":"5069cb9edf83cf34-LAX","level":"debug","msg":"Response content length unknown","time":"2019-08-15T01:28:32-07:00"}
The access.log when coming in via SSH tunnel (which seems to work):
Code:
10.7.12.1 - - [15/08/2019:01:19:01 -0700] "POST /api2/extjs/access/ticket HTTP/1.1" 200 685
10.7.12.1 - - [15/08/2019:01:19:01 -0700] "GET /pve2/ext6/theme-crisp/resources/images/loadmask/loading.gif HTTP/1.1" 200 1849
10.7.12.1 - root@pam [15/08/2019:01:19:01 -0700] "GET /api2/extjs/version?_dc=1565857141693 HTTP/1.1" 200 95
10.7.12.1 - root@pam [15/08/2019:01:19:01 -0700] "GET /api2/extjs/nodes/localhost/subscription?_dc=1565857141694 HTTP/1.1" 200 180
10.7.12.1 - root@pam [15/08/2019:01:19:01 -0700] "GET /api2/json/cluster/resources HTTP/1.1" 200 587
10.7.12.1 - root@pam [15/08/2019:01:19:02 -0700] "GET /api2/json/cluster/tasks HTTP/1.1" 200 2224
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/tools/tool-sprites.png HTTP/1.1" 200 24404
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/shared/icon-warning.png HTTP/1.1" 200 17717
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/toolbar/default-scroll-left.png HTTP/1.1" 200 17863
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/toolbar/default-scroll-right.png HTTP/1.1" 200 18300
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/toolbar/default-scroll-top.png HTTP/1.1" 200 18260
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/images/logo-ceph.png HTTP/1.1" 200 488
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/toolbar/default-scroll-bottom.png HTTP/1.1" 200 18266
10.7.12.1 - - [15/08/2019:01:19:02 -0700] "GET /pve2/ext6/theme-crisp/resources/images/grid/sort_asc.png HTTP/1.1" 200 18239
Compared to via cloudflared daemon:
Code:
127.0.0.1 - - [15/08/2019:01:19:42 -0700] "POST /api2/extjs/access/ticket HTTP/1.1" 200 690
127.0.0.1 - - [15/08/2019:01:19:42 -0700] "GET /api2/extjs/version?_dc=1565857182452 HTTP/1.1" 401 -
127.0.0.1 - - [15/08/2019:01:19:42 -0700] "GET /api2/extjs/nodes/localhost/subscription?_dc=1565857182453 HTTP/1.1" 401 -
 
Last edited:
Instead of Cloudflare access, I also tried with Google IAP as well.

That simply proxies the connections from a load-balancer sitting within GCP.

When I do that, I get an error:
09bhOtp.png

Code:
Connection error 504: Gateway Timeout
In the access.log file, I still see a HTTP 401:
Code:
34.83.155.61 - - [15/08/2019:07:58:45 -0700] "GET / HTTP/1.1" 200 797
34.83.109.226 - - [15/08/2019:07:58:46 -0700] "GET /api2/json/access/domains HTTP/1.1" 200 159
34.83.155.61 - - [15/08/2019:07:58:53 -0700] "POST /api2/extjs/access/ticket HTTP/1.1" 200 684
34.83.155.61 - - [15/08/2019:07:58:53 -0700] "GET /api2/extjs/version?_dc=1565881133874 HTTP/1.1" 401 -
34.83.62.136 - - [15/08/2019:07:58:53 -0700] "GET /api2/extjs/nodes/localhost/subscription?_dc=1565881133874 HTTP/1.1" 401 -
34.83.155.61 - - [15/08/2019:07:58:57 -0700] "GET /pve2/ext6/theme-crisp/resources/images/tools/tool-sprites.png HTTP/1.1" 200 24404
Is there just something funny with Proxmox 6.0 and reverse proxies all of a sudden? Did some behaviour change that breaks them?
 
Also - this is the cloudflared (HTTPS proxy) logs at the same time:
i dont see any pveauthcookies in your requests... are you sure that the proxy relays the cookies ?
 
@dcsapak - Hmm, are you saying that the proxy needs to do something special to pass a cookie on? Or could the cookie somehow be tied to a specific IP address or machine?

(Is there any chance this cookie behaviour has changed in Proxmox 5.4 vs Proxmox 6.0)?

What could I try to diagnose this further?

(I did also purchase our first subscription last week - because of how friendly the staff are here - and I'm hoping to get it for all our machines - but the case seems to currently be stalled there, hoping it gets looked at again soon?)
 
Hmm, are you saying that the proxy needs to do something special to pass a cookie on? Or could the cookie somehow be tied to a specific IP address or machine?
no i am saying the machine making the request has to have set the 'PVEAuthCookie', normally the user interface does this on a api request of /access/ticket and it depends on the config of the proxy
if that is used for the backend connection (should be, but since i have no experience with both the options you tried, i cannot really say if you have to configure something special)

(Is there any chance this cookie behaviour has changed in Proxmox 5.4 vs Proxmox 6.0)?
no, i am not aware of any changes regarding cookie handling on our side, the only change we made is that we use a different hash algorithm for the csrfpreventiontoken, but i can see that in your request, so
that should work fine...

I did also purchase our first subscription last week - because of how friendly the staff are here - and I'm hoping to get it for all our machines -
great :)

but the case seems to currently be stalled there, hoping it gets looked at again soon?)
i will get looked at it for sure, but i think it would be better to concentrate on a single communications channel (so either forum or ticket, your choice)
 
I'm happy to continue communication here - if you are still willing to help me? =) That way hopefully other people can benefit from the knowledge as well!

(I would also love to help improve the Proxmox docs as well, but that's another story).

Got it. So the browser client should be the one setting the cookie.

The strange thing is - our test system is running Proxmox 5.4. When we use this system, it sets the cookie correctly.

The new system is running Proxmox 6.0 - and this does not seem to set the cookie (which I assume explains the HTTP 401 errors).

I've taken the proxy logfiles (cloudflared.log), as well as the pveproxy access.log files, as well as HTTP network captures from both servers, to try to compare. All of them are at https://gist.github.com/victorhooi/0df3ea729e4fc9fa1d900cfc71130b8b.

UPDATE

Woot - I think I figured it out!

For some reason (possibly due to Chrome Omnibox autocomplete, and me typing it wrong once before...lol), I was requesting HTTPS (port 443) for the working Proxmox 5.4 system, and HTTP (port 80) for the non-working Proxmox 6.0 system.

The /access/ticket request still went through for both, and the cloudflared proxy passed it on fine - but for some reason Proxmox then didn't set the cookie when going to the proxy over HTTP?

Is that expected behaviour from Proxmox?
 

Attachments

  • syd1-cloudflared.log
    932.1 KB · Views: 1
  • mtv1-cloudlfared.log
    63 KB · Views: 0
  • mtv1-access.log
    89 KB · Views: 0
  • syd1-access.log
    26.5 KB · Views: 0
EDIT** Corrected my response because I realize the fallacies therein! Edits are denoted either with strikethrough text followed by the new information, if any, in italics.

@LnxBil - Thank you for your reply, your reasoning is correct and the first thing I'll be trying is your suggestion about the hostname.

@LnxBil said:


I realized these errors the next day when my access stopped working through my public hostname, and embarked down a dark journey of trying to bind mount a config.yml to the container. Then decided to trash the docker runtime tunnel config and see if I could gain greater control over the backend with the cloudflared-cli installation on fedora server 38 x86_64 for the tunnel - not the proxmox host. I finally just now was able to repair my firewalld zones and get docker back to its original state.


Original reply:

I stumbled upon this trying to figure out the same thing with cloudflare today.

Running a cloudflared in the lightweight LXD on a separate host from my proxmox server at my home network and have yet to get this to work.

The config for Cloudflare public hostname - per the dashboard - needs to be set to `https://` and under the ’Additional Application Settings’ needs to have the path to the ca.pem certification on the proxmox host `/etc/pve/pve-root-ca.pem` listed in `TLS > Certificate Authority Pool`. Also, the `No TLS Verify` option needs to be turned off.

I can confirm that this setup did work for whatever reason, but the following day - as expected - I was left without remote access

Now that I have my server (not proxmox) back to working order (I think), I can now continue on my quest to try to get this working with Cloudflare tunnels.
 
Last edited:
the path to the ca.pem certification
That sounds strange. Those two settings are normally mutually exclusively working. If you set the CA, you don't need the noverify, because the CA is there exactly for the verification of the certificate. In addition the hostname of the target and the hostname in the signed-by-the-CA certificate must also match, so this can be an additional hurdle.

So setting noverify will overwrite the CA setting and ignore if completely, because it does not make any sense to have both. The CA will not beread. This setting is the general knob to turn of any verification.
 
  • Like
Reactions: jvince11
In addition the hostname of the target and the hostname in the signed-by-the-CA certificate must also match, so this can be an additional hurdle.
Do you happen to know which document explains this about the hostname of the host (I'm assuming by host, you mean the proxmox machine.

When you say "hostname of the target" are you referring to the hostname as it has been set on the proxmox machine, or do you mean the "public hostname" as part of the configuration?
 
Do you happen to know which document explains this about the hostname of the host (I'm assuming by host, you mean the proxmox machine.

When you say "hostname of the target" are you referring to the hostname as it has been set on the proxmox machine, or do you mean the "public hostname" as part of the configuration?
I mean the hostname, that is written in the certificate must match the hostname that you want to access.

Here an example from proxmox.com:

Code:
$ openssl s_client -showcerts -connect www.proxmox.com:443 -servername www.proxmox.com < /dev/null 2>&1 | \
  openssl x509 -noout -text -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_issuer,no_pubkey,no_sigdump,no_aux | \
  grep -i DNS
 
                DNS:newsletter.proxmox.com, DNS:proxmox.com, DNS:www.proxmox.com

Curl will work with all of those DNS names with this certificate, yet if you use the IP, it'll not work:

Code:
$ curl -s https://proxmox.com/
$ echo $?
0

$ curl https://212.224.123.69/
curl: (60) SSL: no alternative certificate subject name matches target host name '212.224.123.69'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


$ echo $?
60

There you have also a link with more information about SSL certificates.
 
EDIT** Corrected my response because I realize the fallacies therein! Edits are denoted either with strikethrough text followed by the new information, if any, in italics.

@LnxBil - Thank you for your reply, your reasoning is correct and the first thing I'll be trying is your suggestion about the hostname.

https://forum.proxmox.com/threads/p...ttp-401-with-valid-password.56974/post-589619

I realized these errors the next day when my access stopped working through my public hostname, and embarked down a dark journey of trying to bind mount a config.yml to the container. Then decided to trash the docker runtime tunnel config and see if I could gain greater control over the backend with the cloudflared-cli installation on fedora server 38 x86_64 for the tunnel - not the proxmox host. I finally just now was able to repair my firewalld zones and get docker back to its original state.


Original reply:

I stumbled upon this trying to figure out the same thing with cloudflare today.

Running a cloudflared in the lightweight LXD on a separate host from my proxmox server at my home network and have yet to get this to work.

The config for Cloudflare public hostname - per the dashboard - needs to be set to `https://` and under the ’Additional Application Settings’ needs to have the path to the ca.pem certification on the proxmox host `/etc/pve/pve-root-ca.pem` listed in `TLS > Certificate Authority Pool`. Also, the `No TLS Verify` option needs to be turned off.

I can confirm that this setup did work for whatever reason, but the following day - as expected - I was left without remote access

Now that I have my server (not proxmox) back to working order (I think), I can now continue on my quest to try to get this working with Cloudflare tunnels.
Ok, now the typical method I use for `https://` - which is to choose the `notlsverify` option - is working. Just to confirm, it wasn't working to begin with.

Unfortunately I can't speculate on the root issue I was having because it could have been related to one, or all of the issues I just fixed in the past week.

I had to basically rebuild many critical systems on my SBC server (host running cloudflared) on my local network (e.g. `firewalld`, docker storage, or `cloudflared` itself).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!