api returned unexpected data - expected json body, got text/html

NomadCF

Active Member
Dec 20, 2017
29
1
43
44
When we're trying to add our cluster or even a node, we're getting the error below. All the nodes have been fully updated & upgraded to pve-manager/8.3.2/3e76eec21c4a14a7 (running kernel: 6.8.12-5-pve). All the nodes are the same hardware & kernel.


api returned unexpected data - expected json body, got text/html
 
Hello, I am receiving the same error message when I try to add nodes.
"api returned unexpected data - expected json body, got text/html"

This happens when I try to use my local domain address to add to PDM. (node1.mydomain.com) (I use HA proxy on an OPNsense to add a self signed cert / not have to deal with "insecure" messages every time I visit the node.) Everything is on the same network, router has a DNS override for my domain that sends it back to my servers so I wonder if it doesn't like sending it "outward" when it is expecting local resources?

From what I can tell DNS is set correctly on both the nodes and PDM. Each of the nodes are upgraded to 8.3.3. I am able to add them directly via IPs (10.0.0.X) with or without ports, but that makes it so that all of the redirect links point to the IPs instead of my nicer to use FQDN addresses that don't have "insecure" warnings.

I understand it's Alpha and I'm excited for the future of this product. Makes perfect sense to use when clustering/reaching quorum is not ideal.

If there's any tests or logs you'd like me to run please let me know and I can try to provide them.
 
Intermediate proxies are currently not supported (and it's not certain they can ever be), as per the FAQ from the initial alpha release post:

Q: What network setups are supported between Proxmox Datacenter Manager and remotes?
A: In general the Proxmox Datacenter Manager needs to be able to connect to all Proxmox VE remotes directly to send API requests and query load and usage metrics. Remotes on the other hand do not need to be able to connect to Datacenter Manager directly. Reverse proxies between Proxmox Datacenter Manager and any of its Proxmox VE remotes are not supported, we recommend using tunneling (for example, WireGuard or OpenVPN) for hosts that must not be exposed directly to a non-private network.
Everything is on the same network, router has a DNS override for my domain that sends it back to my servers so I wonder if it doesn't like sending it "outward" when it is expecting local resources?

Then PDM not support (HA) proxy should not really matter. You could try adding the PVE node using the local IP address directly. Or alternatively test if your DNS indeed resolves correctly to the local address by trying to resolve the node's hostname from the PDM shell (ping could be used for starters).

not have to deal with "insecure" messages every time I visit the node
FWIW, depending on your time and enthusiasm one option here might be to setup a local ACME backed CA service, i.e. what Let's Encrypt provides but locally. Our products support using a local ACME instance (albeit IIRC, one might need to use the CLI once for adding the ACME directory initially). Compared to using DNS based challenge, which also works for private hosts, this would leave you in full control of the CA and avoid leaking some metadata through the transparency logs the public CAs must publish. Just mentioning this as you explicitly noted that you're using that HA Proxy setup for the local CA cert.
 
Intermediate proxies are currently not supported (and it's not certain they can ever be), as per the FAQ from the initial alpha release post:




Then PDM not support (HA) proxy should not really matter. You could try adding the PVE node using the local IP address directly. Or alternatively test if your DNS indeed resolves correctly to the local address by trying to resolve the node's hostname from the PDM shell (ping could be used for starters).


FWIW, depending on your time and enthusiasm one option here might be to setup a local ACME backed CA service, i.e. what Let's Encrypt provides but locally. Our products support using a local ACME instance (albeit IIRC, one might need to use the CLI once for adding the ACME directory initially). Compared to using DNS based challenge, which also works for private hosts, this would leave you in full control of the CA and avoid leaking some metadata through the transparency logs the public CAs must publish. Just mentioning this as you explicitly noted that you're using that HA Proxy setup for the local CA cert.

Appreciate the quick reply! Got it. Must have missed that when I was reading up on this. I thought I was going nuts since some of the examples I could find on Youtube ( https://youtu.be/o0QeU4xMjPc?si=yAPlOOVlIVsEcfsN&t=334 and https://youtu.be/rsguS0hw1PI?si=HX_YWdhJUD3Xbo4Y&t=518 ) seemed to be using domains instead of IPs, but I didn't look closely enough to see they were using exclusively local TLDs (.home / .local).

Pinging the FQDN of the node from PDM receives a response from each of the different services that I have reverse proxied using HA proxy, so no wonder PDM is having trouble connecting properly. Adding the nodes directly from IPs worked flawlessly (besides having to remove the API Token from the node when I was trying to re-add it during troubleshooting.)

I'll add that to my list of potential future projects. Always looking for something new to tinker with and I have appreciated Proxmox's role in my homelab journey. I can't wait to see where PDM goes in the future.

Thanks again, hopefully others running into the same issue will find this thread.