proxmox UI, Caddy and reverse proxy

[ayo91]

I use Caddy Server as reverse proxy in my local network for avoid use specify ports in each of my applications. I have the solution working for all my network except for promox. When open the UI using https://proxmox.lan it shows me an error 502. Any idea? I have several other apps with the same setup and everything works well. It has to be something related to Proxmox...

Caddy config:

proxmox.lan:443 {
reverse_proxy {
to 192.168.1.200:8006
}
}

Any idea?

 

[tom]

See also: https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy

 

[ayo91]

Thanks!!

After setup it, I can access directly using the ip throught https without the port, as expected (https://192.168.1.200)

However, if I open https://proxmox.lan, Google Chrome gets this error: ERR_TOO_MANY_REDIRECTS

 

[ayo91]

See also: https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy

any idea tom?

 

[Artem1231]

I use Caddy Server as reverse proxy in my local network for avoid use specify ports in each of my applications. I have the solution working for all my network except for promox. When open the UI using https://proxmox.lan it shows me an error 502. Any idea? I have several other apps with the same setup and everything works well. It has to be something related to Proxmox...

Caddy config:

proxmox.lan:443 {
reverse_proxy {
to 192.168.1.200:8006
}
}

Any idea?

Hi! Have you been able to configure caddy to work with proxmox? I don't get any errors at all, the page with proxmox is just blank.

 

[oherold]

Hi,

I have only registered in order to post my solution to the problem. The working entry in my Caddy config file is as follows:

proxmox.lan {
reverse_proxy 192.168.1.200:8006 {
transport http {
tls_insecure_skip_verify
}
}
}

 

[stuif1982]

Hi,

I have only registered in order to post my solution to the problem. The working entry in my Caddy config file is as follows:

proxmox.lan {
reverse_proxy 192.168.1.200:8006 {
transport http {
tls_insecure_skip_verify
}
}
}

Thnx! This did it!

 

[ayo91]

Hi,

I have only registered in order to post my solution to the problem. The working entry in my Caddy config file is as follows:

proxmox.lan {
reverse_proxy 192.168.1.200:8006 {
transport http {
tls_insecure_skip_verify
}
}
}

Amazing! It worked like a charm. Without user Nginx_Proxy. Only caddy configuration

 

[stuif1982]

Thnx! This did it!

I noticed this is not completely working in a cluster:

The node which my domain is proxying to is working. But on the others nodes added to the cluster (and so also have other IP-addresses) the consoles stay blank.

 

[eivamu]

I noticed this is not completely working in a cluster:

The node which my domain is proxying to is working. But on the others nodes added to the cluster (and so also have other IP-addresses) the consoles stay blank.

I ran into problems with Proxmox clusters on Caddy too. This was especially apparent when trying to connect to consoles via noVNC. I think I fixed it by selecting a correct load balancing rule to make sure that the client stays on the same backend server.

Here's my working config extracted from my Caddyfile:

proxmox.domain.com {
    reverse_proxy * {
        to server01:8006
        to server02:8006
        to server03:8006

        lb_policy ip_hash     # Makes backend sticky based on client ip
        lb_try_duration 1s
        lb_try_interval 250ms

        health_uri /          # Backend health check path
        # health_port 80      # Default same as backend port
        health_interval 10s
        health_timeout 2s
        health_status 200

        transport http {
            tls_insecure_skip_verify
        }
    }
}

Caddy 2.6.2 on Ubuntu 22.04 LTS.
Proxmox GUI 7.2-11.

 

[kimmer2k]

Is using tls_insecure_skip_verify advised from a security standpoint? If not, does anyone know another workaround to this problem?

 

[billthe4th]

Is using tls_insecure_skip_verify advised from a security standpoint? If not, does anyone know another workaround to this problem?

It does enable a man-in-the-middle attack as the Caddy server will accept any certificate from the backend server(s), but assuming Caddy and Proxmox are on the same LAN, or even the same machine, I'd say the risk is minimal.

If you want to avoid using that option, you've got to make Caddy trust the certificate that Proxmox is providing, either by adding your PVE root cert to the Caddy server's trusted root store, or by adding it to the config something like this (stealing eivamu's example):

proxmox.domain.com {
    reverse_proxy * {
        to server01:8006
        to server02:8006
        to server03:8006

        lb_policy ip_hash     # Makes backend sticky based on client ip
        lb_try_duration 1s
        lb_try_interval 250ms

        health_uri /          # Backend health check path
        # health_port 80      # Default same as backend port
        health_interval 10s
        health_timeout 2s
        health_status 200

        transport http {
            tls_trusted_ca_certs /etc/pve/pve-root-ca.pem # Path to PVE root cert
        }
    }
}

Or you could make PVE use a publicly trusted CA signed cert through the ACME challenge options available in Node -> System -> Certificates.

 

[kimmer2k]

It does enable a man-in-the-middle attack as the Caddy server will accept any certificate from the backend server(s), but assuming Caddy and Proxmox are on the same LAN, or even the same machine, I'd say the risk is minimal.

If you want to avoid using that option, you've got to make Caddy trust the certificate that Proxmox is providing, either by adding your PVE root cert to the Caddy server's trusted root store, or by adding it to the config something like this (stealing eivamu's example):

proxmox.domain.com {
    reverse_proxy * {
        to server01:8006
        to server02:8006
        to server03:8006

        lb_policy ip_hash     # Makes backend sticky based on client ip
        lb_try_duration 1s
        lb_try_interval 250ms

        health_uri /          # Backend health check path
        # health_port 80      # Default same as backend port
        health_interval 10s
        health_timeout 2s
        health_status 200

        transport http {
            tls_trusted_ca_certs /etc/pve/pve-root-ca.pem # Path to PVE root cert
        }
    }
}

Or you could make PVE use a publicly trusted CA signed cert through the ACME challenge options available in Node -> System -> Certificates.

Thanks for this, I was able to figure out adding the pve cert, but didn't think of making it a public trusted CA as an even simpler way!

 

[Robert.H]

Health_uri being simply /, does it exclude hosts in maintenance or perhaps not part of the cluster?
I'm searching for a /live and /ready endpoint on the api but not finding anything close to that...
ref. https://testfully.io/blog/api-health-check-monitoring/#api-health-check-endpoint-types