[TUTORIAL] Using a reverse proxy to limit PBS exposition to API endpoints only

meyergru

Active Member
Jan 28, 2023
303
242
48
www.congenio.de
It always bugged me that when there is no possibility of having a VPN to a remote PBS server, you will have to expose the web UI as a whole.

I even did a feature request to create an "API-only" frontend to PBS to limit the attack surface, see: https://bugzilla.proxmox.com/show_bug.cgi?id=7010

I have this running for a while now. I found that neither Caddy nor Nginx were really suitable, because they lack one critical feature that HAproxy has, namely: "option http-server-close".

If you do not have that option, the setup works partially, but on the server side, you will get log entries with "TASK ERROR: connection error: not connected". That does not happen with HAproxy.

You only need to install HAproxy on PBS via "apt install haproxy" and then create /etc/haproxy/haproxy.cfg like so:

Code:
global
        maxconn 5000
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        nbthread 6
#       hard-stop-after 15m
        tune.ssl.default-dh-param 4096
        nokqueue

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: [URL]https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate[/URL]
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-CDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Parameter
    timeout client  3600s
    timeout server  3600s


# PBS proxy
frontend pbs_tcp
    bind :::18007 name :::18007 ssl crt /etc/haproxy/proxy.pem verify none
    mode http
    option httplog
    option http-server-close
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
#   timeout client 30000

    # Probe direkt beantworten
    acl path_probe path_beg /probe
    http-request return status 200 content-type "text/plain" string "alive" if path_probe

    # Nur /api2 Requests an Backend weiterleiten
    acl path_api2 path_beg /api2 //api2

    # Interaktive Terminal-Sitzungen vollstaendig sperren
    acl path_terminal path_reg -i ^/{1,2}api2/(json|extjs|html)/nodes/[^/]+/(termproxy|vncwebsocket)(/|$)
    http-request deny deny_status 403 if path_terminal

    # Ausstellung neuer Passwort-Tickets verhindern
    acl path_ticket path_reg -i ^/{1,2}api2/(json|extjs|html)/access/ticket(/|$)
    http-request deny deny_status 403 if path_ticket

    use_backend pbs_backend if path_api2

    # Alle anderen Requests ablehnen
    http-request return status 500 unless path_api2 or path_probe


backend pbs_backend
    mode http
    server      pbs 127.0.0.1:8007 ssl verify none

Of course you will have to copy your existing PBS cert via "cat /etc/proxmox-backup/proxy.{pem,key} >/etc/haproxy/proxy.pem", because HAproxy likes the cert and the key in one file.

This will allow only /api2, //api2 and /probe for monitoring purposes on a different port (18007). Any other path will result in an HTTP 500 error.

AFAIK, the API does not allow for any other authorisation type than tokens, so no usernames can be probed and even fingerprinting is limited to the bare minimum should the port be detected.

P.S.: I have changed the configuration to exlude the problematic terminal proxy endpoint as discussed the next posting. Therefore, I am not stating that there can be no circumventing the blocking here - for this, you would need an explicit white-list for all allowable endpoints.
 
Last edited:
  • Like
Reactions: Johannes S
This is gonna get long - so please bear with me, but you will catch my drift.

API-only access, AI-assisted updates, and an uncomfortable discovery about Proxmox security​

A recent experience has exposed what I believe is a fundamental inconsistency in Proxmox's API security architecture. Ironically, it started with AI.

Why AI is changing the update problem​

In a recent discussion about the increasing number of Proxmox kernel updates, I argued that AI-assisted vulnerability discovery is fundamentally changing the situation for system administrators:

https://forum.proxmox.com/threads/p...ualization-platform.187108/page-2#post-876118

AI-assisted code analysis increasingly uncovers vulnerabilities in existing code, some decades old. Their discovery is welcome, but the resulting stream of patches is getting harder to manage.

As the time between disclosure and exploitation shrinks, administrators must increasingly inventory systems, assess exposure, install patches and coordinate reboots without lengthy delays.

The obvious response is: If AI contributes to the growing volume of security maintenance, why not use AI to help manage that maintenance?

An apparent change of mind about AI and hypervisors​

Those who remember my rather critical comments in John Broadway's Proximo thread may find this surprising:

https://forum.proxmox.com/threads/community-project-proximo-—-an-open-source-least-privilege-mcp-api-layer-for-managing-pve-with-an-ai-agent-feedback-wanted.184435/#post-858371

Back in June, I warned against giving AI agents access to critical hypervisors through third-party software I could not sufficiently assess. Due to design flaws, that particular MCP approach also required a cloud AI, raising confidentiality concerns.

Those reservations remain. Yet I now use a locally hosted AI model for infrastructure administration.

My model runs entirely on my own hardware. I developed MCP servers for OPNsense and Proxmox with explicit operation allowlists, separate credentials and narrowly controlled writes.

The AI is supposed to assist with administration, not receive unrestricted control over my infrastructure.

That is where the problem began.

The missing upgrade API​

One of my intended workflows looks very straightforward:
  1. Check installed packages and available security updates.
  2. Refresh APT package lists.
  3. Perform a regular Proxmox package upgrade.
  4. Reboot if a new kernel has been installed.
  5. Verify the result.
The Proxmox API already provides most of the necessary functionality, including package information, refreshing APT metadata, node status and reboot operations.

But there is no native REST API operation for actually installing the updates.

Yet the WebUI can perform an upgrade. How? Not through a self-contained REST operation.

Instead, it starts an interactive terminal session through the API's termproxy mechanism, using cmd=upgrade. The actual upgrade then proceeds through an interactive process connected via WebSocket.

cmd=upgrade merely selects the initial program. Subsequent interactions are not individually defined, validated or authorized REST operations. Worse:

After the upgrade has finished, the user is deliberately dropped into an unrestricted root shell!

This is not speculation. It is documented behavior of pveupgrade --shell:

https://lore.proxmox.com/pve-devel/20200707102001.124244-2-m.almalat@proxmox.com/T/

The session does not end with the upgrade: it continues as a root shell, unlike a bounded API operation with defined permissions and results.

Moreover, the privileged upgrade console is restricted to root@pam. A narrowly scoped API token cannot simply obtain an upgrade privilege and perform the same operation.

To automate the task, I would need another privileged mechanism, such as SSH with a restricted maintenance script: more credentials, another access path and more attack surface.

The attempt to automate security maintenance with minimal privileges is forcing me towards a more powerful access mechanism.

Why this contradicts the argument against a separate API​

This brings me back to the reason I originally opened Bug 7010 and subsequently implemented the HAProxy workaround described in this thread.

https://bugzilla.proxmox.com/show_bug.cgi?id=7010

Dominik Csapak argued that separating the WebUI from the API offers no security benefit because the WebUI merely calls the API.

See: https://forum.proxmox.com/threads/can-i-ditch-pveproxy-i-guess-not.135980/

I can understand that reasoning if the WebUI exclusively uses the same functionally defined and granularly authorizable operations that external REST clients can invoke.

But that is not the architecture we actually have now.

The WebUI also establishes interactive terminal sessions via API endpoints. What happens inside those sessions is not a set of self-contained REST operations. This reinforces my original case for API isolation.

Calling the endpoint that establishes a privileged terminal session an "API" does not make the capabilities inside that session equivalent to a collection of narrowly scoped administrative REST API functions.

Otherwise, by the same reasoning, an SSH server could be considered a perfectly adequate universal management API.

My earlier assumption that /api2 permitted only token authentication was wrong. It also exposes password-ticket authentication and console endpoints. I must amend my HAProxy rules to block termproxy and vncwebsocket, along with password-ticket login, on the public listener. Even then, a path-based proxy is no substitute for a true minimal API.

And why the missing upgrade endpoint is a security issue​

There is another relevant feature request: https://bugzilla.proxmox.com/show_bug.cgi?id=6799

In September 2025, Florian Hoberg submitted a patch implementing a native upgrade API operation.

Thomas Lamprecht rejected it, arguing that unattended upgrades might damage a system and that an interactive shell keeps a human administrator involved.

The discussion is available here: https://lists.proxmox.com/pipermail/pve-devel/2025-September/075184.html

That mixes two different concerns:
  • Operational safety: Can an upgrade fail, damage package dependencies or render a system unusable?
  • Access security: Can we authorize a particular administrator or automation service to perform exactly this operation without granting unnecessarily broad capabilities?
Both matter. A native API could preview changes, require approval, enforce its own privilege and provide an auditable task result, without allowing blind upgrades or granting a root terminal.

The current design does not prevent failed upgrades; it just makes least-privilege automation harder. Proxmox already exposes potentially more destructive actions through its API, subject to permissions. Why not upgrades?

The underlying contradiction​

We are effectively being presented with two design positions:
  • Position A: Separating the WebUI from the API provides no meaningful security benefit because the WebUI only uses the API.
  • Position B: An ordinary administrative operation such as a package upgrade should not be exposed as a native API function because it is safer to perform it through an interactive root terminal.
These positions are difficult to reconcile.

The first assumes functionally equivalent, well-defined management operations. The second requires an interactive execution channel vastly more powerful than the requested operation. Initiating a shell through an API endpoint does not turn its commands into granularly authorized REST calls. Rejecting a native upgrade operation thus reinforces the very distinction invoked to dismiss API isolation.

I consider this a serious security design problem.

Not because it automatically constitutes an exploitable vulnerability, but because it unnecessarily undermines two fundamental security principles: least privilege and attack surface reduction.

What I propose Proxmox should actually change​

At the least, both requests deserve reconsideration. They address distinct parts of the same problem and can be implemented independently and a solution can of course look different, but here is my take on this:

Bug 7010 – A genuinely separate, restricted API interface​

Security objective: Allow administrators to expose only the management functionality required by external clients, without unnecessarily exposing interactive or privileged administrative mechanisms.

A separate listener that merely provides the same unrestricted interface on another port would not be sufficient.

Concrete implementation proposal:
  • Provide an independently configurable API listener, with its own bind address and port, separate from the administrative WebUI listener.
  • Support a server-side allowlist of permitted API operations, ideally matching both HTTP method and API route. Operations not explicitly permitted must be rejected by default.
  • Allow token-only authentication on this listener, without password-ticket login or browser session authentication.
  • Exclude WebUI assets, /nodes/{node}/termproxy, /nodes/{node}/vncwebsocket and all other interactive console paths from the restricted listener.
  • Keep the existing Proxmox ACL and token permission checks fully effective. The listener's allowlist must be an additional security boundary, not a replacement for authorization.
  • Document useful predefined configurations, for example a PBS backup-only interface, including any protocol-specific endpoints required by actual backup clients.
Enforce these restrictions within Proxmox, allowing the administrative UI to remain on a trusted network without requiring fragile reverse-proxy workarounds.

Bug 6799 – A native package-upgrade API​

Security objective: Enable controlled package upgrades using a narrowly authorized API token, without requiring a privileged interactive terminal, SSH access or another general-purpose execution mechanism.

The implementation should not merely provide another way to start termproxy with cmd=upgrade.

Concrete implementation proposal:
  • Add a native upgrade endpoint, for example POST /nodes/{node}/apt/upgrade, backed by the existing Proxmox package-upgrade infrastructure.
  • Introduce a dedicated privilege, such as Sys.Upgrade, rather than requiring broad administrative privileges or full terminal access. The name is illustrative; the important point is a separate authorization boundary.
  • Provide a dry-run or planning operation that reports the proposed package installations, upgrades, removals and other potentially disruptive changes.
  • Require explicit approval of the resulting upgrade plan before installation. A changed or outdated plan should require renewed approval.
  • Execute the upgrade as a managed Proxmox task, returning a UPID and making progress, logs, exit status and errors available through the normal task API.
  • Apply Proxmox's recommended upgrade procedure, including the equivalent of apt-get dist-upgrade, rather than relying on an incomplete plain apt upgrade.
  • Preserve operational safeguards: detect unexpected package removals, handle or reject unresolved interactive configuration decisions, and report failures instead of silently proceeding.
  • Report whether the running kernel differs from the installed boot kernel, allowing an authorized client to decide whether to request a reboot through the existing node API.
The endpoint should not allow arbitrary commands, arbitrary APT arguments or uncontrolled modification of package sources.

It should perform one clearly defined management operation, under one clearly defined permission.

If desired, Proxmox could require an explicit administrator confirmation for execution, without requiring that the administrator interact with a root shell.

This would address T. Lamprecht's operational-safety concerns while avoiding the additional access-security risks created by the current workaround.

Conclusion​

These are not merely convenience features for administrators who want to automate everything. They are complementary security requirements.

Bug 7010 would make it possible to expose a smaller, explicitly controlled management surface.

Bug 6799 would make it possible to perform a routine privileged maintenance operation without granting access to a more general execution mechanism.

The irony is hard to miss:

I started using a locally hosted AI and a deliberately restricted MCP infrastructure to cope with the growing workload of security maintenance, while keeping powerful management capabilities away from the AI itself.

Yet the routine task I wanted to automate is unavailable through a restricted REST operation precisely because Proxmox requires a more powerful interactive mechanism.

A secure management API should not force administrators to bypass its own security model in order to perform ordinary maintenance tasks.

That is why both requests deserve reconsideration as security improvements, not just convenience features.
 
Last edited:
  • Like
Reactions: Johannes S