[TUTORIAL] Howto: WebAuthn Passkeys across Cluster or on single Node

[aj@root]

Live Training Video (40m)

https://www.youtube.com/watch?v=Qhm8NsYq6dc

WARNING: Follow Carefully, You May Only Get 1 Try

As per our experimentation, it is possible to misconfigure WebAuthn in a way that cannot easily be fixed by using the Proxmox Web UI, nor by directly updating the relevant config files.

See the Troubleshooting section below for more help.

If you intend to create a cluster, set up WebAuthn with the parent domain for Datacenter: Options: WebAuthn: ID from the start (NOT Auto-fill)!

DO NOT go through the effort of creating a cluster without testing that WebAuthn works on the first node in the parent domain configuration.

BEFORE YOU BEGIN: Set Recovery Keys & TOTP

WebAuthn is fragile for a a number of reasons:

• easy to misconfigure
• a domain may expire
• certificates may lapse

Under any of these conditions, Passkeys will most likely stop working. So, before you enable WebAuthn, also enable Recovery Keys and TOTP as alternate authentication methods.

WARNING: Requiring TFA in a REALM will DISABLE WebAuthn

If you enable Require TFA in Datacenter: Permissions: Realms, it will disable WebAuthn Passkeys.
(possibly a bug that will be fixed in future versions)

0. Setup & Preconditions

We're on PVE 8.4.1 (Enterprise repo), updated on Apr 21, 2025.

We have been able to get WebAuthn working on both

• a single node, using Auto-fill with the node's full subdomain.
• a cluster, using a parent domain as the ID

We're using these domains for our internal lab to test and document this process:

      lab2.therootcompany.com  # parent domain, Relying Party ID
mplx1.lab2.therootcompany.com  # node 1
mplx2.lab2.therootcompany.com  # node 2
mplx3.lab2.therootcompany.com  # node 3

(we use mplxN rather than pveN to further distinguish our lab from our production cluster)

Note: since WebAuthn follows cookie rules for security, for a cluster we must choose a parent domain which must NOT be on the Public Suffix List - meaning that you CANNOT test with duckdns.org or most other shared domains as the parent domain (although you could use <pveN>.<yourlab>.duckdns.org).

We want the scope of our WebAuthn (the Relying Party ID) to be segmented to a particular subdomain of our main site as to separate it from the WebAuthn of other apps for security and avoid misconfiguration.

Each PVE will be a sibling under the parent domain - this is what allows a single Passkey stored in iCloud Keychain, Google Sync, or the Browser Passkey Manager to be shared between them.

Depending on your configuration, and if other applications might use WebAuthn, or should not be allowed to use WebAuthn to retrieve a key that could be used to login to a PVE, you may wish to further segment PVEs under their own subdomain.

For example, we would use pves.lab2.therootcompany.com as the parent if we wanted to host dashboard.lab2.therootcompany.com directly under lab2.therootcompany.com. Otherwise either it would not be able to use WebAuthn, or it would have to be manually configured using the data from /etc/pve/priv/tfa.cfg.

1. Let's Encrypt via Datacenter: ACME & Node: System: Certificates

Under Datacenter: ACME we set up:

• Let's Encrypt Account
  (no other ACME services are supported as of v8.4.1)
• ACME DNS Challenge Plugin
  (HTTP can only be used for public-facing IPs)

Under Node: System: Certificates: ACME we will:

• Change Using Account to the account we created in Datacenter: ACME: Accounts and Apply
• Add the domain name of our PVE (mplx3.lab2.therootcompany.com) and Order Certificates Now

After the timeout, we have our certificates, which can be confirmed in the PVE config:

#/etc/pve/nodes/mplx3/config

acme: account=letsencrypt-therootcompany
acmedomain0: mplx3.lab2.therootcompany.com,plugin=namecom-therootcompany

Currently only Let's Encrypt (not other ACME services such as BuyPass, ZeroSSL, Sectigo, Google PKI, etc).

2. Datacenter: Options: WebAuthn

At the Datacenter level we set the Name (a comment), Origin (leave unset except to restrict to a single node), and ID (the WebAuthn Relying Party ID, which is a domain as mentioned earlier).

For our use case that looks like this:

Single Node

Name: MPLX3 (friendly name for the iCloud Passkey / WebAuthn storage manager)
Origin: (unset)
ID: mplx3.lab2.therootcompany.com (RPID / FQDN of Relying Party)

For All Nodes in Cluster

Name: Lab2 Cluster (friendly name for the iCloud Passkey / WebAuthn storage manager)
Origin: (unset)
ID: lab2.therootcompany.com (RPID / FQDN of Relying Party)

For a single node we can use Auto-fill to set the ID to the node name as mentioned above, but for a cluster the ID must be set to a parent domain.

After updating the settings we can see it reflected in the config file:

#/etc/pve/datacenter.cfg

webauthn: id=lab2.therootcompany.com,rp=Lab2 Cluster

3. Datacenter: Permissions: Two Factor

We select our root@pam to start. Once we have success with that we can go back and add our admin@pve.

From the Add menu we select WebAuthn, add a Description and Register Webauthn Device.

We will be prompted to use the Chromium Passkey Manager (i.e. Brave Sync, Google Sync, etc), but can back out to choose iCloud Keychain or Security Key (i.e. Yubikey) instead.

After completing the authentication, the TFA is added.

4. Web UI Login on Any Node

At this point we should be able to visit any node in the cluster with a subdomain of lab2.therootcompany.com and login using the user at specified realm.

Since we're prompted for a WebAuthn Passkey, we know we've selected the correct realm.
(if you don't get a Passkey prompt, check your realm)

5. Checking the Audit Logs (Web UI, journald, PVE Access)

Web Inspector

We can check the Web Inspector by Alt+Cmd+i, and in some cases we'll see error messages in more detail than what we might see in the Web UI alert.

In this case, the Web UI only gives a very generic "Login Failed. Please try again.", and there are no additional messages in the JavaScript console.

Systemd Journald
(replaces syslog and auth.log)

#journalctl --since '10 minutes ago' | grep 'auth'

Apr 21 12:35:40 mplx3 pvedaemon[1160]: <root@pam> successful auth for user 'root@pam'
Apr 21 12:35:51 mplx3 pvedaemon[1160]: authentication failure; rhost=::ffff:172.18.0.86 >

PVE Access Logs

#grep -i 'POST .*/ticket' /var/log/pveproxy/access.log

::ffff:172.18.0.86 - - [22/04/2025:12:52:34 -0600] "POST /api2/extjs/access/ticket HTTP/1.1" 200 748
::ffff:172.18.0.86 - - [22/04/2025:13:06:34 -0600] "POST /api2/json/access/ticket HTTP/1.1" 200 740

WebAuthn Origins (not used)

Under some conditions the browser will check the the domain of the Relying Party ID for a list of origins.

In our case that would be https://lab2.therootcompany.com/.well-known/webauthn.

We setup an internal webserver at lab2.therootcompany.com (which our browsers can access) to respond to /.well-known/webauthn with the primary domain and allowed subdomains and tested that it responds as expected.

{
    "origins": [
        "https://lab2.therootcompany.com",
        "https://mplx1.lab2.therootcompany.com",
        "https://mplx2.lab2.therootcompany.com",
        "https://mplx3.lab2.therootcompany.com"
    ]
}

However, the server logs never showed any access, so this doesn't appear to be used in this setup.

Troubleshooting

It's entirely possible that you currently have everything correct and have removed and re-added everything in this process from top to bottom in the Web UI and still end up unable to use WebAuthn.

Here's what we've done in order to wipe things out and (after multiple tries in different orders) finally had success:

# remove the 'webauthn:' line
vi /etc/pve/datacenter.cfg

# remove tfa lines and delete user config
vi /etc/pve/domains.cfg
rm /etc/pve/priv/tfa.cfg

# remove and regenerate the default tls certs
rm /etc/pve/nodes/mplx3/pve-ssl.key
rm /etc/pve/nodes/mplx3/pve-ssl.pem
pvecm updatecerts --force

# remove the acme-related lines
# (in our case we removed the whole file)
vi /etc/pve/nodes/${PVE_N}/config

# remove and regenerate the ACME certs
rm /etc/pve/nodes/mplx3/pveproxy-ssl.key
rm /etc/pve/nodes/mplx3/pveproxy-ssl.pem
pvenode acme cert order

# make sure all related services are restarted
reboot

Also:

• Check that Datacenter: Permissions: Realms does NOT enable Require TFA
• Close the Browser COMPLETELY (destroys all TLS sessions)
• Check your Passkey Manager and delete all keys related to the domains you're using

If all of that fails, it may very well work with a fresh install with the correct settings given on the first try.

It very much seems that it is possible to set invalid configurations that are difficult to diagnose and recover from.

Web UI Configuration

ACME is correctly configured, and certificates have been ordered.



WebAuthn is configured in the Datacenter, Options.


WebAuthn is configured in Datacenter, Permissions, Two Factor.



Passkey is visible in the Apple iCloud Passkey manager.

Login Succeeds or Fails.

 

[weehooey-bh]

Hey @aj@root !

Thank you for creating this tutorial. I have validated the WebAuthn steps in our lab (already had the certs). Works great!

Also, thank you for sharing.