[SOLVED] Undefined Error 1006: Console not working / Websocket Connection Failed (Proxmox VE 8, Nginx Proxy Manager, Authelia)

[bashrag]

Hi everyone, I need some help troubleshooting websockets (at least that's what I think the issue is). If I access Proxmox via 192.168.2.10:8006, the Proxmox console works perfectly. If I access Proxmox via pve1.mydomain.com while the "advanced" tab of of the Proxy Host in the "Edit Proxy Hosts" window in Nginx Proxy Manager is completely empty, the Proxmox console works perfectly.

However, as soon as I add the Authelia code to that "Advanced" section, the Proxmox Console stops working and this is what I see:

Trying to access any of the consoles (regardless of whether it's in pve1 or any of the LXC containers) results in a brief grey banner over the console with "Undefined Error: 1006".

At the moment, after Authelia lets me through, I am logging in as root@pam & not myname@Authelia. I recently found out that only root@pam can access the console. For the sake of completeness, I tried logging in as myname@Authelia as well, and as expected, that did not work:


After spending a few days Googling this, I guess it's something to do with Authelia blocking Websockets, but I do not know nearly enough to know what I should look for, let alone to know how to fix this. As a disclaimer, I am a complete beginner to homelabbing. I have no formal training or experience in anything IT. I literally just started doing this as a hobby in my spare time, becuase I want to try to get some fun stuff set up at home. I am already at the very limits of my knowledge. I'm posting everything that I can think of that might be helpful for troubleshooting, but if you guys require more details, please let me know and I'll post them! Any help would be appreciated

The general workflow I am trying to set up is as follows:
Switch on Wireguard → Open *App* by visiting subdomain.mydomain.com → Get intercepted by Authelia → Login + 2FA → Authelia grants access to subdomain.mydomain.com → Use Authelia's SSO to login to *App*

I have Proxmox 8 running on a Lenovo M720q. To access Proxmox (or anything on that machine for that matter), I need to either be on my Home network, or be connected to Wireguard (I am using a Ubiquiti UniFi Dream Machine SE which has Wireguard built in) and reach it via the IP address (which is annoying). I have a domain with Cloudflare managing the domain's records (they are proxied). I have an access list set up in Nginx Proxy Manager (running as an LXC container thanks to the Helper Scripts) which only allows 192.168.X.X & my public-facing static IP through to whatever apps have subdomains. Nginx Proxy Manager is also handling the SSL Certificates (I have one wildcard Let's Encrypt Certificate with mydomain.com, *.mydomain.com), and the SSL Mode is set to Full (Strict) in Cloudflare.

Authelia is set up in its own LXC Container (thanks to the Helper Scripts). I followed this tutorial to set it up. The files in /etc/authelia/certs/ are Cloudflare Origin Certificates. I tried to follow Authelia's documentation and tried allowing resources to bypass authentication, but I don't know if I did it right.
Here's the config file with Domain/IP/Email details sanitised:

########################################
# authelia config
# source: https://florianmuller.com
# #######################################

theme: dark
default_redirection_url: https://auth.mydomain.com/

server:
  host: 0.0.0.0
  port: 9091
  asset_path: /etc/authelia/.assets/
  tls:
    key: /etc/authelia/certs/mydomain.key
    certificate: /etc/authelia/certs/mydomain.crt

log:
  level: debug
  file_path: /var/log/authelia.log

totp:
  issuer: auth.mydomain.com

webauthn:
  disable: false
  display_name: Authelia
  attestation_conveyance_preference: indirect
  user_verification: discouraged
  timeout: 60s

ntp:
  address: "0.de.pool.ntp.org:123"
  version: 4
  max_desync: 3s
  disable_startup_check: false

authentication_backend:
  file:
    path: /etc/authelia/.users/users_database.yml
    password:
      algorithm: argon2id
      iterations: 1
      key_length: 32
      salt_length: 16
      memory: 1024
      parallelism: 8

access_control:
  default_policy: deny
 
  networks:
  - name: wan-ip
    networks:
    - '193.my.public.ip'
  - name: internal-default
    networks:
    - '192.168.1.0/24'
  - name: internal-trusted
    networks:
    - '192.168.2.0/24'
  - name: internal-iot
    networks:
    - '192.168.3.0/24'
  - name: wireguard
    networks:
    - '192.168.4.0/24'
 
  rules:
   
    #Bypass Proxmox APIs & Resources on all subdomains
    - domain: '*.mydomain.com'
      resources:
        - '\/api.*\/'
      policy: bypass  
      networks:
      - 'wan-ip'
      - 'internal-default'
      - 'internal-trusted'
      - 'wireguard'
   
    #Require 2FA for Proxmox  
    - domain: 'pve1.mydomain.com'
      policy: two_factor
      networks:
      - 'wan-ip'
      - 'internal-default'
      - 'internal-trusted'
      - 'wireguard'

    #Require 2FA for NginxProxyManager  
    - domain: 'npm.mydomain.com'
      policy: two_factor
      networks:
      - 'wan-ip'
      - 'internal-default'
      - 'internal-trusted'
      - 'wireguard'
     
session:
  name: authelia_session
  expiration: 3600  # 1 hour
  inactivity: 300  # 5 minutes
  domain: mydomain.com
  same_site: lax
  remember_me_duration: 7d

regulation:
  max_retries: 3
  find_time: 120
  ban_time: 300

storage:
  local:
    path: /etc/authelia/.db/sqlite3.db

notifier:
  disable_startup_check: false
  smtp:
    host: smtp.mail.yahoo.com
    port: 465
    timeout: 5s
    username: email@yahoo.com
    sender: "mydomain Authentification Server <auth@mydomain.com>"
    subject: "{title}"
    startup_check_address: email@yahoo.com
    disable_require_tls: false
    disable_html_emails: false
    tls:
      skip_verify: false
      minimum_version: TLS1.2

identity_providers:
  oidc:
    ## The other portions of the mandatory OpenID Connect 1.0 configuration go here.
    ## See: https://www.authelia.com/c/oidc
    clients:

    - id: Proxmox
      description: Proxmox
      secret: '$plaintext$supersecureipromise'
      public: false
      authorization_policy: two_factor
      redirect_uris:
        - https://pve1.mydomain.com
      scopes:
        - openid
        - profile
        - email
      userinfo_signing_algorithm: none

    - id: NPM
      description: NginxProxyManager
      secret: '$plaintext$iswearihaventreusedit'
      public: false
      authorization_policy: two_factor
      redirect_uris:
        - https://npm.mydomain.com
      scopes:
        - openid
        - profile
        - email
      userinfo_signing_algorithm: none

In Nginx Proxy Manager, I have one entry for Authelia & one entry for Proxmox.
Both use the same SSL Certificate and have identical "SSL" Tabs:


auth.mydomain.com is set up like so:

With the "Advanced" section having:

#Authelia
location / {
        set $upstream_authelia https://192.168.2.12:9091;
        proxy_pass $upstream_authelia;
        client_body_buffer_size 128k;
 
        #Timeout if the real server is dead
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
 
        # Advanced Proxy Config
        send_timeout 5m;
        proxy_read_timeout 360;
        proxy_send_timeout 360;
        proxy_connect_timeout 360;
 
        # Basic Proxy Config
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Uri $request_uri;
        proxy_set_header X-Forwarded-Ssl on;
        proxy_redirect  http://  $scheme://;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_cache_bypass $cookie_session;
        proxy_no_cache $cookie_session;
        proxy_buffers 64 256k;
 
        # If behind reverse proxy, forwards the correct IP
        set_real_ip_from 10.0.0.0/8;
        set_real_ip_from 172.0.0.0/8;
        set_real_ip_from 192.168.0.0/16;
        set_real_ip_from fc00::/7;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;
    }

pve1.mydomain.com is set up like so:

With the "Advanced" section having:

#Proxmox
location /authelia {
    internal;
    set $upstream_authelia https://192.168.2.12:9091/api/verify; #ADD YOUR IP AND PORT OF AUTHELIA
    proxy_pass_request_body off;
    proxy_pass $upstream_authelia;  
    proxy_set_header Content-Length "";
 
    # Timeout if the real server is dead
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
    client_body_buffer_size 128k;
    proxy_set_header Host $host;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Forwarded-Uri $request_uri;
    proxy_set_header X-Forwarded-Ssl on;
    proxy_redirect  http://  $scheme://;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_cache_bypass $cookie_session;
    proxy_no_cache $cookie_session;
    proxy_buffers 4 32k;
 
    send_timeout 5m;
    proxy_read_timeout 240;
    proxy_send_timeout 240;
    proxy_connect_timeout 240;
}
 
    location / {
        set $upstream_Proxmox https://192.168.2.10:8006;  #CHANGE NAME AND IP AND PORT
        proxy_pass $upstream_Proxmox;  #change name of the service
  auth_request /authelia;
  auth_request_set $target_url $scheme://$http_host$request_uri;
  auth_request_set $user $upstream_http_remote_user;
  auth_request_set $groups $upstream_http_remote_groups;
  proxy_set_header Remote-User $user;
  proxy_set_header Remote-Groups $groups;
  proxy_set_header Access-Control-Allow-Origin *;
  error_page 401 =302 https://auth.mydomain.com/?rd=$target_url; #change YOURDOMAIN.COM to your domain
 
  client_body_buffer_size 128k;
 
  proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
 
  send_timeout 5m;
  proxy_read_timeout 360;
  proxy_send_timeout 360;
  proxy_connect_timeout 360;
 
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-Host $http_host;
  proxy_set_header X-Forwarded-Uri $request_uri;
  proxy_set_header X-Forwarded-Ssl on;
  proxy_redirect  http://  $scheme://;
  proxy_http_version 1.1;
  proxy_set_header Connection "";
  proxy_cache_bypass $cookie_session;
  proxy_no_cache $cookie_session;
  proxy_buffers 64 256k;
 
  #real_ip_header X-Forwarded-For;
  real_ip_header CF-Connecting-IP;
  real_ip_recursive on;
    }

The only thing I can see in the Authelia Log after successful 2FA is this:

root@Authelia:~# tail -f /var/log/authelia.log
time="2023-10-27T10:33:00Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/tasks (method )."
time="2023-10-27T10:33:03Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/nodes/pve1/status (method )."
time="2023-10-27T10:33:03Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/resources (method )."
time="2023-10-27T10:33:03Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/tasks (method )."
time="2023-10-27T10:33:06Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/resources (method )."
time="2023-10-27T10:33:06Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/tasks (method )."
time="2023-10-27T10:33:08Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/nodes/pve1/status (method )."
time="2023-10-27T10:33:09Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/resources (method )."
time="2023-10-27T10:33:09Z" level=debug msg="Check authorization of subject username=myname groups= ip=192.168.2.3 and object https://pve1.mydomain.com/api2/json/cluster/tasks (method )."

What (all) am I doing wrong? Should I provide any other info that may be helpful?

 

[bashrag]

I managed to fix this thanks to a comment on a Reddit thread where OP was having the same problem, but with Authentik.
Adding proxy_set_header Upgrade $http_upgrade; and proxy_set_header Connection "upgrade"; to the location / block in the Host's entry in Nginx Proxy Manager fixed the issue with the container consoles not working.

The resulting stuff that goes into the "Advanced" tab of the entry in Nginx Proxy Manager for my pve.mydomain.com host now looks like this:

#Proxmox
location /authelia {
    internal;
    set $upstream_authelia https://192.168.2.12:9091/api/verify; #ADD YOUR IP AND PORT OF AUTHELIA
    proxy_pass_request_body off;
    proxy_pass $upstream_authelia;
    proxy_set_header Content-Length "";
 
    # Timeout if the real server is dead
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
    client_body_buffer_size 128k;
    proxy_set_header Host $host;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Forwarded-Uri $request_uri;
    proxy_set_header X-Forwarded-Ssl on;
    proxy_redirect  http://  $scheme://;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_cache_bypass $cookie_session;
    proxy_no_cache $cookie_session;
    proxy_buffers 4 32k;
 
    send_timeout 5m;
    proxy_read_timeout 240;
    proxy_send_timeout 240;
    proxy_connect_timeout 240;
}
 
location / {
    set $upstream_Proxmox https://192.168.2.10:8006;  #CHANGE NAME AND IP AND PORT
    proxy_pass $upstream_Proxmox;  #change name of the service
    auth_request /authelia;
    auth_request_set $target_url $scheme://$http_host$request_uri;
    auth_request_set $user $upstream_http_remote_user;
    auth_request_set $groups $upstream_http_remote_groups;
    proxy_set_header Remote-User $user;
    proxy_set_header Remote-Groups $groups;
    proxy_set_header Access-Control-Allow-Origin *;
    error_page 401 =302 https://auth.mydomain.com/?rd=$target_url; #change YOURDOMAIN.COM to your domain

    client_body_buffer_size 128k;

    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

    send_timeout 5m;
    proxy_read_timeout 360;
    proxy_send_timeout 360;
    proxy_connect_timeout 360;
 
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Forwarded-Uri $request_uri;
    proxy_set_header X-Forwarded-Ssl on;
    proxy_set_header Upgrade $http_upgrade; #Problem Fixer Line 1
    proxy_set_header Connection "upgrade"; #Problem Fixer Line 2
    proxy_redirect  http://  $scheme://;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_cache_bypass $cookie_session;
    proxy_no_cache $cookie_session;
    proxy_buffers 64 256k;
 
    real_ip_header CF-Connecting-IP;
    real_ip_recursive on;
}

I have absolutely no idea why this worked, so if anyone wants to explain it, please feel free to.

When logged in as myname@Authelia, LXC Container consoles are supposed to work, but the node console is not. I found a workaround for this on the forums here. I used the script that @Games_Crack wrote:

Same thing as before but automated (creates a backup too), tested on pve 8.0

#!/bin/bash

# Define backup directory
backup_dir="$HOME/pve-backup"

# Create backup directory if it doesn't exist
mkdir -p "$backup_dir"

# Get current date and time
datetime=$(date +"%Y%m%d%H%M%S")

# Backup file
cp /usr/share/perl5/PVE/API2/Nodes.pm "$backup_dir/nodes.pm.$datetime-bak"

# Comment out lines containing "!= pam"
sed -i '/!= pam/s/^/# /' /usr/share/perl5/PVE/API2/Nodes.pm

echo "Backup created and lines commented out."

# Restart services
systemctl restart pvedaemon.service pveproxy.service

echo "Services restarted."

Everything is working as I want it to now.

When I go to pve.mydomain.com, I get intercepted by Authelia where I need to log in & use my second factor, Authelia then sends me to Proxmox, where I have the option to log in using OIDC, and then I can use the LXC Consoles as normal. If I try using the Node console after logging in using OIDC (where I will be logged in as myname@Authelia and not root@pam), then instead of throwing a 403 error, it will ask me to log in inside the console. I can use root as the ID and my root password, and the console works.

Theoretically, this is probably not "secure", but considering that to get to my Proxmox setup, you'd first need to either be on the local network or have Wireguard, and then you'd need to have a user account manually set up inside Authelia, I think I am okay with the risk. I hope that isn't too far unfounded.

 

[LordZNC]

Hello,

Thank you very much. (I created the account to reply this)

I want to add this firstly didn't solved my issue and even made my host appears "Offline" in NPM.
It was due to the proxmox entry > Advanced tab
I added in the end this:

        set_real_ip_from 10.0.0.0/8;
        set_real_ip_from 172.0.0.0/8;
        set_real_ip_from 192.168.0.0/16;
        set_real_ip_from fc00::/7;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;

This make the consle unreachable. So we need to have the exact same location as describe above by having this end:

    real_ip_header CF-Connecting-IP;
    real_ip_recursive on;

Have a good day.

 

[sarpuser]

Oh my god you are a legend!!! This is the 10th forum post I found about this and those two lines absolutely fixed it! Thank you so much!