DANGER: Missing fuse.ko in proxmox-kernel-6.8.12-9-pve Package (Bookworm, No-Subscription Repo)

[ihearvoices]

Missing fuse.ko in proxmox-kernel-6.8.12-9-pve Package (Bookworm, No-Subscription Repo)

Hi Proxmox Team and Community,

I've encountered a critical issue on a single-node Proxmox VE installation where the `pmxcfs` filesystem fails to mount `/etc/pve`. After extensive troubleshooting, the root cause appears to be that the required `fuse.ko` kernel module file is missing from the official kernel packages in the `pve-no-subscription` repository for Bookworm.

**Environment:**
* Proxmox VE Version: 8.4.0 base (pve-manager: 8.4.1, pve-cluster: 8.1.0)
* Running Kernel: `6.8.12-9-pve`
* Repository Used: `deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription`
* System Base: Debian 12 (Bookworm)

**Problem Description:**
The `pmxcfs` FUSE filesystem fails to mount on `/etc/pve`, rendering the Web UI inaccessible (SSL errors) and breaking many management commands (`qm status`, etc.). The `fuse` kernel module cannot be loaded (`modprobe fuse` fails) because the actual module file is missing.

**Evidence / Steps Taken:**
1. Confirmed `/etc/pve` is not mounted via `mount | grep /etc/pve`.
2. Confirmed `fuse` module is not loaded via `lsmod | grep fuse`.
3. Confirmed `fuse.ko` is missing via `ls /lib/modules/6.8.12-9-pve/kernel/fs/fuse/fuse.ko` (returns "No such file or directory").
4. Attempted `apt install --reinstall proxmox-kernel-6.8.12-9-pve-signed`. The command completed successfully, including running postinst scripts (initramfs, bootloader updates), but `fuse.ko` remained missing.
5. Manually downloaded both `proxmox-kernel-6.8.12-9-pve_6.8.12-9_amd64.deb` AND `proxmox-kernel-6.8.12-9-pve-signed_6.8.12-9_amd64.deb` directly from `http://download.proxmox.com/debian/pve/dists/bookworm/pve-no-subscription/binary-amd64/`.
6. Extracted the contents of both `.deb` files using `dpkg-deb -x`.
7. Verified using `ls -lR` on the extracted contents that the directory `./lib/modules/6.8.12-9-pve/kernel/fs/fuse/` exists within both packages, but it **does not contain the `fuse.ko` file**.

**Impact:**
The missing `fuse.ko` module prevents `pmxcfs` from mounting, leading to an unstable system where configuration cannot be reliably accessed, the Web UI is broken, and standard management is impossible. I am DEAD IN THE WATER.

**Request:**
Could the Proxmox team please investigate the `proxmox-kernel-6.8.12-9-pve` and `proxmox-kernel-6.8.12-9-pve-signed` packages (version `_6.8.12-9`) in the `pve-no-subscription` repository for Bookworm? It appears the `fuse.ko` module was omitted during packaging. A corrected package is needed.

Thank you for your time and assistance.

 

[leesteken]

2. Confirmed `fuse` module is not loaded via `lsmod | grep fuse`.
3. Confirmed `fuse.ko` is missing via `ls /lib/modules/6.8.12-9-pve/kernel/fs/fuse/fuse.ko` (returns "No such file or directory").

There is also no fuse.ko in kernel version 6.8.12-8-pve. This is not new. And kernel version 6.8.12-9-pve works fine on other systems.

**Impact:**
The missing `fuse.ko` module prevents `pmxcfs` from mounting, leading to an unstable system where configuration cannot be reliably accessed, the Web UI is broken, and standard management is impossible. I am DEAD IN THE WATER.

I'm sure your issues is not caused by fuse not being a module (it's probably built-in). We'll need to search for another reason.

1. Confirmed `/etc/pve` is not mounted via `mount | grep /etc/pve`.

There can be many reasons why /etc/pve is not mounted. Any error messages in the system log? What is the output of systemctl --failed?

 

[ihearvoices]

The missing fuse.ko was indeed a red herring. The root cause appears to be pmxcfs failing to initialize properly, likely due to issues communicating with corosync or accessing its configuration, potentially triggered or exacerbated by the incorrect ownership and state of the underlying /etc/pve directory. The service starts but is non-functional, leading to cascading "Connection refused" errors across other PVE components.

I'm continuing on this new track.
Thank you for your input!

 

[ihearvoices]

Hi everyone,

Just wanted to follow up and report that the issue with `pmxcfs` failing to mount `/etc/pve` on my Proxmox 8.4.1 node (Kernel `6.8.12-9-pve`) has been **resolved**.

A big thank you to **@leesteken** for pointing out that the missing `fuse.ko` file was likely a red herring, as FUSE is built into this kernel version. You were absolutely correct.

The actual root cause turned out to be the **underlying physical directory `/etc/pve` itself**. For reasons unknown, it had incorrect group ownership (`root:www-data`) and an anomalous timestamp (`Dec 31 1969`). Crucially, even with `pmxcfs` not mounted and services stopped, attempting to `sudo chown root:root /etc/pve` failed with "Operation not permitted". This indicated some form of corruption or inconsistent state preventing modification of the directory itself.

The fix involved the following steps (performed via SSH):

1. Stopped `pve-cluster` and `corosync` services:
* `sudo systemctl stop pve-cluster.service`
* `sudo systemctl stop corosync.service`
2. Renamed the problematic directory:
* `sudo mv /etc/pve /etc/pve.old` (This succeeded where `chown` failed)
3. Created a new, clean directory:
* `sudo mkdir /etc/pve`
* `sudo chown root:root /etc/pve`
* `sudo chmod 755 /etc/pve`
4. Restarted the services:
* `sudo systemctl start corosync.service`
* `sudo systemctl start pve-cluster.service`

After these steps, `pve-cluster` started cleanly without the previous initialization errors (`cpg_initialize failed`, `cmap_initialize failed`, etc.), and `pmxcfs` successfully mounted `/etc/pve` via FUSE (`mount | grep /etc/pve` confirmed).

The Web UI is now fully accessible and correctly displays the node, VMs, storage, etc., indicating the configuration data in `/var/lib/pve-cluster/config.db` was intact and is now being read correctly.

Interestingly, while the Web UI is working, the command-line tools (`qm list`, `pct list`) are currently still showing empty lists immediately after the fix. I suspect this is a minor synchronization delay or inconsistency, and I plan to perform a host reboot shortly to fully resolve it.

Thanks again for the help! Hopefully, this information helps someone else who might run into issues where `pmxcfs` fails to mount despite the FUSE kernel capability being present. The state of the underlying physical `/etc/pve` directory can be critical.

-LGV