lxcfs.service Failed to start FUSE filesystem for LXC.

level3

New Member
Oct 24, 2021
2
2
1
35
Hi,

I have problem with new proxmox host, lxcfs.service is unable to start on it and I can't figure out why.

It is new installation where I connect old disks and tried to copy pct config files and container volumes using rsync, however once I finished I've noticed that containers are not able to start.

So far everything pointing out to lxcfs.service, but I can't figure it out why it is not starting.

Code:
 systemctl status lxcfs.service
● lxcfs.service - FUSE filesystem for LXC
     Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-03-08 21:41:15 CET; 7min ago
       Docs: man:lxcfs(1)
    Process: 22684 ExecStart=/usr/bin/lxcfs /var/lib/lxcfs (code=exited, status=1/FAILURE)
    Process: 22685 ExecStopPost=/bin/fusermount -u /var/lib/lxcfs (code=exited, status=1/FAILURE)
   Main PID: 22684 (code=exited, status=1/FAILURE)
        CPU: 2ms

Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Scheduled restart job, restart counter is at 5.
Mar 08 21:41:15 h7 systemd[1]: Stopped FUSE filesystem for LXC.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Start request repeated too quickly.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: Failed to start FUSE filesystem for LXC.


Code:
 journalctl -u lxcfs.service | tail -n 50
Mar 08 21:41:15 h7 lxcfs[22682]: - sys_cpu_online
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_cpuinfo
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_diskstats
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_loadavg
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_meminfo
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_stat
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_swaps
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_uptime
Mar 08 21:41:15 h7 lxcfs[22682]: - shared_pidns
Mar 08 21:41:15 h7 lxcfs[22682]: - cpuview_daemon
Mar 08 21:41:15 h7 lxcfs[22682]: - loadavg_daemon
Mar 08 21:41:15 h7 lxcfs[22682]: - pidfds
Mar 08 21:41:15 h7 lxcfs[22682]: fuse: device not found, try 'modprobe fuse' first
Mar 08 21:41:15 h7 lxcfs[22682]: Running destructor lxcfs_exit
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 21:41:15 h7 fusermount[22683]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Scheduled restart job, restart counter is at 4.
Mar 08 21:41:15 h7 systemd[1]: Stopped FUSE filesystem for LXC.
Mar 08 21:41:15 h7 systemd[1]: Started FUSE filesystem for LXC.
Mar 08 21:41:15 h7 lxcfs[22684]: Running constructor lxcfs_init to reload liblxcfs
Mar 08 21:41:15 h7 lxcfs[22684]: mount namespace: 5
Mar 08 21:41:15 h7 lxcfs[22684]: hierarchies:
Mar 08 21:41:15 h7 lxcfs[22684]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Mar 08 21:41:15 h7 lxcfs[22684]: Kernel supports pidfds
Mar 08 21:41:15 h7 lxcfs[22684]: Kernel supports swap accounting
Mar 08 21:41:15 h7 lxcfs[22684]: api_extensions:
Mar 08 21:41:15 h7 lxcfs[22684]: - cgroups
Mar 08 21:41:15 h7 lxcfs[22684]: - sys_cpu_online
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_cpuinfo
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_diskstats
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_loadavg
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_meminfo
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_stat
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_swaps
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_uptime
Mar 08 21:41:15 h7 lxcfs[22684]: - shared_pidns
Mar 08 21:41:15 h7 lxcfs[22684]: - cpuview_daemon
Mar 08 21:41:15 h7 lxcfs[22684]: - loadavg_daemon
Mar 08 21:41:15 h7 lxcfs[22684]: - pidfds
Mar 08 21:41:15 h7 lxcfs[22684]: fuse: device not found, try 'modprobe fuse' first
Mar 08 21:41:15 h7 lxcfs[22684]: Running destructor lxcfs_exit
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 21:41:15 h7 fusermount[22685]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Scheduled restart job, restart counter is at 5.
Mar 08 21:41:15 h7 systemd[1]: Stopped FUSE filesystem for LXC.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Start request repeated too quickly.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: Failed to start FUSE filesystem for LXC.

Code:
 pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-7
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
openvswitch-switch: 2.15.0+ds1-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

What can be the problem?
 
seems like I found issue, root partition was mounted two times from new and old disks:

root@h7:/var/log# zfs list
NAME USED AVAIL REFER MOUNTPOINT
<snip>
DATA2/ROOT/pve-1 18.4G 6.24T 14.3G /
<snip>
rpool/ROOT/pve-1 1.81G 877G 1.81G /
<snip>

fixed with:
zfs set mountpoint=legacy DATA2/ROOT/pve-1

which disable automount for old root partition
 
seems like I found issue, root partition was mounted two times from new and old disks:

root@h7:/var/log# zfs list
NAME USED AVAIL REFER MOUNTPOINT
<snip>
DATA2/ROOT/pve-1 18.4G 6.24T 14.3G /
<snip>
rpool/ROOT/pve-1 1.81G 877G 1.81G /
<snip>

fixed with:
zfs set mountpoint=legacy DATA2/ROOT/pve-1

which disable automount for old root partition
Thank you! I had the same problem and almost thought I f***** my system.

For anyone googling, I had this errors:

lxc_spawn: 1731 Operation not permitted - Failed to clone a new set of namespaces__lxc_start: 2107 Failed to spawn container "101"

/var/lib/lxcfs/proc/: No such file or directory

Also simply a double mount on /
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!