lxcfs.service Failed to start FUSE filesystem for LXC.

level3

New Member
Oct 24, 2021
2
2
1
36
Hi,

I have problem with new proxmox host, lxcfs.service is unable to start on it and I can't figure out why.

It is new installation where I connect old disks and tried to copy pct config files and container volumes using rsync, however once I finished I've noticed that containers are not able to start.

So far everything pointing out to lxcfs.service, but I can't figure it out why it is not starting.

Code:
 systemctl status lxcfs.service
● lxcfs.service - FUSE filesystem for LXC
     Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-03-08 21:41:15 CET; 7min ago
       Docs: man:lxcfs(1)
    Process: 22684 ExecStart=/usr/bin/lxcfs /var/lib/lxcfs (code=exited, status=1/FAILURE)
    Process: 22685 ExecStopPost=/bin/fusermount -u /var/lib/lxcfs (code=exited, status=1/FAILURE)
   Main PID: 22684 (code=exited, status=1/FAILURE)
        CPU: 2ms

Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Scheduled restart job, restart counter is at 5.
Mar 08 21:41:15 h7 systemd[1]: Stopped FUSE filesystem for LXC.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Start request repeated too quickly.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: Failed to start FUSE filesystem for LXC.


Code:
 journalctl -u lxcfs.service | tail -n 50
Mar 08 21:41:15 h7 lxcfs[22682]: - sys_cpu_online
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_cpuinfo
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_diskstats
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_loadavg
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_meminfo
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_stat
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_swaps
Mar 08 21:41:15 h7 lxcfs[22682]: - proc_uptime
Mar 08 21:41:15 h7 lxcfs[22682]: - shared_pidns
Mar 08 21:41:15 h7 lxcfs[22682]: - cpuview_daemon
Mar 08 21:41:15 h7 lxcfs[22682]: - loadavg_daemon
Mar 08 21:41:15 h7 lxcfs[22682]: - pidfds
Mar 08 21:41:15 h7 lxcfs[22682]: fuse: device not found, try 'modprobe fuse' first
Mar 08 21:41:15 h7 lxcfs[22682]: Running destructor lxcfs_exit
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 21:41:15 h7 fusermount[22683]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Scheduled restart job, restart counter is at 4.
Mar 08 21:41:15 h7 systemd[1]: Stopped FUSE filesystem for LXC.
Mar 08 21:41:15 h7 systemd[1]: Started FUSE filesystem for LXC.
Mar 08 21:41:15 h7 lxcfs[22684]: Running constructor lxcfs_init to reload liblxcfs
Mar 08 21:41:15 h7 lxcfs[22684]: mount namespace: 5
Mar 08 21:41:15 h7 lxcfs[22684]: hierarchies:
Mar 08 21:41:15 h7 lxcfs[22684]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Mar 08 21:41:15 h7 lxcfs[22684]: Kernel supports pidfds
Mar 08 21:41:15 h7 lxcfs[22684]: Kernel supports swap accounting
Mar 08 21:41:15 h7 lxcfs[22684]: api_extensions:
Mar 08 21:41:15 h7 lxcfs[22684]: - cgroups
Mar 08 21:41:15 h7 lxcfs[22684]: - sys_cpu_online
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_cpuinfo
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_diskstats
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_loadavg
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_meminfo
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_stat
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_swaps
Mar 08 21:41:15 h7 lxcfs[22684]: - proc_uptime
Mar 08 21:41:15 h7 lxcfs[22684]: - shared_pidns
Mar 08 21:41:15 h7 lxcfs[22684]: - cpuview_daemon
Mar 08 21:41:15 h7 lxcfs[22684]: - loadavg_daemon
Mar 08 21:41:15 h7 lxcfs[22684]: - pidfds
Mar 08 21:41:15 h7 lxcfs[22684]: fuse: device not found, try 'modprobe fuse' first
Mar 08 21:41:15 h7 lxcfs[22684]: Running destructor lxcfs_exit
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 21:41:15 h7 fusermount[22685]: /bin/fusermount: failed to unmount /var/lib/lxcfs: Invalid argument
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Scheduled restart job, restart counter is at 5.
Mar 08 21:41:15 h7 systemd[1]: Stopped FUSE filesystem for LXC.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Start request repeated too quickly.
Mar 08 21:41:15 h7 systemd[1]: lxcfs.service: Failed with result 'exit-code'.
Mar 08 21:41:15 h7 systemd[1]: Failed to start FUSE filesystem for LXC.

Code:
 pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-12
pve-kernel-5.13: 7.1-7
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
openvswitch-switch: 2.15.0+ds1-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-6
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

What can be the problem?
 
seems like I found issue, root partition was mounted two times from new and old disks:

root@h7:/var/log# zfs list
NAME USED AVAIL REFER MOUNTPOINT
<snip>
DATA2/ROOT/pve-1 18.4G 6.24T 14.3G /
<snip>
rpool/ROOT/pve-1 1.81G 877G 1.81G /
<snip>

fixed with:
zfs set mountpoint=legacy DATA2/ROOT/pve-1

which disable automount for old root partition
 
seems like I found issue, root partition was mounted two times from new and old disks:

root@h7:/var/log# zfs list
NAME USED AVAIL REFER MOUNTPOINT
<snip>
DATA2/ROOT/pve-1 18.4G 6.24T 14.3G /
<snip>
rpool/ROOT/pve-1 1.81G 877G 1.81G /
<snip>

fixed with:
zfs set mountpoint=legacy DATA2/ROOT/pve-1

which disable automount for old root partition
Thank you! I had the same problem and almost thought I f***** my system.

For anyone googling, I had this errors:

lxc_spawn: 1731 Operation not permitted - Failed to clone a new set of namespaces__lxc_start: 2107 Failed to spawn container "101"

/var/lib/lxcfs/proc/: No such file or directory

Also simply a double mount on /