Lxc start error- subvol-ID-disk -No such file or directory

P

penguin2020

Active Member
Jul 2, 2018
16
1
43
Hi everybady

I have been testing (in addition to being using it in production) an installation with ZFS in root and in the pools where I have the disks of my VM and CT

I have updated to the latest version in the laboratory (cluster with physical equipment for testing and development) and the only container I have, it is not possible to start, something happens like it cannot find the disk that is mounted (or should) in a zfs subvolume...
But if I create a new container, using the same storage, it is created fine, and it starts correctly
With the VM (qemu) there is no problem.
I think the issue is due to the ZFS subvolumes, that the system is not mounting them correctly, but I can not find the solution

Some data of the system and the container that does not start:

When wanting to start it:

TASK ERROR: zfs error: /saspool/containers/subvol-106-disk-0: No such file or directory

PVE Version

Code: 
root@pvedev1:~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.11-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-4
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-10
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-10
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-6
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-19
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

CT Configuration

Code: 
arch: amd64
cores: 4
hostname: odo
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.30.1,hwaddr=EE:EC:99:26:25:ED,ip=192.168.30.190/24,type=veth
ostype: debian
rootfs: containers:subvol-106-disk-0,size=128G
swap: 1024

Storage Configuration


Code: 
root@pvedev1:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,images,backup,vztmpl
        maxfiles 0
        shared 0

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

zfspool: vm-disk
        pool saspool/vm-disk
        content images
        mountpoint /saspool/vm-disk
        nodes pvedev2,pvedev1
        sparse 1

zfspool: containers
        pool saspool/containers
        content images,rootdir
        mountpoint /saspool/containers
        nodes pvedev2,pvedev1
        sparse 0

dir: dump-isos
        path /saspool/dump-isos
        content backup,iso,vztmpl,images,snippets
        maxfiles 3
        nodes pvedev2,pvedev1
        shared 0

nfs: nas_omv11
        export /export/NAS_omv11
        path /mnt/pve/nas_omv11
        server 10.0.3.3
        content images,iso,backup
        maxfiles 4

zfs list

Code: 
root@pvedev1:/saspool# zfs list -r -o name,mountpoint,mounted
NAME                                  MOUNTPOINT                             MOUNTED
rpool                                 /rpool                                     yes
rpool/ROOT                            /rpool/ROOT                                yes
rpool/ROOT/pve-1                      /                                          yes
rpool/data                            /rpool/data                                yes
saspool                               /saspool                                    no
saspool/containers                    /saspool/containers                         no
saspool/containers/subvol-106-disk-0  /saspool/containers/subvol-106-disk-0       no
saspool/dump-isos                     /saspool/dump-isos                          no
saspool/vm-disk                       /saspool/vm-disk                            no
saspool/vm-disk/vm-101-disk-0         -                                            -
saspool/vm-disk/vm-1010-disk-0        -                                            -
saspool/vm-disk/vm-102-disk-0         -                                            -
saspool/vm-disk/vm-1020-disk-0        -                                            -
saspool/vm-disk/vm-103-disk-0         -                                            -
saspool/vm-disk/vm-104-disk-0         -                                            -

File systems


Code: 
root@pvedev1:/# df -HT
Filesystem                 Type      Size  Used Avail Use% Mounted on
udev                       devtmpfs   17G     0   17G   0% /dev
tmpfs                      tmpfs     3.4G  9.6M  3.4G   1% /run
rpool/ROOT/pve-1           zfs       579G  5.0G  574G   1% /
tmpfs                      tmpfs      17G   66M   17G   1% /dev/shm
tmpfs                      tmpfs     5.3M     0  5.3M   0% /run/lock
tmpfs                      tmpfs      17G     0   17G   0% /sys/fs/cgroup
rpool                      zfs       574G  132k  574G   1% /rpool
rpool/ROOT                 zfs       574G  132k  574G   1% /rpool/ROOT
rpool/data                 zfs       574G  132k  574G   1% /rpool/data
/dev/fuse                  fuse       32M   33k   32M   1% /etc/pve
10.0.3.3:/export/NAS_omv11 nfs       2.0T  993G  976G  51% /mnt/pve/nas_omv11
tmpfs                      tmpfs     3.4G     0  3.4G   0% /run/user/0

pvesm status

Code: 
root@pvemgua1:~# pvesm status
Name              Type     Status           Total            Used       Available        %
containers     zfspool     active       946800896          937712       945863184    0.10%
dump-isos          dir     active       564781312         4847744       559933568    0.86%
local              dir     active       564781312         4847744       559933568    0.86%
local-zfs      zfspool     active       559933684              96       559933588    0.00%
nas_omv11          nfs     active      1921554432       969037824       952500224   50.43%
vm-disk        zfspool     active      1125358292       179495108       945863184   15.95%

lxc-start verbose

Code: 
root@pvedev1:~# lxc-start -n 106 -F -l DEBUG -o /tmp/lxc-106.log
lxc-start: 106: conf.c: run_buffer: 323 Script exited with status 2
lxc-start: 106: start.c: lxc_init: 797 Failed to run lxc.hook.pre-start for container "106"
lxc-start: 106: start.c: __lxc_start: 1896 Failed to initialize container "106"
lxc-start: 106: conf.c: run_buffer: 323 Script exited with status 1
lxc-start: 106: start.c: lxc_end: 964 Failed to run lxc.hook.post-stop for container "106"
lxc-start: 106: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 106: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

cat /tmp/lxc-106.log

Code: 
root@pvedev1:~# cat /tmp/lxc-106.log
lxc-start 106 20201111181813.127 INFO     lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor
lxc-start 106 20201111181813.128 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "106", config section "lxc"
lxc-start 106 20201111181813.755 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 106 lxc pre-start produced output: cannot open directory //saspool/containers/subvol-106-disk-0: No such file or directory

lxc-start 106 20201111181813.769 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 2
lxc-start 106 20201111181813.769 ERROR    start - start.c:lxc_init:797 - Failed to run lxc.hook.pre-start for container "106"
lxc-start 106 20201111181813.769 ERROR    start - start.c:__lxc_start:1896 - Failed to initialize container "106"
lxc-start 106 20201111181813.769 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "106", config section "lxc"
lxc-start 106 20201111181814.272 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "106", config section "lxc"
lxc-start 106 20201111181814.990 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 106 lxc post-stop produced output: umount: /var/lib/lxc/106/rootfs: not mounted

lxc-start 106 20201111181814.990 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 106 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/106/rootfs' failed: exit code 1

lxc-start 106 20201111181815.207 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 1
lxc-start 106 20201111181815.218 ERROR    start - start.c:lxc_end:964 - Failed to run lxc.hook.post-stop for container "106"
lxc-start 106 20201111181815.224 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 106 20201111181815.227 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start 106 20201111182737.869 INFO     lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor
lxc-start 106 20201111182737.869 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "106", config section "lxc"
lxc-start 106 20201111182738.605 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 106 lxc pre-start produced output: cannot open directory //saspool/containers/subvol-106-disk-0: No such file or directory

lxc-start 106 20201111182738.616 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 2
lxc-start 106 20201111182738.616 ERROR    start - start.c:lxc_init:797 - Failed to run lxc.hook.pre-start for container "106"
lxc-start 106 20201111182738.616 ERROR    start - start.c:__lxc_start:1896 - Failed to initialize container "106"
lxc-start 106 20201111182738.616 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "106", config section "lxc"
lxc-start 106 20201111182739.119 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "106", config section "lxc"
lxc-start 106 20201111182739.826 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 106 lxc post-stop produced output: umount: /var/lib/lxc/106/rootfs: not mounted

lxc-start 106 20201111182739.826 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 106 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/106/rootfs' failed: exit code 1

lxc-start 106 20201111182739.836 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 1
lxc-start 106 20201111182739.837 ERROR    start - start.c:lxc_end:964 - Failed to run lxc.hook.post-stop for container "106"
lxc-start 106 20201111182739.837 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 106 20201111182739.837 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options


I really appreciate your comments
 
Finally I was can start the container

Code: 
# rm -rf /saspool/containers/
# zfs mount saspool/containers/subvol-106-disk-0

this sounds like Proxmox wanted to mount it, but couldn't
Any ideas to fix it and keep it on every reboot?
 
Hello, dear friends of the community


since i upgrade from version 6.1-x to 6.2-x, my containers are not starting
I have created a new one, and when restarting, it does not start giving this error
Any idea what might be happening
Thanks in advance

Code: 
mount_entry: 1892 No such file or directory - Failed to mount "run" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/run"
lxc_setup: 3256 Failed to setup mount entries
do_start: 1224 Failed to setup container "107"
__sync_wait: 41 An error occurred in another process (expected sequence number 5)
__lxc_start: 1950 Failed to spawn container "107"
TASK ERROR: startup for container '107' failed
 
I'm in this situation

Code: 
root@pvemgua1:~# zfs list -o name,mounted
NAME                            MOUNTED
rpool                               yes
rpool/ROOT                          yes
rpool/ROOT/pve-1                    yes
rpool/data                          yes
saspool                              no
saspool/containers                   no
saspool/dump-isos                    no
saspool/vm-disk                      no
saspool/vm-disk/vm-101-disk-0         -
saspool/vm-disk/vm-1010-disk-0        -
saspool/vm-disk/vm-102-disk-0         -
saspool/vm-disk/vm-1020-disk-0        -
saspool/vm-disk/vm-103-disk-0         -
saspool/vm-disk/vm-104-disk-0         -
 
You must log in or register to reply here.
Top Bottom