Container Help

[Davidtt485]

First want to say thank you to this team, I was an ESXI user for a long time but the features and ability of Proxmox are amazing, Also I feel like I am learning Linux way more with this setup.

I have a 3 node cluster using a NAS for Shared storage, VM's and networking work amazing. I wanted to start using LXC containers but I am struggling with a weird error.

I create the CT fine, but it will not start, the first start prompts an error saying the disk cannot be found, but I do see it in the shared storage. If I resize the disk, it will start fine and work fine. If I shut it down it will not restart,

Host info:

CPU(s) 12 x AMD Ryzen 5 2600 Six-Core Processor (1 Socket)

Kernel Version Linux 5.13.19-1-pve #1 SMP PVE 5.13.19-3 (Tue, 23 Nov 2021 13:31:19 +0100)

PVE Manager Version pve-manager/7.1-6/4e61e21c

below is the log info I could find, any help or direction to what I am doing wrong would be amazing

Nov 25 09:16:29 Gryffindor systemd[1]: Started PVE LXC Container: 110.
Nov 25 09:16:30 Gryffindor kernel: loop0: detected capacity change from 0 to 16777216
Nov 25 09:16:30 Gryffindor kernel: EXT4-fs warning (device loop0): ext4_multi_mount_protect:326: MMP interval 42 higher than expected, please wait.

Nov 25 09:16:59 Gryffindor pveproxy[64443]: proxy detected vanished client connection
Nov 25 09:17:01 Gryffindor CRON[149122]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 25 09:17:01 Gryffindor CRON[149123]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 25 09:17:01 Gryffindor CRON[149122]: pam_unix(cron:session): session closed for user root
Nov 25 09:17:14 Gryffindor kernel: print_req_error: 18 callbacks suppressed
Nov 25 09:17:14 Gryffindor kernel: blk_update_request: I/O error, dev loop0, sector 0 op 0x1WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 25 09:17:14 Gryffindor kernel: Buffer I/O error on dev loop0, logical block 0, lost sync page write
Nov 25 09:17:14 Gryffindor kernel: EXT4-fs (loop0): I/O error while writing superblock
Nov 25 09:17:14 Gryffindor kernel: EXT4-fs (loop0): mount failed
Nov 25 09:17:14 Gryffindor pvestatd[1370]: unable to get PID for CT 110 (not running?)
Nov 25 09:17:14 Gryffindor pvedaemon[1397]: unable to get PID for CT 110 (not running?)
Nov 25 09:17:14 Gryffindor pvedaemon[149048]: startup for container '110' failed
Nov 25 09:17:14 Gryffindor pvedaemon[1396]: <root@pam> end task UPID:Gryffindor:00024638:0058852A:619F9ABD:vzstart:110:root@pam: startup for container '110' failed

 

[oguz]

hi,

could you post:
* the full pveversion -v output (just in case)
* the container config pct config CTID
* your storage config cat /etc/pve/storage.cfg
* container debug log pct start CTID --debug

(please use [code][/code] tags when posting)

I create the CT fine, but it will not start, the first start prompts an error saying the disk cannot be found, but I do see it in the shared storage. If I resize the disk, it will start fine and work fine. If I shut it down it will not restart,

do you have any problems with your shared storage? how fast are the read/write speeds in general with VMs on that storage?

 

[Davidtt485]

Shared Storage is NVME with 10GB networking on a separate network.

root@Gryffindor:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-6 (running version: 7.1-6/4e61e21c)
pve-kernel-5.13: 7.1-4
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
openvswitch-switch: 2.15.0+ds1-2
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.4-3
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

root@Gryffindor:~# pct config 110
arch: amd64
cores: 2
features: nesting=1
hostname: CT110
memory: 512
net0: name=eth0,bridge=vmbr1,firewall=1,hwaddr=AE:4A:AA:CB:E7:84,ip=dhcp,ip6=dhcp,tag=5,type=veth
ostype: ubuntu
rootfs: Proxmox:110/vm-110-disk-0.raw,size=8G
swap: 512
unprivileged: 1

root@Gryffindor:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

cifs: Proxmox
        path /mnt/pve/Proxmox
        server 192.168.7.10
        share Proxmox
        content snippets,vztmpl,images,iso,rootdir,backup
        domain Hogwarts
        prune-backups keep-last=3
        username Proxmox

root@Gryffindor:~# pct start 110 --debug
run_buffer: 316 Script exited with status 255
lxc_init: 816 Failed to run lxc.hook.pre-start for container "110"
__lxc_start: 2007 Failed to initialize container "110"
type g nsid 0 hostid 100000 range 65536
INFO     lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver AppArmor
INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "110", config section "lxc"
DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 110 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: can't read superblock on /dev/loop0.

DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 110 lxc pre-start produced output: command 'mount /dev/loop0 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

ERROR    conf - conf.c:run_buffer:316 - Script exited with status 255
ERROR    start - start.c:lxc_init:816 - Failed to run lxc.hook.pre-start for container "110"
ERROR    start - start.c:__lxc_start:2007 - Failed to initialize container "110"
INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "110", config section "lxc"
startup for container '110' failed

 

[oguz]

thanks. could you post the container configuration as well?

also, does it work if you:
* move the container disk to a local storage
or
* run pct fsck CTID

 

[Davidtt485]

thanks. could you post the container configuration as well?

also, does it work if you:
* move the container disk to a local storage
or
* run pct fsck CTID[/ic [/QUOTE]

I edited the post above to include the config, I pasted one of them twice by accident, also the pct fsck did work, quite well actually.

 

[oguz]

also the pct fsck did work, quite well actually.

great! does it keep working after a reboot of the container?

 

[Davidtt485]

great! does it keep working after a reboot of the container?

Once it is shutdown or poweroff it will not start again unless I do pct fsck again.

 

[oguz]

could you test using NFS instead of CIFS?

 

[Davidtt485]

could you test using NFS instead of CIFS?

CIFS is the issue, same Storage on the Same NAS using NFS works fine.

 

[oguz]

CIFS is the issue, same Storage on the Same NAS using NFS works fine.

thank you for confirming