Setup Proxmox to use NAS for all Node storage?

[Quincy]

Hi,

I rebuilt my Proxmox instance due to /dev/sda3 running at 93% usage with 4 LV's, plus I somehow lost my previous documentation on how/why I set Proxmox up so I thought a fresh start would work, before I reinstalled Proxmox backed up my container hosting Plex to my NAS which I had running under Datacenter > Storage > Cifs
Screenshot1.

With this freshly built Proxmox I checked my LVM and /dev/sda3 is at 93 usage with 3 LV's and i`m not sure why?

I have re-added my NAS via Datacenter > Storage > Cifs and I can access it via Datacenter > NodeName > NAS > Content which shows the backup container and a few templates.

I've restored the container but it fails to start due to the following error "Job for pve-container@100.service failed because the control process exited with error code." i think this is related to the /dev/sda3 being nearly full but i`m not sure.

So, would it be possible to use my NAS as storage for the whole Node by adding it to /dev/sda3? if so will any issues arise from me doing that?

 

[oguz]

hi,

So, would it be possible to use my NAS as storage for the whole Node by adding it to /dev/sda3? if so will any issues arise from me doing that?

i'd suggest against that, but you can store your disk images for the VMs in there (just choose the nas for storage when you're creating the VM, or move your current VM disks from local to there).

I've restored the container but it fails to start due to the following error "Job for pve-container@100.service failed because the control process exited with error code." i think this is related to the /dev/sda3 being nearly full but i`m not sure.

is likely a different problem. following would be useful for debug:

-> pct config CTID
-> pveversion -v
-> debug logs from container start [0]

[0]: https://pve.proxmox.com/pve-docs/chapter-pct.html#_obtaining_debugging_logs

 

[Quincy]

Hi,

I read these articles ended up removing my LVM-Thin storage as it wasnt in use, this ended up freeing the disk space on /dev/sda3 but my main issue of being unable to run containers still exists
#
https://forum.proxmox.com/threads/i-cant-start-lxc.41378/
https://forum.proxmox.com/threads/need-help-troubleshooting-lxc-not-starting-issue.45455/
https://forum.proxmox.com/threads/cant-start-lxc-container.45588/

Somehow by running that debug command I can now run that container but I can't connect to it via IP(but i can ping it from another PC on the same network, Ive set up a new container and re-ran the debug command

lxc-start -n 100 -F -l DEBUG -o /tmp/lxc-ID.log
lxc-start: 100: conf.c: run_buffer: 352 Script exited with status 5
lxc-start: 100: start.c: lxc_init: 897 Failed to run lxc.hook.pre-start for container "100"
lxc-start: 100: start.c: __lxc_start: 2032 Failed to initialize container "100"
Segmentation fault

cat /tmp/lxc-100.log
lxc-start 100 20200306183105.838 INFO     confile - confile.c:set_config_idmaps:2003 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20200306183105.838 INFO     confile - confile.c:set_config_idmaps:2003 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20200306183105.838 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "keyctl errno 38"
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for keyctl action 327718(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for keyctl action 327718(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for keyctl action 327718(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for keyctl action 327718(errno)
lxc-start 100 20200306183105.838 INFO     seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 100 20200306183105.839 INFO     conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20200306183150.683 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: can't read superblock on /dev/loop0.

lxc-start 100 20200306183150.737 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: command 'mount /dev/loop0 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 100 20200306183150.745 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 255
lxc-start 100 20200306183150.745 ERROR    start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20200306183150.745 ERROR    start - start.c:__lxc_start:2032 - Failed to initialize container "100"

root@Node:~# pct config 100
arch: amd64
cores: 4
hostname: Plex
memory: 7629
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=FA:0C:95:72:B7:C8,ip=192.168.1.113/24,type=veth
ostype: ubuntu
rootfs: NasPi:100/vm-100-disk-2.raw,size=60G
swap: 2048
unprivileged: 1

root@Node:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-5
pve-kernel-helper: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.6-pve1
ceph-fuse: 14.2.6-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-12
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-6
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-5
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[oguz]

hi,

i see in your log: lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: can't read superblock on /dev/loop0.

which is usually indicative of i/o errors or corrupted volume.

i also see that this raw volume is on your NasPi.

to rule out problem with NFS, you could move that volume to a local storage (CT -> Resources -> Select Disk -> Move Volume) and see if it works.

if not then it might actually be i/o problem or corrupted.

can you run other containers fine?

 

[Quincy]

Hi Oguz,

I`m not running NFS, my NasPi is using CIFS. Its a raspberry Pi 3B running Samba with a 8TB hard drive connected via USB.

Before I updated I could run containers fine but now they are a bit tricky in the sense that I have to run the container > watch it fail > run pct fsck 10X > then start it up again. but if i need to run that same container again or restart the container, I have to go through the whole motion again.

Running VM's is fine for some reason.

The problem isn't related to CIFS is it? running my network storage on a Pi with 8TB is only a stop gap till i can afford a Synology or Qnap.

I get the following error when i attempt to move the container to local

()
rootfs
local-lvm
TASK ERROR: lvcreate 'pve/vm-100-disk-0' error:   Pool data not found in Volume group pve.

I think the above error is due to me removing lvm-thin as I wasnt using it and it was taking up sda3 space (see top post)

I`m also unable to backup to the NAS, error below:

ERROR: Backup of VM 100 failed - command 'mount /dev/loop1 /mnt/vzsnap0//' failed: exit code 32

I used to be able to with this version of Proxmox so not sure why its throwing that error now

 

[Quincy]

is the main issue, using CIFS as my main storage
Would i have the same issues if I were using a synology nas and NFS?

 

[oguz]

since the rootfs of the container is stored on CIFS this means that I/O errors that cause boot to fail are likely because of some problem with the network connection or the NasPi itself. that's what i'd try to debug in your case

 

[Quincy]

okay, I dont think its the network connection as its directly connect to my dumb switch and i've never had any network issues with any other device.
i'll look into the NasPi itself, any pointers for where i should check first?

 

[cokyc]

Hi,
have you resolved your problem?
I am not able to run a container with rootfs in a cifs storage. I can confirm the problem is not related to rp3 share since I have the same with a synology nas.
No issues running VMs from cifs storage.

I have to run pct fsck ctid each time before start the container.
Also pct mount fail with these errors:

pct mount 105
mount: /var/lib/lxc/105/rootfs: can't read superblock on /dev/loop0.
mounting container failed
command 'mount /dev/loop0 /var/lib/lxc/105/rootfs//' failed: exit code 32

 

[Quincy]

Hi,

My Proxmox instance is stable, for now, what i have done is:

1. Reinstalled Proxmox
2. Recreated containers using local storage
3. Once each container was installed and configured i powered them off then the server, turned server back on and containers work without any issues

I haven't added my rp3 back yet as i'm not sure if its causing the issue, will do some testing later.

i'm just going to stick to not updating proxmox and having the containers storage be on local/local-lvm while i use my NAS for container data i.e. movies that my Plex container will use

I do have the suspicion that i corrupted my containers due to improper shutdown of the server/NAS as my NasPi only appears to spin its drive when i access a resource on it