ZFS Mounting Problems

C

c0mputerking

Renowned Member
Oct 5, 2011
174
5
83
Hello all i have some ongoing issues with ZFS not always mounting properly, after reboot.

I tried some full ZFS (system installed to rpool) installs awhile back but this setup seemed to unstable for me grub errors and disks not found all that sort of thing. Anyways for now i have moved to an ext4 root system installed from the lastest 3.4-102d4547-6 ISO after which I setup some ZFS pools for VZs/VMs and backups.

I have updated to latest version of pve hoping ZFS might mount properly but it still does not here are my version infos

root@pve:~# pveversion -v
proxmox-ve-2.6.32: 3.4-163 (running kernel: 2.6.32-41-pve)
pve-manager: 3.4-11 (running version: 3.4-11/6502936f)
pve-kernel-2.6.32-40-pve: 2.6.32-160
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-41-pve: 2.6.32-163
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-19
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

Anyways i get errors in the boot logs (full log is attached) about pools not mounting because the directories are not empty as seen below, as it is not convient to reboot this machine right now i cannot show everything i have tried, however below is the latest reboot, after which i shutdown into single user mode and deleted the directories created by pve (dump images and that sort of thing)

see https://pve.proxmox.com/wiki/Storage:_ZFS#ZFS_mounting_workaround

Maybe this is a problem because i am mixing ext4 lvm root with ZFS pools, and i am reluctant to mess with the /etc/insserv.conf and /etc/ as it has caused me problems with system updates, but i am willing to try if someone assures me it might help :) Anyways this is quite confusing/scary to have to boot into single user mode deleting stuff. At this point im not sure if i have stuff mounted on top of other stuff or what exactly is goin on please help. NOTE i have attached some more infos to hopefully help get this all sorted and will provide any further information requested

Wed Sep 23 15:41:24 2015: Starting vzeventd:
Wed Sep 23 15:41:24 2015: Starting enhanced syslogd: rsyslogd.
Wed Sep 23 15:41:24 2015: Starting deferred execution scheduler: atd.
Wed Sep 23 15:41:24 2015: Starting NTP server: ntpd.
Wed Sep 23 15:41:24 2015: Starting OpenBSD Secure Shell server: sshd.
Wed Sep 23 15:41:24 2015: Mounting ZFS filesystem(s) cannot mount '/apool': directory is not empty
Wed Sep 23 15:41:24 2015: cannot mount '/bpool': directory is not empty
Wed Sep 23 15:41:24 2015: 1 ... failed!
Wed Sep 23 15:41:26 2015: Starting VMs and Containers
Wed Sep 23 15:41:26 2015: Starting VM 100
Wed Sep 23 15:41:30 2015: Starting CT 101
Wed Sep 23 15:41:32 2015: Starting CT 101 failed: command 'vzctl start 101' failed: exit code 62
Wed Sep 23 15:41:32 2015: Starting CT 102
Wed Sep 23 15:41:35 2015: Starting CT 102 failed: command 'vzctl start 102' failed: exit code 62
Wed Sep 23 15:41:35 2015: Starting CT 105
Wed Sep 23 15:41:38 2015: Starting CT 105 failed: command 'vzctl start 105' failed: exit code 62
Wed Sep 23 15:41:38 2015: Starting VM 106
Wed Sep 23 15:41:42 2015: Starting CT 109
Wed Sep 23 15:41:45 2015: Starting CT 109 failed: command 'vzctl start 109' failed: exit code 62
Wed Sep 23 15:41:45 2015: nohup: redirecting stderr to stdout
 

Attachments

  • bootlog.txt
    4.6 KB · Views: 0
  • mount.txt
    4.3 KB · Views: 3
  • zfslist.txt
    1 KB · Views: 1
  • df.txt
    2.7 KB · Views: 0
This is a long standing Proxmox bug (my definition)/feature (maybe their definition :) ) when the storage plugin is crazy in re-creating the folder structure. If the plugin starts before ZFS mounts, then this problem shows its ugly head. Even more, it looks like it is a race, because it hapens randomly to me (even in 4b2 with systemd).

From my point of view, the best solution is to maybe create the structure at first when the storage endpoint is added and that's it. If somebody messes with it, just throw errors instead of keep retrying to "fix" it all the time.
 
I agree this is a bug, caused by a feature, and is very problematic, causing additional downtime having to resort to IMPI to reboot my servers. However recently i have have discovered the -O (overlay) option for the zfs mount command. Yet i am unsure how safe the overlay mount option is or if it would be proper to implement it at boot time, or if it is even something that should be considered?

PS i have already had to revert from zol root ie rpool and have my root partition on cumbersome old LVM/ext4 due to some boot loader/grub issues, and i do not take my systems back to even more cumbersome and unsupported by the Proxmox team Software RAID/LVM/ext4 ... argh.
 
Hmm this is super annoying problem, and causes me alot of grief, is there really no solution?
 
c0mputerking said:
Hmm this is super annoying problem, and causes me alot of grief, is there really no solution?
Click to expand...

I think I ran into that when I was trying to use zfs with proxmox. The problem (as I recall) is that proxmox insists on populating the directory with the requisite subdirectories, but then the zfs mount fails because the directory is not empty :( I seem to recall working around this by doing a zfs force mount -a in /etc/rc.local or some such.
 
I think the real problem is that you use a 'dir' type storage instead of a 'zfspool' for ZFS. The zfspool plugin
does not even try to create those subdirs.
 
@dietmar, can we disable directory structure auto-creation? It is not needed to do this check and fix periodically. If this is done once when the "dir" endpoint is added in the UI, all of these issues will be gone.
 
Necro-ing without shame :eek:

dietmar said:
Yes, by using a 'zfspool' instead of a 'dir' type storage.
Click to expand...
But what if the storage content needs to be `ISO image, Container template` ?
Those are unavailable for ZFS type storage...
 
Tommmii said:
Necro-ing without shame :eek:


But what if the storage content needs to be `ISO image, Container template` ?
Those are unavailable for ZFS type storage...
Click to expand...

then you can create a separate dataset for this, and configure it as "directory" storage with "mountpoint: 1" to prevent PVE from creating the directories before the dataset got mounted.
 
fabian said:
then you can create a separate dataset for this,
Click to expand...
?like so :
Code: 
root@pve:/# zfs list
NAME                                          USED  AVAIL     REFER  MOUNTPOINT
zfs-pool                                      500G   153G      244K  /zfs-pool
zfs-pool/iso                                 31.0G   153G     31.0G  /zfs-pool/iso

Then, where do i add `mountpoint: 1` ? (Assuming that I already have a dir type storage, and indeed it is creating the directories before the mounting of dataset... this is the issue i'm trying to work around atm.)
 
Tommmii said:
?like so :
Code: 
root@pve:/# zfs list
NAME                                          USED  AVAIL     REFER  MOUNTPOINT
zfs-pool                                      500G   153G      244K  /zfs-pool
zfs-pool/iso                                 31.0G   153G     31.0G  /zfs-pool/iso

Then, where do i add `mountpoint: 1` ? (Assuming that I already have a dir type storage, and indeed it is creating the directories before the mounting of dataset... this is the issue i'm trying to work around atm.)
Click to expand...

you configure a directory storage with '/zfs-pool/iso' as directory, and set "mountpoint:1" in /etc/pve/storage.cfg (for that storage definition).
 
partial of /etc/pve/storage.cfg :
Code: 
dir: zfs-iso
        path /zfs-pool/iso
        content iso,vztmpl
        shared 0
        mountpoint: 1

rebooting now ...
 
nope...
fabian said:
"mountpoint:1" in /etc/pve/storage.cfg
Click to expand...
the directory structure still gets created before the zfs pool get mounted.
Back to the drawing board for me i guess...
Doesn't work either :
Code: 
dir: zfs-iso
        path /zfs-pool/iso
        content iso,vztmpl
        shared 0
        is_mountpoint 1
        mkdir 0
 
Last edited:
yes, sorry it is "is_mountpoint". the above should work - are you sure it's not earlier left-over directories that you are seeing? could you post your full storage.cfg and zfs list output?
 
Code: 
root@pve:/# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content rootdir,iso
        maxfiles 0
        shared 0
lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images
zfspool: zfs-containers
        pool zfs-pool
        content rootdir
        sparse 0
zfspool: vm-disks
        pool zfs-pool/vm-disks
        content images
        sparse 1
dir: usb-backup
        path /mnt/usb-backup
        content backup
        maxfiles 1
        shared 0
dir: zfs-iso
        path /zfs-pool/iso
        content iso,vztmpl
        shared 0
        is_mountpoint 1
        mkdir 0
root@pve:/#
Code: 
root@pve:/# zfs list
NAME                                          USED  AVAIL     REFER  MOUNTPOINT
zfs-pool                                      500G   153G      244K  /zfs-pool
zfs-pool/iso                                 31.0G   153G     31.0G  /zfs-pool/iso
zfs-pool/share                                121G   153G      121G  /zfs-pool/share
zfs-pool/subvol-100-disk-0                   3.22G  4.96G     3.04G  /zfs-pool/subvol-100-disk-0
zfs-pool/subvol-102-disk-0                   1.19G  6.81G     1.19G  /zfs-pool/subvol-102-disk-0
zfs-pool/subvol-103-disk-0                   1.57G  6.84G     1.16G  /zfs-pool/subvol-103-disk-0
zfs-pool/subvol-104-disk-0                   1.10G  7.22G      797M  /zfs-pool/subvol-104-disk-0
zfs-pool/subvol-106-disk-0                   1.76G  6.24G     1.76G  /zfs-pool/subvol-106-disk-0
zfs-pool/subvol-300-disk-0                   3.02G  5.54G     2.46G  /zfs-pool/subvol-300-disk-0
zfs-pool/vm-disks                             336G   153G      140K  /zfs-pool/vm-disks
zfs-pool/vm-disks/vm-101-disk-0              24.3G   153G     24.3G  -
zfs-pool/vm-disks/vm-105-disk-0              14.1G   153G     13.3G  -
zfs-pool/vm-disks/vm-105-state-gdrivebackup  5.88G   153G     5.88G  -
zfs-pool/vm-disks/vm-200-disk-0              8.19G   153G     6.05G  -
zfs-pool/vm-disks/vm-200-state-stable         728M   153G      728M  -
zfs-pool/vm-disks/vm-201-disk-0              25.2G   153G     25.2G  -
zfs-pool/vm-disks/vm-206-disk-0              74.0G   153G     46.6G  -
zfs-pool/vm-disks/vm-333-disk-0              57.3G   153G     57.3G  -
zfs-pool/vm-disks/vm-334-disk-1              76.1G   153G     76.1G  -
zfs-pool/vm-disks/vm-335-disk-0              46.8G   153G     31.7G  -
zfs-pool/vm-disks/vm-335-state-preupdate     3.58G   153G     3.58G  -
root@pve:/#
I don't think there's lingering dirs in the mountpoint, because :
Code: 
root@pve:/# ls -lha /zfs-pool/
total 90K
drwxr-xr-x 12 root root   12 Aug  5 15:21 .
drwxr-xr-x 23 root root 4.0K Oct 10  2018 ..
drwxr-xr-x  3 root root    3 Jun 25 14:12 iso
drwxr-xr-x  4 root root    4 Mar 23 13:43 share
drwxr-xr-x 21 root root   21 Aug  9 13:13 subvol-100-disk-0
drwxr-xr-x 21 root root   21 Aug  8 10:15 subvol-102-disk-0
drwxr-xr-x 22 root root   22 Aug  4 13:37 subvol-103-disk-0
drwxr-xr-x 21 root root   21 Jun 22 11:06 subvol-104-disk-0
drwxr-xr-x 22 root root   22 Aug  9 13:13 subvol-106-disk-0
drwxr-xr-x 22 root root   22 Aug  8 10:16 subvol-300-disk-0
drwxr-xr-x  4 root root    4 Oct 10  2018 template
drwxr-xr-x  2 root root    2 Oct 10  2018 vm-disks
root@pve:/# mount --bind / /mnt/tmp
root@pve:/# ls -lha /mnt/tmp/zfs-pool/
total 8.0K
drwxr-xr-x  2 root root 4.0K Aug  9 13:01 .
drwxr-xr-x 23 root root 4.0K Oct 10  2018 ..
root@pve:/# umount /mnt/tmp
Also, it isn't that i'm seeing dirs in the mount point, but after a reboot no containers start & i see :
Code: 
root@pve:~# zfs list -r -o name,mountpoint,mounted
NAME                                         MOUNTPOINT                   MOUNTED
zfs-pool                                     /zfs-pool                         no
zfs-pool/iso                                 /zfs-pool/iso                     no
zfs-pool/share                               /zfs-pool/share                   no
zfs-pool/subvol-100-disk-0                   /zfs-pool/subvol-100-disk-0       no
zfs-pool/subvol-102-disk-0                   /zfs-pool/subvol-102-disk-0       no
zfs-pool/subvol-103-disk-0                   /zfs-pool/subvol-103-disk-0       no
zfs-pool/subvol-104-disk-0                   /zfs-pool/subvol-104-disk-0       no
zfs-pool/subvol-106-disk-0                   /zfs-pool/subvol-106-disk-0       no
zfs-pool/subvol-300-disk-0                   /zfs-pool/subvol-300-disk-0       no
zfs-pool/vm-disks                            /zfs-pool/vm-disks                no
zfs-pool/vm-disks/vm-101-disk-0              -                                  -
zfs-pool/vm-disks/vm-105-disk-0              -                                  -
zfs-pool/vm-disks/vm-105-state-gdrivebackup  -                                  -
zfs-pool/vm-disks/vm-200-disk-0              -                                  -
zfs-pool/vm-disks/vm-200-state-stable        -                                  -
zfs-pool/vm-disks/vm-201-disk-0              -                                  -
zfs-pool/vm-disks/vm-206-disk-0              -                                  -
zfs-pool/vm-disks/vm-333-disk-0              -                                  -
zfs-pool/vm-disks/vm-334-disk-1              -                                  -
zfs-pool/vm-disks/vm-335-disk-0              -                                  -
zfs-pool/vm-disks/vm-335-state-preupdate     -                                  -
 
Last edited:
but that indicates that not mounting /zfs-pool is the problem.
 
fabian said:
but that indicates that not mounting /zfs-pool is the problem.
Click to expand...
indeed, at first i thought it didn't mount because it there was a dir structure in the mountpoint (/zfs-pool), which i tried to overlay with ""zfs mount -O -a".
But now, I have emptied mountpoint, and it still won't auto mount, but it does mount with a simple "zfs mount -a" from console after boot.
If it matters, this started only after upgrade to 6. No other changes.
 
yikes...that's above my capabilities.
Is this already pushed in upgrades ?

And...why would the pool not mount at reboot? I know there's a fix for when the pool is on root, but this isn't the case here .
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back