proxmox ve zfs does not mount on boot

E

everwake

New Member
Aug 10, 2013
3
0
1
installed proxmox and created a few VM's running on ZFS, worked fine for a couple of reboots, but all of a sudden it stopped mounting the zfs filesystem on boot.

seems pvestatd is creating files in the pool, so when ZFS tries to mount it fails with:
cannot mount '/proxmox1_local_ssd/Containers': directory is not empty

same for all filesystems

the only workaround I found was:

zpool export proxmox1_local_ssd
rm -r /proxmox1_local_ssd
zpool import proxmox1_local_ssd
zfs mount -O proxmox1_local_ssd/Images​
​
​
zfs mount -O proxmox1_local_ssd/Containers
/etc/rc3.d/S22pvestad restart

taht way I can manually start my vm's again.

any idea why it stopped working and what I need to do to get in working again?


​
​
​


 
AFAIR there is a limited number/list of FS supported by OpenVZ-Containers....ZFS is definitely not on that list...IMHO you'll need to migrate to KVM for your VMs.
 
Must say that I fail to see what OpenVZ has do do with my problem?

My issue is that pvestatd for some reason creates the folder structure in the very same place zfs tries to mount its filesystems, thereby causing the mount to fail.

therefore after boot the system "looks" ok , since the filestructure is there, but when you check " zfs get mounted proxmox1_local_ssd/Images" you will se that the actual filesystem is not mounted at all.

If there is any way to force zfs to mount all filesystems at boot with the "-O" overlay flag my problem would go away, or better yet, make sure pvestatd waits until all filesystems are mounted.
will try legacy mounts, to see if pvestatd works better with those
 
I've used zfs on proxmox at home for 3 years.

in /etc/rc.local :
Code: 
# need to make it so zfs will mount even if pve made folders in /data and /bkup
# so put in to rc.local:
zfs mount -O -a

BTW - do you know what to do with a kernel upgrade before rebooting?
 
thank you :p

I did a workaround of setting all filesystems to "mountpoint=legacy" and mounting them via fstab instead, as long as you remember to manually create all mountpoint dirs that also works, but you solution is the one I was looking for.

dont't understand the question about kernel upgrade though (new to proxmox and
Zol)
RobFantini said:
I've used zfs on proxmox at home for 3 years.

in /etc/rc.local :
Code: 
# need to make it so zfs will mount even if pve made folders in /data and /bkup
# so put in to rc.local:
zfs mount -O -a

BTW - do you know what to do with a kernel upgrade before rebooting?
Click to expand...
 
everwake said:
Must say that I fail to see what OpenVZ has do do with my problem?

My issue is that pvestatd for some reason creates the folder structure in the very same place zfs tries to mount its filesystems, thereby causing the mount to fail.

therefore after boot the system "looks" ok , since the filestructure is there, but when you check " zfs get mounted proxmox1_local_ssd/Images" you will se that the actual filesystem is not mounted at all.

If there is any way to force zfs to mount all filesystems at boot with the "-O" overlay flag my problem would go away, or better yet, make sure pvestatd waits until all filesystems are mounted.
will try legacy mounts, to see if pvestatd works better with those
Click to expand...

...thought you were using ZFS *inside* the VM and based on your mount point you gave away, that you were using "containers", AKA OpenVZ.

Still not sure what ypour ZFS organisation is.
For ZFS...you do not use a pool ("root" zfs-folder) directly...create ZFS folders instead.
So if any process wants to write data, it should go into a zfs folder....
If any process interferes until the FS gets mounted, try and change startup sequences ?.
 
Its not big deal to fix it. First make zfs-mount starts before pvestatd.

My runlevel.conf
# cat /etc/runlevel.conf | grep zfs-mount
01 0,1,6 - /etc/init.d/zfs-mount
01 - 2,3,4,5 /etc/init.d/zfs-mount

# cat /etc/runlevel.conf | grep pvestatd
02 0,1,6 - /etc/init.d/pvestatd
07 - 2,3,4,5 /etc/init.d/pvestatd
Click to expand...

To fix on running system: turn off pvestatd -> remove sub folders in mount catalog -> mount zfs -> turn on pvestatd


EDIT: I`m using ZFS for openvz and KVM. Who need openvz disk usage to work can use zvol.
 
Last edited:
why don't you put to wiki how to do that with a copy and paste command? or put commands here and I'll put to wiki.
 
Last edited:
Hm I just tried install fresh proxmox 3.0 with latest updates into KVM. After install debian-zfs and set up /etc/default/zfs to mount on startup everything is working OK. What about auto creating catalogs I can say pvestatd is not the only one who is responsible for this. Pveproxy creates catalogs too.

So if you are in trouble with zfs mount on live system do it:

1. Unmount all zfs volumes
Code: 
zfs unmount -a

or part of zfs subvolumes
Code: 
zfs unmount your_pool/subvolume

2. Stop pvestatd and pveproxy services
Code: 
service pvestatd stop
service pveproxy stop

3. Go to mount point and remove directories and files (make sure you are not deleting needed files)
Code: 
cd /mount/point ; rm * -R

4. Mount zfs
Code: 
zfs mount -a

5. Start pvestatd and pveproxy services
Code: 
service pvestatd start
service pveproxy start



BOOT startup mount fix (or at least way to check is everything OK)

1. Make sure zfs is mount at startup.

a. Using /etc/default/zfs settings
Code: 
# Run `zfs mount -a` during system start?
# This should be 'no' if zfs-mountall or a systemd generator is available.
ZFS_MOUNT='yes'

b. Maybe you are using /etc/fstab (google)

c. Using mountall ? (google)

2. Startup system

a. sysv-rc

Code: 
# ls  /etc/rc2.d/ -l
total 4
-rw-r--r-- 1 root root 677 Feb  8  2013 README
lrwxrwxrwx 1 root root  18 May 23 12:41 S01bootlogs -> ../init.d/bootlogs
lrwxrwxrwx 1 root root  14 May 23 12:41 S01motd -> ../init.d/motd
[B]lrwxrwxrwx 1 root root  19 Aug 13 10:59 S01zfs-mount -> ../init.d/zfs-mount[/B]
lrwxrwxrwx 1 root root  17 Aug 13 10:13 S14rpcbind -> ../init.d/rpcbind
lrwxrwxrwx 1 root root  20 Aug 13 10:13 S15nfs-common -> ../init.d/nfs-common
lrwxrwxrwx 1 root root  18 Aug 13 10:13 S17ksmtuned -> ../init.d/ksmtuned
lrwxrwxrwx 1 root root  19 Aug 13 10:13 S17rrdcached -> ../init.d/rrdcached
lrwxrwxrwx 1 root root  17 Aug 13 10:13 S17rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root  18 Aug 13 10:14 S17vzeventd -> ../init.d/vzeventd
lrwxrwxrwx 1 root root  19 Aug 13 10:59 S17zfs-share -> ../init.d/zfs-share
lrwxrwxrwx 1 root root  13 Aug 13 10:13 S18atd -> ../init.d/atd
lrwxrwxrwx 1 root root  13 Aug 13 10:13 S18ntp -> ../init.d/ntp
lrwxrwxrwx 1 root root  17 Aug 13 10:13 S18postfix -> ../init.d/postfix
lrwxrwxrwx 1 root root  21 Aug 13 10:13 S18pve-cluster -> ../init.d/pve-cluster
lrwxrwxrwx 1 root root  15 Aug 13 10:13 S18rsync -> ../init.d/rsync
lrwxrwxrwx 1 root root  13 Aug 13 10:13 S18ssh -> ../init.d/ssh
lrwxrwxrwx 1 root root  14 Aug 13 10:13 S19cman -> ../init.d/cman
lrwxrwxrwx 1 root root  14 Aug 13 10:13 S19cron -> ../init.d/cron
lrwxrwxrwx 1 root root  14 Aug 13 10:14 S20clvm -> ../init.d/clvm
lrwxrwxrwx 1 root root  19 Aug 13 10:14 S21pvedaemon -> ../init.d/pvedaemon
lrwxrwxrwx 1 root root  21 Aug 13 10:14 S21qemu-server -> ../init.d/qemu-server
lrwxrwxrwx 1 root root  12 Aug 13 10:14 S21vz -> ../init.d/vz
[B]lrwxrwxrwx 1 root root  18 Aug 13 10:14 S22pveproxy -> ../init.d/pveproxy[/B]
[B]lrwxrwxrwx 1 root root  18 Aug 13 10:14 S22pvestatd -> ../init.d/pvestatd[/B]
lrwxrwxrwx 1 root root  19 Aug 13 10:14 S22rgmanager -> ../init.d/rgmanager
lrwxrwxrwx 1 root root  21 Aug 13 10:14 S23pve-manager -> ../init.d/pve-manager
lrwxrwxrwx 1 root root  19 Aug 13 10:14 S24pvebanner -> ../init.d/pvebanner
lrwxrwxrwx 1 root root  18 Aug 13 10:14 S24rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root  19 Aug 13 10:14 S24rmnologin -> ../init.d/rmnologin
lrwxrwxrwx 1 root root  23 Aug 13 10:14 S24stop-bootlogd -> ../init.d/stop-bootlogd

b. file-rc

Code: 
# cat runlevel.conf 
# This file was automatically generated by /usr/share/file-rc/insserv2file.sh.
# You can use update-rc.d(8) to modify it.  Do not edit by hand
# or else your changes will be lost next time update-rc.d is run.
# Read runlevel.conf(5) man page for more information about this file.
#
# Format:
# <sort> <off-> <on-levels>     <command>
01    0,1,6    -        /etc/init.d/atd
01    0,1,6    -        /etc/init.d/ksmtuned
01    0,1,6    -        /etc/init.d/postfix
01    0,1,6    -        /etc/init.d/pve-manager
01    0,1,6    -        /etc/init.d/rgmanager
01    0,1,6    -        /etc/init.d/rrdcached
01    0,1,6    -        /etc/init.d/zfs-mount
01    0,1,6    -        /etc/init.d/zfs-share
01    0,6    -        /etc/init.d/urandom
01    -    1,2,3,4,5        /etc/init.d/bootlogs
01    -    1,2,3,4,5        /etc/init.d/motd
01    -    1        /etc/init.d/killprocs
[B]01    -    2,3,4,5        /etc/init.d/zfs-mount[/B]
01    -    S        /etc/init.d/mountkernfs.sh
02    0,1,6    -        /etc/init.d/pveproxy
02    0,1,6    -        /etc/init.d/pvestatd
02    0,1,6    -        /etc/init.d/qemu-server
02    0,1,6    -        /etc/init.d/vz
02    -    1        /etc/init.d/single
02    -    S        /etc/init.d/udev
03    0,1,6    -        /etc/init.d/pvedaemon
03    0,1,6    -        /etc/init.d/vzeventd
03    -    S        /etc/init.d/mountdevsubfs.sh
04    0,1,6    -        /etc/init.d/clvm
04    -    S        /etc/init.d/bootlogd
05    0,1,6    -        /etc/init.d/cman
05    -    S        /etc/init.d/keymap.sh
06    0,1,6    -        /etc/init.d/pve-cluster
06    0,1,6    -        /etc/init.d/umountiscsi.sh
06    -    S        /etc/init.d/hostname.sh
06    -    S        /etc/init.d/hwclock.sh
06    -    S        /etc/init.d/lvm2
07    0,1,6    -        /etc/init.d/open-iscsi
07    -    S        /etc/init.d/checkroot.sh
08    0,6    -        /etc/init.d/sendsigs
08    -    S        /etc/init.d/checkroot-bootclean.sh
08    -    S        /etc/init.d/kmod
08    -    S        /etc/init.d/mtab.sh
08    -    S        /etc/init.d/pvenetcommit
09    0,1,6    -        /etc/init.d/rsyslog
09    -    S        /etc/init.d/checkfs.sh
10    0,6    -        /etc/init.d/umountnfs.sh
10    -    S        /etc/init.d/mountall.sh
11    0,1,6    -        /etc/init.d/nfs-common
11    0,1,6    -        /etc/init.d/rpcbind
11    -    S        /etc/init.d/mountall-bootclean.sh
12    0,6    -        /etc/init.d/hwclock.sh
12    0,6    -        /etc/init.d/networking
12    -    S        /etc/init.d/procps
12    -    S        /etc/init.d/udev-mtab
12    -    S        /etc/init.d/urandom
13    0,6    -        /etc/init.d/umountfs
13    -    S        /etc/init.d/networking
14    0,6    -        /etc/init.d/umountroot
14    -    2,3,4,5,S        /etc/init.d/rpcbind
15    0    -        /etc/init.d/halt
15    -    2,3,4,5,S        /etc/init.d/nfs-common
15    6    -        /etc/init.d/reboot
16    -    S        /etc/init.d/mountnfs.sh
17    -    2,3,4,5        /etc/init.d/ksmtuned
17    -    2,3,4,5        /etc/init.d/rrdcached
17    -    2,3,4,5        /etc/init.d/rsyslog
17    -    2,3,4,5        /etc/init.d/vzeventd
17    -    2,3,4,5        /etc/init.d/zfs-share
17    -    S        /etc/init.d/mountnfs-bootclean.sh
18    -    2,3,4,5        /etc/init.d/atd
18    -    2,3,4,5        /etc/init.d/ntp
18    -    2,3,4,5        /etc/init.d/postfix
18    -    2,3,4,5        /etc/init.d/pve-cluster
18    -    2,3,4,5        /etc/init.d/rsync
18    -    2,3,4,5        /etc/init.d/ssh
18    -    S        /etc/init.d/console-screen.sh
19    -    2,3,4,5        /etc/init.d/cman
19    -    2,3,4,5        /etc/init.d/cron
19    -    S        /etc/init.d/bootmisc.sh
19    -    S        /etc/init.d/open-iscsi
19    -    S        /etc/init.d/x11-common
20    -    2,3,4,5        /etc/init.d/clvm
20    -    S        /etc/init.d/stop-bootlogd-single
21    -    2,3,4,5        /etc/init.d/pvedaemon
21    -    2,3,4,5        /etc/init.d/qemu-server
21    -    2,3,4,5        /etc/init.d/vz
[B]22    -    2,3,4,5        /etc/init.d/pveproxy[/B]
[B]22    -    2,3,4,5        /etc/init.d/pvestatd[/B]
22    -    2,3,4,5        /etc/init.d/rgmanager
23    -    2,3,4,5        /etc/init.d/pve-manager
24    -    2,3,4,5        /etc/init.d/pvebanner
24    -    2,3,4,5        /etc/init.d/rc.local
24    -    2,3,4,5        /etc/init.d/rmnologin
24    -    2,3,4,5        /etc/init.d/stop-bootlogd
# THE LAST LINE IS NEVER READ
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back