LXC not starting after V5 to V6 upgrade using ZFS for storage

mjoconr

Active Member
Dec 5, 2009
74
1
28
HI All

I just finished upgrading from V5 to V6 of proxmox and have an issue with LXC 's not starting.

The issue seems to be that the LXC is being started with out first mounting the ZFS subvolume.
This results in a dev directory being created which then means ZFS will not mount over anymore because there are files in the mount point

Mount does not work
Code:
root@pve:/rbd# zfs mount rbd/subvol-109-disk-0
cannot mount '/rbd/subvol-109-disk-0': directory is not empty

There is a directory created by the attempt to start the lXC
Code:
root@pve:/rbd# find /rbd/subvol-109*/
/rbd/subvol-109-disk-0/
/rbd/subvol-109-disk-0/dev

Remove the directory
Code:
rm -rf /rbd/subvol-109-disk-0/dev/

Mount the rbd volume
Code:
root@pve:/rbd# zfs mount rbd/subvol-109-disk-0

I can then start the LXC from the web page or via the cli.

Question:
What mounts the zfs subvol ?
Is this a ZFS issue of not mounting the subvol at boot ?
Should Proxmox be mounting the image ?
Should Proxmox not be checking its mounted before starting the LXC ?

I've been able to start the LXC but after a reboot, it seems I have to manual fix the mounts again.

Thanks
 

tonci

Active Member
Jun 26, 2010
58
2
28
Hello to all,
yes, I'm experiencing the same issue ... after upgrade to vers 6 , lxc datasets are not mounted ... directory is not empty etc...
After cleaning subvol directories , I can mount them manually and fire container up.
But reboot resets the situation back to the same ... no subvols are mounted .
There is one more strange thing ... I set nfs server on the host itself up. Created one dataset that is exported through nfs service in order to serve nfs backup server for the main host. This dataset is also gone after reboot. So rpool/data/nfs-export cannot be mounted because of the same reason ... dir is not empty ... after deletion of dump and template directories , I can mount it manually and must restart nfs service too ... but after that everything is up and running .
Everything mentioned above worked correctly on 5.x version ...

Thank you very much in advance for any solution/hint ...

BR
Tonci
 

ramrot

New Member
Aug 2, 2019
28
1
3
33
I would check if the systemd services are enabled for the zfs with systemctl list-unit-files. On my machines i have enabled:
Code:
zfs-import-cache.service                enabled
zfs-import-scan.service                 disabled
zfs-mount.service                       enabled
zfs-share.service                       enabled
zfs-zed.service                         enabled
zfs-import.target                       enabled
zfs.target                              enabled

Have you checked the journal for any error log messages? Does the journalctl -u zfs-mount.service ( or any other zfs service) indicating something about the issue you encounter?
 

mjoconr

Active Member
Dec 5, 2009
74
1
28
I'm sorry I updated the Proxmox mailing list with the answer, but forgot to do the same for the forum

---
The issue was that I had a fstab entry which was doing a bind from
the zfs pool to a normal path location

This was causing the zfs mount service to fail because it thought the
zfs pool had files in it.

Removed this by changing the service to use the rbd directory directly
instead of a bind.
---

I suggest that tonci issue is the NFS path is inside of the zfs pool directory. Mount the NFS via proxmox storage configuration or make sure that the NFS mount point is outside of the zfs pool.
 

tonci

Active Member
Jun 26, 2010
58
2
28
Yes, thank you both for your responses ... I hope I resolved it ... the problem was that I cleared only the content of lxc subvols (i.e. /rpool/data/subvol-xyz ) but not the content of the whole /rpool/ . As soon as I deleted the content of the /rpool/ , subvols were mounted correctly (tested through many reboots).
Regarding nfs ... I figured that out too:) I had kind of "mounting loop" (like you mentioned) ... I deleted the the same fstab line and dataset /rpool/nfs-server was finally ready to be exported through nfs-service for other hosts and , like you said, I mounted it via prox-storage-config as directory for local dumps ... But, I'm using the dataset within the /rpool/ for nfs-exports and it "works" (the question remains how it worked in 5.x ... or maybe something changed after upgrade)
... mjconor, I'm not sure what you meant by "make sure that nfs mount point is outside zfs pool" so I kindly ask for more details ?
.. ramrot, I did check the services you mentioned and they were all dead :) before correcting this mount issue and now they are all active

Thank you once more
BR
Tonci
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!