LXC not starting after V5 to V6 upgrade using ZFS for storage

[mjoconr]

HI All

I just finished upgrading from V5 to V6 of proxmox and have an issue with LXC 's not starting.

The issue seems to be that the LXC is being started with out first mounting the ZFS subvolume.
This results in a dev directory being created which then means ZFS will not mount over anymore because there are files in the mount point

Mount does not work

root@pve:/rbd# zfs mount rbd/subvol-109-disk-0
cannot mount '/rbd/subvol-109-disk-0': directory is not empty

There is a directory created by the attempt to start the lXC

root@pve:/rbd# find /rbd/subvol-109*/
/rbd/subvol-109-disk-0/
/rbd/subvol-109-disk-0/dev

Remove the directory

rm -rf /rbd/subvol-109-disk-0/dev/

Mount the rbd volume

root@pve:/rbd# zfs mount rbd/subvol-109-disk-0

I can then start the LXC from the web page or via the cli.

Question:
What mounts the zfs subvol ?
Is this a ZFS issue of not mounting the subvol at boot ?
Should Proxmox be mounting the image ?
Should Proxmox not be checking its mounted before starting the LXC ?

I've been able to start the LXC but after a reboot, it seems I have to manual fix the mounts again.

Thanks

 

[mjoconr]

Just had to reboot and had the same problem. subvol not mounted before LXC tries to start the guests.
How do I fix this ?

 

[tonci]

Hello to all,
yes, I'm experiencing the same issue ... after upgrade to vers 6 , lxc datasets are not mounted ... directory is not empty etc...
After cleaning subvol directories , I can mount them manually and fire container up.
But reboot resets the situation back to the same ... no subvols are mounted .
There is one more strange thing ... I set nfs server on the host itself up. Created one dataset that is exported through nfs service in order to serve nfs backup server for the main host. This dataset is also gone after reboot. So rpool/data/nfs-export cannot be mounted because of the same reason ... dir is not empty ... after deletion of dump and template directories , I can mount it manually and must restart nfs service too ... but after that everything is up and running .
Everything mentioned above worked correctly on 5.x version ...

Thank you very much in advance for any solution/hint ...

BR
Tonci

 

[ramrot]

I would check if the systemd services are enabled for the zfs with systemctl list-unit-files. On my machines i have enabled:

zfs-import-cache.service                enabled
zfs-import-scan.service                 disabled
zfs-mount.service                       enabled
zfs-share.service                       enabled
zfs-zed.service                         enabled
zfs-import.target                       enabled
zfs.target                              enabled

Have you checked the journal for any error log messages? Does the journalctl -u zfs-mount.service ( or any other zfs service) indicating something about the issue you encounter?

 

[mjoconr]

I'm sorry I updated the Proxmox mailing list with the answer, but forgot to do the same for the forum

---
The issue was that I had a fstab entry which was doing a bind from
the zfs pool to a normal path location

This was causing the zfs mount service to fail because it thought the
zfs pool had files in it.

Removed this by changing the service to use the rbd directory directly
instead of a bind.
---

I suggest that tonci issue is the NFS path is inside of the zfs pool directory. Mount the NFS via proxmox storage configuration or make sure that the NFS mount point is outside of the zfs pool.

 

[tonci]

Yes, thank you both for your responses ... I hope I resolved it ... the problem was that I cleared only the content of lxc subvols (i.e. /rpool/data/subvol-xyz ) but not the content of the whole /rpool/ . As soon as I deleted the content of the /rpool/ , subvols were mounted correctly (tested through many reboots).
Regarding nfs ... I figured that out too I had kind of "mounting loop" (like you mentioned) ... I deleted the the same fstab line and dataset /rpool/nfs-server was finally ready to be exported through nfs-service for other hosts and , like you said, I mounted it via prox-storage-config as directory for local dumps ... But, I'm using the dataset within the /rpool/ for nfs-exports and it "works" (the question remains how it worked in 5.x ... or maybe something changed after upgrade)
... mjconor, I'm not sure what you meant by "make sure that nfs mount point is outside zfs pool" so I kindly ask for more details ?
.. ramrot, I did check the services you mentioned and they were all dead before correcting this mount issue and now they are all active

Thank you once more
BR
Tonci

 

[mjoconr]

NFS export of ZFS pool should be done via the in build nfs feature of ZFS (https://blog.programster.org/sharing-zfs-datasets-via-nfs), but if you mount an external file system inside of the zfs pool (rpool I think in your case) and this mounts before the pool is fully operational zfs will not mount the file system correctly getting the errors both of use had.

 

[ChESch]

I can confirm that the solution from tonci worked for me as well, I wrote everything that solved my issue right here!