Different folder content when bind mounted in a LXC

[brewno]

Hi,

I'm pulling my hair out here. I spent a few days trying to bind mount a folder containing subfolders and files into a Privileged LXC. I'll try my best to explain what I think is happening.

I'm running Cockpit in a Privileged LXC and bind-mounted an Immich folder that contains other subfolders like library, upload, thumb, etc. These were created when I used to run Immich in TrueNAS, which I'm not using anymore. This is the LXC.conf.


When I ls into them from Proxmox, I see the folders that Immich uses, and in particular, the library folder that has two subfolders for two users I've created containing other folders containing .jpegs.

When I view Immich_shared folder in Cockpit I see the same folders I have in /tank/Immich but they are empty:

but the strange thing is the library folder, which is different from what I see in Proxmox:


Now I might know why this is happening - I managed to run Immich in Portainer once and set the upload path to /var/lib/docker/volumes/tank/Immich/library which was pointing to my /tank/Immich in my zfs.
To test the app, I uploaded one image for testing it, but soon I realized that I should not have included the /library at the end of the path, as this created a folder in /Immich called libraryand inside that when uploading something it created library,uploads,thumbs etc... folders.
I think this has overwritten the existing /Immich/library folder in my pool, although I only see this when mounting it in a LXC. (Cockpit in this case)
I've deleted Portainer LXC altogether, but I'm still seeing that image I uploaded if I look in the thumbs folder in Cockpit. (and this happens whenever I try to bind mount the same folder in different LXCs)
[/FONT]

This is probably some kind of cache issue but my questions are:

• How do I clean the cache? (If that's the issue)
• Why the other folders in the LXC are empty i.e uploads?

I hope this makes sense

Thanks in advance.

 

[Lantis]

I just want to say that I am having the same or a very similar problem.
The files are not synchronized between the host directory and the bind mount, it seems that they are 2 different file systems. I tried removing and readding the bind mount but it doesn't work: When the bind mount is not there, the directory is empty (good) but when I readd it, it shows different files.

Other bind mount points work properly.

 

[fabian]

please post "mount" from inside the container and from the host..

also note: https://bugzilla.proxmox.com/show_bug.cgi?id=2762 and https://bugzilla.proxmox.com/show_bug.cgi?id=3493

 

[brewno]

please post "mount" from inside the container and from the host..

also note: https://bugzilla.proxmox.com/show_bug.cgi?id=2762 and https://bugzilla.proxmox.com/show_bug.cgi?id=3493

Thank for replying, do you mean this:

 

[fabian]

yeah - as you can see, you have multiple nested mountpoints, but the bind mount just passes a single one into the container. you need to define bind mounts for all the mountpoints you want to pass in (and also, you seem to have "hidden" directories and files underneath those nested mount points that you might want to clean up)

 

[brewno]

Thanks for the explanation.
So yeah bind mounting each folder in /Immich works!
I'm not sure which "hidden" files you're referring to tho, I noticed tho that for example /Nextcloud it also has many folders inside but from the host I don't see them while for /Immich I see the sub-folders... is that what you're referring to?
Does that mean I need to remove the ones in /Immich so I only see the parent folder?

 

[fabian]

I am referring to your screenshots when you only had the top-level dir bind-mounted. you showed that the container has "different contents". that's not true, you just don't see those contents on the PVE host because they are "shadowed" by the nested mountpoints. probably you (or some application) created them at some point when the nested mountpoints didn't yet exist, or at least, were not mounted.

e.g., think about the following situation:

/A/B/C/D
/X/Y/Z

if you now mount /X/Y into /A/B yo will only see:

/A/B/Z
/X/Y/Z

but t hat doesn't mean that the original C/D sub dir of /A/B is not still there, you just no longer see it.

 

[brewno]

I am referring to your screenshots when you only had the top-level dir bind-mounted. you showed that the container has "different contents". that's not true, you just don't see those contents on the PVE host because they are "shadowed" by the nested mountpoints. probably you (or some application) created them at some point when the nested mountpoints didn't yet exist, or at least, were not mounted.

e.g., think about the following situation:

/A/B/C/D
/X/Y/Z

if you now mount /X/Y into /A/B yo will only see:

/A/B/Z
/X/Y/Z

but t hat doesn't mean that the original C/D sub dir of /A/B is not still there, you just no longer see it.

Understood

But how can I revert that?
I ran zfs unmount /tankImmich/library and all the other subfolders and when I run mount that's the result:


I no longer see the subfolders but still when checking in Cockpit can't see the content of each subfolder. This doesn't happen with /Nextcloud for example which it seems now the same, at least for me

 

[fabian]

well your Nextcloud dataset doesn't have any sub-datasets/mountpoints

you can just do a bind mount on the host into some (empty) dir of your choice if you want to see the "hidden" contents.

 

[brewno]

I'm sorry to be such a pain but I don't understand. It seems to me that now neither /tank/Immich or /tank/Nextcloud have sub-datasets/mountpoints?

Update: So after running the zfs unmount I couldn't see any sub-datasets under /tank/Immich but after I restarted Proxmox they're back. I'm getting even more confused

 

[Lantis]

Thanks for the help.
My problem was solved, I also had nested mount points.

 

[fabian]

I'm sorry to be such a pain but I don't understand. It seems to me that now neither /tank/Immich or /tank/Nextcloud have sub-datasets/mountpoints?

Update: So after running the zfs unmount I couldn't see any sub-datasets under /tank/Immich but after I restarted Proxmox they're back. I'm getting even more confused

maybe let's start with the following (please as text in [code] tags, not as screenshot)

zfs list -o name,mountpoint,canmount

and the question - what do you want to achieve?

do you want to get rid of your nested datasets/mountpoints? then just rename the (child) dataset or change its mountpoint, copy the things you want to keep to the "parent" and you are done.

do you want to clean up the hidden dirs and files? then just bind-mount the parent somewhere else and clean it up.

do you want something else? then please say what, hopefully we can help you get there

 

[brewno]

maybe let's start with the following (please as text in [code] tags, not as screenshot)
zfs list -o name,mountpoint,canmount

and the question - what do you want to achieve?

do you want to get rid of your nested datasets/mountpoints? then just rename the (child) dataset or change its mountpoint, copy the things you want to keep to the "parent" and you are done.

do you want to clean up the hidden dirs and files? then just bind-mount the parent somewhere else and clean it up.

do you want something else? then please say what, hopefully we can help you get there

Sorry for the slow reply but I think I understood now.

What I wanted was to remove the nested datasets and only keep the parent and be able to view all nested content whenever I bind mounted into an LXC.
So your first suggestion worked perfectly I had to ask for help AI for the commands as I'm new to ZFS and Linux in general, but I got there eventually