What's the difference between creating an LXC on a storage of type directory with a specific path vs. a storage type ZFS without a path?

[backdoc]

I'm still very confused by Proxmox's use of storage. I have probably made it more complicated it by importing a pool that already existed. Since importing, I have created 2 LXC containers on a storage type "ZFS" and 1 was created on a storage type "directory" (both created on the Datacenter level). I noticed when looked on the command line, the 2 ZFS types are mounted to the filesystem and the directory type is just a subdirectory of an existing dataset on the pool. I didn't do these differently intentionally. I've just been experimenting trying to understand the storage concepts.

One of these containers is intended to be a samba server for media, backups and whatever else I need to dump there from other computers on my network. So, I can't predict the final size. Does the storage type (ZFS vs. directory) have an impact on this? I have 6 4TB drives in a single pool. I'd like for the samba server LXC to be able to use as much of that as it needs.

Also, I've noticed that I can browse files that existed on the pool before I imported the pool from the command line. But, I don't see how to access them otherwise. What's the strategy if I wanted to make those available to my containers?

 

[Dunuin]

I'm still very confused by Proxmox's use of storage. I have probably made it more complicated it by importing a pool that already existed. Since importing, I have created 2 LXC containers on a storage type "ZFS" and 1 was created on a storage type "directory" (both created on the Datacenter level). I noticed when looked on the command line, the 2 ZFS types are mounted to the filesystem and the directory type is just a subdirectory of an existing dataset on the pool. I didn't do these differently intentionally. I've just been experimenting trying to understand the storage concepts.

When using a ZFS storage each LXC will create its own ZFS dataset and a dataset is not just a folder, its a complete filesystem. What your LXC can do and what not also depends on the storage you choose. ZFS for example support snapshots. A "Directory storage" for example don't support snapshots (or only for VMs when using qcow2 as format) and is just a folder on any filesystem.
See here what storage type support what features: https://pve.proxmox.com/wiki/Storage#_storage_types

One of these containers is intended to be a samba server for media, backups and whatever else I need to dump there from other computers on my network. So, I can't predict the final size. Does the storage type (ZFX vs. directory) have an impact on this? I have 6 4TB drives in a single pool. I'd like for the samba server LXC to be able to use as much of that as it needs.

If both are pointing to the same pool both should be using thin-provisioning in case you enabled that for your ZFS pool.

Also, I've noticed that I can browse files that existed on the pool before I imported the pool from the command line. But, I don't see how to access them otherwise. What's the strategy if I wanted to make those available to my containers?

Can you give some examples?

 

[backdoc]

First of all, thanks for your reply. I've been resisting asking. But, I can't seem to figure it out via searching online. I'm quite experienced with Linux, a little familiar with ZFS since I've been running a FreeNAS box for many years. But, I'm totally new to some of the concepts with Proxmox. So, thanks for your patience.

When using a ZFS storage each LXC will create its own ZFS dataset and a dataset is not just a folder, its a complete filesystem. What your LXC can do and what not also depends on the storage you choose. ZFS for example support snapshots. A "Directory storage" for example don't support snapshots (or only for VMs when using qcow2 as format) and is just a folder on any filesystem.
See here what storage type support what features: https://pve.proxmox.com/wiki/Storage#_storage_types

I have read that. I'm just having trouble digesting it. It sounds like ZFS is probably preferred.

If both are pointing to the same pool both should be using thin-provisioning in case you enabled that for your ZFS pool.

I see that I did not have thin provisioning selected. From what I understand, it sounds like, if you have thin provisioning selected, the container sets a "max" storage size when it is created. So, if I want all of the available storage available to my samba server, would I still have to set a container size close to what I expect it to be? My other 2 containers aren't likely to grow at all. One is a vpn gateway and one is hobby nginx web server. Most likely, t's not going to grow by more than a few kilobytes.

Can you give some examples?

Sure. Let me see if I can make this make sense. If I go to my Proxmox shell and run mount, I see one of my datasets that I had created on the Pool in TrueNAS. My pool name is "p0". The dataset is TIME_MACHINE. The directory under it is "backdoc". I can see all of that if I do an ls.

root@proxmox:~# ls /p0/TIME_MACHINE/backdoc/
'Cindy’s Mac mini (2).sparsebundle'  'Backdoc’s MacBook Air.sparsebundle'

But, I don't see those files in Proxmox anywhere and if I wanted to make that available to my samba server, I'm not sure how I'd do that.

 

[LnxBil]

It sounds like ZFS is probably preferred.

Yes, it is superior to anything else in the context of LX(C) containers.

But, I don't see those files in Proxmox anywhere and if I wanted to make that available to my samba server, I'm not sure how I'd do that.

Yes, the sparsebundle can IMHO only be opened directly from MacOS, so you cannot access those Mac files on any system other then MacOS. In MacOS thse sparsebundle is mounted and then accessed, so you have another layer of filesystem on top of that.

 

[backdoc]

Yes, it is superior to anything else in the context of LX(C) containers.

Okay. If I use thin provisioning on my ZFS storage types, will the LXC containers be allowed to use all of the available space in the pool?

Yes, the sparsebundle can IMHO only be opened directly from MacOS, so you cannot access those Mac files on any system other then MacOS. In MacOS thse sparsebundle is mounted and then accessed, so you have another layer of filesystem on top of that.

I can traverse those directories on the command line. So, I'm not sure any limitations of sparsebundle is my problem. I think I just picked a bad example.

I think my real question or confusion here is that my instincts tell me that when I add storage to the Datacenter and I see it visible in the "Server View" navigator pane, it seems like I should be able to browse its contents.

Here's a better example. I don't see the ISO listed in the Samba storage, but I see it on the command line on line 5. I think I first noticed this when I tried copying an ISO (via command line) I had on other storage to the local storage and thought I could use it to create a VM.

 

[LnxBil]

Okay. If I use thin provisioning on my ZFS storage types, will the LXC containers be allowed to use all of the available space in the pool?

If you want that, sure. Per default it is only the amount you supplied while creating the container.

I think my real question or confusion here is that my instincts tell me that when I add storage to the Datacenter and I see it visible in the "Server View" navigator pane, it seems like I should be able to browse its contents.

No, PVE is a virtualization environment, not a NAS software. You will not see anything besides the stuff that it needs to.

Here's a better example. I don't see the ISO listed in the Samba storage, but I see it on the command line on line 5. I think I first noticed this when I tried copying an ISO (via command line) I had on other storage to the local storage and thought I could use it to create a VM.

Yes, that is normal. Configuring a storage in the PVE sense is setting up a directory storage which a special structure and only the iso's that are present in

<storage>/template/iso

will be shown.

 

[backdoc]

If you want that, sure. Per default it is only the amount you supplied while creating the container.

Thin provisioning takes priority over this value? I can't recall from memory and I'm away from my server atm. But, I believe the amount defaulted to 8 Gb. So, this value will be ignored when thin provisioning is selected? Or, am I too late and need to turn on thin provisioning before creating the container?

As for the rest of the storage questions, I guess it makes some better sense. I'm sure the reasons and all will become clearer over time. One reason I'm asking these questions about storage is that I will most likely create a VM to host Dockers and I'm trying to plan how all of the containers and VMs will share data. Based on what I understand about storage, I don't guess Proxmox has any way to create a common storage that they can all access. I suppose I just need to handle that with normal file sharing strategies within the containers and VMs. Is that the correct approach?

 

[Dunuin]

Thin provisioning takes priority over this value? I can't recall from memory and I'm away from my server atm. But, I believe the amount defaulted to 8 Gb. So, this value will be ignored when thin provisioning is selected? Or, am I too late and need to turn on thin provisioning before creating the container?

Without thin-provisioning it will reserve the full space. Lets say you got 3 LXCs with 100GB in size each but each one one filled with 20GB of data. Without thin-provisioning this would use 300GB of your storage. With thin-provisioning it just would use 60GB of storage.
See the "refreservation" ZFS attribute: https://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html

As for the rest of the storage questions, I guess it makes some better sense. I'm sure the reasons and all will become clearer over time. One reason I'm asking these questions about storage is that I will most likely create a VM to host Dockers and I'm trying to plan how all of the containers and VMs will share data. Based on what I understand about storage, I don't guess Proxmox has any way to create a common storage that they can all access. I suppose I just need to handle that with normal file sharing strategies within the containers and VMs. Is that the correct approach?

Jup, you need to setup a mix of bind-mounts and NFS/SMB for that by yourself. VMs won't allow bind-mounts and require network shares like SMB/NFS. And a unprivileged LXC won't allow you to mount a NFS/SMB inside the LXC, so you would need to make use of bind-mounts to directly bind-mount the folder into the LXC or to mount the NFS/SMB share on the host first and then bind-mount that SMB/NFS mountpoint from the host to the LXC. Privileged LXC on the other hand can mount SMB/NFS shares, but should be avoided because of bad isolation.

 

[backdoc]

Without thin-provisioning it will reserve the full space. Lets say you got 3 LXCs with 100GB in size each but each one one filled with 20GB of data. Without thin-provisioning this would use 300GB of your storage. With thin-provisioning it just would use 60GB of storage.
See the "refreservation" ZFS attribute: https://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html

Jup, you need to setup a mix of bind-mounts and NFS/SMB for that by yourself. VMs won't allow bind-mounts and require network shares like SMB/NFS. And a unprivileged LXC won't allow you to mount a NFS/SMB inside the LXC, so you would need to maje use of bind-mounts to directly bind-mount the folder into the LXC or to mount the NFS/SMB share on the host first and then bind-mount that SMB/NFS mountpoint from the host to the LXC. Privileged LXC on the other hand can mount SMB/NFS shares, but should be avoided because of bad isolation.

I'm back . What a rabbit hole!! I got a lot more ZFS experience than I had planned. But, it was good. As I learned more about what I needed to do, I didn't like some of the naming conventions I had used initially. So, I did a lot of cleanup of my ZFS datasets on the command line and now feel hyper organized. I used Proxmox's backup and restore to relocate some containers. That worked great.

I made use of the bind mount points per your suggestion and have gotten my Samba Time Machine (and other file sharing) LXC container up and functioning. As far as I understand, this was the best way to allow my file shares to use as much disk space as they need and not be limited by the size of the container. As the Proxmox documentation predicted, I did run into some permissions issues. I could have mapped users. But, this is a home setup and I'm the only user. In the end, I could have left the dataset owned by root or another user and just mapped those id's within the container. But, I have no reason to restrict access. So, I just make that dataset 777. That was easy. In a production setting, that might not be the best solution. But, for me, I don't have any security concerns.

As a side note, through my reorganizing, I could see how the containers were on ZFS storage were images mounted as datasets and the one on the directory storage was all readable on the command line. That was interesting.

Long story short, I just wanted to follow up to this thread for closure and thank you for your help so you'd know that your suggestion didn't fall on deaf ears. I'm slow because I want to plan it out right and I have just a few minutes here and there to work on it.

I have one more question that really doesn't fit in this thread. So, I'll be posting separately soon.

 

[LnxBil]

As a side note, through my reorganizing, I could see how the containers were on ZFS storage were images mounted as datasets and the one on the directory storage was all readable on the command line. That was interesting.

The wording is a little bit wrong: You have a ZFS filesystem, there is no image mounted and this is a very good thing. The image-way with ZFS are zvol (block device) with another filesystem (normally not ZFS) on top, like with LVM+EXT4 for LX(C) containers.

 

[backdoc]

The wording is a little bit wrong: You have a ZFS filesystem, there is no image mounted and this is a very good thing. The image-way with ZFS are zvol (block device) with another filesystem (normally not ZFS) on top, like with LVM+EXT4 for LX(C) containers.

Thanks for pointing out my error. I paused for a min when I wrote that because I wasn’t 100% sure I was phrasing it correctly.