Understanding the disk layout automatically created by Proxmox VE installer

[H25E]

Hello everybody,

I have "inherited" a simple proxmox cluster with only one cluster, and I'm doing my best to catch up, reading the wiki, doc and forum threads. But there is something I still can't understand, that is the ZFS and proxmox storages layout.

In the system there are two 1TB SSD and four 4TB HDD. The first ones in a mirror zpool (rpool) and the second ones in a raidz1 zpool (HDD)

zpool list output:


And here pvesm status:



I understand the zfs pool concept, but I don't understand what proxmox storages are (Datas, local and local-zfs).

1. What are proxmox storages actually are? Is Datas equivalent to HDD and local-zfs to rpool?
2. What happens with local? I know that this is the default configuration, but why there are two proxmox storages sharing the same zpool space? I know that local storage is dir type and local-zfs is zfs, but which are the real implications? Both are over the same filesystem (ZFS). Could I merge both storages?
3. Why local and local-zfs have different Total and Used bytes but the "same" Available ones?
4. Do proxmox storages expand to the maximum zpool space available without allocating it for themselves?
5. Why local total size has disminished with time (see picture below)? Local-zfs has a similar behavior, but not the same.



In the other hand, here is my df output:


6. Why HDD zpool appears as 688G/128k and the rpool/ROOT/pve-1 appears "correctly" as 753G/154G
7. Why rpool, rpool/ROOT and rpool/data appear as 600G/0?
8. Why all this different filesystems (rpool, rpool/data, rpool/ROOT, rpool/ROOT/pve-1) if there is only one "partition".

I know that the post it's a little bit long. If there is any problem with the post message me and I will edit it.

Thanks for reading,


Hector

 

[aaron]

I'll try to answer your questions.

1. What are proxmox storages actually are? Is Datas equivalent to HDD and local-zfs to rpool?

Most likely. If you go to Datacenter -> Storage and edit the `Datas` storage, it should show you which ZFS pool is used for that storage.

2. What happens with local? I know that this is the default configuration, but why there are two proxmox storages sharing the same zpool space? I know that local storage is dir type and local-zfs is zfs, but which are the real implications? Both are over the same filesystem (ZFS). Could I merge both storages?
3. Why local and local-zfs have different Total and Used bytes but the "same" Available ones?
4. Do proxmox storages expand to the maximum zpool space available without allocating it for themselves?
5. Why local total size has disminished with time (see picture below)? Local-zfs has a similar behavior, but not the same.

`local` is a directory located at /var/lib/vz. It is meant as a first storage after installation to store ISO images and backups.

If you install Proxmox VE not on a ZFS root but with ext4 or xfs file system, the installer will set up an LVM underneath it and you will have a similar storage arrangement. `local` as directory storage and a `local-lvm` which can be compared to `local-zfs` in order to store VM disks on the LVM.

VM disks on ZFS are not stored as files but on ZFS datasets of the type `volume` which are exposed as block devices.
The way ZFS works though is that all datasets share the free space between them. Thus, you see a similar pattern of free space going down.

6. Why HDD zpool appears as 688G/128k and the rpool/ROOT/pve-1 appears "correctly" as 753G/154G
7. Why rpool, rpool/ROOT and rpool/data appear as 600G/0?

Don't use `df` to account for usage on a ZFS pool as it will do the accounting wrong. The zfs list command will give you the full picture and also show the datasets for VM disks.

8. Why all this different filesystems (rpool, rpool/data, rpool/ROOT, rpool/ROOT/pve-1) if there is only one "partition".

ZFS creates datasets on the pool which are hierarchically ordered. They can be mounted freely in the FS tree. As you can see in the df output and if you run zfs list is that the root FS of the OS is located in rpool/ROOT/pve-1 for example.

The rpool/data dataset is the root dataset for containers (which are file system datasets that show up in the FS tree) and the VM disks (volume datasets, block devices which are not present in the FS tree but in /dev/).

One reason why one might want to order ZFS datasets hierarchically, besides keeping some structure, is so that properties set will (for most properties) be applied to all child datasets as well.

If you are interested in how ZFS works, a first good read are the man pages to zpool for everything related to the pool and data redundancy and the zfs man page for everything related to the different types of datasets.

man zpool
man zfs

 

[H25E]

Hello Aaron,

Thanks for your answer. I think I have writted my questions too early. A lot of them were about ZFS fs and it was my first time with it. I have spent some days reading ZFS documentation and some of the questions have been resolved, some still stay and new ones have been created. Thank you anyway for the answer, it has solved me also some missconceptions.

I suppose you will read more about me if I find myself in a dead end or to search for specific advice.