Organizing storage
I decided to try out Proxmox on my VM server. So far I like Proxmox
lot, it seems to be a really great product. The only thing that gives me
serious headaches is proper management of storage. Either I
fundamentally do not understand how storage managment in
Proxmox is supposed to work, or storage management in Proxmox is
very chaotic/buggy.
I use a ZFS mirror on two SATA SSDs to host the OS and a RAID-Z1
of four 8TB drives with a 1 TB NVMe SSD as L2ARC cache.
All disposable data OS and ISO images are stored in the /rpool.
The actual containers and VMs are supposed to go to the zpool0.
While this sounds like an easy enough setup, I've run into a lot of
road blocks.
To provide a better overview, things look roughly like this:
First attempt:
Initially I did plan to create multiple ZFS datasets to separate
disk images, container images, backups. This would allow me
to define individual quotas and reserve space for each type
of data.
While I was not able create ZFS datasets using the UI, it is possible
to create them via shell. Once I create a storage location based on
a manually created dataset via the Datacenter -> Storage UI, Proxmox
implicitly creates its own subfolders, and I end up with redundant paths
(e.g. /zpool0/images/images). Where zpool0/images is the ZFS dataset
and images is the auto-generated subfolder To avoid such a messy folder
structure I took a different approach...
Second attempt:
... gave up the ability to define individual quotas and used a
global zpool without extra datasets instead. Unfortunately it is
not possible to store "container templates" or "VZDUmp backup files"
directly into the root dataset of a zpool, is there any reason for
that?
Third attempt:
Since I can only store disk images and containers into the "zpool0"
root dataset I have to create sub directories inside zpool0, using
Datacenter -> Storage -> Add -> Directory. Once a directory is
created that way, I can assign this directory to store anything:
Disk images, ISO images, Container templates, VZDump backup files,
Containers, Snippets.
However for each directory I add under zpool0, Proxmox also adds the
available free space in zpool0 to the total amount of available
space. I.e. I have ~20TB of actual usable storage in zpool0 if I add
one sub directory as storage location the Datacenter -> Summary
suddenly shows 40TB of total storage, if I add another directory it
becomes 60TB of storage and so on.
Storage with one sub directory
Storage with two subdirectories:
With distinct sub directories set up for every type of data (one
for disk images, one for Containers ...), the root dataset zpool0
itself should not be a destination for any data. Unfortunately it
is impossible to assign "no content" to the zpool0 in the Proxmox UI.
It has to either store "disk images" or "containers". The only way I
found to remove the root dataset "zpool0" from the list of storage
locations was to disable it. Once disabled zpool0 will not come online
anymore after a reboot. Instead an empty directory structure will be
created in a folder /zpool0 on the root fs. This prevents the /zpool0
dataset from being mounted, all data appears to be gone. This
can be amended by deleting the empty directory structure,
adding the "immutable" attribute to the /zpool0 and then mounting
the zfs root dataset manually.
After three failed attempts I'm out of ideas. Is there any way to organize
storage in an orderly fashion when using a ZFS pool as main data store?
I decided to try out Proxmox on my VM server. So far I like Proxmox
lot, it seems to be a really great product. The only thing that gives me
serious headaches is proper management of storage. Either I
fundamentally do not understand how storage managment in
Proxmox is supposed to work, or storage management in Proxmox is
very chaotic/buggy.
I use a ZFS mirror on two SATA SSDs to host the OS and a RAID-Z1
of four 8TB drives with a 1 TB NVMe SSD as L2ARC cache.
All disposable data OS and ISO images are stored in the /rpool.
The actual containers and VMs are supposed to go to the zpool0.
While this sounds like an easy enough setup, I've run into a lot of
road blocks.
To provide a better overview, things look roughly like this:
Code:
zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_500GB-part3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_500GB-part3 ONLINE 0 0 0
errors: No known data errors
pool: zpool0
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zpool0 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4 ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4 ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4 ONLINE 0 0 0
ata-HGST_HUS728T8TALE6L4 ONLINE 0 0 0
cache
nvme-Samsung_SSD_970_EVO_1TB ONLINE 0 0 0First attempt:
Initially I did plan to create multiple ZFS datasets to separate
disk images, container images, backups. This would allow me
to define individual quotas and reserve space for each type
of data.
While I was not able create ZFS datasets using the UI, it is possible
to create them via shell. Once I create a storage location based on
a manually created dataset via the Datacenter -> Storage UI, Proxmox
implicitly creates its own subfolders, and I end up with redundant paths
(e.g. /zpool0/images/images). Where zpool0/images is the ZFS dataset
and images is the auto-generated subfolder To avoid such a messy folder
structure I took a different approach...
Second attempt:
... gave up the ability to define individual quotas and used a
global zpool without extra datasets instead. Unfortunately it is
not possible to store "container templates" or "VZDUmp backup files"
directly into the root dataset of a zpool, is there any reason for
that?
Third attempt:
Since I can only store disk images and containers into the "zpool0"
root dataset I have to create sub directories inside zpool0, using
Datacenter -> Storage -> Add -> Directory. Once a directory is
created that way, I can assign this directory to store anything:
Disk images, ISO images, Container templates, VZDump backup files,
Containers, Snippets.
However for each directory I add under zpool0, Proxmox also adds the
available free space in zpool0 to the total amount of available
space. I.e. I have ~20TB of actual usable storage in zpool0 if I add
one sub directory as storage location the Datacenter -> Summary
suddenly shows 40TB of total storage, if I add another directory it
becomes 60TB of storage and so on.
Storage with one sub directory
Storage with two subdirectories:
With distinct sub directories set up for every type of data (one
for disk images, one for Containers ...), the root dataset zpool0
itself should not be a destination for any data. Unfortunately it
is impossible to assign "no content" to the zpool0 in the Proxmox UI.
It has to either store "disk images" or "containers". The only way I
found to remove the root dataset "zpool0" from the list of storage
locations was to disable it. Once disabled zpool0 will not come online
anymore after a reboot. Instead an empty directory structure will be
created in a folder /zpool0 on the root fs. This prevents the /zpool0
dataset from being mounted, all data appears to be gone. This
can be amended by deleting the empty directory structure,
adding the "immutable" attribute to the /zpool0 and then mounting
the zfs root dataset manually.
After three failed attempts I'm out of ideas. Is there any way to organize
storage in an orderly fashion when using a ZFS pool as main data store?
Last edited:
) when first starting to work with it.