ZFS storage configuration

penguin2020

Active Member
Jul 2, 2018
16
1
43
Hello


I use Proxmox since version 4, I have always used an installation with a hardware controller with Raid10, and the installation with local (dir), and local LVM-Thin.
I have a new server and I want to use ZFS now :
( OVH USA Dedicate Server Service)
2x Intel E5-2650v2 - 16C/32T - 2.6GHz/3.4GHz - RAM 256GB - 2x 240GB SSD + 4x 2TB HDD SATA

I recently installed (from scratch) proxmox 6.0-1 (and updated to 6.0-5)
I have installed the system on 2 x 240 GB datacenter-grade ssd drives in ZFS mirror, using the installer's default values

I also have 4x 2TB HDD SATA disks, with which I have created (using the proxmox GUI) a zfs pool in raid10
It automatically created proxmox storage type ZFS, with the same name of the zfs pool

Code:
root@pve1:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                                                     STATE     READ WRITE CKSUM
        rpool                                                    ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            ata-SAMSUNG_MZ7LM240HCGR-00003_S1YFNX0H401329-part3  ONLINE       0     0     0
            ata-SAMSUNG_MZ7LM240HCGR-00003_S1YFNX0H601823-part3  ONLINE       0     0     0

errors: No known data errors

  pool: satapool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        satapool    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdf     ONLINE       0     0     0

errors: No known data errors
root@pve1:~# zfs list
NAME               USED  AVAIL     REFER  MOUNTPOINT
rpool             1.38G   214G      104K  /rpool
rpool/ROOT        1.38G   214G       96K  /rpool/ROOT
rpool/ROOT/pve-1  1.38G   214G     1.38G  /
rpool/data          96K   214G       96K  /rpool/data
satapool          1020K  3.51T       96K  /satapool

Any other work with ZFS (with commands from console) is necessary? If yes, I thank you for indicating the "what" and the "how"

It is recommended to create some other type of Storage to store the images of my CT, VM, or backup, or host some type of templates, in this last pool zfs that I created?


i read here ----> https://pve.proxmox.com/wiki/Storage:_ZFS


Code:
It is recommended to create an extra ZFS file system to store your VM images:

# zfs create tank/vmdata
To enable compression on that newly allocated file system:

# zfs set compression=on tank/vmdata
You can get a list of available ZFS filesystems with:

# pvesm zfsscan

I appreciate your recommendations, I want to start migrating VM and CT from my local servers, to my new dedicated server (the one that detailed the previous installation) in OVH USA
Thank you!
 
Any other work with ZFS (with commands from console) is necessary? If yes, I thank you for indicating the "what" and the "how"

Necessary is a hard word, it's usable the way it is.

Enabling compression is always a good idea. The extra CPU load is negligible and in the best case you can get quite a bit more out of the storage. Database servers can have a compression ratio of up to 2 or 3.

If you want to create an extra dataset to store your VM data is up to you. If you do, you have to add it in the storage configuration on the datacenter level and probably want to remove the one automatically created.

Compression can also be enabled directly on the pool
Code:
zfs set compression=on satapool
in your case.

Besides that everything should be good to go. Containers are stored in ZFS datasets and VM disks use ZVOLs (blockdevices in the pool).
 
  • Like
Reactions: penguin2020
Hi aaron, thanks for your kind comments

Besides that everything should be good to go. Containers are stored in ZFS datasets and VM disks use ZVOLs (blockdevices in the pool).

the ZVOLs , are automatically make when i make a VM ?

And

If you want to create an extra dataset to store your VM data is up to you. If you do, you have to add it in the storage configuration on the datacenter level and probably want to remove the one automatically created.

To create a new dataset , is for this way ? for ex.

Code:
# zfs create satapool/vmdata

and later in storage configuration level on the datacenter, add--> ZFS ?

and probably want to remove the one automatically created.

Why should I remove this storage? Could you use it for CT?

I need more space (in addition to the local storage that was create on the installation, on my SSD disks. "local (pve1)" ) to store templates, iso, backup, etc.
Should I create a storage of the type directory mounted on / satapool / backup, for example?

sorry if I'm lost in this aspect ... I have read all the documentation in Oracle of Zfs, and I have some doubts even of how it is the best way to have everything in the best way (with ZFS) in Proxmox
 
Hi aaron, thanks for your kind comments
the ZVOLs , are automatically make when i make a VM ?
Yes, and datasets for containers. Go ahead, create a test VM and container (CT) with it's disk on the `satapool` storage. Do a `zfs list` before and after to see what is happening. The datasets of the containers are regular file systems which you can browse from your PVE node. The zvols are block devices and therefore don't show up as file system.

To create a new dataset , is for this way ? for ex.

Code:
# zfs create satapool/vmdata

and later in storage configuration level on the datacenter, add--> ZFS ?
Yes.

Why should I remove this storage? Could you use it for CT?
The automatically created storage for the `satapool` has `satapool` as root where everything is happening, the `vmdata` storage is in `satapool/vmdata`.

If you want some hierarchy create a `satapool/containers` or something like it. By default the whole pool and all datasets you create like this share the same available storage space of the pool.
So for a clean hierarchy I would remove the automatically create storage if you create a hierarchy in `satapool`. Technically you don't have to.

I need more space (in addition to the local storage that was create on the installation, on my SSD disks. "local (pve1)" ) to store templates, iso, backup, etc.
Should I create a storage of the type directory mounted on / satapool / backup, for example?
That can work, I would create a new dataset `zfs create satapool/isos` or whatever you want to call it. The pool `satapool` should be mounted at `/satapool`. Therefore the path for the directory storage would be `/satapool/isos`.
 
a newbee question:

> It is recommended to create an extra ZFS file system to store your VM images:
> # zfs create tank/vmdata
> To enable compression on that newly allocated file system:
> # zfs set compression=on tank/vmdata
> You can get a list of available ZFS filesystems with:
> # pvesm zfsscan

after, there is no "tank/vmdata" in /etc/pve/storage.cfg
nor in the GUI

then I tried:
> pvesm import tank zfs tank/vmdata
=> open(tank/vmdata): No such file or directory

"df" and "zfs list" show the filesystem and path of tank/vmdata

Do I have to add it manually to /etc/pve/storage.cfg (?) like:
> zfspool: vmdata
pool tank/vmdata
content rootdir,images
sparse

Or can I create or add it in another recommended way by CLI or GUI?

Thanx
 
After you created the pool and file system you can add it via GUI or CLI, whichever you like better.
You can probably also create the zfs somewhere in the GUI (never did that, to be honest), but creating it on the command line is just as good.
 
  • Like
Reactions: ake855
Like phox said you need to tell Proxmox that there is a new storage you created by yourself.
IF you want to do that using GUI you can find it here: Datacenter -> Storage -> Add -> ZFS

If your pool is using raidz remember to calculate and increase the volblocksize there of you will waste alot of space if you just keep the default value of 8K. You need to do that before creating your first VMs virtual disk because that can't be (easily) changed later.
 
  • Like
Reactions: ake855

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!