zfs local storage setup help

[juju01]

Hello - I just added a local storage disk in my server. I'd like to setup a zfs pool on it - Please let me know if its possible to do all of the following on one zfs pool :

1. create a partition for scheduled backups ( vzdumps of all containers and vms)
2. create a partition for database storage - can I mount this to an lxc container which has a postgres db?
3. create a partition to use as a persistent storage volume for my kubernetes cluster i.e. setup a kubernetes persistent volume using a local zfs pool ( will appreciate if anyone has a tutorial on how to do this!)

How do I setup the zfs pool to do all the above? I have one 960gb disk for now. I intend to add another 960gb disk to create a zfs raid1 mirror.

 

[LnxBil]

create a partition for scheduled backups ( vzdumps of all containers and vms)

sure

create a partition for database storage - can I mount this to an lxc container which has a postgres db?

use bind mount via the GUI

create a partition to use as a persistent storage volume for my kubernetes cluster i.e. setup a kubernetes persistent volume using a local zfs pool ( will appreciate if anyone has a tutorial on how to do this!)

You may need this to get it to work. I have no experience, but it is on my todo list.

How do I setup the zfs pool to do all the above? I have one 960gb disk for now. I intend to add another 960gb disk to create a zfs raid1 mirror.

No problem, just attach it correctly.

 

[juju01]

sure

How ? Not sure how to setup the partition for this

 

[LnxBil]

How ? Not sure how to setup the partition for this

First, partition is the wrong word, you need a dataset and then setup is as a directory, e.g. your pool is named rpool, just run:

zfs create rpool/backups

and add the directory /rpool/backups as a directory storage in your dataset configuration and select the type vzdump.

 

[juju01]

First, partition is the wrong word, you need a dataset and then setup is as a directory, e.g. your pool is named rpool, just run:

zfs create rpool/backups

and add the directory /rpool/backups as a directory storage in your dataset configuration and select the type vzdump.

So I created a zfs pool called datastore on the new disk. Has 1 tb of storage. Does creating a new dataset called backups at datastore/backup assign all the 1 tb to this dataset? Is there a way to limit how much space the backup dataset can have?

 

[LnxBil]

So I created a zfs pool called datastore on the new disk. Has 1 tb of storage. Does creating a new dataset called backups at datastore/backup assign all the 1 tb to this dataset?

Yes, all datasets share all the space.

Is there a way to limit how much space the backup dataset can have?

Yes, quota.

 

[juju01]

Yes, all datasets share all the space.



Yes, quota.

Something like this ?

zfs create -o refquota=100G datastore/backups

 

[jayg30]

Something like this ?

zfs create -o refquota=100G datastore/backups

Probably will work. It depends on what type of behavior you actually want.

You should read the following;
https://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html or https://illumos.org/books/zfs-admin/gavwq.html#gazvb

ZFS supports quotas and reservations at the file system level. You can use the quota property to set a limit on the amount of space a file system can use. In addition, you can use the reservation property to guarantee that some amount of space is available to a file system. Both properties apply to the dataset they are set on and all descendents of that dataset.

That is, if a quota is set on the tank/home dataset, the total amount of space used by tank/home and all of its descendents cannot exceed the quota. Similarly, if tank/home is given a reservation, tank/home and all of its descendents draw from that reservation. The amount of space used by a dataset and all of its descendents is reported by the used property.

In addition to the quota and reservation property, the refquota and refreservation properties are available to manage file system space without accounting for space consumed by descendents, such as snapshots and clones.

Consider the following points to determine which quota and reservations features might better manage your file systems:

• The quota and reservation properties are convenient for managing space consumed by datasets.
• The refquota and refreservation properties are appropriate for managing space consumed by datasets and snapshots.
• Setting refquota or refreservation higher than quota or reservation have no effect. If you set the quota or refquota properties, operations that try to exceed either value fail. It is possible to a exceed a quota that is greater than refquota. If some snapshot blocks are dirtied, you might actually exceed the quota before you exceed the refquota.

 

[juju01]

Hello @LnxBil and @jayg30 thanks for the info. I am all set with the bind mounts now.

Now need to tackle the issue of setting up persistent storage volumes for my kubernetes cluster ( dynamically provisioned ). There is almost nothing out there for doing this on proxmox. My kubernestes boxes are setup on vms instead of lxc containers. I'd like to use the same datastore zfs pool to dynamically allocate storage to pods in the cluster but have no idea how.

 

[LnxBil]

Now need to tackle the issue of setting up persistent storage volumes for my kubernetes cluster ( dynamically provisioned ). There is almost nothing out there for doing this on proxmox. My kubernestes boxes are setup on vms instead of lxc containers. I'd like to use the same datastore zfs pool to dynamically allocate storage to pods in the cluster but have no idea how.

For ZFS, there is this project. AFAIK provisioning of hosts itself needs also a machine driver, there is one one for PVE and Docker, but not yet for kubernetes.

 

[juju01]

Hello @LnxBil - Apologies for reviving an old thread. I have a question related to our discussion above. This time around I am looking to setup my postgresdb with storage directly on a zfs volume I have on my proxmox server. The postgresdb is setup on an ubuntu 21.10 VM. How do I assign storage for it on the zfs volume? Any pointers how I can do that? Seems rather rudimentary but not sure how! Thanks.

 

[LnxBil]

storage directly on a zfs volume I have on my proxmox server.

Why a volume instead of a filesystem? Please have a look here.

 

[juju01]

Why a volume instead of a filesystem? Please have a look here.

Thanks for that link. It was helpful. But I am missing how I can assign and connect storage to the postgres VM. I have done the following so far:

zfs create databank/db -o mountpoint=/db
zfs set quota=48G databank/db 

# create one dataset for every database
zfs create databank/db/db01-pg14
chown postgres:postgres /db/db01-pg14
zfs set reservation=1G databank/db/db01-pg14

So I have a dataset at databank/db/db01-pg14. How do I get the postgres instance to use this for storage?

 

[Dunuin]

You can't use ZFS datasets inside a VM. That would only be possible if you run a SMB/NFS server and use SMB/NFS to bring the mountpoint of that dataset into your VM. But then you got the overhead and latency of the NFS/SMB protocol and network stack so thats not great for running a DB.

Best would be to use a LXC instead of a VM where bind-mounts are possible so you could bind-mount a dataset directly into your LXC.

Or if you want to stick with a VM I would create a virtual disk (zvol with 8k volblocksize) for your DB as this should give way better performance compared to a SMB/NFS share.

 

[juju01]

create a virtual disk (zvol with 8k volblocksize) for your DB as this should give way better performance compared to a SMB/NFS share.

Can you point me in the right direction how to do this? I will stick with the VM

 

[Dunuin]

PVE only allows the blocksize to defined for the complete ZFS storage and default already should be 8K. So if you add a new virtual disk to your VM (YourVM -> Hardware -> Add -> Harddisk) it should create a new zvol with a 8k volblocksize for you.
Also keep in mind that a mirror or 4 disk striped mirror is the only ZFS pool layout (when using ashift=12) that can handle 8K blocksizes needed for a posgres DB without alot of overhead.

 

[juju01]

PVE only allows the blocksize to defined for the complete ZFS storage and default already should be 8K. So if you add a new virtual disk to your VM (YourVM -> Hardware -> Add -> Harddisk) it should create a new zvol with a 8k volblocksize for you.

Do I need to do anything in postgres to make sure the data is saved on the new virtual disk?

Which disk type do I use ? SCSI, VirtIO, SATA, IDE ? I am looking to use the ZFS storage drive I have on the proxmox machine.

 

[Dunuin]

Do I need to do anything in postgres to make sure the data is saved on the new virtual disk?

If you want to use the new virtual disk for posgres you would need to partition it , format it and mount it inside your guest OS and configure posgres to use it.

Which disk type do I use ? SCSI, VirtIO, SATA, IDE ? I am looking to use the ZFS storage drive I have on the proxmox machine.

"Virtio SCSI single" as controller and "SCSI" as protocol usually gives the best performance/features.

 

[juju01]

If you want to use the new virtual disk for posgres you would need to partition it , format it and mount it inside your guest OS and configure posgres to use it.

"Virtio SCSI single" as controller and "SCSI" as protocol usually gives the best performance/features.

Are there any guides how to do this? Apologies, this is a bit over my head. Learning.

 

[Dunuin]

Just google for some ubuntu 21.10 tutorials as this all is just normal stuff totally done in the guest OS. There should be a ton out there that explains how to partition/format/mount disks or how to manage posgres. Maybe you also find some info on what filesystem to best use for posgres.