ZFS Pool Handling

linedan

Member
Feb 8, 2020
46
1
13
26
I have a new HP-Server, with a "fake" Software-RAID-Controller.
So I often read that the software-raid and Proxmox are not the best friends, so I decided to install Proxmox with ZFS-Storage-Pool.

Everything alright so far, but I usually used to install Proxmox on a Hardware-Raid, so no idea how to use it the right way.
I so the documentations and so on but I think I need some more input.


This is the actual configuration:
  1. 4 Harddrives with 4TB each
  2. ZFS RAID 1+0
1621253934097.png



No i have one VM (with nearly the whole free disk space), but the "local-zfs" and "local" are both more or less empty:

1621254001986.png


Questions:
  • I can't understand where the disk space, which the VM should occupy, is used?!
  • I can't find the VM-Disk-File (raw or qcow2)?!
  • Can I use ZFS with qcow2?
  • Can I bind ZFS-Pool like a directory?
  • Is a ZFS-RAID Hot-Swap capable? So if the Bays are, is there a possibility to change a single hard drive while the server is running?

Maybe somebody can share their experience with me or give me a best-practice-guide how to safely use this type of RAID!
Thank you very much!
 

Dunuin

Famous Member
Jun 30, 2020
8,075
2,014
149
Germany
  • I can't understand where the disk space, which the VM should occupy, is used?!
ZFS is thin provisioned. If you for example create a 1TB virtual disk and only write 50GB to it, ZFS will only use 50GB.
  • I can't find the VM-Disk-File (raw or qcow2)?!
There is no file. ZFS isn't working with folders and files the usual way. Its a "zvol" and just part of the filesystem and not a file in directory.
  • Can I use ZFS with qcow2?
No, RAW will be used.
  • Can I bind ZFS-Pool like a directory?
You can create datasets. Each dataset is its own separate filesystem. You can mount datasets and store files and folders in it.
  • Is a ZFS-RAID Hot-Swap capable? So if the Bays are, is there a possibility to change a single hard drive while the server is running?
Yes. There are ZFS command to add/remove/replace drives.
 
Last edited:

linedan

Member
Feb 8, 2020
46
1
13
26
ZFS is thin provisioned. If you for example create a 1TB virtual disk and only write 50GB to it, ZFS will only use 50GB.

There is no file. ZFS isn't working with folders and files the usual way. Its a "zvol" and just part of the filesystem and not a file in directory.

No, RAW will be used.

You can create datasets. Each dataset is its own separate filesystem. You can mount datasets and store files and folders in it.

Yes. There are ZFS command to add/remove/replace drives.
  • Ok thank you, understood.

  • So I can't for example copy the whole machine? How about Full-Backups? How about Snapshots?

  • Ok, thank you, understood.

  • Sounds little bit complicated, if there are no disk-files or something like that it also would be more or less needless.

  • Ok perfect, that sounds good. I have to give that a try!
------------------------------------------------------------------------------------------------------------------------------------------------------

Is there something else to observe or do, after the normal proxmox-zfs-raid install?
Configuration looks like this no:

RPOOL Status

1621258005683.png


/etc/fstab

1621258128497.png


zfs list

1621258162689.png



zpool list

1621258261355.png




Thank you!
 

Dunuin

Famous Member
Jun 30, 2020
8,075
2,014
149
Germany
So I can't for example copy the whole machine?
No, like I said ZFS isn't like your typical filesystem where file/folders are stored on specific parts on some disks. Its more like a journal where nothing can be changed after it is written and every new entry or change will be just added at the end of the book. So a file is somewhere in that journal, every edit of the file is a new entry somewhere in that journal and you always need the complete journal to "merge" the initial file and all changes to that file to get the file at its latest version.
How about Full-Backups?
You can use the build in backup function of proxmox to export a full backup of the VM as a compressed file. Or you can install the Proxmox Backup Server to some host to be able to send incremental backups to that host.
How about Snapshots?
ZFS is supporting snapshots and proxmox will use the native snapshots of ZFS. But unlike with qcow2 you can't switch between snapshots. Its a oneway road. If you rollback a snapshot you can't return to a newer state later. Everything newer than the snapshot you rollback to will be deleted.
 

linedan

Member
Feb 8, 2020
46
1
13
26
No, like I said ZFS isn't like your typical filesystem where file/folders are stored on specific parts on some disks. Its more like a journal where nothing can be changed after it is written and every new entry or change will be just added at the end of the book. So a file is somewhere in that journal, every edit of the file is a new entry somewhere in that journal and you always need the complete journal to "merge" the initial file and all changes to that file to get the file at its latest version.

You can use the build in backup function of proxmox to export a full backup of the VM as a compressed file. Or you can install the Proxmox Backup Server to some host to be able to send incremental backups to that host.

ZFS is supporting snapshots and proxmox will use the native snapshots of ZFS. But unlike with qcow2 you can't switch between snapshots. Its a oneway road. If you rollback a snapshot you can't return to a newer state later. Everything newer than the snapshot you rollback to will be deleted.
Ok understood, no I have a better overview how it works.

One more question, is there a possibility to backup this journal?
If I need more or less the whole journal, this would be important I think!

With a Full-Backup, made with the proxmox-onboard-backup, I could restore the whole machine (if same configuration)?
So this could be more or less a possibility to copy.
 

Dunuin

Famous Member
Jun 30, 2020
8,075
2,014
149
Germany
One more question, is there a possibility to backup this journal?
You can backup a pool/dataset/zvol by replicating it to another pool (for example zfs send ... | zfs receive ...) or use "zfs send" and pipe the output to a file and store the file where you want.
With a Full-Backup, made with the proxmox-onboard-backup, I could restore the whole machine (if same configuration)?
So this could be more or less a possibility to copy.
Yes, it will restore the complete VM.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!