ZFS Pool Handling

[linedan]

I have a new HP-Server, with a "fake" Software-RAID-Controller.
So I often read that the software-raid and Proxmox are not the best friends, so I decided to install Proxmox with ZFS-Storage-Pool.

Everything alright so far, but I usually used to install Proxmox on a Hardware-Raid, so no idea how to use it the right way.
I so the documentations and so on but I think I need some more input.


This is the actual configuration:

1. 4 Harddrives with 4TB each
2. ZFS RAID 1+0

No i have one VM (with nearly the whole free disk space), but the "local-zfs" and "local" are both more or less empty:




Questions:

• I can't understand where the disk space, which the VM should occupy, is used?!
• I can't find the VM-Disk-File (raw or qcow2)?!
• Can I use ZFS with qcow2?
• Can I bind ZFS-Pool like a directory?
• Is a ZFS-RAID Hot-Swap capable? So if the Bays are, is there a possibility to change a single hard drive while the server is running?

Maybe somebody can share their experience with me or give me a best-practice-guide how to safely use this type of RAID!
Thank you very much!

 

[Dunuin]

• I can't understand where the disk space, which the VM should occupy, is used?!

ZFS is thin provisioned. If you for example create a 1TB virtual disk and only write 50GB to it, ZFS will only use 50GB.

• I can't find the VM-Disk-File (raw or qcow2)?!

There is no file. ZFS isn't working with folders and files the usual way. Its a "zvol" and just part of the filesystem and not a file in directory.

• Can I use ZFS with qcow2?

No, RAW will be used.

• Can I bind ZFS-Pool like a directory?

You can create datasets. Each dataset is its own separate filesystem. You can mount datasets and store files and folders in it.

• Is a ZFS-RAID Hot-Swap capable? So if the Bays are, is there a possibility to change a single hard drive while the server is running?

Yes. There are ZFS command to add/remove/replace drives.

 

[linedan]

ZFS is thin provisioned. If you for example create a 1TB virtual disk and only write 50GB to it, ZFS will only use 50GB.

There is no file. ZFS isn't working with folders and files the usual way. Its a "zvol" and just part of the filesystem and not a file in directory.

No, RAW will be used.

You can create datasets. Each dataset is its own separate filesystem. You can mount datasets and store files and folders in it.

Yes. There are ZFS command to add/remove/replace drives.

• Ok thank you, understood.

• So I can't for example copy the whole machine? How about Full-Backups? How about Snapshots?

• Ok, thank you, understood.

• Sounds little bit complicated, if there are no disk-files or something like that it also would be more or less needless.

• Ok perfect, that sounds good. I have to give that a try!

------------------------------------------------------------------------------------------------------------------------------------------------------

Is there something else to observe or do, after the normal proxmox-zfs-raid install?
Configuration looks like this no:

RPOOL Status



/etc/fstab




zfs list





zpool list






Thank you!

 

[Dunuin]

So I can't for example copy the whole machine?

No, like I said ZFS isn't like your typical filesystem where file/folders are stored on specific parts on some disks. Its more like a journal where nothing can be changed after it is written and every new entry or change will be just added at the end of the book. So a file is somewhere in that journal, every edit of the file is a new entry somewhere in that journal and you always need the complete journal to "merge" the initial file and all changes to that file to get the file at its latest version.

How about Full-Backups?

You can use the build in backup function of proxmox to export a full backup of the VM as a compressed file. Or you can install the Proxmox Backup Server to some host to be able to send incremental backups to that host.

How about Snapshots?

ZFS is supporting snapshots and proxmox will use the native snapshots of ZFS. But unlike with qcow2 you can't switch between snapshots. Its a oneway road. If you rollback a snapshot you can't return to a newer state later. Everything newer than the snapshot you rollback to will be deleted.

 

[linedan]

No, like I said ZFS isn't like your typical filesystem where file/folders are stored on specific parts on some disks. Its more like a journal where nothing can be changed after it is written and every new entry or change will be just added at the end of the book. So a file is somewhere in that journal, every edit of the file is a new entry somewhere in that journal and you always need the complete journal to "merge" the initial file and all changes to that file to get the file at its latest version.

You can use the build in backup function of proxmox to export a full backup of the VM as a compressed file. Or you can install the Proxmox Backup Server to some host to be able to send incremental backups to that host.

ZFS is supporting snapshots and proxmox will use the native snapshots of ZFS. But unlike with qcow2 you can't switch between snapshots. Its a oneway road. If you rollback a snapshot you can't return to a newer state later. Everything newer than the snapshot you rollback to will be deleted.

Ok understood, no I have a better overview how it works.

One more question, is there a possibility to backup this journal?
If I need more or less the whole journal, this would be important I think!

With a Full-Backup, made with the proxmox-onboard-backup, I could restore the whole machine (if same configuration)?
So this could be more or less a possibility to copy.

 

[Dunuin]

One more question, is there a possibility to backup this journal?

You can backup a pool/dataset/zvol by replicating it to another pool (for example zfs send ... | zfs receive ...) or use "zfs send" and pipe the output to a file and store the file where you want.

With a Full-Backup, made with the proxmox-onboard-backup, I could restore the whole machine (if same configuration)?
So this could be more or less a possibility to copy.

Yes, it will restore the complete VM.