Proxmox 5.1 - ZFS: guidelines

[zeuxprox]

Hi to all,

I would try to use ZFS as storage system and, after reading many "how to" and tutorial, I'm still confused...

My questions are:

1. For VMs (linux based, Windows based, etc..) is better to use volume or dataset/filesystem?
2. For LCX container is better to use volume or dataset/filesystem?
3. Image format: RAW or qcow?
4. Cache: which is the best compromise between performance and security?
5. Snapshot: Can I create a snapshot with: zfs snapshot ?

Thank you very much

 

[wolfgang]

Hi,

For VMs (linux based, Windows based, etc..) is better to use volume or dataset/filesystem?

IMHO zvols. It is true that the zfs block device emulation is up to 10% slower than zfilesystem. But anyway the fs have no O_Direct and all our tool chain is optimized to zvols.

For LCX container is better to use volume or dataset/filesystem?

Only file-system is supported.

Image format: RAW or qcow?

LXC can't use qcow2 format and will not use any of this formats. It will use the filesystem direct.
KVM can use both but qcow2 will be very slow because it write every io twice as zfs do. This result in a four io's on the disk for one io in the VM.

Cache: which is the best compromise between performance and security?

zvols do not profit from any cache, to be exact you will slow down a bit, because you must write the io twice in memory.
filesystem do not support no cache so you should use write back.

Snapshot: Can I create a snapshot with: zfs snapshot ?

Yes

 

[zeuxprox]

Thank you wolfgang

Another question. Consider having the following scenario

I have a pool named storage and within it a volume named VM100. In this volume I have a VM with ID 100.
Now, at time t0 I do a snapshot of VM 100 with zfs snapshot storage/VM100@vm100-t0

Then I continue to use my VM100 until I want to use, for any reasons, my backup at time t0. So I do a clone with:
zfs clone storage/VM100@vm100-t0 storage/VM100/VM100-clone-t0

Now I have a clone of VM 100 at time t0, right? After that I stop VM 100 and how can I use the clone at time t0 of VM 100?

Thank you

 

[wolfgang]

You have to edit the vm config and change the vdisk.
But the easier way would be, to create a linked clone on the GUI.

 

[zeuxprox]

Hi wolfgang,

at my question n.1 (For VMs (linux based, Windows based, etc..) is better to use volume or dataset/filesystem?) you answered that is better to use Volume because tool chain is optimized to zvols and FS has not O_DIRECT.
Now I have read this proxmox wiki:
https://pve.proxmox.com/wiki/Storage:_ZFS#Adding_ZFS_root_file-system_as_storage_with_Plugin

Under Storage Features it says:
"... The backend uses ZFS datasets for both VM images (format raw) and container data (format subvol) ..."

And under Examples:
"It is recommended to create an extra ZFS file system to store your VM images"
# zfs create tank/vmdata

Now I'm a bit confused because you, but also my instinct, say me that, for VMs, Zvol should be better than dataset...

Another question:
If at the end I will use Zvol, when I create a VM, Proxmox automatically creates a Zvol for each VM?

Than you very much

 

[wolfgang]

Now I'm a bit confused because you, but also my instinct, say me that, for VMs, Zvol should be better than dataset...

I think this will help you from the zfs docu

A dataset can be one of the following:

     file system  A ZFS dataset of type filesystem can be mounted within the standard system namespace and behaves like other file systems.  While ZFS file systems are designed to be POSIX compliant, known
                  issues exist that prevent compliance in some cases.  Applications that depend on standards conformance might fail due to non-standard behavior when checking file system free space.

     volume       A logical volume exported as a raw or block device.  This type of dataset should only be used under special circumstances.  File systems are typically used in most environments.

     snapshot     A read-only version of a file system or volume at a given point in time.  It is specified as filesystem@name or volume@name.

     bookmark     Much like a snapshot, but without the hold on on-disk data. It can be used as the source of a send (but not for a receive). It is specified as filesystem#name or volume#name.

If at the end I will use Zvol, when I create a VM, Proxmox automatically creates a Zvol for each VM?

Yes for every VM/CT we will create a own dataset. If you use the file system with images(raw, qcow2) you must do it on your own or use a single one.

 

[zeuxprox]

Ok thank you.
Now I'm in testing phase and my Test Server has 4 old HDD: 2 for OS and the other 2 for storage (mirror)

I create a pool with:
zpool create -f -m /zfsFS/zstorage zstorage -o ashift=9 mirror /dev/disk/by-id/wwn-0x6842b2b05711e7002170862a0e0d42c8 /dev/disk/by-id/wwn-0x6842b2b05711e7002170865010492b8f

Then within that pool I create a dataset/filesystem for containers:
zfs create zstorage/container

and another dataset/filesystem for storage:
zfs create zstorage/storage

In the Proxmox GUI, under Storage section (ZFS), I add:

• For VMs:
  
  • ID: Zstorage-VM
    
  • ZFS pool: zstorage
  • Content: Disk image
  • Thin provision: checked
• For Containers:
  • ID: Zstorage-Container
  • ZFS pool: zstorage/container
  • Content: Container
    
  • Thin provision: checked
• For General Storage:
  • ID: Zstorage-Storage
  • ZFS pool: zstorage/storage
  • Content: Disk image, Container
  • Thin provision: checked

Is this correct?

From now, when I want to create a VM, under Hard Disk section, I will select:

• Bus/Device: VirtioO Block
• Storage: Zstorage-VM
• Disk size (GiB): 100
• Cache: Default (No Cache)
• IO thread: checked or unchecked ?

Instead for Container, under Root Disk section:

• Storage: Zstorage-Container

Is it correct?

Thank you very much

 

[wolfgang]

zpool create -f -m /zfsFS/zstorage zstorage -o ashift=9 mirror /dev/disk/by-id/wwn-0x6842b2b05711e7002170862a0e0d42c8 /dev/disk/by-id/wwn-0x6842b2b05711e7002170865010492b8f

Please use shift=12 not 9 because 9 is very slow.

The rest is ok

 

[guletz]

Hi,

Thin provision: checked

Thin provision, is OK, ONLY if you will be sure, that you do not running out-of-space. In out of space situation, many bad thing could happend, even if you solve the out of space situation.