Quastions about ZFS

[Dunuin]

Hi,

I'm running 2 FreeNAS Boxes and use them also to run my VMs/jails but they only got the maximum of 32GB of RAM and my jails/VMs now need 22GB RAM so I bought a new Server and want to run Proxmox on it so it can host the VMs instead.

The Server is a Supermicro X10SRM-F with a Xeon E5-2620v4 and 2x 16GB ECC RAM which I want to upgrade to 4x16 GB. I like ZFS so I would like to use it for Proxmox too. But I have some questions:

1.) Once I updated FreeNAS and it asked me if I want to upgrade my pools too. With the hint that a newer version von ZFS on the pools could make them incompatible with ZFS on lower versions. Are FreeNAS and Proxmox using the same ZFS version? I would like to "zfs send" my backups of the Proxmox pools to my FreeNAS backup server.

2.) If it is possible to send snaphots to a FreeNAS server, is there something I can do to ensure it keeps working and not getting incompatible?

3.) I would like to encrypt my pools and I've read in the wiki article "ZFS on Linux" that Proxmox can't boot from encrypted pools. So my idea was to use ZFS to mirror two unencrypted SATA SSDs as Proxmox boot pool and two mirrored 500GB NVMe M2 SSDs as encrypted pool for my VMs.

4.) How much space does the boot pool need? Is 120GB enough?

5.) How is Proxmox handling the ZFS caching and journaling? I would think that will happen on the boot pool like FreeNAS does. Are the caching and journaling datasets encrypted atleast?

6.) What could I move from the bootpool to the faster encrypted pool to increase speed and security?

 

[aaron]

1.) Once I updated FreeNAS and it asked me if I want to upgrade my pools too. With the hint that a newer version von ZFS on the pools could make them incompatible with ZFS on lower versions. Are FreeNAS and Proxmox using the same ZFS version? I would like to "zfs send" my backups of the Proxmox pools to my FreeNAS backup server.

I cannot answer this question fully but when I migrated my personal server at home from FreeNAS to PVE it worked fine.

2.) If it is possible to send snaphots to a FreeNAS server, is there something I can do to ensure it keeps working and not getting incompatible?

The OpenZFS project aligned the source repositories in the last few months and all implementations use the same source code. Thus I expect there to be only slight differences in the rollout of new ZFS feature flags.

Someone else might have a better idea on how to deal with it, if even necessary.

3.) I would like to encrypt my pools and I've read in the wiki article "ZFS on Linux" that Proxmox can't boot from encrypted pools. So my idea was to use ZFS to mirror two unencrypted SATA SSDs as Proxmox boot pool and two mirrored 500GB NVMe M2 SSDs as encrypted pool for my VMs.

You can setup a basic Debian installation with encryption if you want to encrypt the root FS. You can then install PVE on top of Debian. See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_install_proxmox_ve_on_debian

4.) How much space does the boot pool need? Is 120GB enough?

Definitely more than enough. Keep in mind that the `local` storage is a directory storage on the root pool. Thus if you want to use it (to store ISOs for example) you have that to keep in ming.

5.) How is Proxmox handling the ZFS caching and journaling? I would think that will happen on the boot pool like FreeNAS does. Are the caching and journaling datasets encrypted atleast?

ZFS caches are usually in the RAM. If you define an L2ARC it is up to you on which device you do that.

The logs are located on the boot device by default.

 

[Dunuin]

You can setup a basic Debian installation with encryption if you want to encrypt the root FS. You can then install PVE on top of Debian. See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_install_proxmox_ve_on_debian

But then I will have to use some other filesystem instead of zfs and need to use onboard raid if I want redundancy? Like ext4 with luks?

 

[guletz]

Once I updated FreeNAS and it asked me if I want to upgrade my pools too. With the hint that a newer version von ZFS on the pools could make them incompatible with ZFS on lower versions. Are FreeNAS and Proxmox using the same ZFS version? I would like to "zfs send" my backups of the Proxmox pools to my FreeNAS backup server.

If you want to play at lottery, yes, is possible, but nobody will guaranteed that you will win for each time Maybe you will win during thih moth, ... next yaear?

 

[guletz]

would like to encrypt my pools and I've read in the wiki article "ZFS on Linux" that Proxmox can't boot from encrypted pools

Encryption on zfs is very cool, but if want to be safe, wait one year, and then you can try it. In my own oppinion now zfs encryption could be rysky Even now could be some corner cases where you can have problems.

 

[Dunuin]

My new server finally arrived. But I'm still unsure what drives to buy and how to setup them.

The mainboard only got 1 M.2 slot and only PCIe 2.0 x4. But it supports splitting the PCIe 3.0 8x Slot into 4x4x so I should be able to buy a "Supermicro 2-Port NVme HBA AOC-SLG3-2M2-O" (40€) adapter to be able to add 2 NVMe PCIe 3.0 SSDs. There I could add 2 "500GB Samsung 970 Evo" as ZFS mirror as storage for my VMs. Unencrypted first and add encryption later. For the boot pool I would just use 2 "120GB Crucial BX500" SATA SSDs mirrored with ZFS and they will stay unencrypted. I plan to run emby, LAMP, some MySQL DBs, graylog, minecraft server, pihole, zabbix, ComiXed, jDownloader and deluge in the VMs.

1.) Is that hardware ok or do I not need NVMe SDDs for that? Except for the Minecraft server I'm the only person who uses the VMs so it shouldn't be that much IO. So I don't know if it is a bit overkill.
2.) What is the best alternative to backup the server if sending zfs snapshots isn't reliable between different OSs? I would bet Proxmox is able to snapshot the VMs and I could just use rsync to send the snapshots to my backup server? Is there a better way?
3.) What data will be stored on the boot pool? Is there anything worth encrypting? Does it cache/log any data from inside the VMs for example?

 

[guletz]

Hi,

There I could add 2 "500GB Samsung 970 Evo" as ZFS mirror

Do not do use any consummer SSD/NVME on any zfs pool. Use enetrprise class devices. I have use somthink like Samsung 970 Evo PRO, only as cache+slog, and during 1-2 weeks, the weare-level was 4%.

For the boot pool I would just use 2 "120GB Crucial BX500" SATA SSDs

Use this for ext4/LVM with some hardware mirror for PMX OS. From time to time boot-loader grub can be broken(not PMX fault), and is not so easy to recover.

2.) What is the best alternative to backup the server if sending zfs snapshots isn't reliable between different OSs? I would bet Proxmox is able to snapshot the VMs and I could just use rsync to send the snapshots to my backup server? Is there a better way?

Not sure if would be better, but for sure it will be relaible: use backup localy(compressed files) and then rsync to your NAS. At some point in the future will be also available a differential backup system.

Is there anything worth encrypting?

I think not! Anyway encryption is very cool and trending, but in reality is very difficult to understood and to manage in the wright way.

Does it cache/log any data from inside the VMs for example?

No. Only some data in case of swap using.

My recommandation is to use a simple setup, accomodate with PMX(learn, watch how it works), and after you have basic understandings about all of this(encryption included) the go on next level!

Good luck / Bafta!

 

[Dunuin]

Do not do use any consummer SSD/NVME on any zfs pool. Use enetrprise class devices. I have use somthink like Samsung 970 Evo PRO, only as cache+slog, and during 1-2 weeks, the weare-level was 4%.

Why? Can you give me some more details? I've read TRIM is now supported and the 970 EVO does atleast use a MLC cache. I don't have much data that will be written to the SSD. Maybe 2-10GB per day and I wanted to lower the swappiness so no SSD is used for caching.