PVE Storage Replication and PVE-zsync

[ozgurerdogan]

I am very unfamiliar to zfs but I learnt to I need to use it for pve-zsync (I prefered to use to pve storage replication but my servers are not on same subnet/network so I go with pve-zysnc)

So I have only one disk, do I have to create swap volume or any special volume on disk ? All I did is:

zpool create -f -o ashift=12 local2 /dev/sdb

And create vms on local2. Is this enough? With one disk, do I need to create log, cache partitions?

 

[harvie]

create pool called tank ("tank" is most commonly used and reccomended name for zfs pool)

zpool create tank mirror /dev/sda /dev/sdb
or without raid (not reccomended):
zpool create tank /dev/sda

i am not using ashift, because after some reading on this topic i got impression, that defaults are more than sane.

useful defaults to improve performance:

zfs set atime=off tank
zfs set compression=on tank

create filesystem on tank pool for your vps

zfs create tank/vps

now in proxmox web ui add new ZFS(!) storage and enter tank/vps in there. This is important. Don't just use /tank/vps as local directory. You have to add storage of ZFS type. That way proxmox will be able to use all ZFS specific features.

i also like to create two more

zfs create tank/backup
zfs create tank/local

which i setup for backups and local storage (that one must be symlinked), so i have everything on ZFS and can delete other storages (i have proxmox installed on SSDs that are relatively small, so i don't really want to put backups and other data there).

 

[ozgurerdogan]

Thank you I have more questions please;
1. Why can not I use tanks for iso and backup? When choosing ZFS storage only "Disk Image, Contatiner" options are avaliable.
2. When using ssh, I can not see any files in tank folders. How to copy, upload files with ssh?
3. When doing any compression, would not it make load on memory and cpu?

 

[harvie]

1.) when you want iso and backup, you have to add storage not as ZFS, but as regular directory. but don't do that for disk images and containers, that would be bad.

2.) don't know what you mean. everything is mounted in /tank/ directory

3.) No. the compression is ZFS best practice. It defaults to very efficient algorithm, so don't worry. Also ZFS is mostly slowed down by accessing the disk, so it has plenty of time to do the (de)compression while waiting for the disk as CPU is much faster than disk. It also means that you don't need to write/read so much data to slow disk (because it's compressed), which also alleviates some problems related to slow disk access. However do not enable "deduplication" (it's not enabled by default) it takes lots of RAM.

 

[ozgurerdogan]

When I ssh to server and enter tank directory after creating some vm in gui, I see no files in there. I must see disk images but nothing in it.

 

[harvie]

Oh sure. CTs in /tank, but VMs go to /dev/zvol.
This is realy cool about ZFS. It can create virtual block devices in it (similar to how LVM works), so you don't have owerhead of two filesystems stacked. So if you create VM it creates blockdev rather than just file. For CTs it creates new ZFS dataset (like sub-filesystem), which is similary cool feature, but it looks like plain directory in /tank.

Note that you can't just copy image to /dev/zvol/tank/vps/something... You have to create it first and then dd image to it...

This is why you have to add ZFS as ZFS rather than plain directory. In that case it would just create regular image file on ZFS and you would lose performance and all of cool features and flexibility that ZFS offers.

 

[ozgurerdogan]

I am currently using ext4 but for PVE-zsync, I need to go with zfs. It seems little compliated. Do you suggest it over ext4? Is it stable and durable enough?
I see high cpu usage also, is there something to check? Cpu is around %70 with empty vm. Only isntalled windows 2016 for test.

 

[harvie]

I don't think it's complicated. It's pretty straightforward once you get grip for it. ZFS sacrifices some of IO performance for it's features (after all it's copy-on-write filesystem). But it balances this out by using extensive RAM caching. (Well it's good idea to buy enough RAM).

I use ZFS on Proxmox machine with 30 Linux containers (two drives in ZFS mirror + SSD cache) and i guess i can fit 30 more containers (60 total) if i add some RAM and another ZFS mirror with another two drives. I don't use Proxmox to host Windows, so i can't really tell about that.

If you don't want to run containers, but only VMs it might be better (performance-wise) to use LVM instead of ZFS. And handle syncing in another way. Not sure. If you don't know much about Linux it might be hard to figure this out for you...

 

[ozgurerdogan]

thank you for all your helps.