Too slow windows startup on zfs?

[ozgurerdogan]

Overall performance is good. But if I shutdown and start windows guest, startup is extremely slow.
I already set that is mentioned here
echo 1073741824 > /sys/module/zfs/parameters/zfs_arc_min
echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_max

What could be problem? Server is 32 GB ram and have 5 vms each have 4 gb of ram. So Aroun 12 GB of ram is generally free.

I have 5 servers and all are same.

 

[LnxBil]

Please post more information about your system, especially about your zfs configuration (e.g. zpool status -v)

 

[killmasta93]

what is you pveperf?

 

[ozgurerdogan]

I saw a suggestion which was telling to set:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb     ONLINE       0     0     0

CPU BOGOMIPS:      49596.96
REGEX/SECOND:      1261878
HD SIZE:           1812.70 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     15753.01
DNS EXT:           18.29 ms
DNS INT:           126.21 ms

 

[LnxBil]

The numbers look good, almost too good: You're not using harddisks, are you?

 

[ozgurerdogan]

What do you mean "using harddisks" ? The problem is only with windows guests. Not always but most of time, startup seems to me taking hours to complete. But after setting;

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

seems normal now. Not sure if those setting would harm in future as one says "defrag".

 

[LnxBil]

What do you mean "using harddisks" ?

You have two devices in your vdev on your zpool, your fsync/sec value suggests that you use SSDs, is this correct? If not, there is something wrong with your numbers.

 

[ozgurerdogan]

yes I only use two sata disks.

 

[LnxBil]

Have you disabled sync on your pool? The numbers from pveperf are bogus and cannot be real.

 

[ozgurerdogan]

Yes I usually set these:

zfs set compression=on rpool
zfs set sync=disabled rpool
zfs set primarycache=all rpool
zfs set atime=off rpool
zfs set checksum=off rpool
zfs set dedup=off rpool

 

[Nemesiz]

You have pool from 2 disks. Don`t expect fast read, except your pool are from SSD.

 

[mir]

Yes I usually set these:

zfs set compression=on rpool
zfs set sync=disabled rpool
zfs set primarycache=all rpool
zfs set atime=off rpool
zfs set checksum=off rpool
zfs set dedup=off rpool

This must by far be one of the worst advices ever given!
disable sync: Do not honour sync requests from applications and instead give applications the impression that data is safely persisted. I hope no database server is running on this disk
disable checksum: Effectively break ZFS' self healing feature which prevents bit rot etc. Also more or less makes scrubbing useless.

 

[ozgurerdogan]

My problem here is that windows gueest's startups' are too slow sometimes. I am really happy overall performance. And I am aware of those performance settings.

 

[mir]

My problem here is that windows gueest's startups' are too slow sometimes. I am really happy overall performance. And I am aware of those performance settings.

I you consider your customers data of just modest value you would behave professionally and upgrade your hardware instead of gambling with the validity of there data!

 

[guletz]

disable sync: Do not honour sync requests from applications and instead give applications the impression that data is safely persisted.

{ This is not true. ZFS can not do sync writes. It only tell to the appl that ... yes, I do a sync}. In any case data on disk for zfs is ok with/without zfs sync on/off. But if you have some doubts about this, you can reduce the zfs flush time to disk from 5 seconds by default to 2-3 sec.
But I am agree with checksums=off it is a very bad ideea. This is the worst problem, and not the zfs sync on/off.

Late edit - I was wrong. I should say about faking O_DIRECT, and not about sync! My mistake, so thx. @mir

 

[mir]

ZFS can not do sync writes.

Come on, ZFS is a filesystem!

 

[tuonoazzurro]

My problem here is that windows gueest's startups' are too slow sometimes. I am really happy overall performance. And I am aware of those performance settings.

same problem here. excellent performance, but if the vms is shut down and next started, the boot can take also 30 min (windows 2016/10)

 

[Nemesiz]

Build ZFS pool from 8 or more disk if you want fast VM startup, or use SSD only.

 

[tuonoazzurro]

Build ZFS pool from 8 or more disk if you want fast VM startup, or use SSD only.

i understand that a mirror of 2 drives is not the best setupm but i've shut down 1 vm (30 sec). Restarted and took exactly more than 30 min to boot a clean windows installation. I don't think this is normal.

 

[Nemesiz]

It is normal. My ZFS pool with 2 hdd mirror or another pool with 3 hdd raidz VM startup takes some time. But in ZFS pool with 3 SSD raidz VM boots instantly.