Proxmox ZFS RAIDZ1 pool on SSD drives - sync parameter

[medeis_JAR]

Hi i have proxmox 4.3 host witch zfs raidz1 pool on 2 ssd.

I use proxmox host for VMs Windows OS.

I have UPS backup.

I have question about zfs "sync" parameter, it improve alot write performance but is it really unsafe to use ?

How sync=disbaled afect Widnows VMs ?

How much unsafe it is? what can i lose ?

 

[gkovacs]

Hi i have proxmox 4.3 host witch zfs raidz1 pool on 2 ssd.

I use proxmox host for VMs Windows OS.

I have UPS backup.

I have question about zfs "sync" parameter, it improve alot write performance but is it really unsafe to use ?

How sync=disbaled afect Widnows VMs ?

How much unsafe it is? what can i lose ?

Well, I'm not sure you can create a RAIDZ1 pool out of 2 SSDs (you need at least 3 drives), and even if you could, there is no advantage to a RAID1 (mirror) when using 2 drives, only disadvantage (much higher CPU usage due to parity calculation), and you also lose an entire disk (like with the mirror).

Regarding your question: here are the three possible sync behaviors of ZFS.

sync=standard
  This is the default option. Synchronous file system transactions
  (fsync, O_DSYNC, O_SYNC, etc) are written out (to the intent log)
  and then secondly all devices written are flushed to ensure
  the data is stable (not cached by device controllers).

sync=always
  For the ultra-cautious, every file system transaction is
  written and flushed to stable storage by a system call return.
  This obviously has a big performance penalty.

sync=disabled
  Synchronous requests are disabled.  File system transactions
  only commit to stable storage on the next DMU transaction group
  commit which can be many seconds.  This option gives the
  highest performance.  However, it is very dangerous as ZFS
  is ignoring the synchronous transaction demands of
  applications such as databases or NFS.
  Setting sync=disabled on the currently active root or /var
  file system may result in out-of-spec behavior, application data
  loss and increased vulnerability to replay attacks.
  This option does *NOT* affect ZFS on-disk consistency.
  Administrators should only use this when these risks are understood.

So in the event of a power failure, you would only lose the last couple of seconds of sync disk writes, since ZFS commits all your writes to disk in transaction groups at every 5 seconds by default. You can check this yourself:

root@proxmox:~# cat /sys/module/zfs/parameters/zfs_txg_timeout
5

I have no experience on how this would affect Windows guests.

 

[Nemesiz]

In [ power lost | software crash ] all your not writed async data will be lost - it is normal. All your data in ZIL (sync data) will be recovered and writed to pool.

In any case ( sync = standard | disabled ) your data will be not corrupted. I had never so any Windows VM doing recovering or fixing its hdd.

When to use sync = standard - then you have mix programs like SQL servers (if you don`t want to loose transaction) and they use sync flag on data write and some programs which not request sync writes.

Then you need to make sure all data are writed ASAP - sync = all

sync = disabled - when you do not have external ZIL and you dont want to have double writes in the same pool.

You can control ZFS write buffer with zfs_txg_timeout ( how long to buffer data before flush ) and zfs_dirty_data_max ( how big write cache if I`m not wrong ). Reaching write buffer ZFS hold accepting new writes while writes to data to pool.

 

[medeis_JAR]

Wow @gkovacs and @Nemesiz
thanks alot for you answears.

Please tell me, i have already two ssd's drive in mirror on ZFS, and use Windows VM with mysql database, so according to your informations i shouldn't use sync=disabled to make sure that SQL data will be ok?

I am aware that after power loss i always lose data but is there any difference between settings sync standard/disabled in loosing data ?

 

[fabian]

I am aware that after power loss i always lose data but is there any difference between settings sync standard/disabled in loosing data ?

yes. if you set sync=disabled, sync writes will be handled like async ones (buffered in RAM, writing application does not know whether data was actually written to disk already, on power loss everything that was not yet written out to disk is gone). with sync=standard, sync writes will be written to ZIL and only then the write will be returned as successful (on power loss, everything that has return as successful was written to ZIL and will be replayed on pool import). applications that write synchronously usually rely on this behaviour (e.g., to ensure consistency).

 

[medeis_JAR]

yes. if you set sync=disabled, sync writes will be handled like async ones (buffered in RAM, writing application does not know whether data was actually written to disk already, on power loss everything that was not yet written out to disk is gone). with sync=standard, sync writes will be written to ZIL and only then the write will be returned as successful (on power loss, everything that has return as successful was written to ZIL and will be replayed on pool import). applications that write synchronously usually rely on this behaviour (e.g., to ensure consistency).

Thanks now i fully understand differences ;]