The backups arn't written to ZFS but to another EXT3 drive. So only the reads are from ZFS. I believe ZFS tries to put the most frequently used data in ARC, so when a backup is made from every VM each night, maybe this ends up in ARC, poluting the normal reads?? Just a tought
Poor performance with ZFS
- Thread starter Silvos
- Start date
The backups arn't written to ZFS but to another EXT3 drive. So only the reads are from ZFS. I believe ZFS tries to put the most frequently used data in ARC, so when a backup is made from every VM each night, maybe this ends up in ARC, poluting the normal reads?? Just a tought
I think we are talking two different things here.
You state "But my performance is way slower:" and you show pveperf output. In that output the only FS performance parameter is "FSYNCS/SECOND: 128.89
That parameter is pretty low at ~129 for a sync activity using a slog device.
Other than that you don't present any evidence on slow FS performance, so there is nothing that can be debugged, sorry.
You state "But my performance is way slower:" and you show pveperf output. In that output the only FS performance parameter is "FSYNCS/SECOND: 128.89
That parameter is pretty low at ~129 for a sync activity using a slog device.
Other than that you don't present any evidence on slow FS performance, so there is nothing that can be debugged, sorry.
Yes 129 is pretty low, compared to the 2999.24 on -almost- the same system in the post of tom on 03-25:
Code:
root@proxmox:~# pveperf /space/
CPU BOGOMIPS: 57595.44
REGEX/SECOND: 1804289
HD SIZE: 14601.65 GB (space)
FSYNCS/SECOND: 2999.24
DNS EXT: 35.32 ms
DNS INT: 0.52 ms (proxmox.com)I seen more threads with this problem and if I read them it looks like they all use ZFS as root system.
Or can we drop any comparisson with pvepref's FSYNCS/SEC and ZFS?
I have a post above where I suggest running a iostat command during pveperf and see where is the bottleneck.
You mean this:
I can create the two windows, "zpool iostat -v 1" runs in one of them, but when I run "iostat -kx 1" iostat command doesn't exist nor is "zpool iostat -kx 1" (invalid option "k")
apt-get install sysstat (that package provides iostat)
And you need to run pveperf along each of "zpool iostat" and "iostat", not the iostats together.
Basically, you need to check with "zpool iostat" and/or "iostat" what disk(s) take the pveperf fsync load.
And you need to run pveperf along each of "zpool iostat" and "iostat", not the iostats together.
Basically, you need to check with "zpool iostat" and/or "iostat" what disk(s) take the pveperf fsync load.
Thank you. Was already searching for a package. Will try that tomorrow and post the results here.
Here is the first output, FSYNC was even worse.
Code:
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 68 93 6.41M 1.65M
raidz2 885G 15.4T 68 78 6.41M 989K
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 14 11 1.04M 381K
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 15 11 1.12M 379K
ata-ST3000DM001-1ER166_W500TF38-part3 - - 16 11 1.22M 379K
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 14 11 1.03M 381K
ata-ST3000DM001-1ER166_W500THXN-part3 - - 15 11 1.12M 379K
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 16 11 1.22M 379K
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 4.04M 37.2G 0 15 4 703K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 1 10 88.8K 970K
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 14 0 210K 7.99K
raidz2 885G 15.4T 14 0 210K 0
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 4 0 40.0K 0
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 1 0 20.0K 0
ata-ST3000DM001-1ER166_W500TF38-part3 - - 5 0 52.0K 0
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 1 0 28.0K 0
ata-ST3000DM001-1ER166_W500THXN-part3 - - 2 0 32.0K 0
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 10 0 79.9K 0
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 4.04M 37.2G 0 0 0 7.99K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 17 25 133K 201K
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 55 293 237K 1.58M
raidz2 885G 15.4T 55 282 237K 1.45M
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 4 41 36.0K 843K
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 1 49 24.0K 891K
ata-ST3000DM001-1ER166_W500TF38-part3 - - 28 49 136K 891K
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 7 50 48.0K 891K
ata-ST3000DM001-1ER166_W500THXN-part3 - - 3 41 20.0K 859K
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 24 41 104K 843K
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 6.85M 37.2G 0 10 0 124K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 61 24 215K 285K
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 22 0 116K 0
raidz2 885G 15.4T 22 0 116K 0
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 2 0 12.0K 0
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 1 0 7.99K 0
ata-ST3000DM001-1ER166_W500TF38-part3 - - 12 0 60.0K 0
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 1 0 16.0K 0
ata-ST3000DM001-1ER166_W500THXN-part3 - - 1 0 7.99K 0
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 13 0 56.0K 0
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 6.85M 37.2G 0 0 0 0
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 26 33 116K 197K
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 0 0 2.50K 0
raidz2 885G 15.4T 0 0 2.50K 0
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 0 0 0 0
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 0 0 0 0
ata-ST3000DM001-1ER166_W500TF38-part3 - - 0 0 4.00K 0
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 0 0 0 0
ata-ST3000DM001-1ER166_W500THXN-part3 - - 0 0 0 0
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 0 0 0 0
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 6.85M 37.2G 0 0 0 0
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 0 6 3.00K 37.0K
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 52 5 1.20M 116K
raidz2 885G 15.4T 52 0 1.20M 0
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 43 0 384K 0
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 29 0 280K 0
ata-ST3000DM001-1ER166_W500TF38-part3 - - 26 0 248K 0
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 41 0 296K 0
ata-ST3000DM001-1ER166_W500THXN-part3 - - 13 0 99.9K 0
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 8 0 75.9K 0
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 6.85M 37.2G 0 5 0 116K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 26 0 460K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 43 364 536K 2.98M
raidz2 885G 15.4T 43 359 536K 2.92M
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 18 39 112K 1.35M
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 8 37 48.0K 1.36M
ata-ST3000DM001-1ER166_W500TF38-part3 - - 13 38 75.9K 1.36M
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 19 37 156K 1.35M
ata-ST3000DM001-1ER166_W500THXN-part3 - - 10 39 87.9K 1.37M
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 10 39 83.9K 1.36M
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 6.85M 37.2G 0 4 0 56.0K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 31 0 257K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 28 292 259K 803K
raidz2 885G 15.4T 28 261 259K 584K
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 16 52 99.9K 588K
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 3 53 36.0K 616K
ata-ST3000DM001-1ER166_W500TF38-part3 - - 3 52 28.0K 608K
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 17 53 99.9K 612K
ata-ST3000DM001-1ER166_W500THXN-part3 - - 5 52 40.0K 600K
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 3 52 32.0K 588K
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 260K 37.2G 0 30 0 220K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 6 0 69.5K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 27 564 78.9K 1.37M
raidz2 885G 15.4T 27 564 78.9K 1.37M
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 12 119 56.0K 1.28M
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 1 128 12.0K 1.31M
ata-ST3000DM001-1ER166_W500TF38-part3 - - 0 126 7.99K 1.29M
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 16 126 72.0K 1.29M
ata-ST3000DM001-1ER166_W500THXN-part3 - - 0 124 4.00K 1.31M
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 0 119 4.00K 1.28M
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 708K 37.2G 0 0 0 0
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 10 0 165K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 41 606 48.5K 1.62M
raidz2 885G 15.4T 41 421 48.5K 920K
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 11 87 48.0K 971K
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 0 83 0 979K
ata-ST3000DM001-1ER166_W500TF38-part3 - - 6 83 28.0K 959K
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 11 83 48.0K 959K
ata-ST3000DM001-1ER166_W500THXN-part3 - - 0 87 4.00K 983K
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 10 87 44.0K 975K
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 260K 37.2G 0 184 0 739K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 3 0 28.5K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 64 245 341K 731K
raidz2 885G 15.4T 64 149 341K 295K
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 23 32 132K 336K
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 5 29 59.9K 336K
ata-ST3000DM001-1ER166_W500TF38-part3 - - 16 29 95.9K 332K
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 25 29 108K 332K
ata-ST3000DM001-1ER166_W500THXN-part3 - - 6 32 32.0K 344K
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 10 32 79.9K 336K
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 136K 37.2G 0 95 0 436K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 21 0 245K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 77 477 425K 1.64M
raidz2 885G 15.4T 77 466 425K 1.31M
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 39 72 180K 1.09M
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 16 87 79.9K 1.12M
ata-ST3000DM001-1ER166_W500TF38-part3 - - 8 87 40.0K 1.11M
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 43 87 196K 1.11M
ata-ST3000DM001-1ER166_W500THXN-part3 - - 13 72 68.0K 1.09M
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 7 72 36.0K 1.09M
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 1.79M 37.2G 0 10 0 332K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 39 0 341K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
--------------------------------------------------- ----- ----- ----- ----- ----- -----
rpool 885G 15.4T 42 570 334K 1.52M
raidz2 885G 15.4T 42 565 334K 1.44M
ata-ST3000DM001-1ER166_W500V5P2-part3 - - 28 111 128K 1.34M
ata-ST3000DM001-1ER166_W500V5P4-part3 - - 11 119 56.0K 1.34M
ata-ST3000DM001-1ER166_W500TF38-part3 - - 2 116 20.0K 1.32M
ata-ST3000DM001-1ER166_W500TF0G-part3 - - 22 118 120K 1.31M
ata-ST3000DM001-1ER166_W500THXN-part3 - - 9 109 64.0K 1.35M
ata-ST3000DM001-1ER166_W500TEVZ-part3 - - 7 109 44.0K 1.32M
logs - - - - - -
ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1 260K 37.2G 0 4 0 79.9K
cache - - - - - -
scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part2 278G 16.0E 4 0 162K 0
--------------------------------------------------- ----- ----- ----- ----- ----- -----
CPU BOGOMIPS: 57600.72
REGEX/SECOND: 1879422
HD SIZE: 10876.31 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 99.93To me it doesn't look like your slog device is used at all. I assume that the system was completely idle while doing these benchmarks right?
Using sync writes and a slog device should show steady I/O on slog device and bursts on main drives.
Using sync writes and a slog device should show steady I/O on slog device and bursts on main drives.
No the system isn't idle. I have VM's running, but as you can see they don't use much IO
Do you have any idea why the slog device isn't used? Strange thing is that I added the SLOG with:
"scsi-SATA_INTEL_SSDSC2BA2BTHV50530396200MGN-part1" and now it shows up as:
"ata-INTEL_SSDSC2BA200G4_BTHV50530396200MGN-part1"
Seems to be working. My FSYNCS were still pretty slow though even with the SSD cache/logs drive at around 400.
Actual performance seems to be okay, busy restoring a backup from a USB 2 external drive. Will only know when I start working with the VMs.
While copying from external:
Actual performance seems to be okay, busy restoring a backup from a USB 2 external drive. Will only know when I start working with the VMs.
While copying from external:
Code:
root@proxmox0:~# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
------------------------------------------------ ----- ----- ----- ----- ----- -----
zfs0 285G 3.35T 0 5.12K 408 29.7M
raidz2 285G 3.35T 0 5.12K 56 29.7M
scsi-SATA_ST1000VX000-1CU_Z1D8RSG0 - - 0 219 503 26.3M
scsi-SATA_ST1000VX000-1CU_Z1D8RRRT - - 0 219 327 26.4M
scsi-SATA_ST1000VX000-1CU_Z1D8SBR9 - - 0 220 359 26.4M
scsi-SATA_ST1000DM003-9YN_S1D1V9LC - - 0 218 340 26.4M
logs - - - - - -
scsi-SATA_Samsung_SSD_850S21TNSAG223774B-part1 4K 18.5G 0 0 352 2.78K
cache - - - - - -
scsi-SATA_Samsung_SSD_850S21TNSAG223774B-part2 8.04G 66.5G 0 26 117 2.95M
------------------------------------------------ ----- ----- ----- ----- ----- -----The most dangerous aspect of sync=disabled is that the client guest can issue a write barrier if it is doing metadata-related things on that FS, and won't continue until the write is ACK'ed. But here we are lying and saying 'we wrote it!'. It's entirely possible to cause filesystem corruption (for the guest OS, not ZFS...)
I think FSYNCS measurement are just wrong with zfs .
zfs 'raid-10' shows a lot lower then 3ware raid-10 system. like 15-20 times less .
However for our main business application the speed is close to the same.
Currently we use zfs / drbd / lvm with one drbd per kvm. Our goal is high data availability with users having zero keyboard lags and zero unscheduled downtime.
I figure there will be improvements to pve/zfs in the future , just like with ceph and other storage.
Our mission critical app uses openvz. I find it is more reliable then kvm for that system. Our main app runs old debian etch. most other systems use jessie and have no kvm issues.
zfs 'raid-10' shows a lot lower then 3ware raid-10 system. like 15-20 times less .
However for our main business application the speed is close to the same.
Currently we use zfs / drbd / lvm with one drbd per kvm. Our goal is high data availability with users having zero keyboard lags and zero unscheduled downtime.
I figure there will be improvements to pve/zfs in the future , just like with ceph and other storage.
Our mission critical app uses openvz. I find it is more reliable then kvm for that system. Our main app runs old debian etch. most other systems use jessie and have no kvm issues.
Would you mind elaborating a bit?
Do you mean that HW raid is supposed to be faster or that the test is skewed?
If you are measuring fsync's then having a cache on your RAID controller (assuming it is battery backed up and gets actually used safely) is going to make it faster. Having an SSD SLOG device on your ZFS is the way to speed up that operation in ZFS.