Test result: Disk performance

mir

Famous Member
Apr 14, 2012
3,568
127
133
Copenhagen, Denmark
Hi all,

I have made some disk performance test under pve-2.2 and the overall conclusion is as follows using EXT4 in a Debian Wheezy client:

  • raw gives better performance than qcow2
  • iSCSI gives better performance than NFSv3
  • And the big surprise: SATA seems to give the overall best performance of all the available Bus options.

Given the combination iSCSI, RAW and EXT4 the cache method to choose is write-through since this gives a performance boost in and order of magnitude between 2 - 4 times above the other available cache options. Detailed measures below.

Anybody who can confirm these test results?

Code:
bonnie++ -b -u root -d /tmp (NFSv3, ext4, writeback)


Version  1.96  Bus          ------Sequential Output------ --Sequential Input- --Random-
Concurrency             1   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine                Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
debian7-raw    virtio   2G   547  98 23306   6 19116   4  1570  94 80531   8 272.4   7
debian7-qcow2  virtio   2G   244  99 16649   5 18962   5  1335  95 66210   8 263.0   6
debian7-raw    sata     2G   281  99 21712   7 18895   6  1297  94 76488  11 197.7   9
debian7-qcow2  sata     2G   264  99 20075   7 18485   6  1298  92 83979  12 187.0   6
debian7-raw    ide      2G   227  99 15071   4 18158   7   992  74 81723  17 129.6  10
debian7-qcow2  ide      2G   265  99 16801   4 18858   7  1625  97 77032  16 134.3   9
debian7-raw    scsi     2G   226  99 20613   6 17915   6  1190  89 68363  10 281.4  14
debian7-qcow2  scsi     2G   557  99 19452   6 17681   6  1259  89 76981  12 256.1  29


Version  1.96               ------Sequential Create------ --------Random Create--------
                            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
debian7-raw    virtio   16   283   1 +++++ +++   358   2   274   1 +++++ +++   360   1
debian7-qcow2  virtio   16   266   1 +++++ +++   352   1   263   1 +++++ +++   345   1
debiant-raw    sata     16   581   5 +++++ +++   790   4   589   5 +++++ +++   788   4
debian7-qcow2  sata     16   290   2 +++++ +++   376   2   291   2 +++++ +++   375   2
debian7-raw    ide      16   209   1 +++++ +++   290   2   210   1 +++++ +++   290   1
debian7-qcow2  ide      16   204   1 +++++ +++   277   1   204   1 +++++ +++   278   1
debian7-raw    scsi     16   240   1 +++++ +++   326   1   243   1 +++++ +++   324   1
debian7-qcow2  scsi     16   233   1 +++++ +++   313   1   233   1 +++++ +++   294   1


bonnie++ -b -u root -d /tmp (iscsi, ext4, writeback)


Version  1.96  Bus          ------Sequential Output------ --Sequential Input- --Random-
Concurrency             1   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine                Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
wheezy-raw     virtio   2G   111  99 29793   6 17130   3   295  99  81120   8 756.3  23
wheezy-raw     sata-wb  2G   110  99 38680  12 16544   5  1539  76 103913  15 719.9  74
wheezy-raw     ide      2G   111  99 28716   9 21751   7   331  99 329385  60  2023 132
wheezy-raw     scsi     2G   111  99 27811   9 24284   9   342  99 148113  93  1195 109
wheezy-raw     sata-wt  2G   112  99 25472   7 23068   6   329 100 406291  70  1903 192
wheezy-raw     sata-ds  2G   111  99 28809   8 15397   4   535  99  52301   7 145.9   7
wheezy-raw     sata-nc  2G   112  99 28645   7 15710   4   350  99  53315   7 137.2   6


Version  1.96               ------Sequential Create------ --------Random Create--------
                            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
                      files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
wheezy-raw     virtio   16   295   1 +++++ +++   376   1   302   2 +++++ +++   379   1
wheezy-raw     sata-wb  16   557   4 +++++ +++   680   4   541   4 +++++ +++   703   4
wheezy-raw     ide      16   221   1 +++++ +++   308   2   234   1 +++++ +++   318   1
wheezy-raw     scsi     16   256   2 +++++ +++   335   1   265   2 +++++ +++   346   1
wheezy-raw     sata-wt  16   557   4 +++++ +++   680   4   541   4 +++++ +++   703   4
wheezy-raw     sata-ds  16   310   1 +++++ +++   388   1   320   1 +++++ +++   403   1
wheezy-raw     sata-nc  16   274   2 +++++ +++   356   2   280   2 +++++ +++   366   2
 
View attachment 1158View attachment 1159

Writeback will be faster but AFAIK is not that safe against hardware failures.

cache=writeback or cache=none, are not safe, as you can loose datas (some seconds of write) in case of powerfailure.
Now you really need to use barriers if your kernel is < 2.6.37. (or it can break your filesystem in case of powerfailure, and not only loose last datas seconds)
 
SATA seems to be better on writes, but it uses alot more CPU power and IO/s seems to lower then with VirtIO.
iscsi-test.jpg

Test on iSCSI @ 1Gbps (iometer)
4K READ 100% SEQ512K WRITE 100% SEQ4K WRITE 100% SEQ
BUSIO/sCPU %READ SPEEDBUSIO/sCPU %WRITE SPEEDBUSIO/sCPU %WRITE SPEED
VIRTIO2500037100VIRTIO90045VIRTIO180002270
SATA28009010SATA1701090SATA2000958
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!