I was interested in trying to virtualise a FreeNAS instance which is only responsible for some temporary storage for security camera recording/backups etc and found performance under Proxmox to be awful. Im curious if anyone else has observed this or of theres a magic CPU flag etc which fixes this.
Hardware is Supermicro X9SRL-F, 64GB, Xeon e5-1620v2, LSI 9300-8i, Intel 520 10gig network card.
Running Proxmox 4.3-12/6894c9d9
FreeNAS-9.10.1-U4 (ec9a7d3)
Theres no other VMs on this system running during these tests.
Am testing with a simple 'dd' command
Have tested and confirm identical performance with following options
and
With a compressed (lz4) array (this blows with circa 10MB/s from the same SSD drives)
and
Performance drops dramatically form 500MB/s to 10MB/s on the SSD array and from 120MB/s to 10MB/s on the WD RED due to CPU bottleneck but adding cores doesn't help so is there another solution or is it just a case of FreeNAS doesn't play well with Proxmox?
Hardware is Supermicro X9SRL-F, 64GB, Xeon e5-1620v2, LSI 9300-8i, Intel 520 10gig network card.
Running Proxmox 4.3-12/6894c9d9
FreeNAS-9.10.1-U4 (ec9a7d3)
Theres no other VMs on this system running during these tests.
Am testing with a simple 'dd' command
Code:
dd if=/dev/zero of=/mnt/tankssd/dd.tst bs=2048000 count=262000
Have tested and confirm identical performance with following options
- 2 and 4 cores
- CPU=host and QEMU64
- 16GB and 32GB RAM allocated
- WD Red 4TB and Samsung 850 Pro 256GB
- 10% free vs 100% free pool space
Code:
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
tankssd 14.0G 222G 0 3.90K 0 496M
mirror 14.0G 222G 0 3.90K 0 496M
gptid/f090ced5-bb64-11e6-97fc-171bf529e9b8 - - 0 3.91K 0 496M
gptid/f0ce17b9-bb64-11e6-97fc-171bf529e9b8 - - 0 3.90K 0 496M
-------------------------------------- ----- ----- ----- ----- ----- -----
and
Code:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
2 0 0 3903M 27G 6094 0 1 0 10833 134 10333 10333 111 7107 1439 5 10 86
1 0 0 3903M 26G 4735 0 0 0 8892 126 4022 4016 7740 3795 56611 4 53 43
1 0 0 3903M 26G 9677 0 0 0 17761 126 3635 3643 6971 5057 38518 6 77 17
2 0 0 3903M 25G 4684 0 0 0 8883 126 3981 3978 7697 3609 55220 3 61 36
1 0 0 3903M 25G 489 0 0 0 0 126 3998 3994 7763 2138 56854 1 55 44
1 0 0 3903M 24G 5016 0 0 0 8882 125 4005 4000 7856 3732 56925 4 57 39
2 0 0 3896M 23G 7917 0 0 0 17659 126 3933 3945 7745 4565 52515 5 60 34
0 0 0 3903M 23G 11038 0 0 0 17857 126 3125 3124 5771 5521 29260 11 78 12
1 0 0 3903M 22G 5554 0 0 0 9140 126 3999 3998 7869 5962 56800 5 59 37
1 0 0 3903M 22G 4723 0 0 0 8945 126 3989 3983 7754 3575 55717 3 53 44
1 0 0 3903M 21G 5020 0 0 0 8882 126 3998 3994 7727 3834 55828 4 56 41
0 0 0 3903M 20G 5016 0 0 0 8881 126 4022 4019 7708 3702 54599 4 50 46
2 0 0 3896M 20G 7855 0 0 0 17653 126 3945 3962 7734 3990 42821 5 41 54
1 0 0 3903M 20G 2274 0 0 0 286 126 3930 3926 7416 2950 39975 2 69 29
1 0 0 3903M 20G 4891 0 0 0 8880 126 4027 4022 7751 3450 51036 4 40 56
0 0 0 3903M 19G 4735 0 0 0 8881 126 4030 4023 7762 3499 52647 3 44 53
0 0 0 3903M 19G 5014 0 0 0 8881 126 3995 3990 7736 3520 53165 4 47 49
0 0 0 3903M 18G 14347 0 0 0 26658 124 3946 3944 7579 6812 49687 11 51 38
1 0 0 3903M 18G 5151 0 0 0 8893 126 3789 3809 7297 5343 46307 4 57 39
1 0 0 3903M 17G 5022 0 0 0 8881 126 3919 3914 7368 3564 43008 3 70 26
1 0 0 3903M 17G 5015 0 0 0 8881 126 3986 3982 7680 3672 52283 3 44 53
2 0 0 3889M 16G 318 0 0 0 8397 126 4016 4010 7761 2286 54775 0 49 51
1 0 0 3903M 16G 9458 0 0 0 9363 125 3991 3986 7551 4684 50820 7 49 44
1 0 0 3894M 15G 5423 0 0 0 9567 125 3926 3941 7494 4071 46480 3 67 29
0 0 0 3894M 15G 4640 0 0 0 8880 125 3765 3772 7240 3555 44864 4 69 28
0 0 0 3894M 15G 5017 0 0 0 8882 125 3997 4003 7771 3559 53724 4 39 57
1 0 0 3894M 14G 5033 0 0 0 8882 125 3976 3974 7723 3555 53041 4 44 52
4 0 0 3895M 14G 13998 0 0 0 26644 125 3938 3928 7589 7194 49667 11 47 42
1 0 0 3894M 13G 5734 0 0 0 8923 125 3515 3509 6758 4998 41630 4 65 32
With a compressed (lz4) array (this blows with circa 10MB/s from the same SSD drives)
Code:
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
tankssd 2.91M 236G 0 2.71K 0 10.8M
mirror 2.91M 236G 0 2.71K 0 10.8M
gptid/f090ced5-bb64-11e6-97fc-171bf529e9b8 - - 0 651 0 11.2M
gptid/f0ce17b9-bb64-11e6-97fc-171bf529e9b8 - - 0 650 0 11.2M
-------------------------------------- ----- ----- ----- ----- ----- -----
and
Code:
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
2 0 0 3912M 29G 5030 0 0 0 8885 125 631 632 1282 6575 8299 3 85 13
2 0 0 3905M 29G 3811 0 0 0 8777 126 479 479 972 5983 6623 1 91 8
1 0 0 3912M 29G 1322 0 0 0 106 125 593 589 1160 5180 7682 4 87 9
2 0 0 3894M 29G 5644 0 0 0 17658 125 606 604 1177 7213 7714 4 88 8
3 0 0 3912M 29G 8778 0 0 0 9005 125 411 408 797 7034 5715 7 87 5
2 0 0 3912M 29G 5032 0 0 0 8884 122 637 636 1238 6326 8550 3 87 10
2 0 0 3912M 29G 4895 0 0 0 8884 125 575 572 1178 6473 8044 4 87 9
1 0 0 3912M 29G 9681 0 0 0 17767 125 390 386 762 7943 5466 6 86 8
1 0 0 3912M 29G 5322 0 0 0 8892 125 557 556 1083 7985 7547 3 86 11
2 0 0 3905M 29G 8752 0 0 0 17659 125 576 574 1115 7220 7593 5 86 9
1 0 0 3912M 29G 1346 0 0 0 105 125 736 736 1439 5313 9613 2 86 12
1 0 0 3912M 29G 4733 0 0 0 8884 130 575 574 1111 6172 7770 4 85 11
1 0 0 3912M 29G 5015 0 0 0 8883 125 637 635 1222 6336 8452 5 84 11
2 0 0 3912M 29G 4735 0 0 0 8884 122 486 485 955 6425 6608 4 88 9
2 0 0 3912M 29G 9642 0 0 0 17764 125 412 412 805 7852 5746 7 84 9
1 0 0 3912M 29G 5036 0 0 0 8885 125 623 621 1207 6485 8155 4 85 11
2 0 0 3903M 29G 3126 0 0 0 8778 125 631 631 1223 5781 8357 1 88 11
1 0 0 3912M 29G 6653 0 0 0 9005 125 531 530 1005 6613 7158 6 85 9
2 0 0 3989M 29G 11428 0 0 0 12861 125 606 604 1193 14699 7999 12 83 5
2 0 0 3989M 29G 9357 0 0 0 17776 128 695 693 1381 7415 8860 30 70 0
2 0 0 3989M 29G 5049 0 0 0 8886 128 810 810 1620 5632 10132 32 68 0
2 0 0 3989M 29G 5016 0 0 0 8883 129 723 723 1448 6001 9186 23 77 0
3 0 0 3989M 29G 4729 0 0 0 8883 129 721 721 1449 6064 9290 23 77 0
2 0 0 3989M 29G 5024 0 0 0 8883 128 826 825 1660 7290 10556 37 63 0
3 0 0 3983M 29G 9417 0 0 0 17663 128 821 823 1618 6571 9923 39 61 0
2 0 0 3989M 29G 400 0 0 0 104 128 949 948 1885 4330 11694 38 62 0
2 0 0 3989M 29G 5041 0 0 0 8891 128 832 828 1657 5647 10438 35 65 0
2 0 0 3989M 29G 9635 0 0 0 17768 130 738 738 1467 7358 9316 33 67 0
Performance drops dramatically form 500MB/s to 10MB/s on the SSD array and from 120MB/s to 10MB/s on the WD RED due to CPU bottleneck but adding cores doesn't help so is there another solution or is it just a case of FreeNAS doesn't play well with Proxmox?