Hi All,
I have been working on optimizing our cluster performance and I noticed that using krbd will give us approx 10X read/write performance gain compared to librbd.
And while I am thinking to switch I want to understand why ?.
I know krbd is running in the kernel so it in not userspace process which gives it advantages when dealing with memory allocation and other stuff but for me that does not explain 10X performance gain .
if the gain was 30 to 50 % more it would have been understandable but we are talking about 1000% gain here.
I need to understand before switching to krbd as to switch i have to disable all the rbd features on the images except layering (without object-map deleting big rbd images is slow)
for the test mythology and how i confirmed it was only switching between krbd and librbd that makes the difference i did this
1- i tested on new ceph pool with new proxmox server and found krbd is approx 8X better
2- i created new proxmox server and created new vm on it using librbd and tested disk performance
then on same vm on same server i switched to krbd , removed some rbd image features ,stopped and started the vm and i got 10X gain
3- on the same vm on step 2 which now work with krbd i switched it back to librbd and i lost the performance gains
4- on our production pool and our production ceph cluster I did this
A. create vm normally using proxmox librbd
B. test vm performance
C. stop vm
D. remove extra rbd image features that dont work with krbd
E- switch the whole proxmox cluster rbd storage to krbd (this means if at that moment any vm reboot it cannot boot but that was just for 1 min till i do the test)
F- test performance on the same vm after switching and we got 10X gain
G- revert the rbd storage in proxmox to non krbd
H- reboot the vm back which has some features missing like object map and retest which gave bad results again
I- reading the rbd image features that was removed manually and rebuilding the object map for the rbd image then retest again and no clear gain .
Note the images we are testing on are 2TB+ maybe that is a factor which give advantage to krbd not sure.
So does anyone have explanation as i plan to switch everything to krbd but need to understand first
Thanks
I have been working on optimizing our cluster performance and I noticed that using krbd will give us approx 10X read/write performance gain compared to librbd.
And while I am thinking to switch I want to understand why ?.
I know krbd is running in the kernel so it in not userspace process which gives it advantages when dealing with memory allocation and other stuff but for me that does not explain 10X performance gain .
if the gain was 30 to 50 % more it would have been understandable but we are talking about 1000% gain here.
I need to understand before switching to krbd as to switch i have to disable all the rbd features on the images except layering (without object-map deleting big rbd images is slow)
for the test mythology and how i confirmed it was only switching between krbd and librbd that makes the difference i did this
1- i tested on new ceph pool with new proxmox server and found krbd is approx 8X better
2- i created new proxmox server and created new vm on it using librbd and tested disk performance
then on same vm on same server i switched to krbd , removed some rbd image features ,stopped and started the vm and i got 10X gain
3- on the same vm on step 2 which now work with krbd i switched it back to librbd and i lost the performance gains
4- on our production pool and our production ceph cluster I did this
A. create vm normally using proxmox librbd
B. test vm performance
C. stop vm
D. remove extra rbd image features that dont work with krbd
E- switch the whole proxmox cluster rbd storage to krbd (this means if at that moment any vm reboot it cannot boot but that was just for 1 min till i do the test)
F- test performance on the same vm after switching and we got 10X gain
G- revert the rbd storage in proxmox to non krbd
H- reboot the vm back which has some features missing like object map and retest which gave bad results again
I- reading the rbd image features that was removed manually and rebuilding the object map for the rbd image then retest again and no clear gain .
Note the images we are testing on are 2TB+ maybe that is a factor which give advantage to krbd not sure.
So does anyone have explanation as i plan to switch everything to krbd but need to understand first
Thanks