Performance problems with DRBD - slow i/o writes

F

fluke

Guest
Hi,

I have done a fresh install of Proxmox 1.4, then performed shrink of the volume group, so I could free space for a drbd device. Installation went fine, but I am having terrible performance problems - with only one machine running I have 10-30% IO delays. Here are some basic performance results (drbd was in primary/secondary mode and no VM's were running):

1. /dev/drbd0 - created according to the HOWTO, published on the proxmox site
Code:
time sh -c "dd if=/dev/zero of=/dev/drbdvg/vm-101-disk-1 bs=8k count=200000 &&sync"
200000+0 records in
200000+0 records out
1638400000 bytes (1.6 GB) copied, 123.873 s, 13.2 MB/s
real    2m3.878s
user    0m0.016s
sys     0m1.788s

Here speed is 13.2 MB/s

2. /dev/drbd1 - created on top of /dev/pve/data and mounted on /var/lib/vz with ext3 filesystem
Code:
time sh -c "dd if=/dev/zero of=/var/lib/vz/test3.raw bs=8k count=200000 &&sync"
200000+0 records in
200000+0 records out
1638400000 bytes (1.6 GB) copied, 6.71476 s, 244 MB/s
real    0m41.549s
user    0m0.024s
sys     0m4.312s

Here speed is 39 MB/s

3. Regular volume
Code:
time sh -c "dd if=/dev/zero of=/root/test3.raw bs=8k count=200000 &&sync"
200000+0 records in
200000+0 records out
1638400000 bytes (1.6 GB) copied, 4.20871 s, 389 MB/s
real    0m11.528s
user    0m0.012s
sys     0m2.948s

Here speed is 142MB/s

Any help and share of experience will be very much appreciated.
 
I really don't have any idea what's going on but when I have drbd on top of lvm I have no problems with i/o write speed (it's around 100MB/s), but when I have lvm on top of drbd (the required setup) speed is 10 times slower!

Anyone has any idea?
 
I really don't have any idea what's going on but when I have drbd on top of lvm I have no problems with i/o write speed (it's around 100MB/s), but when I have lvm on top of drbd (the required setup) speed is 10 times slower!

I guess it is best to ask such questions on the drbd list.
 
I guess it is best to ask such questions on the drbd list.

Yes, but just to mention one more thing:
I have done a fresh install of debian lenny on both servers and then installed drbd8 modules and utils, drbd version was 8.0.14 and a simple write test produced:

Code:
proxmox2:~# dd if=/dev/zero of=/dev/drbd0 bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 38.625 s, 111 MB/s

Then on the first server installed the pve kernel and upgraded drbd to 8.3.2. Now we have:

Code:
proxmox1:~# dd if=/dev/zero of=/dev/drbd0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 131.256 s, 8.2 MB/s

And on the second one, which still runs the standard debian kernel and old drbd version we have:

Code:
proxmox2:~# dd if=/dev/zero of=/dev/drbd0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 20.9044 s, 51.4 MB/s

It seems, that the pve kernel 2.6.24-9 and drbd 8.3.2 slow things down.
 
Yes, but just to mention one more thing:
I have done a fresh install of debian lenny on both servers and then installed drbd8 modules and utils, drbd version was 8.0.14 and a simple write test produced:

Code:
proxmox2:~# dd if=/dev/zero of=/dev/drbd0 bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 38.625 s, 111 MB/s
Then on the first server installed the pve kernel and upgraded drbd to 8.3.2. Now we have:

Code:
proxmox1:~# dd if=/dev/zero of=/dev/drbd0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 131.256 s, 8.2 MB/s
And on the second one, which still runs the standard debian kernel and old drbd version we have:

Code:
proxmox2:~# dd if=/dev/zero of=/dev/drbd0 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 20.9044 s, 51.4 MB/s
It seems, that the pve kernel 2.6.24-9 and drbd 8.3.2 slow things down.

the current pvetest repository have DRBD 8.3.4 for all kernel branches (2.6.18, 2.6.24 and 2.6.32), including the 8.3.4 tools.
maybe you can test it?
 
the current pvetest repository have DRBD 8.3.4 for all kernel branches (2.6.18, 2.6.24 and 2.6.32), including the 8.3.4 tools.
maybe you can test it?

Initial tests show that on 2.6.24 drbd (8.3.4) is slow and on 2.6.32 drbd is OK. I will now reinstall from the proxmox installer and do some more tests.
 
Last edited by a moderator:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!