Extremly slow LXC using CEPH huge difference comparated with kvm

[egoistdream]

Hi,

I configured Proxmox 5.3 with ceph in a 3 node cluster. When I start to test the setup I see a huge difference between the kvm and lxc using the same ceph pools, the pools have the size of 2 and min size 1.
The LXC is very very slow even if I try to update, KVM run ok.

So, I start to run some tests because it seems strange and I never have problems like this, here are the results:


Sequential Reads – Async mode – 8K block size – Direct IO – 100% Reads
LXC
read: IOPS=2081, BW=16.3MiB/s (17.1MB/s)(8192MiB/503717msec)
iops : min= 2, max= 1616, avg=277.15, stdev=370.69, samples=7547
KVM
read: IOPS=39.9k, BW=311MiB/s (327MB/s)(8192MiB/26309msec)
iops : min= 3116, max= 7928, avg=6168.21, stdev=922.54, samples=335

Sequential Writes – Async mode – 32K block size – Direct IO – 100% Writes
LXC
write: IOPS=80, BW=2589KiB/s (2651kB/s)(1517MiB/600061msec)
iops : min= 1, max= 42, avg=20.71, stdev= 9.79, samples=4684
KVM
write: IOPS=13.0k, BW=406MiB/s (426MB/s)(8192MiB/20163msec)
iops : min= 2188, max= 4184, avg=3295.68, stdev=459.19, samples=157

Random Reads – Async mode – 8K block size – Direct IO – 100% Reads
LXC
write: IOPS=77, BW=4958KiB/s (5077kB/s)(2911MiB/601201msec)
iops : min= 1, max= 20, avg=11.76, stdev= 5.16, samples=7916
KVM
read: IOPS=57.8k, BW=452MiB/s (474MB/s)(16.0GiB/36277msec)
iops : min= 2720, max= 5238, avg=4163.64, stdev=422.26, samples=990

Random Writes – Async mode – 64K block size – Direct IO – 100% Writes
LXC
write: IOPS=77, BW=4958KiB/s (5077kB/s)(2911MiB/601201msec)
iops : min= 1, max= 20, avg=11.76, stdev= 5.16, samples=7916
KVM
write: IOPS=17.7k, BW=1104MiB/s (1158MB/s)(4096MiB/3709msec)
iops : min= 2108, max= 2490, avg=2264.95, stdev=105.60, samples=56

The test was made on LXC with centos 7, official LXC template, and on KVM with Centos 7.


What is wrong with LXC?


Later edit:

I disabled disk cache for kvm and I get about the same result, but why is so slow?

 

[egoistdream]

Hi,

I there any way to set data=writeback on /dev/rbd0 for root lxc hdd ?


I am using ceph and is very very slow, is this normal or my hdd are slow?

I am using 8 osd per host in the ceph cluster.
I can buy 24 ssd but I do not know if this will solve the problem.

 

[Klaus Steinberger]

from your post I suspect you have HDD's as OSD'S ? No SSD caching?

CEPH is very latency sensitive. A setup with HDD's only may have a very good bandwith, but it is bad in terms of latency, which will kill you in the lxc i/O patterns.

SSD's will be definitly faster, but to get decent results consider using enterprise grade SSD's
(the consumer ones have issues with latencies and especially shaking latencies)
you could instead also use SSD's for caching, one SSD could serve multiple OSD's

Next point: Which type of network do you run? For CEPH you should have at least 10GBit/s in the backend (GBit is again too slow in latencies)

 

[egoistdream]

I have sshd... and no ssd for caching, I was thinking sshd will help but it seems no, I have 10GBit/s network and I can live with data=writeback for kvm and the kvm run relativ ok, but the lxc is very very slow, i didn't find a way to cache the disks for lxc....