Extremly slow LXC using CEPH huge difference comparated with kvm


Dec 11, 2018

I configured Proxmox 5.3 with ceph in a 3 node cluster. When I start to test the setup I see a huge difference between the kvm and lxc using the same ceph pools, the pools have the size of 2 and min size 1.
The LXC is very very slow even if I try to update, KVM run ok.

So, I start to run some tests because it seems strange and I never have problems like this, here are the results:

Sequential Reads – Async mode – 8K block size – Direct IO – 100% Reads

read: IOPS=2081, BW=16.3MiB/s (17.1MB/s)(8192MiB/503717msec)
iops : min= 2, max= 1616, avg=277.15, stdev=370.69, samples=7547
read: IOPS=39.9k, BW=311MiB/s (327MB/s)(8192MiB/26309msec)
iops : min= 3116, max= 7928, avg=6168.21, stdev=922.54, samples=335

Sequential Writes – Async mode – 32K block size – Direct IO – 100% Writes

write: IOPS=80, BW=2589KiB/s (2651kB/s)(1517MiB/600061msec)
iops : min= 1, max= 42, avg=20.71, stdev= 9.79, samples=4684
write: IOPS=13.0k, BW=406MiB/s (426MB/s)(8192MiB/20163msec)
iops : min= 2188, max= 4184, avg=3295.68, stdev=459.19, samples=157

Random Reads – Async mode – 8K block size – Direct IO – 100% Reads

write: IOPS=77, BW=4958KiB/s (5077kB/s)(2911MiB/601201msec)
iops : min= 1, max= 20, avg=11.76, stdev= 5.16, samples=7916
read: IOPS=57.8k, BW=452MiB/s (474MB/s)(16.0GiB/36277msec)
iops : min= 2720, max= 5238, avg=4163.64, stdev=422.26, samples=990

Random Writes – Async mode – 64K block size – Direct IO – 100% Writes

write: IOPS=77, BW=4958KiB/s (5077kB/s)(2911MiB/601201msec)
iops : min= 1, max= 20, avg=11.76, stdev= 5.16, samples=7916
write: IOPS=17.7k, BW=1104MiB/s (1158MB/s)(4096MiB/3709msec)
iops : min= 2108, max= 2490, avg=2264.95, stdev=105.60, samples=56

The test was made on LXC with centos 7, official LXC template, and on KVM with Centos 7.

What is wrong with LXC?

Later edit:

I disabled disk cache for kvm and I get about the same result, but why is so slow?
Last edited:

I there any way to set data=writeback on /dev/rbd0 for root lxc hdd ?

I am using ceph and is very very slow, is this normal or my hdd are slow?

I am using 8 osd per host in the ceph cluster.
I can buy 24 ssd but I do not know if this will solve the problem.
from your post I suspect you have HDD's as OSD'S ? No SSD caching?

CEPH is very latency sensitive. A setup with HDD's only may have a very good bandwith, but it is bad in terms of latency, which will kill you in the lxc i/O patterns.

SSD's will be definitly faster, but to get decent results consider using enterprise grade SSD's
(the consumer ones have issues with latencies and especially shaking latencies)
you could instead also use SSD's for caching, one SSD could serve multiple OSD's

Next point: Which type of network do you run? For CEPH you should have at least 10GBit/s in the backend (GBit is again too slow in latencies)
I have sshd... and no ssd for caching, I was thinking sshd will help but it seems no, I have 10GBit/s network and I can live with data=writeback for kvm and the kvm run relativ ok, but the lxc is very very slow, i didn't find a way to cache the disks for lxc....


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!