KVM vs LXC for web server

[proxbob]

Hello, I have proxmox 6 installed on a Xeon CPU E3-1245 V2 @ 3.40GHz, 8 cores, 32Gb RAM, SSD.

I want to use it for web hosting, I will create two guest os (debian, now that CentOs is "Gone").

I can't decide what to use, LXC or KVM ? I tested both and each has it's advantages and disadvantages.

My question is, from a pure performance perspective, for the fastest website response, which one should I use ?

both have the same configuration : 8Gb ram, 8 cores, host CPU for the VM.

The websites don't have lot for trafic, almost static, but they are importent, and need to be the fastest possible.

Thanks in advance.

 

[oguz]

hi,

containers are generally less overhead, if you notice any performance issues you can allocate more resources to it.

 

[proxbob]

hi,

containers are generally less overhead, if you notice any performance issues you can allocate more resources to it.

sur, containers are easier to manage, but I want the best perfomence.

I did some testing:
I setup a debian 10 VM and LXC with both 4 cores - 4Go RAM.
results :

VM :

CPU speed:
    events per second:  4500.58

RAM speed:
102400.00 MiB transferred (11655.43 MiB/sec)

File operations:
     writes/s:                     18572.66
    fsyncs/s:                     23823.36

LXC :

CPU speed:
    events per second:  3912.70

RAM speed:
102400.00 MiB transferred (10576.33 MiB/sec)

File operations:
    writes/s:                     10907.49
    fsyncs/s:                     14001.10

VM seems to perform better in all tests, it's this results enough to go for it ?

 

[bobmc]

I'm really surprised by those results - intuitively I would have said containers would win, but the numbers tell a different story

 

[oguz]

i'm curious how are you getting these statistics?

 

[mailinglists]

Suprising!
FYI I never had KVM VM outperform LXC (unless LXC was limited .

 

[proxbob]

I did more tests to be sure.

I created a new Debian 10 container, from the template, with 8 cores, 8Gb RAM, no limitations at all.
a Debian 10 VM (installed with the latest Debian 10 stable iso), 8 cores, 8Gb RAM, cpu = host, numa enable, virIO everywhere it's possible.
note: the host is a new, unused server, the only running guests are this two ones, tests are made one at a time.

I tested using sysbench command, and also 7zip built-in benchmark feature :

CPU TEST : sysbench --test=cpu run --threads=(number of cores)
RAM TEST : sysbench --test=memory run --threads=(number of cores)
FILE r/w TEST : sysbench --test=fileio --file-test-mode=seqwr run --threads=(number of cores)
7zip 1core: 7z b -mmt1
7zip 8core: 7z b

results :

VM :

CPU 1core : 1194.78
CPU 4core : 4536.25
CPU 8core : 6526.97

RAM 1core : 5302.51
RAM 4core : 11691.60
RAM 8core : 17133.52

FILE r/w 1core : 13310.64 17049.91
FILE r/w 4core : 26065.81 33413.38
FILE r/w 8core : 40254.04 51616.85


7zip 1core : 4851 3915
7zip 8core : 22646 20069

LXC :

CPU 1core : 1198.87
CPU 4core : 4505.24
CPU 8core : 6533.60

RAM 1core : 5019.34
RAM 4core : 11660.22
RAM 8core : 16212.28

FILE r/w 1core : 7579.83 9705.17
FILE r/w 4core : 10371.45 13324.67
FILE r/w 8core : 11716.02 15085.95


7zip 1core : 4455 3865
7zip 8core : 21163 19938

this did not surprised me as, as I understand, VM is more hardware assisted virtualization when containers are more software virtualization.
however, the file io performance difference is huge on my system.
can anyone make similar tests and share the results ?

 

[datdenkikniet]

Wouldn't it make much more sense to do benchmarks with workload similar to the one you plan to run on it (e.g. wrk, a HTTP benchmarking tool which seems to be quite well tailored for benchmarking how quickly it can fetch static files), possibly for a longer duration, to test the performance for this specific purpose?

While I agree that the numbers you show definitely seem to favor the VMs (numbers are hard to argue with lol), are file writes part of the workload you expect them to receive?

 

[mailinglists]

How is disk cache option set in KVM VM?

 

[macleod]

LXC vs KVM it's a long discussion, there is no perfect answer, you must think about you needs and decide.
- VM it's a little "safer" (i.e. better isolation, no shared kernel) - but with the neverending list of bugs from intel & others it's very arguable
- LXC comes with a little overhead (1-3%), VM overhead is greater (some say 5-10%)
- on VM you can use a different kernel (or even a different operating system), with LXC you share the host kernel
- ... many other differences
- and most important: you can do live migrations only with VM, for the moment container migrations needs downtime (but, with k8s containers there is another philosophy, container failure is not an exception, but a normal case, so you must plan your applications to expect this behaviour)

 

[9bitbyte]

I am in a similar situation, but for me security is #1, then #2 is convenience (easy configure+manage). (Reasonable) performance difference (+-10%) is not so important.

I am researching and considering 3 options for web/app servers (e.g urbackup, vaultwarden, plex, nextcloud) in:

1. LXC on bare-metal proxmox,
2. VM on bare-metal prxoxmox
3. LXC on VM on bare-metal proxmox (is that too paranoid)?

I see pros and cons with all of them of course!

What are your thoughts/experience?


My thoughts:

#1: I really like the idea of being able to simply spin LXCs directly on proxmox and manage them together with my other VMs, from the (great) pve ui. I also have a ZFS pool so I could utilize snapshots etc. But I am really doubting if LXC offers that good isolation/protection for my proxmox host...

# 2: Well, this is the traditional approach, installing all (or some) of these app/web servers in a guest VM that better isolates the proxmox host better. Even if I mess-up with some server configuration or some of the apps/servers have bugs, breaking out to the proxmox host is very unlikely. Server cross-contamination (ie. app to app or from user to user) is off course more possible.

# 3: I like the idea of further isolating the servers/apps in LXC containers, making it even more difficult to break out to the VM guest and to other apps/servers and data. Also, I find this better in terms of separating the configuration and managing the servers/apps (e.g can have different dependencies, updating, deleting etc). However, this seems to be the most cumbersome to configure and install.

Btw, I evaluated docker containers for some time, but came to understand they are not good for my use cases. Its a great project, very helpful community but it has a very specific use-cases and limitations. And also, security is not top-of-mind. That's why I am now considering if there's any use utilizing LXC instead..

Ramblings of a mad man