LXC IO performance

[Mark Verboom]

Hi,

Since migrating to Proxmox 4, I've seen a degradation in IO performance on my 2 systems. Especially on a larger box which houses around 40 LXC containers. I've seen quite a few posts about this and it seemed that it should have something to do with the way the containers have their data stored. Moving over to a directory tree like with OpenVZ seems to give a performance increase. The only thing I could think of that might give an issue is that each of the mounted ext4 filesystems (one for each container and also the filesystem holding the containers) has barriers enabled. So I decided to do a test to see if disabling barriers for the ext4 filesystems in the containers (not the base filesystem that's on the main storage) would show a difference. Below is the graph of my results:



It looks like the IO wait has dropped significantly since the barriers are disabled at around 15:15. I'm not 100% sure, but I don't think this should pose a huge risk in consistency if the base ext4 filesystem keeps barriers enabled.

The biggest problem I'm having at the moment is how I can make a container boot with barriers disabled on it's filesystem? Currently I'm doing it runtime after the container is booted by remounting the root filesystem, which isn't a very elegant solution..

Regards,

Mark

 

[fabian]

I'm not 100% sure, but I don't think this should pose a huge risk in consistency if the base ext4 filesystem keeps barriers enabled.

writes inside containers do not cause writes to the host (ext4) file system in many setups (e.g., LVM-(Thin)). better aim for "100% sure" when data conistency is concerned

 

[Mark Verboom]

writes inside containers do not cause writes to the host (ext4) file system in many setups (e.g., LVM-(Thin)). better aim for "100% sure" when data conistency is concerned

I'm not sure I understand. When a process writes to a loopback mounted filesystem, the data isn't written out the the underlying host filesystem? Are those writes significantly different from other data written to a file because of the extra filesystem layer?

Regards,

Mark

 

[fabian]

I'm not sure I understand. When a process writes to a loopback mounted filesystem, the data isn't written out the the underlying host filesystem? Are those writes significantly different from other data written to a file because of the extra filesystem layer?

Many setups don't have FS on FS with a loopback-style setup, but FS on LV or even (Z)FS directly. I just wanted to point this out.

 

[ohmer]

I'm using LVM-Thin with raw image for my LXC containers. Currently testing proxmox on a desktop PC with a single SATA drive, but the prod will be on SAS or SSD drives in RAID10 on a hw raid controller with battery. So, I understand this would be safe to use barrier=0.

It's easy to change that settings for the root fs in /etc/fstab on the host, but how can I configure my containers to set barrier=0 when they mount their raw image?

 

[LnxBil]

Maybe create your own LXC template with has a hook or something that fixes /etc/fstab after deployment. I often use my own debian packages for such things.

Have you considered using ZFS? You have the "real" files there like it was with OpenVZ on a directory.

 

[ohmer]

Didn't used ZFS. I have read that ZFS require a lot of RAM and preferred to keep the RAM for my containers. I'm still evaluating proxmox (I use if for years for my personal server, but first time I use it at work and first time with 4.x/LXC), so if my thinking is wrong, it's not too late to use ZFS instead.

 

[LnxBil]

Yes, ZFS uses a lot of ram (half your ram at most per default), but it's worth it.

 

[Mark Verboom]

Many setups don't have FS on FS with a loopback-style setup, but FS on LV or even (Z)FS directly. I just wanted to point this out.

Ah, sorry, I thought you were referencing the ext4 on ext4 setup

Is there a way to specify that barriers need to be disabled when mounting rootfs on an LXC container?

I've already tried specifying lxc.rootfs.options, but that is not supported:

vm 104 - lxc.rootfs.options: lxc.rootfs.options is not supported, please use mountpoint options in the "rootfs" key

And the rootfs key doesn't seem to support this option.

I also tried tune2fs to set the the nobarrier option as default:

tune2fs -o nobarrier /var/lib/vz/images/104/vm-104-disk-1.raw
tune2fs -l /var/lib/vz/images/104/vm-104-disk-1.raw | grep -i barr
Default mount options:  user_xattr acl nobarrier
pct start 104
pct exec 104 mount | grep ext4
/var/lib/vz/images/104/vm-104-disk-1.raw on / type ext4 (rw,relatime,data=ordered)

But that also doesn't seem to get honorred..

Hope you have a solution for this.

Regards,

Mark

 

[gkovacs]

Yes, ZFS uses a lot of ram (half your ram at most per default), but it's worth it.

Your statement is false, and contributes to the stupid but unkillable myth of ZFS using up all your RAM.

Fact 1: ZFS only uses RAM that the system doesn't need. And no, it won't use half of your RAM by default.

Fact 2: ZFS does not use a lot of RAM at all even if there is a lot of free memory.
On one of our Proxmox VE 4.2 nodes, running 13 KVM guests in 42 GB out of 72 GB of memory, ZFS usually uses around 3 GB for ARC, resulting in over 75% of cache hits..

root@proxmox4:~# arcstat.py
  time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz  c
01:50:59  0  0  0  0  0  0  0  0  0  3.0G  3.0G
root@proxmox4:~# arc_summary.py

------------------------------------------------------------------------
ZFS Subsystem Report  Sat Aug 27 01:51:02 2016
ARC Summary: (HEALTHY)
  Memory Throttle Count:  0

ARC Misc:
  Deleted:  926.12m
  Mutex Misses:  152.63k
  Evict Skips:  152.63k

ARC Size:  8.49%  3.00  GiB
  Target Size: (Adaptive)  8.49%  3.00  GiB
  Min Size (Hard Limit):  0.09%  32.00  MiB
  Max Size (High Water):  1132:1  35.39  GiB

ARC Size Breakdown:
  Recently Used Cache Size:  6.17%  189.64  MiB
  Frequently Used Cache Size:  93.83%  2.82  GiB

ARC Hash Breakdown:
  Elements Max:  5.13m
  Elements Current:  8.18%  420.22k
  Collisions:  104.50m
  Chain Max:  7
  Chains:  3.09k

ARC Total accesses:  3.78b
  Cache Hit Ratio:  75.27%  2.85b
  Cache Miss Ratio:  24.73%  935.68m
  Actual Hit Ratio:  68.99%  2.61b

  Data Demand Efficiency:  90.46%  1.84b
  Data Prefetch Efficiency:  24.12%  988.95m

  CACHE HITS BY CACHE LIST:
  Anonymously Used:  7.45%  212.24m
  Most Recently Used:  69.36%  1.97b
  Most Frequently Used:  22.31%  635.13m
  Most Recently Used Ghost:  0.71%  20.27m
  Most Frequently Used Ghost:  0.17%  4.77m

  CACHE HITS BY DATA TYPE:
  Demand Data:  58.33%  1.66b
  Prefetch Data:  8.38%  238.49m
  Demand Metadata:  33.29%  947.79m
  Prefetch Metadata:  0.00%  135.04k

  CACHE MISSES BY DATA TYPE:
  Demand Data:  18.71%  175.10m
  Prefetch Data:  80.20%  750.46m
  Demand Metadata:  1.06%  9.90m
  Prefetch Metadata:  0.02%  227.29k