LXC memory configuration

[Faris Raouf]

I'm having problems limiting memory in an LXC container and would appreciate some help please.

I have created a Centos 7 Container with the following settings:

arch: amd64
cores: 2
cpuunits: 10
hostname: [redacted]
memory: 512
mp0: /backup2/pm2/dump,mp=/mnt/pm2dump,ro=1
net0: name=[redacted]
ostype: centos
rootfs: backup2:555/vm-555-disk-0.raw,size=8G
swap: 512
unprivileged: 1

I'm using it purely to run the AWS CLI so that I can backup the content of /backup2/pm2 to S3.

The problem I'm seeing is that when I use the "aws s3 cp" or similar command to copy a file or directory, the Node's swap (max 8GB) usage starts to go from very little used to 100% used over a period of around 10 minutes during the S3 upload process.

In the meantime, the Node's physical memory usage does not change - there is plenty of headroom and it is not being exhausted or anything like that.

Within the Container, "top" or "ps" shows the aws process using maybe 90% CPU and some reasonable amount of memory - typically 20%.

And yet it is somehow consuming swap on the Node like it has no limit on it.

Is there some other memory parameter that I should be setting to limit the Container's impact on the Node? It was my impression that unlike KVM with balooning memory, a Container should not exceed the memory limit set or the swap set in the config file. But this doesn't seem to be the case, at least to this newbie's eyes.

I know "swap" in Linux is more than just "disk based memory". I know it is far more complex. But even so, I can't understand why a Container with a 512M limit on swap could cause the Node's swap to get completely used.

Suggestions, pointers, explanations etc would be appreciated!

 

[Faris Raouf]

I'm still struggling with this. I find it alarming that Node swap goes to 100% when Container RAM and swap are pretty much unused when I'm backing up to S3.

When I say "alarming", I mean I'm worried something may go wrong when I do my backups to S3.

Any re-assurance would be appreciated.

 

[Faris Raouf]

I though I'd made some progress. By limiting the number of concurrent threads that the AWS S3 command uses, at first I saw no swap usage at all. Load was minimal. CPU and memory usage minimal. Sure, the upload speed was limited to 30Mbytes/sec but that's OK by me for this application.

However, after about 40GB of an 80GB upload, swap usage on the node started going up, and up and reached 100% again.

This is driving me crazy.

 

[Marshalleq]

Just saying I am reading your thread with interest. No replies doesn't mean people don't want to help, just that we might not know how to! I haven't started a Journey to LXC yet, (only one test container) but plan on doing it soon. So this is great understanding for me. Hopefully someone else will be able to chime in!

 

[Faris Raouf]

Thank you Marshalleq.

I've found a large number of threads with people complaining about swap being used excessively on the node for no obvious reason. The general consensus seems to be to try setting swappiness to 10 (or less) on the node. In fact swapiness of 10 is recommended in the docs, at least if you use ZFS (I don't). I wonder why it isn't set at 10 by default?

I will try that next, but first I want to try with 0 swap configured on the Container itself. Effectively disabling swap in the Container.

At the end of the day, the 100% swap use does not seem to do any harm other than to my sanity. But I instinctively feel it should not happen as a result of something running in a Container that (a) has plenty of RAM that doesn't get used, (b) has its own swap that doesn't get used.

 

[Marshalleq]

I was reading up on swap just yesterday because I figure if memory is being paged via the host, it should be able to make it's own swap to supplement any required memory. Then I read (and I might agree) that swap in a guest can be managed better by a guest because it supposedly knows better when and what kind of swap it needs than the host, which probably doesn't know about the applications. However, I would have thought the ballooning driver would also pass this requirement, effectively from the guest to the host. I would have thought that would be good design, but maybe I'm wrong. I also did read that at least one person disabled the swap internally in the guest and got a huge performance boost. Probably need to consider a lot of other factors first to understand which is best ultimately.

 

[Faris Raouf]

Well, I've solve the problem, but still don't understand why it was happening.

At the end of the day, I was seeing Node swap rapidly reaching 100% full when using the aws s3 cli in an LXC Container. Within the Container, little or no swap was being used, and little memory was being used. The Node has plenty of free RAM.

When using the aws s3 cp commands, Node swap would only increase if the file was reasonably large - over 10GB. And I'm backing up files from 2GB to 300GB.

I tried disabling swap in the Container to no avail.

I also adjusted some of the AWS settings : https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

My initial tests were with max_queue_size reduced from 10,000 to 100, and with max_concurrent_requests reduced from the default 10 to 4. But swap would still be used.

Eventually I found the right one. The critical setting in my case appears to be multipart_chunksize

The default is 8Mb. And there are a LOT of 8Mbs in 300GB.

Setting this to 5GB seems to have solved the problem - not one byte of swap is being used, at least for a 60GB file which I just tested on.

The behaviour of the system is somewhat different with this chunk size, but nothing that causes me alarm: With 8MB, the upload starts very quickly, and speed is pretty consistent. With 5GB, there is a long delay before the upload starts, and the speed takes a little while to ramp up.

Load average and I/O delay is also constantly quite high right at the start (10% I/O delay on my systems and a load average of 4 - due to the /o delay no doubt) then drops to very normal after a short while with the 5GB option. With 8MB is generally fluctuates up and down a lot throughout.

I still don't understand how a process running in a Container can cause the Node to use all that swap, but at least I've stopped it happening for the application that was causing it.