Help understanding LXC RAM usage

[ropeguru]

Hi all,

I am having an issue when running rsync in an LXC pulling files in from a remote system where the remote has a rsync daemon running.

So my command line in my LXC would look something like, "rsync -av --progress rsync://192.168.196.11/backup/ /mnt/restore/"

Initially the restore will saturate the remote side's 100Mbps link but then the throughput dies off. With the help of a redditor, I was able to determine that rsync was caching data and in the LXC, the caching would take free memory down to zero at which time the transfer rate would go down. Testing the same scenario on a VM, again, rsync would eat up memory and cache would go up, but once the memory was around 110KB or so, it wouldn't go any lower and rsync would hold its transfer rates.

Can someone help me understand this issue in an LXC? I found one article which talked about dirty cache and flushing, but I am still trying to understand first how the whole LXC memory works and why it would be allowed to go to zero free memory and basically denile of service a running application.

 

[flatpzero]

The problem here is your LXC doesn't have enough memory to execute the rsync task you're asking it to.
Two choices:
1) give the LXC more memory
2) make the work required by rsync smaller

 

[ropeguru]

The problem here is your LXC doesn't have enough memory to execute the rsync task you're asking it to.
Two choices:
1) give the LXC more memory
2) make the work required by rsync smaller

So the working VM has 4GB RAM associated and so does the LXC.

So what is the difference between the two? That is what I am trying to understand.

 

[datschlatscher]

Could you clarify whether the problem exists both on LXC and the VM, or only on LXC?

The container seems to be running out of RAM after which it would naturally change to using swap. Swap, being located on the filesystem, is many times slower than RAM and would present a major bottleneck in your setup. From your first post it sounds like you are experiencing the same problem on your VM.
If that is indeed not the case, please tell us what kind of VM you are running as the swapping behaviour might be different.
Anyway, a possible fix, like ropeguru said, would be to split up your syncing load into lots of small operations. This should keep memory usage in check.

 

[ropeguru]

Could you clarify whether the problem exists both on LXC and the VM, or only on LXC?

The container seems to be running out of RAM after which it would naturally change to using swap. Swap, being located on the filesystem, is many times slower than RAM and would present a major bottleneck in your setup. From your first post it sounds like you are experiencing the same problem on your VM.
If that is indeed not the case, please tell us what kind of VM you are running as the swapping behaviour might be different.
Anyway, a possible fix, like ropeguru said, would be to split up your syncing load into lots of small operations. This should keep memory usage in check.

Only on the LXC...

What I am seeing on the LXC is the free mem dropping to zero as the cache memory inscreases. When the free mem hits zero, the rsync transfer goes from 100Mbps to less than 5Mbps... Nothing ever swaps..

On the VM, when free memory hits around 110KB, it just bounces around and never goes to zero and never uses swap. Transfer rate continues @ 100Mbps until I kill the test.

The VM is an Ubuntu install and the LXC is a Turnkey file server template based off Debian 10.