Lxc container does not free up cached memory

[Gerk]

I have found the real culprit in my case. It was never a kernel or LXC problem, it was my network interface settings. I'm using a Mellanox ConnectX-3 Pro, and I guess I blindly maxed out the RX queues on ethtool to 128 (as my system has 128 threads). Lowering that down to 32 (matching the max TX queues) reduced memory usage dramatically.

ethtool -L "$INTERFACE" rx 32

Just to add to this conversation, that's definitely not an issue with our setup. Luckily the ERP I'm deploying is python based and I can hard limit the memory usage, so as long as I stay under 90% my LXCs don't go into this loop. If I allow them to go over 90% then it happens every time.