We have an error similar to the one in https://forum.proxmox.com/threads/high-java-memory-assignment-on-debian-9-containers.48782/.
Java VMs decide, that they should have more than 20G of heap. This is because they use the physical memory of the host system and not the memory assigned to the container.
The container themselves have much less memory assigned.
Normally we use Ubuntu. In my tests I updated the used Ubuntu 18.04 template and created new test container based on them (ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz).
But the same behaviour occurs when using older version (16.04, older 18.04 template) or another distro like Debian 9.
The behaviour is the same with all newer JDK versions. They _have_ container support, so one would expect that they should work.
But when started inside a lxc container in proxmox we got the following result when running with logging (java -Xlog
s=trace,os+container=trace -version)
That is the content of the memory.limit_in_bytes file inside the vm:
The JVM decides that the information is useless (maximum value meaning it is unlimited) and falls back to the also useless sysinfo call. The problem is, that that value shouldn't be the maximum, but the one assigned to the container.
The problem first occured with an older version of proxmox. But i updated to the current 5.4 release and also have finished the update to 6.0 on one server in the cluster. The error problem is the same on all versions.
lxcfs works and the values under /proc/meminfo and read with e.g. free are correct. It's only the value under /sys/fs/cgroup/memory/memory.limit_in_bytes that is incorrect.
In the following I add some output, which may be useful to understand the problem. The output is from the system with 5.4, which already has corosync 3.
Also on the HOST system, the cgroup seems to be correct !
And the Debug output from a container start is attached.
I hope someone can help!
Thanks in advance,
Stephan
Java VMs decide, that they should have more than 20G of heap. This is because they use the physical memory of the host system and not the memory assigned to the container.
The container themselves have much less memory assigned.
Normally we use Ubuntu. In my tests I updated the used Ubuntu 18.04 template and created new test container based on them (ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz).
But the same behaviour occurs when using older version (16.04, older 18.04 template) or another distro like Debian 9.
The behaviour is the same with all newer JDK versions. They _have_ container support, so one would expect that they should work.
But when started inside a lxc container in proxmox we got the following result when running with logging (java -Xlog
s=trace,os+container=trace -version)
Code:
[0.002s][trace][os,container] Path to /memory.limit_in_bytes is /sys/fs/cgroup/memory/memory.limit_in_bytes
[0.002s][trace][os,container] Memory Limit is: 9223372036854771712
[0.002s][trace][os,container] Memory Limit is: Unlimited
[0.002s][debug][os,container] container memory limit unlimited: -1, using host value
[0.002s][trace][os ] total system memory: 101267460096That is the content of the memory.limit_in_bytes file inside the vm:
Code:
root@meatiest2:~# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712The JVM decides that the information is useless (maximum value meaning it is unlimited) and falls back to the also useless sysinfo call. The problem is, that that value shouldn't be the maximum, but the one assigned to the container.
The problem first occured with an older version of proxmox. But i updated to the current 5.4 release and also have finished the update to 6.0 on one server in the cluster. The error problem is the same on all versions.
lxcfs works and the values under /proc/meminfo and read with e.g. free are correct. It's only the value under /sys/fs/cgroup/memory/memory.limit_in_bytes that is incorrect.
In the following I add some output, which may be useful to understand the problem. The output is from the system with 5.4, which already has corosync 3.
Code:
# pveversion -v
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-6
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.4.117-2-pve: 4.4.117-110
pve-kernel-4.4.117-1-pve: 4.4.117-109
pve-kernel-4.4.98-5-pve: 4.4.98-105
pve-kernel-4.4.98-3-pve: 4.4.98-103
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.4.35-1-pve: 4.4.35-77
corosync: 3.0.2-pve2~bpo9
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-54
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.5-1~bpo9+2
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2
Code:
# pct config CTID
arch: amd64
cpulimit: 1
hostname: meatiest2.innermedia.de
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=10.10.11.240,hwaddr=9A:0E:7A:CF:DB:49,ip=10.10.8.78/22,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-111-disk-0,size=8G
swap: 512Also on the HOST system, the cgroup seems to be correct !
Code:
root@host:~# cat /sys/fs/cgroup/memory/lxc/111/memory.limit_in_bytes
536870912And the Debug output from a container start is attached.
I hope someone can help!
Thanks in advance,
Stephan
