Installing gitlab into LXC container (sysctl kernel.shmmax)

[dlb]

Hi,

I tried to install gitlab into LXC CT, but during the installation I received an error that PostgreSQL requires kernel.shmmax = 17179869184

I verify kernel.shmmax in CT:
# cat /proc/sys/kernel/shmmax
18446744073692774399

If I try set this parameter in CT, I get error:
sysctl -w kernel.shmmax=17179869184
sysctl: setting key "kernel.shmmax": Read-only file system

Ok, I set kernel.shmmax in Hardware Node (HN/PVE), restarted
Checking on HN:
# sysctl kernel.shmmax
kernel.shmmax = 17179869184

I started LXC CT
Checking on CT:
# sysctl kernel.shmmax
kernel.shmmax = 18446744073692774399
If I try to set this parameter again in CT, I get the error message again: "Read-only file system"

As I understand it, sysctl parameters can be set in LXC CT, starting from LXC 3.0.0, through lxc.sysctl.[kernel parameters name] (linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html)

I tried add string lxc.sysctl.kernel.shmmax = 17179869184
in /etc/pve/lxc/<VMID>.conf or in /var/lib/lxc/<VMID>/config
but all without success

How can I set the parameters /proc/sys/kernel/shm* in LXC CT ?
Thanks!

 

[Trigger00]

Hi,

h t t p s://docs.gitlab.com/omnibus/common_installation_problems/#failed-to-modify-kernel-parameters-with-sysctl

[EDIT]: Tested on Proxmox 5.2.11
[EDIT2]: I had to do that twice before relaunching gitlab reconfiguration

Cheers,

 

[dlb]

This document actually describes only two ways to solve this issue, in simple words:

1. "You can patch the github installation script. But you can have unexpected side effects on performance of your GitLab server so it is not recommended to do so"

2. "You can change these parameters in the host OS, after which you can read the values of these parameters inside the virtual machines"

Trigger00, which method you use?
I tried method 2. It worked in old PVE with OpenVZ. But it does not work in my PVE with LXC
I tested on 2 PVE-hosts and on proxmox templates: Debian 8,9; Ubuntu 17,18; Centos 7
The result is the same: in fact, the parameters /proc/sys/kernel/shm* cannot be read in LXC containers under PVE (value 18446744073692774399 is error in fact, IMHO, "value is not set")

My pveversion:
proxmox-ve: 5.2-3 (running kernel: 4.15.18-9-pve)
pve-manager: 5.2-12 (running version: 5.2-12/ba196e4b)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-3-pve: 4.15.18-22
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-2
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-42
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-32
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-30
pve-docs: 5.2-10
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-14
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-41
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[Trigger00]

I tried that:

This error is also reported to occur in virtual machines only, and the recommended workaround is to set the values in the host. The values needed for GitLab can be found inside the file /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf in the virtual machine. After setting these values in /etc/sysctl.conf file in the host OS, run cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - on the host. Then try running gitlab-ctl reconfigure inside the virtual machine. It should detect that the kernel is already running with the necessary settings, and not raise any errors.

Also note you may need to repeat this process for a couple other lines, e.g. reconfigure will fail 3 times and you will eventually have added something like this to /etc/sysctl.conf:

for info :

root@ns123546:~# pveversion --versbose
root@ns359658:~# pveversion --verbose
proxmox-ve: 5.2-2 (running kernel: 4.15.18-9-pve)
pve-manager: 5.2-11 (running version: 5.2-11/13c2da63)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-1
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-41
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-31
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-3
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-30
pve-docs: 5.2-10
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-14
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-40
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
root@ns359658:~#

 

[dlb]

Could you please write in this discussion result of command "sysctl kernel.shmmax" in your gitlab LXC container?

 

[Trigger00]

sure:

root@gitlab:~# sysctl kernel.shmmax
kernel.shmmax = 18446744073692774399

 

[outlander]

Same here any solution?

 

[liquidox]

For others struggling to install Gitlab within LXC on Proxmox:

https://github.com/lxc/lxc/issues/989

Setting "lxc.mount.auto: proc:rw" was the solution for me.

 

[TheT1tanius]

Adding to liquidox' answer: yes this works, however only for privileged containers.

 

[c.weilguny]

Is there anything else to be done? Setting "lxc.mount.auto: proc:rw" in the container config in "/etc/pve/lxc/example.conf" (privileged container) didn't change anything. There always appears the shmmax error.

 

[Francesco Piraneo G.]

Is there anything else to be done? Setting "lxc.mount.auto: proc:rw" in the container config in "/etc/pve/lxc/example.conf" (privileged container) didn't change anything. There always appears the shmmax error.

It seems that the only solution is: Don't use container, use a VM instead.

FPG

 

[onlime]

GitLab runs perfectly fine in an LXC container on Proxmox VE! I am running several rather large GitLab instances (both CE and EE) in this environment, some of them for years and also some fresh installs, so can confirm it is working 100% fine.

For those who followed above advice and feel a bit lost, still don't get it running, here's the easiest way I would recommend:

On your host (Proxmox VE), put the following into /etc/sysctl.conf (you can lookup those values on your GitLab container with $ cat /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.*), my recommended setup:

kernel.sem=250 32000 32 262
kernel.shmall=4194304
kernel.shmmax=17179869184
net.core.somaxconn=1024

load it with:

$ sysctl -p

then, on your LXC container with GitLab installed (Omnibus package), add the following to /etc/gitlab/gitlab.rb:

##! Attempt to modify kernel paramaters. To skip this in containers where the
##! relevant file system is read-only, set the value to false.
package['modify_kernel_parameters'] = false

now GitLab reconfigure should run through fine:

$ gitlab-ctl reconfigure

Cheers, Philip

 

[rokyo]

GitLab runs perfectly fine in an LXC container on Proxmox VE! I am running several rather large GitLab instances (both CE and EE) in this environment, some of them for years and also some fresh installs, so can confirm it is working 100% fine.

For those who followed above advice and feel a bit lost, still don't get it running, here's the easiest way I would recommend:

On your host (Proxmox VE), put the following into /etc/sysctl.conf (you can lookup those values on your GitLab container with $ cat /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.*), my recommended setup:

kernel.sem=250 32000 32 262
kernel.shmall=4194304
kernel.shmmax=17179869184
net.core.somaxconn=1024

load it with:

$ sysctl -p

then, on your LXC container with GitLab installed (Omnibus package), add the following to /etc/gitlab/gitlab.rb:

##! Attempt to modify kernel paramaters. To skip this in containers where the
##! relevant file system is read-only, set the value to false.
package['modify_kernel_parameters'] = false

now GitLab reconfigure should run through fine:

$ gitlab-ctl reconfigure

Cheers, Philip

Will this affect ALL containers running on my Proxmox host?

There are ~20 and I'm a little reluctant to set this if it could affect any other containers. Will it? Negatively?

 

[rokyo]

GitLab runs perfectly fine in an LXC container on Proxmox VE! I am running several rather large GitLab instances (both CE and EE) in this environment, some of them for years and also some fresh installs, so can confirm it is working 100% fine.

For those who followed above advice and feel a bit lost, still don't get it running, here's the easiest way I would recommend:

On your host (Proxmox VE), put the following into /etc/sysctl.conf (you can lookup those values on your GitLab container with $ cat /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.*), my recommended setup:

kernel.sem=250 32000 32 262
kernel.shmall=4194304
kernel.shmmax=17179869184
net.core.somaxconn=1024

load it with:

$ sysctl -p

then, on your LXC container with GitLab installed (Omnibus package), add the following to /etc/gitlab/gitlab.rb:

##! Attempt to modify kernel paramaters. To skip this in containers where the
##! relevant file system is read-only, set the value to false.
package['modify_kernel_parameters'] = false

now GitLab reconfigure should run through fine:

$ gitlab-ctl reconfigure

Cheers, Philip

Alright, I tried it and the parameters are immediately set on the host, I can see them changed with `cat /proc/sys/kernel/xyz`.

However, inside the container there are still the old values set, even after a reboot of the container. Is there anything else needed here?

 

[onlime]

Will this affect ALL containers running on my Proxmox host?

There are ~20 and I'm a little reluctant to set this if it could affect any other containers. Will it? Negatively?

Yes, it affects ALL containers running on that same Proxmox host! I also have this running in production on hosts with 5-10 other LXC containers, for over 7 yrs and never, and have never experienced any negative side effects.

 

[onlime]

Alright, I tried it and the parameters are immediately set on the host, I can see them changed with `cat /proc/sys/kernel/xyz`.

However, inside the container there are still the old values set, even after a reboot of the container. Is there anything else needed here?

The changed values should be immediately visible in the container. But you're right, nowadays it does not anymore. Remember, my post dates back to Oct 2022 and I can't tell what has changed since then. Maybe it was also changed in previous LXC or kernel versions.

You are probably just fine with setting the package['modify_kernel_parameters'] = false in your gitlab.rb and go with the defaults of these params.