Docker on LXC container faster than on VM

[dan.ger]

Hello,

we ran docker on LXC container and vm on a proxmox 8.x.x three node cluster with NVME ceph storage (24 NVMEs) on Dell R740XD servers. Docker runs on Debian booworkm latest version. Hypervisor nesting is activated for LXC and VM. We put our monitoring in a docker container in LXC and VM. On VM hosted docker container we got timeouts multiple times on a day. In LXC we do not get this time outs.

VM Config:

agent: 1,fstrim_cloned_disks=1
boot: order=scsi0
cores: 12
cpu: host
memory: 32768
name: Docker
net0: virtio=56:4B:00:41:23:F3,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsi0: san:vm-107-disk-0,discard=on,iothread=1,size=160G
scsihw: virtio-scsi-pci
smbios1: uuid=0f4782a8-297b-436a-bc69-ee9493e1d090
sockets: 2
vmgenid: 1605a64e-4ac1-434e-b442-4a7dfa1e7e7d

lxc config:

arch: amd64
cores: 16
features: nesting=1
hostname: docker
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,gw=xxx.xxx.xxx.xxx,hwaddr=CA:DD:9D:32:F2:21,ip=xxx.xxx.xxx.xxx/xx,type=veth
ostype: debian
rootfs: san:vm-114-disk-0,size=80G
swap: 0
unprivileged: 1

So do I miss asetting in vm?

We had also issues before with a postgresql container that is in heavy use. At the end we migrated the postgresql container to a seperate vm and had no issues.

 

[Chris]

Hi,
what timeout are you referring to, what services are you hosting inside the VM? Please check the systemd journal from inside the VM for errors.

journalctl -b -r

gives you a paginated view of the logs since boot in reverse order.

Are there other tasks running at the time you notice the issue (backups, other VMs running heavy workloads). Check system load and storage IO.

 

[Guillaume Delanoy]

Hello,

Your KVM configuration shoes on one hand 12 vCPUs on two socket and on the other hand 32 GB RAM, while your LSC configuration displays 16 vCPUs, and 16 GB RAM.

Is it possible to harmonise those settings, at least for the vCPUs then check again if you still get those timeouts ?

Also, could you install

diffstat

on both environments and send back the results of

iostat -c

, especially CPU statistics %system
and %iowait ?

Best regards,

G. Delanoy

 

[dan.ger]

Hello,

The vm has twice power, double cpus and double amount of ram and I get the timeouts. The lxc container wirft docker works as expected without timeouts. So I believe to upgrade lxc container‘s resources will have no effect.

I checked the stats and send it tomorr.

 

[dan.ger]

I checked the system load and the disk is, this is not the problem bare metal is not heavy utilized round 5-10 %.

And yes sometimes it occurs off backups are running. The docker container I ran was uptime kuma. On lxc that is placed on the same bare metal does not have time outs in kuna. I know sqlite3 is not the best Rebstöcker, but they did not offer other Deb’s like postgresql, MySQL, Maria db,…

 

[LnxBil]

The docker container I ran was uptime kuma.

I dropped it after using it a while. Had always a high cpu load.

 

[dan.ger]

I found only this error: kex_exchange_identification: Connection closed by remote host

Seems to be a sqlite3 problem within the container

 

[dan.ger]

@LnxBil: yxou are right it depends on the container and I guess it's the sqlite db, cause I get a lot of knex timeouts in both setups after a while (db is growing, we monitored more then 500 instances). Something happend within the connection pooling of the node.js server application. Connection cannot be scheduled in a short time if they are required.