Disk I/O graph is empty

[Crogdor]

Hi there!

I've got Proxmox VE 8.0.4. I'm running 4 LXCs and 2 VMs. I'm trying to diagnose/solve a problem where one of the LXC containers is being a noisy neighbor and starving out the other containers/VMs of I/O. I'd first like to get some insight into which container is doing the offending (I have an educated guess...), but unfortunately, all my LXCs have an empty Disk I/O graph.

The graph works fine on the VMs. And it may be important to note that the LXC containers are all running Docker for most of their work.

Any thoughts on how to get this telemetry to show up? (And what would be amazing, is knowing how I can limit how much disk I/O an LXC container is permitted to use, since that's potentially my end goal.)

I've posted a sample of one of the LXC config files below.

arch: amd64
cores: 4
features: nesting=1
hookscript: local:snippets/wait-for-nfs-mounts.pl
hostname: docker-dl
memory: 4096
mp0: /mnt/homelab,mp=/mnt/homelab
mp1: /mnt/media,mp=/mnt/media
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=AE:12:1F:0C:EE:FF,ip=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-zfs:subvol-190-disk-0,size=500G
swap: 0
tags: ubuntu-23.04;docker
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir

Thanks,
Dave

 

[fiona]

Hi,

Hi there!

I've got Proxmox VE 8.0.4. I'm running 4 LXCs and 2 VMs. I'm trying to diagnose/solve a problem where one of the LXC containers is being a noisy neighbor and starving out the other containers/VMs of I/O. I'd first like to get some insight into which container is doing the offending (I have an educated guess...), but unfortunately, all my LXCs have an empty Disk I/O graph.

since you are using ZFS, it's probably this bug: https://bugzilla.proxmox.com/show_bug.cgi?id=2135

The graph works fine on the VMs. And it may be important to note that the LXC containers are all running Docker for most of their work.

It's recommended to use VMs to run docker. We had quite a few reports about issues when using docker inside containers (IIRC incompatibilities after an upgrade, etc.).

Any thoughts on how to get this telemetry to show up? (And what would be amazing, is knowing how I can limit how much disk I/O an LXC container is permitted to use, since that's potentially my end goal.)

You could try doing it via cgroup, but since the stats are missing just there, I'm not sure it would work.

 

[pwl]

I have the same problem even when I don't use ZFS. Container based on LVMThin disk. As soon as I start the docker daemon, and reboot contaner, the RRD graphics are empty !

How to explore this issue? what additional logs are needed?

I use 6.5.11-4-pve kernel with latest PVE