[SOLVED] Unable to update a docker container - no space left on device

[leemarrett]

Hi,

I have proxmox running a few vms - one of them is ubuntu running some docker containers of some web apps.

I'm trying to update one of them and getting "
Pulling 38.1s
failed to register layer: write /usr/share/X11/xkb/geometry/digital_vndr/pc: no space left on device"

but it's a vm with 200gb allocated space and i really don't have that many things on there, nothing that should take up that much space...

How do I figure out what's going on?

when i run df -h i get

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              392M  1.9M  390M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   15G   12G  2.1G  86% /
tmpfs                              2.0G     0  2.0G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock

and then a bunch of NFS mounts...

 

[gfngfn256]

As you can see from within your Ubuntu VM, you are only using 15GB for the created LVM (of the 200GB you claim that VM has).


Assuming your VM actually has that 200GB, the issue is that the Ubuntu OS is not "using" that space in its' LVM.
There are many guides online showing how to get the LVM to do this.
I'll give you a quick overview of the necessary steps required. Note all these actions are from within the VM:

• Firstly show output of lsblk to see the actual disk/partition size.
• Next you may need to repartition/extend that partition to use all of that space. (You can use cfdisk for this).
• Next you will need to extend the Physical Volume (PV) to use that extra space. (You use pvresize & pvdisplay for this).
• Then you will check the Volume Group (VG) for the free space. (Using vgdisplay).
• Next you will check the Logical Volume (LV) for free space & then extend it to use that space. (You will use lvdisplay & lvextend for this).
• Finally you will resize the file system (FS) to those extents. (resize2fs will be used for this).
• When you now check with df -h you should be showing the full size available.

For such actions, make sure you have a full & restorable backup of the VM.

Good luck.

 

[leemarrett]

Thanks so much for your help - here's the output of lsblk:

NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                         8:0    0  182G  0 disk
├─sda1                      8:1    0    1M  0 part
├─sda2                      8:2    0    2G  0 part /boot
└─sda3                      8:3    0   30G  0 part
  └─ubuntu--vg-ubuntu--lv 252:0    0   15G  0 lvm  /
sr0                        11:0    1  3.1G  0 rom

i had a quick look at cfdisk to see what it looked like and when i quit it gave me these messages, not sure if it's useful:

GPT PMBR size mismatch (67108863 != 381681663) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.

 

[gfngfn256]

So as you can see the partition sda3 that has the VG, only uses 30G of full ~179G (182-sda1-sda2) & the VG only uses 15G of that.

So you need to perform all the above actions listed above, starting with extending the sda3 partition.

Good luck & remember my backup advice!

 

[leemarrett]

Thank you, I really appreciate your help!

I can confirm that df -h now reports the full size available.

I just wanted to confirm though something that wasn't on the output of df -h before i started this. After the list of NFS mounts I mentioned in my original posts, I have a list of about 10 of these:

overlay 178G 12G 158G 7% /var/lib/docker/overlay2/823891f0dd98e06bc528ff1eaad3422107881203c83d867fb832d7f076ad90d4/merged

I can see these are related to docker but not sure why they would appear now.

Again, thank you so much for your help, I really liked how you told me what I would have to consider without just giving me commands.

 

[gfngfn256]

/var/lib/docker/overlay2/823891f0dd98e06bc528ff1eaad3422107881203c83d867fb832d7f076ad90d4/merged

I've only dabbled with docker in the past, but as far as I know, these are overlays associated with the docker image/container system.
Try shutting down docker & they will probably disappear. If not you may need to clean up your docker instance.

It seems like you got fully sorted. Maybe mark this thread as Solved. At the top of the thread, choose the Edit thread button, then from the (no prefix) dropdown choose Solved.

Happy Proxmoxing!