Docker Build fails, but only in an LXC

LordRatner

Member
Jun 20, 2022
50
12
8
Hello all,

First off, I know that Docker in an LXC is discouraged. Believe me, I'd love for that to be the end of it, but there are simply too many projects that use it. Since I am sharing a GPU across several isolated services, a VM is not an option.

For whatever reason, I can't get a docker image to build (using `docker build`) in an LXC. Every single time it fails at the same point, no matter what variables I change.

Code:
0 40.81 Successfully built audioread ffmpy lit pathtools wavedrom
#0 41.73 Installing collected packages: wcwidth, tokenizers, sentencepiece, safetensors, rfc3986, pytz, pyston-lite, pydub, pathtools, msgpack, mpmath, lit, ifaddr, ffmpy, cmake, av, zipp, xxhash, websockets, uvloop, urllib3, ujson, uc-micro-py, tzdata, tqdm, toolz, threadpoolctl, svgwrite, soxr, sniffio, smmap, six, setproctitle, semantic-version, scipy, ruamel.yaml.clib, regex, PyYAML, python-multipart, python-dotenv, pyston-lite-autoload, pyrsistent, pyparsing, Pygments, pyee, pydantic, pycparser, pyarrow, psutil, protobuf, prompt-toolkit, pkgutil_resolve_name, packaging, orjson, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, networkx, multidict, mdurl, markdown2, llvmlite, lazy_loader, kiwisolver, joblib, itsdangerous, httptools, h11, gunicorn, google-crc32c, fsspec, frozenlist, fonttools, filelock, entrypoints, dnspython, dill, decorator, cycler, contourpy, colorlog, click, charset-normalizer, audioread, attrs, async-timeout, aiofiles, yarl, wavedrom, uvicorn, sentry-sdk, scikit-learn, ruamel.yaml, requests, python-dateutil, nvidia-cusolver-cu11, nvidia-cudnn-cu11, multiprocess, markdown-it-py, linkify-it-py, importlib-resources, importlib-metadata, gitdb, email-validator, docker-pycreds, ctranslate2, cffi, anyio, aiosignal, aioice, watchfiles, starlette, soundfile, rich, responses, pylibsrtp, pooch, pandas, numba, mdit-py-plugins, matplotlib, jsonschema, HyperPyYAML, huggingface-hub, httpcore, GitPython, cryptography, aiohttp, wandb, transformers, pyOpenSSL, librosa, httpx, fastapi, altair, gradio, datasets, aiortc, accelerate, speechbrain, peft, fschat
#0 42.06   Attempting uninstall: mpmath
#0 42.06     Found existing installation: mpmath 1.2.1
#0 42.07     Uninstalling mpmath-1.2.1:
#0 42.07 ERROR: Could not install packages due to an OSError: [Errno 22] Invalid argument: '/usr/local/lib/python3.8/dist-packages/mpmath-1.2.1.dist-info/'
#0 42.07
#0 42.13
#0 42.13 [notice] A new release of pip available: 22.3.1 -> 23.1.2
#0 42.13 [notice] To update, run: python -m pip install --upgrade pip
------
Dockerfile:16
--------------------
  14 |     COPY requirements.txt .
  15 |     # Run pip install with cache so we speedup subsequent rebuilds
  16 | >>> RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt
  17 |     
  18 |     # Install auto-gptq
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1

Now when I try this in a VM, it works without a hitch, and other people have no problem building the image. The author was even able to build it in a LXC container, but not using Proxmox.

I've tried privileged and unprivileged, debian vs ubuntu, nesting and FUSE on and off, building as root and as another user, tons of RAM and storage, but it always fails at the exact same point.

This is the project: https://github.com/toverainc/willow-inference-server#getting-started if anyone wants to give it a try. You can do it without a GPU, which is how I built it in the VM and it built successfully. But in the LXC it always fails.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!