Hello all,
First off, I know that Docker in an LXC is discouraged. Believe me, I'd love for that to be the end of it, but there are simply too many projects that use it. Since I am sharing a GPU across several isolated services, a VM is not an option.
For whatever reason, I can't get a docker image to build (using `docker build`) in an LXC. Every single time it fails at the same point, no matter what variables I change.
Now when I try this in a VM, it works without a hitch, and other people have no problem building the image. The author was even able to build it in a LXC container, but not using Proxmox.
I've tried privileged and unprivileged, debian vs ubuntu, nesting and FUSE on and off, building as root and as another user, tons of RAM and storage, but it always fails at the exact same point.
This is the project: https://github.com/toverainc/willow-inference-server#getting-started if anyone wants to give it a try. You can do it without a GPU, which is how I built it in the VM and it built successfully. But in the LXC it always fails.
First off, I know that Docker in an LXC is discouraged. Believe me, I'd love for that to be the end of it, but there are simply too many projects that use it. Since I am sharing a GPU across several isolated services, a VM is not an option.
For whatever reason, I can't get a docker image to build (using `docker build`) in an LXC. Every single time it fails at the same point, no matter what variables I change.
Code:
0 40.81 Successfully built audioread ffmpy lit pathtools wavedrom
#0 41.73 Installing collected packages: wcwidth, tokenizers, sentencepiece, safetensors, rfc3986, pytz, pyston-lite, pydub, pathtools, msgpack, mpmath, lit, ifaddr, ffmpy, cmake, av, zipp, xxhash, websockets, uvloop, urllib3, ujson, uc-micro-py, tzdata, tqdm, toolz, threadpoolctl, svgwrite, soxr, sniffio, smmap, six, setproctitle, semantic-version, scipy, ruamel.yaml.clib, regex, PyYAML, python-multipart, python-dotenv, pyston-lite-autoload, pyrsistent, pyparsing, Pygments, pyee, pydantic, pycparser, pyarrow, psutil, protobuf, prompt-toolkit, pkgutil_resolve_name, packaging, orjson, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, networkx, multidict, mdurl, markdown2, llvmlite, lazy_loader, kiwisolver, joblib, itsdangerous, httptools, h11, gunicorn, google-crc32c, fsspec, frozenlist, fonttools, filelock, entrypoints, dnspython, dill, decorator, cycler, contourpy, colorlog, click, charset-normalizer, audioread, attrs, async-timeout, aiofiles, yarl, wavedrom, uvicorn, sentry-sdk, scikit-learn, ruamel.yaml, requests, python-dateutil, nvidia-cusolver-cu11, nvidia-cudnn-cu11, multiprocess, markdown-it-py, linkify-it-py, importlib-resources, importlib-metadata, gitdb, email-validator, docker-pycreds, ctranslate2, cffi, anyio, aiosignal, aioice, watchfiles, starlette, soundfile, rich, responses, pylibsrtp, pooch, pandas, numba, mdit-py-plugins, matplotlib, jsonschema, HyperPyYAML, huggingface-hub, httpcore, GitPython, cryptography, aiohttp, wandb, transformers, pyOpenSSL, librosa, httpx, fastapi, altair, gradio, datasets, aiortc, accelerate, speechbrain, peft, fschat
#0 42.06 Attempting uninstall: mpmath
#0 42.06 Found existing installation: mpmath 1.2.1
#0 42.07 Uninstalling mpmath-1.2.1:
#0 42.07 ERROR: Could not install packages due to an OSError: [Errno 22] Invalid argument: '/usr/local/lib/python3.8/dist-packages/mpmath-1.2.1.dist-info/'
#0 42.07
#0 42.13
#0 42.13 [notice] A new release of pip available: 22.3.1 -> 23.1.2
#0 42.13 [notice] To update, run: python -m pip install --upgrade pip
------
Dockerfile:16
--------------------
14 | COPY requirements.txt .
15 | # Run pip install with cache so we speedup subsequent rebuilds
16 | >>> RUN --mount=type=cache,target=/root/.cache pip install -r requirements.txt
17 |
18 | # Install auto-gptq
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1Now when I try this in a VM, it works without a hitch, and other people have no problem building the image. The author was even able to build it in a LXC container, but not using Proxmox.
I've tried privileged and unprivileged, debian vs ubuntu, nesting and FUSE on and off, building as root and as another user, tons of RAM and storage, but it always fails at the exact same point.
This is the project: https://github.com/toverainc/willow-inference-server#getting-started if anyone wants to give it a try. You can do it without a GPU, which is how I built it in the VM and it built successfully. But in the LXC it always fails.