docker: failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /var/log/apt: invalid argument.

[andersdd]

Hey,
I had the same issue today - My fix work as well!
Switch back to VFS storage:
https://docs.docker.com/storage/storagedriver/vfs-driver/

Stop docker service

systemctl stop docker

Create or edit docker daemon config

nano /etc/docker/daemon.json

Add storage-driver:

{
  "storage-driver": "vfs"
}

And start docker

systemctl start docker

Found after some troubleshooting and help with this post:
https://forum.proxmox.com/threads/t...-update-has-really-messed-up-my-boxes.119933/

Thank you for posting this. Saved me a tonne of heartache

 

[Wod]

Hello,
Since I updated to Proxmox 7.3-4, I'm getting the following error when pulling some docker images, inside LXC containers.

What should I do? This worked fine before updating.

I have the same issue in a similiar situation and I like to give some additional information on my environment.
Non privileged Debian (Bookworm) LXC with Rootdisk on a ZFS drive. Latest docker, docker-compose and portainer running. Several Docker Container running without any issues.
I then created a stack for Paperless NGX and the first installation was fine, but I had some minor issues with files outside the container.
So I stopped the stack, deleted volumes, images and containers and made my changes to the stack. After Deploying the stack I had the same issue as reported in post1. Even stopping the LXC and restore it from a backup never let me deploying the stack again. Watchtower did update other dockerfiles fine, though.

I then set up a new LXC and this time the root-fs is on a thin-lvm (ext4, nvme drive). Docker, docker-compose, portainer, Debian-version are all identical to original LXC. Anyhow this time I have been able to create and deploy the stack without any issues.

Next step: Use Backup of original LXC (ZFS), restore to LVM and redeploy stack. There might be somesthing strange with docker and ZFS. I understand Posts #15 to #19 are pointing in that direction as well, allthough Docker says the issue is fixed.

Btw. I am on Proxmox 8.0.4, but obviously the ZFS has been setup under Proxmox 6.xxx.

 

[Wod]

I have the same issue in a similiar situation and I like to give some additional information on my environment.
Non privileged Debian (Bookworm) LXC with Rootdisk on a ZFS drive. Latest docker, docker-compose and portainer running. Several Docker Container running without any issues.
I then created a stack for Paperless NGX and the first installation was fine, but I had some minor issues with files outside the container.
So I stopped the stack, deleted volumes, images and containers and made my changes to the stack. After Deploying the stack I had the same issue as reported in post1. Even stopping the LXC and restore it from a backup never let me deploying the stack again. Watchtower did update other dockerfiles fine, though.

I then set up a new LXC and this time the root-fs is on a thin-lvm (ext4, nvme drive). Docker, docker-compose, portainer, Debian-version are all identical to original LXC. Anyhow this time I have been able to create and deploy the stack without any issues.

Next step: Use Backup of original LXC (ZFS), restore to LVM and redeploy stack. There might be somesthing strange with docker and ZFS. I understand Posts #15 to #19 are pointing in that direction as well, allthough Docker says the issue is fixed.

Btw. I am on Proxmox 8.0.4, but obviously the ZFS has been setup under Proxmox 6.xxx.

I now backed up the whole lxc (root-fs located on ZFS) and restored it to local-lvm (which is ext4). Now the stack deployed nicely, the system is up and running. So still there might be an issue with Docker overlay-fs on ZFS. At least for me that fixed the issue reported in #1:

docker: failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /var/log/apt: invalid argument.
See 'docker run --help'.

 

[unreal1]

I was able to fix this by using the enabling the container features 'keyctl', 'Nesting', and 'FUSE' under the options tab. Then running

apt install -y fuse-overlayfs ln -s /usr/bin/fuse-overlayfs /usr/local/bin/fuse-overlayfs

 

[glassman]

I was able to fix this by using the enabling the container features 'keyctl', 'Nesting', and 'FUSE' under the options tab. Then running

apt install -y fuse-overlayfs ln -s /usr/bin/fuse-overlayfs /usr/local/bin/fuse-overlayfs

Yea, so that’s already been suggested here in a few different posts. I think people want ZFS to work with overlay2, but currently there are still some features missing from ZFS that would make this 100% possible, that’s why some images fail to pull. FUSE is just a band-aid.

 

[sirebral]

I think this is supposed to work now, interestingly enough I still get some complaints about unsupported configs if I use Overlay2, it doesn't SEEM to cause issues, just makes me curious as to why it's still throwing errors.

 

[LordRatner]

I've upgraded with the new zfs version, yet I'm still getting the error in unprivileged container.

failed to register layer: unlinkat /usr/lib/locale/C.UTF-8/LC_MESSAGES: invalid argument

This is trying to install Frigate with the following compose.yml file:

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: false # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "2048mb" # update for your cameras based on calculation above
    devices:
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      - /dev/apex_1:/dev/apex_1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1 # number of GPUs
              capabilities: [gpu]
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/appdata/frigate/config:/config
      - /mnt/nvr/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 2000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"