exec format error when recreating containers inside vm

[losarbitan]

Hello all,

Yesterday I decided to recreate my bare metal Rocky Linux install inside a VM on Proxmox. I have gotten everything up and running with gpu passthrough for the most part. Curiously I have a few containers which refuse to start. Specifically this is about:

• binhex/arch-delugevpn
• ghcr.io/haveagitgat/tdarr
• lscr.io/linuxserver/overseerr
• lscr.io/linuxserver/radarr
• lscr.io/linuxserver/lidarr
• lscr.io/linuxserver/prowlarr
• lscr.io/linuxserver/readarr

As you can see it's mostly the *arr stack. I have been running these on bare metal for about 2 years now. I installed Proxmox on the exact machine which was running it bare metal. My VM config file is as follows:

balloon: 0
bios: ovmf
boot: order=sata0;ide2;net0
cores: 10
cpu: x86-64-v2-AES
efidisk0: sapphire_local:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:08:00
ide2: templates:iso/Rocky-9.3-x86_64-dvd.iso,media=cdrom,size=10267584K
machine: q35
memory: 49152
meta: creation-qemu=8.1.2,ctime=1706632301
name: sapphire
net0: virtio=BC:24:11:CE:1D:3D,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
sata0: sapphire_local:vm-100-disk-1,discard=on,size=512G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=e4700abb-56b2-4405-b81f-dbfd94b274b5
sockets: 1
vmgenid: 828f4bcc-3d8b-4655-937b-4b3fbc00e65b

This is the hardware it runs on:
amd ryzen 7 3700x
asus x570-p
corsair 64gb ddr4
proxmox installed on 2x 256gb samsung 850 pro ssd
vm installed on a local samsung 960 pro 1tb m.2 ssd (i call the vm sapphire).

The docker container log for the containers spits out these strange (to me) errors:

• readarr
  • exec /init: exec format error
• delugevpn
  • exec /usr/bin/dumb-init: exec format error
• tdarr
  • Starting Tdarr_Server
  • s6-envuidgid: fatal: unknown user: abc
  • Starting Tdarr_Node
  • s6-envuidgid: fatal: unknown user: abc
• overseerr
  • s6-rc-compile: fatal: invalid /etc/s6-overlay/s6-rc.d/init-overseerr-config/type: must be oneshot, longrun, or bundle
    20
  • s6-rc: fatal: unable to take locks: No such file or directory

Radarr, Lidarr and Prowlarr don't give any logs. Strangely enough Sonarr does work, also as a Linuxserver container image.
Now I have read online that the exec format error is a problem that occurs when the container image is incompatible with the host machine. but it ran fine bare metal. I guess there's some weird thing going on in VM translation or something.

I also tried running in cpu host mode, but that changed nothing.

Can anybody please help me out here?
Thanks in advance!

 

[losarbitan]

Strangely Overseerr now works after a reboot. I changed nothing (I switched cpu to host and then back to x86-64-v2-AES) in the settings. Simply reboot after switching some settings to and fro and it suddenly sterted the container.

 

[losarbitan]

Checing Lidarr, Radarr, Prowlarr and Readarr logs with Grafana Loki reveals that they also give a "exec /init: exec format error". Still curious why Sonarr does work though. I did a bit of snooping and saw that Sonarr is built upon alpine:3.19 while the others are built upon alpine:3.18. I dont know if this could cause something like this, I simply noticed it.
Oh and curiously Bazarr also works and is also built upon alpine:3.19.

 

[losarbitan]

Checing Lidarr, Radarr, Prowlarr and Readarr logs with Grafana Loki reveals that they also give a "exec /init: exec format error". Still curious why Sonarr does work though. I did a bit of snooping and saw that Sonarr is built upon alpine:3.19 while the others are built upon alpine:3.18. I dont know if this could cause something like this, I simply noticed it.
Oh and curiously Bazarr also works and is also built upon alpine:3.19.

redeploying sonarr with lscr.io/linuxserver/sonarr:develop-4.0.0.740-ls18 (which is I think the latest version built on alpine:3.18) still works. Sonarr was up and running with this tag.

 

[losarbitan]

I stopped my docker stacks and restarted them. Almost everything went fine, but updating Immich to 1.94.1 caused all the Immich containers to spit out this error:

exec /usr/bin/tini: exec format error

I still can not seem to find why some containers work fine and some don't.

 

[BobhWasatch]

Exec format error usually means that the binary is for a different cpu or different libc. Like arm vs x86 or glibc vs musl.

ETA: It can also happen if the bit size is wrong. For example if you have 32-bit x86 binaries running on a 64-bit amd64 system but don't have the 32-bit libraries installed.

 

[losarbitan]

Exec format error usually means that the binary is for a different cpu or different libc. Like arm vs x86 or glibc vs musl.

But wouldn't that problem go away if I put the cpu in host mode?

 

[BobhWasatch]

But wouldn't that problem go away if I put the cpu in host mode?

Not if the stuff you are running is a for completely different CPU architecture like arm aarch64 and you are running on intel amd64. The libc issue should not be an issue inside the container but it can be for the container environment itself.

 

[losarbitan]

Exec format error usually means that the binary is for a different cpu or different libc. Like arm vs x86 or glibc vs musl.

ETA: It can also happen if the bit size is wrong. For example if you have 32-bit x86 binaries running on a 64-bit amd64 system but don't have the 32-bit libraries installed.

How could I install the 32-bit libraries on the Rocky Linux?

The thing is. I have rebuilt my bare metal Rocky Linux install inside a VM. I simply copied all my container data and compose files over and recreated them as it was on the bare metal install. I've done this before when migrating containers from 1 server to another. So I thought that putting the cpu in host mode for the vm would alleviate the cpu issue for format error.

Guessing it must be a lib thing than? I'm not really sure what that exactly means tbh, sorry. I'm no professional nor educated in IT stuff. Simply a self-taught home server owner.

Thanks for the help, I appreciate it.

 

[losarbitan]

Not if the stuff you are running is a for completely different CPU architecture like arm aarch64 and you are running on intel amd64. The libc issue should not be an issue inside the container but it can be for the container environment itself.

Yeah but it worked before. So it can not be that the cpu is incompatible.

 

[losarbitan]

I completely assume that I've done something stupid here. Just hoping to get some help with finding out what it is that I did wrong

 

[BobhWasatch]

How could I install the 32-bit libraries on the Rocky Linux?'t c

Probably with dnf? I don't know that is even the problem.

Guessing it must be a lib thing than? I'm not really sure what that exactly means tbh, sorry. I'm no professional nor educated in IT stuff. Simply a self-taught home server owner.

Most linux programs link with a shared library conventionally known as libc. Some are "statically compiled" and include their own libc so they depend only on the kernel. The vast majority of Linux-en use gnu libc or glibc. Alpine uses an alternative called musl libc. They aren't compatible at the binary level.

Inside a container you can use musl even if the host is glibc and vice versa. Outside a container it is...tricky. So one thing to check would be to make sure you have the right version of Docker installed.

Anyway, that was a digression. If the containers don't start then you might have a problem with the container system itself. If they do start and you get those errors from inside then the problem is inside the container. Since it seems to affect multiple containers in the same way outside seems like the most likely place to look.

 

[losarbitan]

Probably with dnf? I don't know that is even the problem.


Most linux programs link with a shared library conventionally known as libc. Some are "statically compiled" and include their own libc so they depend only on the kernel. The vast majority of Linux-en use gnu libc or glibc. Alpine uses an alternative called musl libc. They aren't compatible at the binary level.

Inside a container you can use musl even if the host is glibc and vice versa. Outside a container it is...tricky. So one thing to check would be to make sure you have the right version of Docker installed.

Anyway, that was a digression. If the containers don't start then you might have a problem with the container system itself. If they do start and you get those errors from inside then the problem is inside the container. Since it seems to affect multiple containers in the same way outside seems like the most likely place to look.

Okay yes that does seem to make sense. But I am still confused about one thing. I built the VM exactly as I built the bare metal install of Rocky Linux. I used the same config files etc. because I have those backed up. The VM is on a Proxmox install on the exact same hardware (the exact same machine) that I had the bare metal install of Rocky Linux on. Basically it is the same machine, but just with a proxmox virtualisation layer inbetween Rocky Linux and the hardware.
So I do not understand how there can be a difference in environment. I guessed that setting CPU to Host mode would basically make it so that the OS would see no difference between the bare metal install and the VM install.

Is there something that happens on the VM layer that could change the environment in such a way? Does it emulate the hardware differently from how it works in bare metal mode? I would guess that it could not be so different right?