I was wondering if anyone has been able to find a solution to running redroid as a pve lxc?
currently redroid offers instructions to run redroid in lxc directly, instructions are here:
binder is included in my kernel so I ran:
works fine then lxc-create:
(I am using a pre-built docker image that has magisk precompiled and working for redroid11)
I run into these issues which after checking those lines in lxc-oci script, seems can be ignored maybe?:
trying to run the lxc will results in error due to incorrect bridge adapter which can be fixed by
nano /var/lib/lxc/redroid/config
changing out lxcbr0 to vmbr0 or whatever your networking bridge setup is
will successfully start, and I can continue with the rest of the instructions, incloudng logging into container via adb running the script directly as instructed:
I can verify that redroid is running fully without issue direclty as an lxc on my debian12 proxmox ve host, so I proceed to convert this from lxc to pve lxc:
the lxc template is now availabe in my CT Template option, so now I try to create with pct. I go through the setup process which seems fine but then complains about setting the arch type which should be arm64. I use:
pct set 4000 --arch arm64
I try to start up again and get errors:
I tried to get more details so I can with debug:
I'm honestly not sure how to go from here or solve the init binary issue. It works fine when i run redroid via lxc on debian12 host, it just doesn't work fine once I attempt to convert it to pve lxc. Any hints would be great or any help. The goal here is to get a working redroid pve lxc template. As there are overhead savings compared to running a vm > docker > redroid
Also I am using Proxmox VE 8.1.7 custom for ARM64 Ampere, since Proxmox does not natively support arm yet or may not never plan to. I have also tried creating a debian12 ct from a standard template, then merging the redroid rootfs with that template pct, and still no joy.
currently redroid offers instructions to run redroid in lxc directly, instructions are here:
https://github.com/remote-android/redroid-doc/blob/master/deploy/lxc.mdbinder is included in my kernel so I ran:
modprobe binder_linux devices="binder,hwbinder,vndbinder"works fine then lxc-create:
(I am using a pre-built docker image that has magisk precompiled and working for redroid11)
Code:
lxc-create -n redroid -t oci -- -u docker://docker.io/abing7k/redroid:a11_magisk_arm
Getting image source signatures
Copying blob 46c685d80da2 skipped: already exists
Copying blob 6a3e3d211f69 skipped: already exists
Copying config d4c3358f44 done
Writing manifest to image destination
Storing signatures
Unpacking the rootfs
/usr/share/lxc/templates/lxc-oci: line 384: /var/lib/lxc/redroid/rootfs/etc/hostname: No such file or directory
/usr/share/lxc/templates/lxc-oci: line 389: /var/lib/lxc/redroid/rootfs/etc/hosts: No such file or directoryI run into these issues which after checking those lines in lxc-oci script, seems can be ignored maybe?:
trying to run the lxc will results in error due to incorrect bridge adapter which can be fixed by
nano /var/lib/lxc/redroid/config
changing out lxcbr0 to vmbr0 or whatever your networking bridge setup is
Code:
lxc-start -l debug -o redroid.log -n redroid
lxc-info
Name: redroid
State: RUNNING
PID: 1476327
Link: vethZmXNf9
TX bytes: 848 bytes
RX bytes: 700 bytes
Total bytes: 1.51 KiBwill successfully start, and I can continue with the rest of the instructions, incloudng logging into container via adb running the script directly as instructed:
Code:
nsenter -t `lxc-info redroid -p | awk '{print $2}'` -a sh
:/ # ls
acct cache data_mirror dsp init.environ.rc metadata persist sdcard system_ext
apex config debug_ramdisk etc ipc mnt proc storage vendor
bin d default.prop firmware linkerconfig odm product sys
bugreports data dev init lost+found oem sbin system
:/ #I can verify that redroid is running fully without issue direclty as an lxc on my debian12 proxmox ve host, so I proceed to convert this from lxc to pve lxc:
Code:
lxc-stop
tar --exclude=dev --exclude=sys --exclude=proc -czvf redroid11.magisk.arm.tar.gz -C /var/lib/lxc/redroid/rootfs .
mv redroid11.magisk.arm.tar.gz /var/lib/vz/template/cachethe lxc template is now availabe in my CT Template option, so now I try to create with pct. I go through the setup process which seems fine but then complains about setting the arch type which should be arm64. I use:
pct set 4000 --arch arm64
I try to start up again and get errors:
Code:
sync_wait: 34 An error occurred in another process (expected sequence number 7)
__lxc_start: 2107 Failed to spawn container "4000"
TASK ERROR: startup for container '4000' failed
failed waiting for client: timed out
TASK ERROR: command '/usr/bin/termproxy 5900 --path /vms/4000 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole4000 -r winch -z lxc-console -n 4000 -e -1' failed: exit code 1I tried to get more details so I can with debug:
Code:
pct start 4000 --debug &> pct.4000.debug.log
cat pct.4000.debug.log
sync_wait: 34 An error occurred in another process (expected sequence number 7)
__lxc_start: 2107 Failed to spawn container "4000"
script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "4000", config section "lxc"
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1227 - Running privileged, not using a systemd unit
DEBUG seccomp - ../src/lxc/seccomp.carse_config_v2:656 - Host native arch is [3221225655]
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "[all]"
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "kexec_load errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[104:kexec_load] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[104:kexec_load] action[327681:errno] arch[1073741864]
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "open_by_handle_at errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[265pen_by_handle_at] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[265pen_by_handle_at] action[327681:errno] arch[1073741864]
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "init_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[105:init_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[105:init_module] action[327681:errno] arch[1073741864]
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "finit_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[273:finit_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[273:finit_module] action[327681:errno] arch[1073741864]
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:807 - Processing "delete_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[106:delete_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[106:delete_module] action[327681:errno] arch[1073741864]
INFO seccomp - ../src/lxc/seccomp.carse_config_v2:1017 - Merging compat seccomp contexts into main context
INFO start - ../src/lxc/start.c:lxc_init:881 - Container "4000" is initialized
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1391 - The monitor process uses "lxc.monitor/4000" as cgroup
DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1499 - The container process uses "lxc/4000/ns" as inner and "lxc/4000" as limit cgroup
INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNS
INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWPID
INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWUTS
INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWIPC
INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNET
INFO start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWCGROUP
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/1516871/fd/18
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/1516871/fd/19
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/1516871/fd/20
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/1516871/fd/21
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 22 and stashed path as net:/proc/1516871/fd/22
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 23 and stashed path as cgroup:/proc/1516871/fd/23
WARN cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits_legacy:3155 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3251 - Limits for the unified cgroup hierarchy have been setup
INFO conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "4000", config section "net"
DEBUG network - ../src/lxc/network.c:netdev_configure_server_veth:852 - Instantiated veth tunnel "veth4000i0 <--> veths49XQT"
DEBUG conf - ../src/lxc/conf.c:lxc_mount_rootfs:1437 - Mounted rootfs "/var/lib/lxc/4000/rootfs" onto "/usr/lib/aarch64-linux-gnu/lxc/rootfs" with options "(null)"
INFO conf - ../src/lxc/conf.c:setup_utsname:876 - Set hostname to "testdroid"
DEBUG network - ../src/lxc/network.c:setup_hw_addr:3821 - Mac address "BC:24:11:61:3C:BD" on "eth0" has been setup
DEBUG network - ../src/lxc/network.c:lxc_network_setup_in_child_namespaces_common:3962 - Network device "eth0" has been setup
INFO network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4019 - Finished setting up network devices with caller assigned names
INFO conf - ../src/lxc/conf.c:mount_autodev:1220 - Preparing "/dev"
INFO conf - ../src/lxc/conf.c:mount_autodev:1281 - Prepared "/dev"
DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:736 - Invalid argument - Tried to ensure procfs is unmounted
DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:759 - Invalid argument - Tried to ensure sysfs is unmounted
DEBUG conf - ../src/lxc/conf.c:mount_entry:2445 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/aarch64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
DEBUG conf - ../src/lxc/conf.c:mount_entry:2464 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
DEBUG conf - ../src/lxc/conf.c:mount_entry:2508 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/aarch64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
DEBUG cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroupfs_mount:1909 - Mounted cgroup filesystem cgroup2 onto 20((null))
INFO conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "4000", config section "lxc"
INFO conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-autodev-hook" for container "4000", config section "lxc"
INFO conf - ../src/lxc/conf.c:lxc_fill_autodev:1318 - Populating "/dev"
DEBUG conf - ../src/lxc/conf.c:lxc_fill_autodev:1327 - Created device node "full"
DEBUG conf - ../src/lxc/conf.c:lxc_fill_autodev:1327 - Created device node "null"
DEBUG conf - ../src/lxc/conf.c:lxc_fill_autodev:1327 - Created device node "random"
DEBUG conf - ../src/lxc/conf.c:lxc_fill_autodev:1327 - Created device node "tty"
DEBUG conf - ../src/lxc/conf.c:lxc_fill_autodev:1327 - Created device node "urandom"
DEBUG conf - ../src/lxc/conf.c:lxc_fill_autodev:1327 - Created device node "zero"
INFO conf - ../src/lxc/conf.c:lxc_fill_autodev:1406 - Populated "/dev"
INFO conf - ../src/lxc/conf.c:lxc_transient_proc:3804 - Caller's PID is 1; /proc/self points to 1
DEBUG conf - ../src/lxc/conf.c:lxc_setup_devpts_child:1780 - Attached detached devpts mount 21 to 19/pts
DEBUG conf - ../src/lxc/conf.c:lxc_setup_devpts_child:1866 - Created "/dev/ptmx" file as bind mount target
DEBUG conf - ../src/lxc/conf.c:lxc_setup_devpts_child:1873 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
DEBUG conf - ../src/lxc/conf.c:lxc_allocate_ttys:1105 - Created tty with ptx fd 23 and pty fd 24 and index 1
DEBUG conf - ../src/lxc/conf.c:lxc_allocate_ttys:1105 - Created tty with ptx fd 25 and pty fd 26 and index 2
INFO conf - ../src/lxc/conf.c:lxc_allocate_ttys:1110 - Finished creating 2 tty devices
DEBUG conf - ../src/lxc/conf.c:lxc_setup_ttys:1029 - Bind mounted "pts/1" onto "/dev/lxc/tty1"
DEBUG conf - ../src/lxc/conf.c:lxc_setup_ttys:1029 - Bind mounted "pts/2" onto "/dev/lxc/tty2"
INFO conf - ../src/lxc/conf.c:lxc_setup_ttys:1073 - Finished setting up 2 /dev/tty<N> device(s)
INFO conf - ../src/lxc/conf.c:setup_personality:1946 - Set personality to "0lx0"
DEBUG conf - ../src/lxc/conf.c:capabilities_deny:3229 - Dropped mac_admin (33) capability
DEBUG conf - ../src/lxc/conf.c:capabilities_deny:3229 - Dropped mac_override (32) capability
DEBUG conf - ../src/lxc/conf.c:capabilities_deny:3229 - Dropped sys_time (25) capability
DEBUG conf - ../src/lxc/conf.c:capabilities_deny:3229 - Dropped sys_module (16) capability
DEBUG conf - ../src/lxc/conf.c:capabilities_deny:3229 - Dropped sys_rawio (17) capability
DEBUG conf - ../src/lxc/conf.c:capabilities_deny:3232 - Capabilities have been setup
NOTICE conf - ../src/lxc/conf.c:lxc_setup:4511 - The container "4000" is set up
INFO apparmor - ../src/lxc/lsm/apparmor.c:apparmor_process_label_set_at:1189 - Set AppArmor label to "lxc-4000_</var/lib/lxc>//&:lxc-4000_<-var-lib-lxc>:"
INFO apparmor - ../src/lxc/lsm/apparmor.c:apparmor_process_label_set:1234 - Changed AppArmor profile to lxc-4000_</var/lib/lxc>//&:lxc-4000_<-var-lib-lxc>:
DEBUG terminal - ../src/lxc/terminal.c:lxc_terminal_peer_default:696 - No such device - The process does not have a controlling terminal
NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1367 - Dropped supplimentary groups
NOTICE start - ../src/lxc/start.c:start:2194 - Exec'ing "/sbin/init"
ERROR start - ../src/lxc/start.c:start:2197 - No such file or directory - Failed to exec "/sbin/init"
ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)
INFO network - ../src/lxc/network.c:lxc_delete_network_priv:3680 - Removed interface "veth4000i0" from ""
DEBUG network - ../src/lxc/network.c:lxc_delete_network:4173 - Deleted network devices
ERROR start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "4000"
WARN start - ../src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 17 for process 1516890
startup for container '4000' failed
Code:
nano /var/lib/lxc/4000/config
GNU nano 7.2 /var/lib/lxc/4000/config
lxc.cgroup.relative = 0
lxc.cgroup.dir.monitor = lxc.monitor/4000
lxc.cgroup.dir.container = lxc/4000
lxc.cgroup.dir.container.inner = ns
lxc.arch = arm64
lxc.include = /usr/share/lxc/config/common.conf
lxc.apparmor.profile = generated
lxc.apparmor.raw = deny mount -> /proc/,
lxc.apparmor.raw = deny mount -> /sys/,
lxc.monitor.unshare = 1
lxc.tty.max = 2
lxc.environment = TERM=linux
lxc.uts.name = testdroid
lxc.cgroup2.memory.max = 17179869184
lxc.cgroup2.memory.high = 17045651456
lxc.cgroup2.memory.swap.max = 34359738368
lxc.rootfs.path = /var/lib/lxc/4000/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth4000i0
lxc.net.0.hwaddr = BC:24:11:61:3C:BD
lxc.net.0.name = eth0
lxc.net.0.mtu = 1500
lxc.net.0.script.up = /usr/share/lxc/lxcnetaddbr
lxc.cgroup2.cpuset.cpus = 5,10,25,36,40,46,50,76
GNU nano 7.2 /etc/pve/lxc/4000.conf
arch: arm64
cores: 8
hostname: testdroid
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:61:3C:BD,type=veth
ostype: unmanaged
rootfs: local:4000/vm-4000-disk-0.raw,size=128G
swap: 32768I'm honestly not sure how to go from here or solve the init binary issue. It works fine when i run redroid via lxc on debian12 host, it just doesn't work fine once I attempt to convert it to pve lxc. Any hints would be great or any help. The goal here is to get a working redroid pve lxc template. As there are overhead savings compared to running a vm > docker > redroid
Also I am using Proxmox VE 8.1.7 custom for ARM64 Ampere, since Proxmox does not natively support arm yet or may not never plan to. I have also tried creating a debian12 ct from a standard template, then merging the redroid rootfs with that template pct, and still no joy.
Last edited: