Hi all,
I have had this issue for almost a week now, and my Google-fu is exhausted. Hoping someone can help.
I had to reinstall Proxmox onto my host, however, before I completed the reinstall, I was able to rescue the lxc raw disk images. After rebuilding Proxmox, I recreated the containers with as similar specifications as I could remember and copied the raw disk image over the top of the one newly created. I ran pct rescan --vmid <cid> for each container and added the 2 config lines to mount /dev/net/tun into the container and configure the cgroups.
Everything seemed to be happy again, except for tailscale. I have tried with privileged and unprivileged containers to no avail. I have gone as far as setting 777 perms on the tun on the host. Still nothing.
If I modprobe tun on the host, it returns success, but on the containers, I always get the error: modprobe: FATAL: Module tun not found in directory /lib/modules/5.15.30-2-pve
If I execute the tailscale daemon with the --tun=userspace-networking flag, it works fine. However when I attempt to run it not in the userspace I get the following output:
On the host, if I run cat /dev/net/tun on the host I get the message 'File descriptor in bad state' which is expected. If I do the same thing on the container I just get Operation Not Permitted
I have tried with a freshly downloaded vanilla template of Debian, and the same issue occurs. This leads me to believe it is the host not the containers perhaps?
Any help is greatly appreciated. Relevant logs below:
/etc/pve/lxc/111.conf
Output of lxc-start -n 111 -F -lDEBUG -o ~/lxc-111.log:
I have had this issue for almost a week now, and my Google-fu is exhausted. Hoping someone can help.
I had to reinstall Proxmox onto my host, however, before I completed the reinstall, I was able to rescue the lxc raw disk images. After rebuilding Proxmox, I recreated the containers with as similar specifications as I could remember and copied the raw disk image over the top of the one newly created. I ran pct rescan --vmid <cid> for each container and added the 2 config lines to mount /dev/net/tun into the container and configure the cgroups.
Everything seemed to be happy again, except for tailscale. I have tried with privileged and unprivileged containers to no avail. I have gone as far as setting 777 perms on the tun on the host. Still nothing.
If I modprobe tun on the host, it returns success, but on the containers, I always get the error: modprobe: FATAL: Module tun not found in directory /lib/modules/5.15.30-2-pve
If I execute the tailscale daemon with the --tun=userspace-networking flag, it works fine. However when I attempt to run it not in the userspace I get the following output:
Code:
root@media .../systemd/system # /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port=41641
logtail started
Program starting: v1.34.0-tbb6e746f3-g8d1edab6f, Go 1.19.2-ts3fd24dee31: []string{"/usr/sbin/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/run/tailscale/tailscaled.sock", "--port=41641"}
LogID: ea897e0cca884b02606a456c0a9f3d51a69932f613fe012010359722ee6c2e34
logpolicy: using system state directory "/var/lib/tailscale"
wgengine.NewUserspaceEngine(tun "tailscale0") ...
Linux kernel version: 5.15.30-2-pve
is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/5.15.30-2-pve
tun module not loaded nor found on disk
wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): operation not permitted
flushing log.
logger closing down
getLocalBackend error: createEngine: tstun.New("tailscale0"): operation not permittedOn the host, if I run cat /dev/net/tun on the host I get the message 'File descriptor in bad state' which is expected. If I do the same thing on the container I just get Operation Not Permitted
I have tried with a freshly downloaded vanilla template of Debian, and the same issue occurs. This leads me to believe it is the host not the containers perhaps?
Any help is greatly appreciated. Relevant logs below:
/etc/pve/lxc/111.conf
Code:
arch: amd64
cores: 2
features: nesting=1
hostname: media
memory: 2048
mp0: mp=/srv/docker,/mnt/disks/rd0/docker
mp1: mp=/downloads,/mnt/disks/rd0/downloads
mp2: mp=/media,/data/media
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.5.254,hwaddr=3E:63:1B:FD:7F:0E,ip=192.168.5.111/24,type=veth
onboot: 1
ostype: debian
rootfs: local:111/vm-111-disk-0.raw,size=32G
swap: 1024
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
lxc.cgroup.devices.allow: c 10:200 rwmOutput of lxc-start -n 111 -F -lDEBUG -o ~/lxc-111.log:
Code:
...
lxc-start 111 20221202111449.536 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 23 and stashed path as cgroup:/proc/169185/fd/23
lxc-start 111 20221202111449.536 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2735 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start 111 20221202111449.536 INFO cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2831 - Limits for the unified cgroup hierarchy have been setup
lxc-start 111 20221202111449.539 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "111", config section "net"
lxc-start 111 20221202111449.883 DEBUG network - network.c:netdev_configure_server_veth:851 - Instantiated veth tunnel "veth111i0 <--> vethK80WFh"
lxc-start 111 20221202111449.883 DEBUG conf - conf.c:lxc_mount_rootfs:1432 - Mounted rootfs "/var/lib/lxc/111/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start 111 20221202111449.883 INFO conf - conf.c:setup_utsname:875 - Set hostname to "media"
lxc-start 111 20221202111449.935 DEBUG network - network.c:setup_hw_addr:3807 - Mac address "3E:63:1B:FD:7F:0E" on "eth0" has been setup
lxc-start 111 20221202111449.935 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3948 - Network device "eth0" has been setup
lxc-start 111 20221202111449.935 INFO network - network.c:lxc_setup_network_in_child_namespaces:4005 - Finished setting up network devices with caller assigned names
lxc-start 111 20221202111449.935 INFO conf - conf.c:mount_autodev:1215 - Preparing "/dev"
lxc-start 111 20221202111449.936 INFO conf - conf.c:mount_autodev:1276 - Prepared "/dev"
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:lxc_mount_auto_mounts:735 - Invalid argument - Tried to ensure procfs is unmounted
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:lxc_mount_auto_mounts:758 - Invalid argument - Tried to ensure sysfs is unmounted
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2412 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2431 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2475 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2412 - Remounting "/dev/net/tun" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/net/tun" to respect bind or remount options
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2431 - Flags for "/dev/net/tun" were 4098, required extra flags are 2
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2475 - Mounted "/dev/net/tun" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/net/tun" with filesystem type "none"
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2475 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
lxc-start 111 20221202111449.936 DEBUG conf - conf.c:mount_entry:2475 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
lxc-start 111 20221202111449.936 DEBUG cgfsng - cgroups/cgfsng.c:__cgroupfs_mount:1541 - Mounted cgroup filesystem cgroup2 onto 20((null))
lxc-start 111 20221202111449.936 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "111", config section "lxc"
lxc-start 111 20221202111449.970 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-autodev-hook" for container "111", config section "lxc"
lxc-start 111 20221202111450.287 INFO conf - conf.c:lxc_fill_autodev:1313 - Populating "/dev"
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "full"
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "null"
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "random"
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "tty"
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "urandom"
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_fill_autodev:1322 - Created device node "zero"
lxc-start 111 20221202111450.288 INFO conf - conf.c:lxc_fill_autodev:1401 - Populated "/dev"
lxc-start 111 20221202111450.288 INFO conf - conf.c:lxc_transient_proc:3771 - Caller's PID is 1; /proc/self points to 1
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_setup_devpts_child:1747 - Attached detached devpts mount 21 to 19/pts
lxc-start 111 20221202111450.288 DEBUG conf - conf.c:lxc_setup_devpts_child:1833 - Created "/dev/ptmx" file as bind mount target
lxc-start 111 20221202111450.289 DEBUG conf - conf.c:lxc_setup_devpts_child:1840 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
lxc-start 111 20221202111450.289 DEBUG conf - conf.c:lxc_allocate_ttys:1101 - Created tty with ptx fd 23 and pty fd 24 and index 1
lxc-start 111 20221202111450.290 DEBUG conf - conf.c:lxc_allocate_ttys:1101 - Created tty with ptx fd 25 and pty fd 26 and index 2
lxc-start 111 20221202111450.290 INFO conf - conf.c:lxc_allocate_ttys:1106 - Finished creating 2 tty devices
lxc-start 111 20221202111450.290 DEBUG conf - conf.c:lxc_setup_ttys:1065 - Bind mounted "" onto "tty1"
lxc-start 111 20221202111450.290 DEBUG conf - conf.c:lxc_setup_ttys:1065 - Bind mounted "" onto "tty2"
lxc-start 111 20221202111450.290 INFO conf - conf.c:lxc_setup_ttys:1072 - Finished setting up 2 /dev/tty<N> device(s)
lxc-start 111 20221202111450.292 INFO conf - conf.c:setup_personality:1913 - Set personality to "0lx0"
lxc-start 111 20221202111450.292 DEBUG conf - conf.c:capabilities_deny:3196 - Dropped mac_admin (33) capability
lxc-start 111 20221202111450.292 DEBUG conf - conf.c:capabilities_deny:3196 - Dropped mac_override (32) capability
lxc-start 111 20221202111450.292 DEBUG conf - conf.c:capabilities_deny:3196 - Dropped sys_time (25) capability
lxc-start 111 20221202111450.292 DEBUG conf - conf.c:capabilities_deny:3196 - Dropped sys_module (16) capability
lxc-start 111 20221202111450.292 DEBUG conf - conf.c:capabilities_deny:3196 - Dropped sys_rawio (17) capability
lxc-start 111 20221202111450.292 DEBUG conf - conf.c:capabilities_deny:3199 - Capabilities have been setup
lxc-start 111 20221202111450.293 NOTICE conf - conf.c:lxc_setup:4464 - The container "111" is set up
lxc-start 111 20221202111450.293 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1186 - Set AppArmor label to "lxc-111_</var/lib/lxc>//&:lxc-111_<-var-lib-lxc>:"
lxc-start 111 20221202111450.293 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set:1231 - Changed AppArmor profile to lxc-111_</var/lib/lxc>//&:lxc-111_<-var-lib-lxc>:
lxc-start 111 20221202111450.299 DEBUG terminal - terminal.c:lxc_terminal_peer_default:702 - Using terminal "/dev/tty" as proxy
lxc-start 111 20221202111450.300 DEBUG terminal - terminal.c:lxc_terminal_winsz:59 - Set window size to 130 columns and 68 rows
lxc-start 111 20221202111450.300 NOTICE utils - utils.c:lxc_drop_groups:1365 - Dropped supplimentary groups
lxc-start 111 20221202111450.301 NOTICE start - start.c:start:2161 - Exec'ing "/sbin/init"
lxc-start 111 20221202111450.304 NOTICE start - start.c:post_start:2172 - Started "/sbin/init" with pid "169207"
lxc-start 111 20221202111450.305 NOTICE start - start.c:signal_handler:449 - Received 17 from pid 169203 instead of container init 169207
lxc-start 111 20221202111450.675 DEBUG commands - commands.c:lxc_cmd_get_tty_fd_callback:1237 - Send tty to client
lxc-start 111 20221202112441.399 DEBUG start - start.c:signal_handler:467 - Container init process 169207 exited
lxc-start 111 20221202112441.410 DEBUG start - start.c:__lxc_start:2104 - UNSUPPORTED(2) - Container "111" is halting
lxc-start 111 20221202112441.410 INFO error - error.c:lxc_error_set_and_log:34 - Child <169207> ended on signal UNSUPPORTED(2)
lxc-start 111 20221202112441.483 INFO network - network.c:lxc_delete_network_priv:3666 - Removed interface "veth111i0" from ""
lxc-start 111 20221202112441.483 DEBUG network - network.c:lxc_delete_network:4159 - Deleted network devices
lxc-start 111 20221202112441.688 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "111", config section "lxc"
lxc-start 111 20221202112442.190 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "111", config section "lxc"