[SOLVED] Problems migrating LXC from Ubuntu2004 to Proxmox

[ninjabes]

Hey guys!
I recently started using Proxmox and I am really happy with it. Thanks for the great work so far!

I tried to migrate some of my LXCs from Ubuntu 2004 to Proxmox 7.1 but I am having problems with the UID/GID mapping. What I did so far:

1. I mounted the LXC storage of my Ubuntu instace
2. Packed a tar.gz archive of the rootfs of my container
3. Created a new LXC in Proxmox by using the tar.gz archive with the following command:

pct create 102 checkmk.tar.gz \
  -description checkmk -hostname checkmk -memory 65536 -nameserver 8.8.8.8 \
  -net0 name=eth0,hwaddr=****,ip=dhcp,bridge=vmbr0 \
  -storage rfci-lxc-zfs -rootfs rfci-lxc-zfs:120 \
  -password ****

Everything worked fine, the containers are created with any errors and boot up. But the problem is I cant use any user in my containers. I think the problem is down to the UID/GID mapping inside of the containers. For example when I do:

root@checkmk:/home/ninjab3s# ls -alh
total 44K
drwxr-xr-x 5 1001001 1001001   11 Dec  1 14:10 .
drwxr-xr-x 4 1000000 1000000    4 Dec  1 10:13 ..
-rw------- 1 1001001 1001001  645 Dec  1 14:22 .bash_history
-rw-r--r-- 1 1001001 1001001  220 Dec  1 10:13 .bash_logout
-rw-r--r-- 1 1001001 1001001 3.7K Dec  1 10:13 .bashrc
drwx------ 2 1001001 1001001    3 Dec  1 10:19 .cache
drwxrwxr-x 3 1001001 1001001    3 Dec  1 10:13 .local
-rw-r--r-- 1 1001001 1001001  807 Dec  1 10:13 .profile
drwxrwxr-x 2 1001001 1001001    3 Dec  1 10:14 .ssh
-rw-r--r-- 1 1001001 1001001    0 Dec  1 10:15 .sudo_as_admin_successful
-rw-rw-r-- 1 1001001 1001001  31K Nov 24 08:39 check-mk-agent_2.0.0p16-1_all.deb

I have read https://pve.proxmox.com/wiki/Unprivileged_LXC_containers and tried to set the mapping of the UID/GID in the /etc/pve/lxc/102.conf but without any success. I am still new to Proxmox and LXC and I am not sure how to fix this problem. May someone point me to the right resource or give me a helping hand on this? Every help is appreciated!

Best!
ninajb3s

 

[ninjabes]

I managed to get it to work! The error was that I was packing the rootfs directly on the host instead of in the LXC and I didnt create an unprivileged CT. Here is my solution in case someone has the same problem:

1. Pack the rootfs inside the LXC excluding /dev /sys /proc
2. Copy file out of the LXC and scp to Proxmox
3. Create new CT with the following command:

pct create 110 checkmk.tar.gz \
    -description checkmk -hostname checkmk -memory 65536 -nameserver 8.8.8.8 \
    -net0 name=eth0,hwaddr=00:16:3e:88:ff:ff,ip=dhcp,bridge=vmbr0 \
    -storage local-lvm -rootfs local-lvm:120 \
    -unprivileged 1

I attached the script I wrote to move all my LXCs to my Proxmox server. It was just a quick solution but I hope it will be helpful to someone.

Cheers!

 

[flove]

Hi @ninjabes
thanks for your workaround and script.

I also struggle migrating from ubuntu 20.04 lxd / lxc to proxmox and have the same permission issues.

Thanks to your script and the approach I'm able to migrate containers and start them.

There is one small issue left halt & reboot commands:

When I enter "halt" or "reboot" from the CT console also nothing happens.
If I run "halt" again I get this error from the container:

╭─root at bookstack in ~ 22-10-17 - 11:16:08
╰─○ halt
[1]    247 hangup     halt
╭─root at bookstack in ~ 22-10-17 - 11:16:09
╰─○ halt
Failed to connect to bus: Connection refused
Failed to open initctl fifo: No such device or address
Failed to talk to init daemon.

Now only the "stop" command from webui works...

Any ideas?

 

[ninjabes]

Hey @flove!

I think its easier to achieve the same by using rsync. I think the problem in my old script was that I had to exclude some more directories. You can try something like this:

rsync -aAXv '--exclude=/dev/*' --exclude=/proc/ --exclude=/sys/ '--exclude=/tmp/*' '--exclude=/usr/tmp/*' '--exclude=/run/*' '--exclude=/mnt/*' '--exclude=/media/*' '--exclude=/var/cache/*' --exclude=/ --exclude=/lost+found --exclude=/boot/ / root@10.1.1.249:/

1. create container in proxmox
2. start and rsync
3. restart

Not sure if the syntax is correct for the rsync command and also not sure about the folders. Make sure that the you use the root user on the container side, otherwise you will be lacking permissions to overwrite files.

 

[flove]

Thanks a lot. So you think the shutdown issues relate to some not excluded directories?

I already excluded via

tar --exclude=dev --exclude=sys --exclude=proc -czvf /${container_name}.tar.gz ./*"

I will try excluding via tar first. With rsync the whole script needs to be refactored and I like the approach of a "real import" instead of overwriting the contents of a fresh container using rsync...

I will keep you posted...

[Update]
With this tar command

 tar --exclude='/dev/*' --exclude='/proc/' --exclude='/sys/' --exclude='/tmp/*' --exclude='/usr/tmp/*' --exclude='/run/*' --exclude='/mnt/*' --exclude='/media/*' --exclude='/var/cache/*' --exclude='/lost+found' --exclude='/boot/' -czvf /${container_name}.tar.gz ./*

I get this error:

pct create 111 /var/lib/vz/template/cache/bookstack.tar.gz -description bookstack -hostname bookstack -memory 1024 -net0 name=eth0,hwaddr=00:16:3e:75:dd:92,ip=dhcp,ip6=dhcp,bridge=vmbr0 -storage local-zfs --rootfs local-zfs:120 -unprivileged 1 --ostype ubuntu -password 'ChangeMEEEEE'
extracting archive '/var/lib/vz/template/cache/bookstack.tar.gz'
tar: ./proc/sys/kernel/random/boot_id: Cannot hard link to './dev/.lxc-boot-id': No such file or directory
tar: ./sys/kernel/security/apparmor/.null: Cannot mknod: Operation not permitted
Total bytes read: 2293596160 (2.2GiB, 150MiB/s)
tar: Exiting with failure status due to previous errors
unable to create CT 111 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - -z --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/111/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code

With --ignore-unpack-errors I can create the container.

Unfortunately halt/reboot commands don't work from within the container.

There must be a better way - Taking backups from within the container is a little ugly and will result in "crash consistent" backup
- databases could be inconsistent
- unfinished file io

Any ideas how we can migrate lxd from ubuntu to proxmox with preserving correct container internal users and permissions?

 

[flove]

Little more testing here and there:

1. created a blank new lxd container on ubuntu 20.04 source host > migrated to proxmox = halt command wokrs fine
   must be some issue in the older images I used for the existing containers (IIRC the already cycled through one or two do-release-upgrades
2. tuned the tar command and now I have no issues with import
   

   lxc exec "$container_name" -- bash -c "cd / && sudo tar --exclude=${container_name}.tar.gz --exclude=proc/* --exclude='tmp/*' --exclude='mnt/*' --exclude='dev/*' --exclude='sys/*' --exclude='run/*' --exclude='media/*' --exclude='var/cache/apt/archives/*' -czpf /${container_name}.tar.gz ./*"

[Update]
Strange things happen, patience is key: as soon as I exit "pct console" the (older) container stops...

 

[ninjabes]

Another approach you could do is mounting the rootfs of the LXC. I didnt go that way the last time because there was a difference in the GID/UID mapping and I had to manually correct that for all files. Maybe that works for you? There you can tar the whole thing or work with rsync. But have a look at the formats first. I recall that there was some differences between Ubuntu and Proxmox formats for containers.

 

[flove]

I tuned your script to my needs and published it on Github: https://gist.github.com/florian-obradovic/07ddaa37a614a028661f683279c25c1e

Please let me know If you're ok with this Thanks.

 

[flove]

Another approach you could do is mounting the rootfs of the LXC. I didnt go that way the last time because there was a difference in the GID/UID mapping and I had to manually correct that for all files. Maybe that works for you? There you can tar the whole thing or work with rsync. But have a look at the formats first. I recall that there was some differences between Ubuntu and Proxmox formats for containers.

Yep, same for me - When mounting the rootfs the IDs won't match. There should be a way to chroot with correct UID mappings, create tar, import on target...

Currently I'm running the migration (looks promising) and will keep the source containers offline just in case.
Thanks a lot for your help!

 

[flove]

Updated to pigz for faster compression and progress bar:

 lxc exec "$container_name" -- bash -c "cd / && tar --exclude=${container_name}.tar.gz --exclude='proc/*' --exclude='tmp/*' --exclude='mnt/*' --exclude='dev/*' --exclude='sys/*' --exclude='run/*' --exclude='media/*' --exclude='var/cache/apt/archives/*' -cpf - ./* | pv | pigz > ${container_name}.tar.gz