[SOLVED] Changing the config for an LXC backup

[jordanplayz158]

I am running Proxmox VE 8.0.4 and trying to restore a LXC backup but the config has issues with `lxc.idmap` as this was used with my old Proxmox setup so I went through some threads, found out the config is stored in `/etc/vzdump/pct.conf` in the backup so I edited the file and removed those lines and re-uploaded the backup (also tried to delete the backup and then upload rather than overwrite, no difference) and try to restore and it is still using the old config and the show config button also confirms this. Given the speed of the extraction (compared to when I first tried to restore), I have to assume there is an uncompressed copy of the backup on Proxmox but I have tried finding a file with the contents of "lxc.idmap" and I don't see one with the id I set it to restore as (I have other lxcs and vms so it does find those configs but those are not relevant). I also checked my zfs pools and there doesn't seem to be a dataset for that lxc either so I am out of ideas. Thank you for reading my thread and any help is appreciated!

 

[jordanplayz158]

I have looked over the codebase and there doesn't seem to be any caching (at least explicitly specified in the code) so I am not sure where this copy of the old `pct.conf` is stored. I tried to restart `pveproxy`, no change. Given that the extractconfig function grabs the config from the archive, I have to assume there is some intermediary cache as that old `pct.conf` should not be shown and the `show config` button and endpoint respond too quickly to be reading and extracting the file from the archive each time.

 

[Chris]

Hi,
the container config is part of the backup and will be restored together with the backup, putting it into /etc/pve/lxc/<VMID>.config after restoring. You can then edit the config there.

 

[jordanplayz158]

I see... What happens when you restore a backup and it fails to restore due to a bad config? Where is the config and/or rootfs stored? In memory? On disk? As I have made the change to the pct.conf inside the tar and tried to restore and it won't use the config inside the tar, just the config it first failed with

 

[Chris]

I see... What happens when you restore a backup and it fails to restore due to a bad config? Where is the config and/or rootfs stored? In memory? On disk? As I have made the change to the pct.conf inside the tar and tried to restore and it won't use the config inside the tar, just the config it first failed with

At what point does the restore of the container fail? Please share the task log as well as the exact error message you encounter. The container is restored by fetching it's config from backup, setting up the rootfs and mountpoints accordingly, mount them and write the backed up files from the tar archive (or pxar archive if PBS is used) to disk.

 

[jordanplayz158]

At what point does the restore of the container fail? Please share the task log as well as the exact error message you encounter. The container is restored by fetching it's config from backup, setting up the rootfs and mountpoints accordingly, mount them and write the backed up files from the tar archive (or pxar archive if PBS is used) to disk.

Oh, I apologize for not providing the log initially in the post, I thought I just missed something obvious to reload the config, here is the log on restore

recovering backed-up configuration from 'local:backup/vzdump-lxc-115-2023_10_21-12_18_42.tar.zst'
restoring 'local:backup/vzdump-lxc-115-2023_10_21-12_18_42.tar.zst' now..
extracting archive '/var/lib/vz/dump/vzdump-lxc-115-2023_10_21-12_18_42.tar.zst'
lxc 20231127193118.956 ERROR    conf - ../src/lxc/conf.c:lxc_map_ids:3701 - newuidmap failed to write mapping "newuidmap: uid range [1100-2000) -> [1100-2000) not allowed": newuidmap 952308 0 100000 1100 1100 1100 900 2000 101200 63535
Failed to write id mapping for child process
lxc 20231127193118.958 ERROR    utils - ../src/lxc/utils.c:lxc_drop_groups:1365 - Operation not permitted - Failed to drop supplimentary groups
lxc 20231127193118.958 ERROR    utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1340 - Invalid argument - Failed to switch to gid 0
TASK ERROR: unable to restore CT 115 - command 'lxc-usernsexec -m u:0:100000:1100 -m g:0:100000:1100 -m u:1100:1100:900 -m g:1100:1100:900 -m u:2000:101200:63535 -m g:2000:101200:63535 -- tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/115/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 1

Essentially, the error is due to `lxc.idmap` entries in the config and I removed these entries at least half a month ago and the "Show Configuration" button for the config still shows

lxc.idmap: u 0 100000 1100
lxc.idmap: g 0 100000 1100
lxc.idmap: u 1100 1100 900
lxc.idmap: g 1100 1100 900
lxc.idmap: u 2000 101200 63535
lxc.idmap: g 2000 101200 63535

 

[fabian]

if the "show configuration" button for that backup shows the entries, then they must still be contained in the backup's config..

 

[jordanplayz158]

I can understand where you are coming from, having looked at the code as well, it seems like that would be the case, but unless it is extracted on disk (and not in the archive) or in some kind of cache (at this point, maybe ZFS has something to do with this given it has caching?). This does not seem to be the case, I have the archive on my machine, I open the archive, go to /etc/vzdump/pct.conf and open the file, it shows the config with the portion removed (lxc.idmap lines) and I do an sha512sum on both the server backup and the backup on my machine, they match, so something weird is going on.

 

[fabian]

no, it's not cached anywhere by PVE.. did you maybe add a second /etc/vzdump/pct.conf to the archive (yes, this is possible with tar )? PVE will only look at the first one..

 

[jordanplayz158]

That is possible, I will check in the morning as it is 3:40 AM my time, I really do hope that is the issue as then I have a solution. And thank you and Chris for looking at my issue and taking the time to respond to help me find the issue.

 

[jordanplayz158]

I have checked and unfortunately, nope there is not a second `/etc/vzdump/pct.conf` as far as I can tell, I run

tar -xpOf vzdump-lxc-115-2023_10_21-12_18_42.tar.zst etc/vzdump/pct.conf
arch: amd64
cores: 2
features: nesting=1
hostname: jellyfin
memory: 4096
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.101,hwaddr=[REDACTED],ip=192.168.1.25/24,ip6=auto,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-115-disk-0,size=8G
swap: 0
tags: 
unprivileged: 1
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'

and even for good measure I run

tar -xpOf vzdump-lxc-115-2023_10_21-12_18_42.tar.zst etc/vzdump/pct.conf --occurrence=2
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.user.crtime_usec'
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.capability'
tar: etc/vzdump/pct.conf: Required occurrence not found in archive
tar: Exiting with failure status due to previous errors

I have also restarted the machine and the config is still showing as the old config despite the above commands and their respective outputs.
The only thing I can guess is that this backup was made in Proxmox 7 before I clean installed Proxmox 8 so maybe the restore logic changed?

 

[Chris]

Are you trying to restore to a new container or are you trying to restore on top of the existing container? If you only tested the latter, try also the first.

 

[jordanplayz158]

I am trying to restore to a new container, I have not tested on top of the existing container as I don't have an existing container for this backup so I don't know if I can test that.

 

[fabian]

(possibly) stupid question - are you sure you are checking and restoring the same tar file? your manual tar invocations don't show the context where they are executed, is it possible the repacked version is not on the storage, but somewhere else?

 

[jordanplayz158]

Often the most obvious questions are the solution but I don't believe this is the case in my case unless I read this wrong.



Sorry for blur, had some sensitive information in the backup

 

[fabian]

can you describe in more detail how you edited and repacked the backup archive?

and can you try the following? "tar tf vzdump-lxc-115-2023_10_21-12_18_42.tar.zst | grep vzdump" ?

 

[Chris]

Hmm, a shot into the wild: Do you maybe have some lingering configs for the same VMID you are trying to restore to located under /var/lib/lxc/<VMID>?

 

[jordanplayz158]

can you describe in more detail how you edited and repacked the backup archive?

I edited and repacked it on a separate machine using Ark, I opened it in Ark, went to etc/vzdump, double clicked on pct.conf, it opens up a temporary file, I edit it, save it, it puts it in the archive and re-compresses the tar and puts it in place, to ensure it goes through, I close out Ark and open the archive again, go to etc/vzdump and open pct.conf and verify the contents, I moved it to the server, did a `sha512sum` on both ends, the checksums are identical (and the commands seem to also show the changes were saved).

SOLUTION:

and can you try the following? "tar tf vzdump-lxc-115-2023_10_21-12_18_42.tar.zst | grep vzdump" ?

Ah ha, I think you found it

root@proxmox:/var/lib/vz/dump# tar tf vzdump-lxc-115-2023_10_21-12_18_42.tar.zst | grep vzdump
etc/vzdump/pct.conf
./etc/vzdump/pct.conf
./etc/vzdump/pct.fw

And... `etc/vzdump/pct.conf` is my modified config while `./etc/vzdump/pct.conf` is the old config and seems to be used by proxmox instead, very fascinating, it is completely invisible in Ark and perhaps this is a double bug to be reported in Ark where it puts an edited file in the wrong location and cannot see `./` files in tar archives, or some variation, fascinating...

Hmm, a shot into the wild: Do you maybe have some lingering configs for the same VMID you are trying to restore to located under /var/lib/lxc/<VMID>?

To answer this, there is only a blank rootfs there that seems to be created each time it is restored (whether it fails or not).

But thank you guys so much, CLI tools for the win I guess, sorry for all the confusion and time used but I do very much appreciate the time and effort you guys put into helping, I may get a subscription when I get the money to afford it.

 

[fabian]

glad it got cleared up and I now know the next time someone repacks one of our tar archives, I'll ask for ./ vs '' specifically