Backup not working for lxc containers.

[rickv55]

I moved 4 containers to a new server with the latest release of Proxmox 5.1 . They were on an older Proxmox 3 server . I backed them up to a nfs storage location I have been using to store backups . They were converted to LXC containers when I restored then to the new server. The new containers seem to be working fine . the problem is when I try to backup the containers , two of the container backup fine to the nfs storage . The other two seem to just hang up . A tmp dir is created on the nfs storage and theyt just seem to just hang during. I have tried backing up to local storage and the same thing happens.I have tried different options like suspend and stop . They just seem to stop during the backup. I have waited several hours and then stop them.
Not sure what to check . I have been using Proxmox 3 and backing up to the NFS storage for several years with no problems.

 

[Alwin]

Many things have changed since PVE 3.x, also NFS has gone through some changes. Eg. the LXC container might have a lot of files, the NFS server might be too slow. Try to set the tempdir to a local storage.

man vzdump

       Example vzdump.conf Configuration.

           tmpdir: /mnt/fast_local_disk
           storage: my_backup_storage
           mode: snapshot
           bwlimit: 10000

 

[rickv55]

I tried creating a local tmp dir and changing vzdump.conf. I also added a new storage on a usb drive.

My CPU usage for the node is under 1 percent of 12 CPU's and ram is 5 g out of 64 g. My Disks have 4 Terabytes available.

It still keep hanging at :

INFO: create storage snapshot 'vzdump'

The two containers that backup ok are 10G each.
The two that do not are 40G and 100G .

 

[Alwin]

The two that do not are 40G and 100G .

This might take some time, depending on the storage speed.

manpage vzdump

snapshot mode
This mode uses the snapshotting facilities of the underlying storage. First, the container
will be suspended to ensure data consistency. A temporary snapshot of the container’s
volumes will be made and the snapshot content will be archived in a tar file. Finally, the
temporary snapshot is deleted again.

 

[rickv55]

I created a new 350G container and it does a snapshot backup to the NFS storage in about 50 minutes.

The restored container seems to hang starting the snapshot backup. Is there any log or status I could check if it is running? I let it run for 24 hours and nothing iseems to be happening..

When I restored the container is converted to a LXC container ,Could something in the container be preventing the backup?

 

[fabian]

I created a new 350G container and it does a snapshot backup to the NFS storage in about 50 minutes.

The restored container seems to hang starting the snapshot backup. Is there any log or status I could check if it is running? I let it run for 24 hours and nothing iseems to be happening..

When I restored the container is converted to a LXC container ,Could something in the container be preventing the backup?

yes. e.g., a mounted fuse file system or similar.

 

[kobuki]

Sorry for resurrecting an old thread, but I've just run into the same problem as OP on a new 5.1 upgrade (from 3.4). All containers are running fine, but the snapshot backup just freezes at the line "INFO: create storage snapshot 'vzdump'".

So you say it's not possible to backup containers with vzdump when there's an active fuse mount? The CT in particular has 2 fuse bindfs mounts.

 

[fabian]

Sorry for resurrecting an old thread, but I've just run into the same problem as OP on a new 5.1 upgrade (from 3.4). All containers are running fine, but the snapshot backup just freezes at the line "INFO: create storage snapshot 'vzdump'".

So you say it's not possible to backup containers with vzdump when there's an active fuse mount? The CT in particular has 2 fuse bindfs mounts.

yes (or at least, yes, it is not possible to back it up reliably).

 

[rickv55]

I never was able to resolve my issues. My problem is the same as yours.

 

[kobuki]

yes (or at least, yes, it is not possible to back it up reliably).

OK, I understand, but is there any info on what blocks the process and why, when a fuse mount is present? Considering the confidence in your statement you must have investigated it more thoroughly.

 

[kobuki]

Here's an idea. Would it be possible to add an option to skip lxc freeze at snapshot? When using a sufficient snapshot mechanism like in zfs or lvm-thin, it's enough that a consistent snapshot is made and that gets backed up. Manually making the snapshot and backing up works without problems, and restoring these archives are always fine and working. For the few problematic containers this would be an easy and adequate solution.

 

[fabian]

OK, I understand, but is there any info on what blocks the process and why, when a fuse mount is present? Considering the confidence in your statement you must have investigated it more thoroughly.

IIRC it was something along the lines of racy deadlocking because the user space part gets frozen, but the kernel part does not - but it's been a while...

Here's an idea. Would it be possible to add an option to skip lxc freeze at snapshot? When using a sufficient snapshot mechanism like in zfs or lvm-thin, it's enough that a consistent snapshot is made and that gets backed up. Manually making the snapshot and backing up works without problems, and restoring these archives are always fine and working. For the few problematic containers this would be an easy and adequate solution.

freezing to stop I/O is the safe and sane choice. your snapshots are not consistent otherwise if you have more than one volume/mountpoint for example. the backup mode without freezing is called 'stop' - it also ensures consistency

 

[kobuki]

freezing to stop I/O is the safe and sane choice. your snapshots are not consistent otherwise if you have more than one volume/mountpoint for example. the backup mode without freezing is called 'stop' - it also ensures consistency

I'm aware of that but the vast majority of CTs are single rooted, with no additional volumes attached. Stop is not a viable method for understandable reasons. So while I understand you strive to provide a tool that's always consistent but in some (admittedly rare) situations it simply does not work at all. This way we're forced to go out of the loop and use other methods besides vzdump. I'm planning to do so anyway (based on zfs send/recv) but not everyone is able or willing to.

 

[fabian]

I'm aware of that but the vast majority of CTs are single rooted, with no additional volumes attached. Stop is not a viable method for understandable reasons. So while I understand you strive to provide a tool that's always consistent but in some (admittedly rare) situations it simply does not work at all. This way we're forced to go out of the loop and use other methods besides vzdump. I'm planning to do so anyway (based on zfs send/recv) but not everyone is able or willing to.

the vast majority of containers don't use FUSE inside. it's explicitly stated that it is strongly advised against due to the problems it causes (https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pct_container_storage), and there are already two existing workarounds (stop mode backup or bind-mounting a FUSE mount established on the host).

freezing also serves a purpose for single-volume containers, because the whole sequence is:

1. freeze container (to stop I/O)
2. sync all mountpoints from within the container's mount namespace (to increase consistency)
3. snapshot all volumes
4. unfreeze container

such a snapshot has a lot more consistency beyond what ZFS/LVMThin give use with a plain snapshot.

 

[kobuki]

I know all that, but... You can't really stop people from using their stuff by stating it's not well supported when they have been using it before for long. When we need it, we use it. And we want to back up those CTs as well. If it's not vzdump, then it's something else. It's just less streamlined, patching together with other solutions.

 

[fabian]

I know all that, but... You can't really stop people from using their stuff by stating it's not well supported when they have been using it before for long. When we need it, we use it.

Nobody is stopping from "using it" - but you have to deal with the fallout if you run an explicitly unsupported setup.

And we want to back up those CTs as well. If it's not vzdump, then it's something else. It's just less streamlined, patching together with other solutions.

You are free to backup your containers however you want.

 

[kobuki]

You are free to backup your containers however you want.

No sh*t Anyway don't get me wrong, vzdump is a great tool, we use it all the time without issues, but as everything else, it has its limitations. Also you're very active here, answering quickly, I appreciate and thank you for that, don't take my dissecting of this single issue an insult against anyone or anything, please. I repeat, it's a great to have utility. Without it our live would be a lot harder, no irony involved.

 

[Dorin]

I had a CT container until i made a backup and tried to restore it after a while.
The container has disappeared during the restore process after the process has ended up with a fail message.
How can i search for the backup file and restore the container (id 200) from cli?

 

[Alwin]

How can i search for the backup file and restore the container (id 200) from cli?

Click on one of you storage entries, there you will find the backup file.

 

[Dorin]

Click on one of you storage entries, there you will find the backup file.

Thank you Alwin. I also found useful the following thread: https://forum.proxmox.com/threads/e...-lxc-container-backup-with-pct-restore.34241/