Backup not working for lxc containers.

May 11, 2016
7
1
3
66
I moved 4 containers to a new server with the latest release of Proxmox 5.1 . They were on an older Proxmox 3 server . I backed them up to a nfs storage location I have been using to store backups . They were converted to LXC containers when I restored then to the new server. The new containers seem to be working fine . the problem is when I try to backup the containers , two of the container backup fine to the nfs storage . The other two seem to just hang up . A tmp dir is created on the nfs storage and theyt just seem to just hang during. I have tried backing up to local storage and the same thing happens.I have tried different options like suspend and stop . They just seem to stop during the backup. I have waited several hours and then stop them.
Not sure what to check . I have been using Proxmox 3 and backing up to the NFS storage for several years with no problems.
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
453
88
Many things have changed since PVE 3.x, also NFS has gone through some changes. Eg. the LXC container might have a lot of files, the NFS server might be too slow. Try to set the tempdir to a local storage.

man vzdump
Code:
       Example vzdump.conf Configuration.

           tmpdir: /mnt/fast_local_disk
           storage: my_backup_storage
           mode: snapshot
           bwlimit: 10000
 
May 11, 2016
7
1
3
66
I tried creating a local tmp dir and changing vzdump.conf. I also added a new storage on a usb drive.

My CPU usage for the node is under 1 percent of 12 CPU's and ram is 5 g out of 64 g. My Disks have 4 Terabytes available.

It still keep hanging at :

INFO: create storage snapshot 'vzdump'

The two containers that backup ok are 10G each.
The two that do not are 40G and 100G .
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
453
88
The two that do not are 40G and 100G .
This might take some time, depending on the storage speed.

manpage vzdump
snapshot mode
This mode uses the snapshotting facilities of the underlying storage. First, the container
will be suspended to ensure data consistency. A temporary snapshot of the container’s
volumes will be made and the snapshot content will be archived in a tar file. Finally, the
temporary snapshot is deleted again.
 
May 11, 2016
7
1
3
66
I created a new 350G container and it does a snapshot backup to the NFS storage in about 50 minutes.

The restored container seems to hang starting the snapshot backup. Is there any log or status I could check if it is running? I let it run for 24 hours and nothing iseems to be happening..

When I restored the container is converted to a LXC container ,Could something in the container be preventing the backup?
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,912
1,523
164
I created a new 350G container and it does a snapshot backup to the NFS storage in about 50 minutes.

The restored container seems to hang starting the snapshot backup. Is there any log or status I could check if it is running? I let it run for 24 hours and nothing iseems to be happening..

When I restored the container is converted to a LXC container ,Could something in the container be preventing the backup?

yes. e.g., a mounted fuse file system or similar.
 

kobuki

Well-Known Member
Dec 30, 2008
470
26
48
Sorry for resurrecting an old thread, but I've just run into the same problem as OP on a new 5.1 upgrade (from 3.4). All containers are running fine, but the snapshot backup just freezes at the line "INFO: create storage snapshot 'vzdump'".

So you say it's not possible to backup containers with vzdump when there's an active fuse mount? The CT in particular has 2 fuse bindfs mounts.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,912
1,523
164
Sorry for resurrecting an old thread, but I've just run into the same problem as OP on a new 5.1 upgrade (from 3.4). All containers are running fine, but the snapshot backup just freezes at the line "INFO: create storage snapshot 'vzdump'".

So you say it's not possible to backup containers with vzdump when there's an active fuse mount? The CT in particular has 2 fuse bindfs mounts.

yes (or at least, yes, it is not possible to back it up reliably).
 

kobuki

Well-Known Member
Dec 30, 2008
470
26
48
yes (or at least, yes, it is not possible to back it up reliably).

OK, I understand, but is there any info on what blocks the process and why, when a fuse mount is present? Considering the confidence in your statement you must have investigated it more thoroughly.
 

kobuki

Well-Known Member
Dec 30, 2008
470
26
48
Here's an idea. Would it be possible to add an option to skip lxc freeze at snapshot? When using a sufficient snapshot mechanism like in zfs or lvm-thin, it's enough that a consistent snapshot is made and that gets backed up. Manually making the snapshot and backing up works without problems, and restoring these archives are always fine and working. For the few problematic containers this would be an easy and adequate solution.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,912
1,523
164
OK, I understand, but is there any info on what blocks the process and why, when a fuse mount is present? Considering the confidence in your statement you must have investigated it more thoroughly.

IIRC it was something along the lines of racy deadlocking because the user space part gets frozen, but the kernel part does not - but it's been a while...

Here's an idea. Would it be possible to add an option to skip lxc freeze at snapshot? When using a sufficient snapshot mechanism like in zfs or lvm-thin, it's enough that a consistent snapshot is made and that gets backed up. Manually making the snapshot and backing up works without problems, and restoring these archives are always fine and working. For the few problematic containers this would be an easy and adequate solution.

freezing to stop I/O is the safe and sane choice. your snapshots are not consistent otherwise if you have more than one volume/mountpoint for example. the backup mode without freezing is called 'stop' - it also ensures consistency ;)
 

kobuki

Well-Known Member
Dec 30, 2008
470
26
48
freezing to stop I/O is the safe and sane choice. your snapshots are not consistent otherwise if you have more than one volume/mountpoint for example. the backup mode without freezing is called 'stop' - it also ensures consistency ;)

I'm aware of that but the vast majority of CTs are single rooted, with no additional volumes attached. Stop is not a viable method for understandable reasons. So while I understand you strive to provide a tool that's always consistent but in some (admittedly rare) situations it simply does not work at all. This way we're forced to go out of the loop and use other methods besides vzdump. I'm planning to do so anyway (based on zfs send/recv) but not everyone is able or willing to.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,912
1,523
164
I'm aware of that but the vast majority of CTs are single rooted, with no additional volumes attached. Stop is not a viable method for understandable reasons. So while I understand you strive to provide a tool that's always consistent but in some (admittedly rare) situations it simply does not work at all. This way we're forced to go out of the loop and use other methods besides vzdump. I'm planning to do so anyway (based on zfs send/recv) but not everyone is able or willing to.

the vast majority of containers don't use FUSE inside. it's explicitly stated that it is strongly advised against due to the problems it causes (https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pct_container_storage), and there are already two existing workarounds (stop mode backup or bind-mounting a FUSE mount established on the host).

freezing also serves a purpose for single-volume containers, because the whole sequence is:
  1. freeze container (to stop I/O)
  2. sync all mountpoints from within the container's mount namespace (to increase consistency)
  3. snapshot all volumes
  4. unfreeze container
such a snapshot has a lot more consistency beyond what ZFS/LVMThin give use with a plain snapshot.
 

kobuki

Well-Known Member
Dec 30, 2008
470
26
48
I know all that, but... You can't really stop people from using their stuff by stating it's not well supported when they have been using it before for long. When we need it, we use it. And we want to back up those CTs as well. If it's not vzdump, then it's something else. It's just less streamlined, patching together with other solutions.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,912
1,523
164
I know all that, but... You can't really stop people from using their stuff by stating it's not well supported when they have been using it before for long. When we need it, we use it.

Nobody is stopping from "using it" - but you have to deal with the fallout if you run an explicitly unsupported setup.

And we want to back up those CTs as well. If it's not vzdump, then it's something else. It's just less streamlined, patching together with other solutions.

You are free to backup your containers however you want.
 

kobuki

Well-Known Member
Dec 30, 2008
470
26
48
You are free to backup your containers however you want.

No sh*t :) Anyway don't get me wrong, vzdump is a great tool, we use it all the time without issues, but as everything else, it has its limitations. Also you're very active here, answering quickly, I appreciate and thank you for that, don't take my dissecting of this single issue an insult against anyone or anything, please. I repeat, it's a great to have utility. Without it our live would be a lot harder, no irony involved.
 

Dorin

Member
Sep 11, 2017
33
2
13
32
I had a CT container until i made a backup and tried to restore it after a while.
The container has disappeared during the restore process after the process has ended up with a fail message.
How can i search for the backup file and restore the container (id 200) from cli?
upload_2018-10-23_0-19-39.png
 

Alwin

Proxmox Retired Staff
Retired Staff
Aug 1, 2017
4,617
453
88
How can i search for the backup file and restore the container (id 200) from cli?
Click on one of you storage entries, there you will find the backup file.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!