[SOLVED] LXC Backup randomly hangs at suspend
- Thread starter pa657
- Start date
manu, I use local storage and I have this issue. So far, the issue has only cropped up on one of the 3 test lxc containers I'm using (backing up a 300 gb disk, very sparsely populated). I have destroyed and re-created this vm (a splunk server), but not configured it yet. I have not had backup issues since doing so.
I'm on a local storage aswell (thought i had mentioned that somewhere, sorry!)
Just an ext3 file system on a different hdd
Oh yeah, would also be good if it would handle the following error more gracefully (just restarting the backup makes it work normally)
Just an ext3 file system on a different hdd
Oh yeah, would also be good if it would handle the following error more gracefully (just restarting the backup makes it work normally)
Code:
INFO: Starting Backup of VM 102 (lxc)
INFO: status = running
INFO: found old vzdump snapshot (force removal)
Use of uninitialized value in string eq at /usr/share/perl5/PVE/LXC.pm line 1872.
ERROR: Backup of VM 102 failed - Can't delete snapshot: 102 vzdump zfs error: could not find any snapshots to destroy; check snapshot names.
Last edited:
Detailled output of tools like ps, iotop, atop and iftop (before, during and after a failed and successful freeze) as well as the tasks and cgroup.procs files in /sys/fs/cgroup/freezer/lxc/<CTID>/ might help with troubleshooting. So far, we have not been able to reproduce this issue at all.
@iMer please open a new thread or a bug report if you can reproduce that
We were not able to reproduce the problem here on a variety of file systems so we need more debugging information
so for people having the problem with their lxc backup not finising in snapshot/suspend mode, can you please post the following debugging information, preferably on pastebin or something similar and link it from here if you can
you need to install the package iotop on the host
1. while the backup process hangs ( not before, not after ), on the host
a) enter the command
ps faxo f,uid,pid,ppid,pri,ni,vsz,rss,wchan:20,stat,tty,time,comm
(don't grep the output we want everything !)
b) enter the command iotop --batch --only --quiet -qq, let it run five seconds, stop it with Ctrl-C and copy the ouput
c) from the host copy the last five lines of the vzdump command output
2. At any time check that the host kernel do not see system wide problem I/O problem with
zgrep 'blocked for more' /var/log/*
you can also add at the end of the document the content of
tasks and cgroup.procs files in /sys/fs/cgroup/freezer/lxc/<CTID>
as recommended by fabian
in the end you should have a document similar to this
https://gist.github.com/EmmanuelKasper/39fe1f64a5420858a266
We were not able to reproduce the problem here on a variety of file systems so we need more debugging information
so for people having the problem with their lxc backup not finising in snapshot/suspend mode, can you please post the following debugging information, preferably on pastebin or something similar and link it from here if you can
you need to install the package iotop on the host
1. while the backup process hangs ( not before, not after ), on the host
a) enter the command
ps faxo f,uid,pid,ppid,pri,ni,vsz,rss,wchan:20,stat,tty,time,comm
(don't grep the output we want everything !)
b) enter the command iotop --batch --only --quiet -qq, let it run five seconds, stop it with Ctrl-C and copy the ouput
c) from the host copy the last five lines of the vzdump command output
2. At any time check that the host kernel do not see system wide problem I/O problem with
zgrep 'blocked for more' /var/log/*
you can also add at the end of the document the content of
tasks and cgroup.procs files in /sys/fs/cgroup/freezer/lxc/<CTID>
as recommended by fabian
in the end you should have a document similar to this
https://gist.github.com/EmmanuelKasper/39fe1f64a5420858a266
Last edited:
I have the same problem as iMer.
Backup output:
strace -p <pid of lxc-freeze>
output of ps axl | awk '$10 ~ /D/'
When I kill lxc-freeze it moves on to the next container:
ERROR: Backup of VM 100 failed - command 'lxc-freeze -n 100' failed: got signal 15
However this leaves the container (100) in a very confused state, ssh processes freeze and won't connect. I have to force stop the container from the web ui.
Backup output:
strace -p <pid of lxc-freeze>
output of ps axl | awk '$10 ~ /D/'
When I kill lxc-freeze it moves on to the next container:
ERROR: Backup of VM 100 failed - command 'lxc-freeze -n 100' failed: got signal 15
However this leaves the container (100) in a very confused state, ssh processes freeze and won't connect. I have to force stop the container from the web ui.
I am having the same issue on a brand new Proxmox setup with containers. I am using suspend backup mode and the backing filesystem is local ext4. Next time it happens I will post a full gist. I need automated backups on my containers; one of them is running our VPN service so if that hangs indefinitely, it's pretty hard to remote manage.
I could migrate to ZFS to get snapshots but it sounds like even people using snapshots are having this problem. Do snapshot backups also suspend the container?
I could migrate to ZFS to get snapshots but it sounds like even people using snapshots are having this problem. Do snapshot backups also suspend the container?
They should not suspend the container, but they briefly (if everything goes right, not noticable) freeze the container to create the snapshot volume
. If stuff goes wrong.. it hangs indefinitely trying to freeze the container
They are no container suspend if snasphot is used
(you can check the code in /usr/share/perl5/PVE/VZDump/LXC.pm)
Sounds about right - this is waiting in FUSE code, which is potentially buggy because the kernel does not know about dependency chains here (it might freeze something which the FUSE call depends on, then the FUSE call does not return until until the dependency is unfrozen again, and lxc-freeze hangs). Can you describe what this ps process / cron job is doing? What are you using FUSE for in this container? If possible, can you try disabling the cron job and the FUSE mount and see if the issue persists?
Suspension is not the issue, freezing is. lxc-freeze is used for both suspend and snapshot backup (in order to get a consistent view of the file system that is backed up / snapshotted).
From what I gather it's spinning in a loop waiting for processes to finish
There's a isProcessStillRunning function.. which gets all pids via some ps/awk trickery and then searches for the pid in there - probably the best way that could feasably be solved *sigh*
an exists(/proc/$PID) call would be too difficult - it's even being checked if proc is available for other stuff in the code
I.. do not know I use FUSE? It's a standard debian container
I just looked up what that even is and it seems to be an abstraction to emulate file systems for containers? Is that correct?
What info do you need/where do I look it up?
I'll try patching the code so it doesnt constantly call ps - that should do the trick, correct?
From what I gather it's spinning in a loop waiting for processes to finish
There's a isProcessStillRunning function.. which gets all pids via some ps/awk trickery and then searches for the pid in there - probably the best way that could feasably be solved *sigh*
an exists(/proc/$PID) call would be too difficult - it's even being checked if proc is available for other stuff in the code
I.. do not know I use FUSE? It's a standard debian container
I just looked up what that even is and it seems to be an abstraction to emulate file systems for containers? Is that correct?
What info do you need/where do I look it up?
I'll try patching the code so it doesnt constantly call ps - that should do the trick, correct?
Could you provide the output of "mount" and "ls -la /dev" ? Is it possible to give us a copy of the php script you use (the one that calls ps)? If not, could you at least give us those snippets where "ps" is called? If you already have a non-productive copy of the container that exhibits the problem, you could also try wrapping the ps calls with strace and collect detailled information about what ps actually does when it messes with lxc-freeze..
While I can reproduce the issue with explicit FUSE mounts in a container (e.g., sshfs), I cannot reproduce it using a vanilla wheezy or jessie container using ps or accessing files in /proc/ ...
One more question: does the problem also occur if you setup a new debian wheezy container from scratch and only setup this cron job / php script?
Last edited:
https://gist.github.com/MatthaeusHarris/128e53aacd74d83c6d3a
The containers in question are fairly stock ubuntu 14.04 containers. One is empty and doing virtually nothing; the one that is frozen during the data collection in the gist was running a Splunk server.
The containers in question are fairly stock ubuntu 14.04 containers. One is empty and doing virtually nothing; the one that is frozen during the data collection in the gist was running a Splunk server.
Code:
# mount
local/subvol-102-disk-1 on / type zfs (rw,relatime,xattr,posixacl)
none on /dev type tmpfs (rw,relatime,size=100k,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,size=12k,mode=755)
tmpfs on /sys/fs/cgroup/cgmanager type tmpfs (rw,mode=755)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/blkio type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/cpu,cpuacct type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/cpuset type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/devices type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/freezer type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/hugetlb type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/memory type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/systemd type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/net_cls,net_prio type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/fs/cgroup/perf_event type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty1 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty2 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=6594316k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5138020k)
# ls -la /dev
total 9
drwxr-xr-x 3 root root 400 Feb 8 08:39 .
drwxr-xr-x 23 root root 23 Feb 8 08:39 ..
crw------- 1 root tty 136, 3 Feb 8 08:39 console
lrwxrwxrwx 1 root root 13 Feb 8 08:39 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Feb 8 08:39 full
srw-rw-rw- 1 root root 0 Feb 8 08:39 log
crw-rw-rw- 1 root root 1, 3 Feb 8 08:39 null
lrwxrwxrwx 1 root root 13 Feb 8 08:39 ptmx -> /dev/pts/ptmx
drwxr-xr-x 2 root root 0 Feb 8 08:39 pts
crw-rw-rw- 1 root root 1, 8 Feb 8 08:39 random
lrwxrwxrwx 1 root root 8 Feb 8 08:39 shm -> /run/shm
lrwxrwxrwx 1 root root 15 Feb 8 08:39 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Feb 8 08:39 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Feb 8 08:39 stdout -> /proc/self/fd/1
crw-rw-rw- 1 root root 5, 0 Feb 8 08:39 tty
crw-rw---- 1 root tty 136, 0 Feb 8 08:39 tty1
crw-rw---- 1 root tty 136, 1 Feb 8 08:39 tty2
crw-rw-rw- 1 root root 1, 9 Feb 8 08:39 urandom
prw-r----- 1 root adm 0 Feb 9 11:54 xconsole
crw-rw-rw- 1 root root 1, 5 Feb 8 08:39 zerohttps://github.com/piwik/piwik/blob/master/core/CliMulti/Process.php#L140 and https://github.com/piwik/piwik/blob/master/core/CliMulti/Process.php#L246 for the function itself
I'll try and set one up
Me neither, trying to freeze the container with the ps command running in a loop freezes it (takes a few more seconds than without it running, but thats it)
I'll give that a try aswell
I manage 3 Proxmox VE servers and have seen this problem at all of them. I have attached the desired files/output in lxc103_desired_files.zip. I have followed the instructions and given you the output/files (the tasks and cgroup.procs files were taken after I killed vzdump, and the rest with the failed vzdump running). I switched all 3 Proxmox VE servers to version 4 during the christmas vacation. And this problem have been troubling me ever since. About half of the virtual machines are LXC (converted from OpenVZ), and they are all located at local ZFS-partitions where we can take snapshots of the filesystem during backup.
@iMer, @Jacob: Thank you for your detailled output. It seems there is a chance that a process accessing certain files in /proc (which are provided by lxcfs) become unfreezable because the lxcfs process does not respond. We will report this upstream, but it would help if you could provide lxcfs related debugging output.
install the packages gdb lxcfs-dbg libc6-dbg on the host (# apt-get install gdb lxcfs-dbg libc6-dbg), and when an lxc-freeze call hangs, attach gdb to the lxcfs processes and collect backtraces.
A (shortened) example (replace 22928 with the PID of your lxcfs process and repeat for all of them):
Please attach the full gdb output and the full output of manu's ps command to this thread. Thanks in advance!
install the packages gdb lxcfs-dbg libc6-dbg on the host (# apt-get install gdb lxcfs-dbg libc6-dbg), and when an lxc-freeze call hangs, attach gdb to the lxcfs processes and collect backtraces.
A (shortened) example (replace 22928 with the PID of your lxcfs process and repeat for all of them):
Code:
# gdb
[gdb version output removed]
(gdb) attach 22928
Attaching to process 22928
[symbol reading output removed]
(gdb) bt
#0 __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1 0x00007f9552b816db in _L_lock_11305 () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f9552b7f838 in __GI___libc_realloc (oldmem=0x7f9552ea8620 <main_arena>, bytes=bytes@entry=567) at malloc.c:3025
#3 0x00007f9552b7524b in _IO_vasprintf (result_ptr=0x7f95520fc748, format=<optimized out>, args=args@entry=0x7f95520fc628) at vasprintf.c:84
#4 0x00007f9552b54027 in ___asprintf (string_ptr=string_ptr@entry=0x7f95520fc748, format=format@entry=0x7f9552c6a688 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n") at asprintf.c:35
#5 0x00007f9552b311c2 in __assert_fail_base (fmt=0x7f9552c6a688 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x7f9552c6d9a0 "({ __typeof (self->tid) __value; if (sizeof (__value) == 1) asm volatile (\"movb %%fs:%P2,%b0\" : \"=q\" (__value) : \"0\" (0), \"i\" (__builtin_offsetof (struct pthread, tid))); else if (sizeof (__value) == "..., file=file@entry=0x7f9552c6d968 "../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c", line=line@entry=141,
function=function@entry=0x7f9552c681b2 <__PRETTY_FUNCTION__.11206> "__libc_fork") at assert.c:57
#6 0x00007f9552b31312 in __GI___assert_fail (
assertion=0x7f9552c6d9a0 "({ __typeof (self->tid) __value; if (sizeof (__value) == 1) asm volatile (\"movb %%fs:%P2,%b0\" : \"=q\" (__value) : \"0\" (0), \"i\" (__builtin_offsetof (struct pthread, tid))); else if (sizeof (__value) == "..., file=0x7f9552c6d968 "../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c", line=141, function=0x7f9552c681b2 <__PRETTY_FUNCTION__.112
06> "__libc_fork") at assert.c:101
#7 0x00007f9552bbd385 in __libc_fork () at ../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c:141
#8 0x00007f9552ebf425 in __fork () at ../nptl/sysdeps/unix/sysv/linux/pt-fork.c:25
#9 0x0000000000404dd8 in write_task_init_pid_exit (sock=4, target=<optimized out>) at lxcfs.c:2530
#10 0x0000000000406fac in get_task_reaper_pid (task=<optimized out>) at lxcfs.c:2565
#11 get_reaper_busy (task=<optimized out>) at lxcfs.c:2581
#12 proc_uptime_read (fi=<optimized out>, offset=<optimized out>, size=<optimized out>, buf=<optimized out>) at lxcfs.c:2613
#13 proc_read (path=<optimized out>, fi=<optimized out>, offset=<optimized out>, size=<optimized out>, buf=<optimized out>) at lxcfs.c:2891
#14 lxcfs_read (path=0xfffffffffffffe00 <error: Cannot access memory at address 0xfffffffffffffe00>, buf=0x80 <error: Cannot access memory at address 0x80>, size=8191, offset=0, fi=0x7f953c002eb0) at lxcfs.c:2980
#15 0x00007f95530d8574 in fuse_fs_read_buf () from /lib/x86_64-linux-gnu/libfuse.so.2
#16 0x00007f95530d8732 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#17 0x00007f95530e108e in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#18 0x00007f95530e1895 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#19 0x00007f95530de394 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#20 0x00007f9552eb60a4 in start_thread (arg=0x7f95520fd700) at pthread_create.c:309
#21 0x00007f9552beb9cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111Please attach the full gdb output and the full output of manu's ps command to this thread. Thanks in advance!