Container storages gone?

[StefanC]

Hi. I just experienced something really weird and I hope someone here may be able to shed some light on this.

I run proxmox 4.4-5 on a zfs mirror and use that as boot disk and primary vm storage. I noticed just now that a container would not come up after a reboot attempt. The log shows: Error: command ‘lxc-start -n 105’ failed: exit code 1. So I attempted to start the container manually in foreground mode with "lxc-start -n 105 -F -lDEBUG". The log shows “cannot open directory //rpool/data/subvol-105-disk-1: No such file or directory”. Sure enough. “ls /rpool/data/“ shows that all but one disk (the one I created yesterday) is gone??? The pool seems to be just fine according to a "zpool status".

I just tried and can restore the storage from backup just fine. What logs can hold some info on what happened that erased the disks?

 

[StefanC]

I think I'm about to go crazy. I restored four of seven containers from backup last night. The backups were only half a day old so I didn't loose much. I had installed the Ubuntu Landscape client on all the Ubuntu containers since the last backup so the restored machines showed up as missing in the Landscape server just as expected.

I sat down this morning with the intention to restore the remaining three containers and redo the Landscape config. I tried to start one of the remaining containers just for fun before I restored it and to my surprise it came right up.

I listed the content of //rpool/data/ and now all subvols are back!? The ones I restored from backup yesterday evening are now in the state they should have been prior to the backup! It is as if nothing happened in the first place and Landscape lists all machines as online. The system is exactly as it should be. No data lost.

I'm stumped. The only thing I can come to think of is that the last backup job failed somehow and that it triggered some weird zfs snapshot behaviour. I didn't receive any successful backup notification email on Sunday morning but all backups were available and worked fine yesterday when I restored four containers.

Do you guys have any idea about what happened here? Has anybody seen something similar before?

 

[viruslab]

I've just experienced a similar issue.
I had a perfectly normal working container with NFS and SMB share + mounted folders from proxmox host.
I shutdown the container.
Got the error when tried to boot it again.
When I looked at the ZFS storage there was no this container storage. It just disappeared!