Hi! Back again soon, I'm afraid.
The problem still exists, but this time I have more information.
We've upgraded to Pacific in the mean time, so the problem was not related to Octopus.
In the most recent occurrence I made checksums of the images and of the snapshots in between the actions that...
I've attached a screenshot of the related tasks. Backup was made Jan 24 22:00:04 - Jan 24 22:02:18.
I can't trigger the issue by running all the actions straight after one another. I'll have to work with my colleagues to ensure we can make checksums after each action when they do their thing...
The checksums of the first MB are identical, the checksums of the entire image are not. But that's probably due to the fact that I've tried to start the VM in the meantime. If I revert the snapshot, and don't start the VM, then the checksums of the whole image are identical.
What seems to be...
Un(?)fortunately we hit the same problem again today. This is the same set of VMs as I reported about last time. Again, the actions that were done on the VMs were the same, i.e.:
(Yesterday)
- VM stopped
- Snapshot created
- VM started
- VM stopped
- Snapshot restored
- VM started (OK)
-...
I did search through the ceph bug tracker but couldn't find anything that looked like our problem. An upgrade to Pacific has already been planned, but at the same time I'm reluctant to do upgrades in case this problem is a symptom of a problem with our cluster somehow.
Well, thanks for your time anyway! Next time we'll preserve the faulty state. With the rate of this occurring, I'll report back in a year or two. :)
Thanks again and best regards, Roel
Hi Fabian, unfortunately the original disks are gone, but I do still have a PBS backup of the disk that was made after the issue occurred and before the restore was done. Is that of any use? I reckon we should have kept the original images so we could trace into its rbd/ceph properties.
Best...
Hi Fabian, thanks for your time!
Here's in short what happened with this VM:
Jan 11 14:01:37 snapshot created
Jan 11 16:11:02 first rollback -> state okay
Jan 11 22:00:04 vzdump backup -> backup okay
Jan 12 11:39:00 2nd rollback -> disk image corrupt, doesn't boot
Jan 12 11:52:42 3rd rollback...
Hi!
I'm sorry to say that a few days back this problem happened again. The actions that have preceded the problem were similar to last time, e.g. making a snapshot of a VM and doing a rollback multiple times. At first the rollback resulted in a working system, but a later rollback resulted in a...
Great, thanks for clarifying! Upgrades to 7.x are already planned, but glad to know that we don't have to go to Pacific immediately after!
Best regards, Roel
Thanks for the new release!
I see that Ceph Octopus is no longer officially supported in this version. Since the upgrade guide for 6to7 mentions that you first have to update to Proxmox 7 and then to Ceph Pacific, I assume that Octopus still works, right? Is it necessary to update to Pacific...
Hi Dominik,
Thanks for your answer!
We're absolutely planning to go to 7.x, but we'll have to go to Octopus first before we can do that.
Would it be possible to disclose the reason why 15.2.16 has not yet moved to main, for buster? That might help us decide what choice would be better for us...
Hello!
Last year while upgrading Ceph from Nautilus to Octopus we were bitten by two Ceph bugs: https://tracker.ceph.com/issues/51619 and https://tracker.ceph.com/issues/51682. While both bugs have been fixed, one is backported to 15.2.14, but the other is backported to 15.2.16. We have one...
After initially scaring me nearly to death, it seems it's not that bad. If I read this post correctly https://forum.proxmox.com/threads/pbs-server-full-two-days-later-almost-empty.83274/post-366927 then @dcsapak says that GC updates the atime explicitly (e.g. it doesn't rely on a read access to...
Hi!
In the past days, we've had three separate VMs where the disk of the VM was corrupt after reverting to a snapshot. On one of these, we had reverted to this snapshot before, succesfully. All three VM's are on Ceph storage. The settings are mixed - two Windows VMs with a virtio driver, one...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.