Sheepdog rollback deletes disk

[athompso]

We've just encountered a situation where we took a snapshot of a sheepdog-based VDI, then later rolled it back - but on trying to roll back, failed, and not only didn't roll back the disk, but deleted the base disk, too.

Can anyone else report on their success or failure using snapshots with Sheepdog?

There's an interesting commit message in another project (iSCSI tgt) that seems to imply that this is a known problem with Sheepdog: https://github.com/fujita/tgt/commit/07ab5ae690b977aa3b31482812d39d298535d4c1 ...

Thanks,
-Adam

 

[wolfgang]

Hi,

I can't reproduce it here.
Can you give the exact procedure to reproduce it?

 

[athompso]

As far as we can determine:

• We took a live snapshot of a powered-on VM running on Sheepdog storage (not including memory)
• A little while later, powered off the VM and attempted to roll back.
• Rollback failed complaining that the disk already existed
• Looks like we attempted to rollback more than once: first error was "TASK ERROR: sheepdog snapshot vm-228-disk-1' error: Failed to create VDI vm-228-disk-1: VDI exists already".
• Second attempt produced: "TASK ERROR: VM is locked (rollback)"
• Presumably my colleague did a "qm unlock" at that point
• Third attempt produced: "TASK ERROR: sheepdog snapshot vm-228-disk-1' error: Failed to create VDI vm-228-disk-1: Failed to read from base VDI"
• When I got involved after that, the base disk had simply vanished.

 

[wolfgang]

I try several scenarios but I can't reproduce it.

I have a look of this post and see the corresponding commit in sheepdog,
but it affect only iscsi multipath with tgtd.

Do you use multipath iscsi with this VM?

 

[athompso]

I try several scenarios but I can't reproduce it.

I have a look of this post and see the corresponding commit in sheepdog,
but it affect only iscsi multipath with tgtd.

Do you use multipath iscsi with this VM?

No, it just had the sound of a potential root-cause, but if you're sure the sheepdog behaviour described therein only affects iSCSI, then it's probably not relevant here.

Also, additional testing reveals that sheepdog is the problem, not PVE: even doing "dog rollback" from the CLI unexpectedly deletes the disk after complaining "Failed to create VDI vm-228-disk-1: failed to read from base VDI".
*sigh*
Time to rebuild the cluster. Again.

 

[wolfgang]

Can you post me you exact setup like
pveversion --v
Do you use a non mainstream hardware like 4k HDD?
How many nodes do you have?

 

[tbowerman]

Can you post me you exact setup like
pveversion --v
Do you use a non mainstream hardware like 4k HDD?
How many nodes do you have?

Hi Wolfgang,

Not OP, but I am the colleague mentioned above. Here is the information you requested:

proxmox-ve: 4.4-77 (running kernel: 4.4.35-1-pve)
pve-manager: 4.4-5 (running version: 4.4-5/c43015a5)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-102
pve-firmware: 1.1-10
libpve-common-perl: 4.0-85
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-71
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-1
pve-qemu-kvm: 2.7.0-10
pve-container: 1.0-90
pve-firewall: 2.0-33
pve-ha-manager: 1.0-38
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.6-5
lxcfs: 2.0.5-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
pve-sheepdog: 1.0.0-1
ceph: 10.2.5-1~bpo80+1

No we don't use 4k HDD's, and the PVE node in question is part of a 4 node cluster.