Issues with Ceph snapshots in PVE 4.3-66?

[acidrop]

Hello,

I have the following situation after the last upgrade to proxmox-ve 4.3-66.

proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.3-3 (running version: 4.3-3/557191d3)
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-91
pve-firmware: 1.1-9
libpve-common-perl: 4.0-75
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-66
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.2-2
pve-container: 1.0-78
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80
openvswitch-switch: 2.5.0-1
ceph: 0.94.9-1~bpo80+1

I have a 3 node pve|ceph cluster. When creating a snapshot including RAM on any vm located at Ceph storage I get:

Task viewer: VM 100 - Snapshot
Output
Status
Stop
snapshot create failed: starting cleanup
Removing all snapshots: 100% complete...done.
Removing image: 1% complete...
Removing image: 2% complete...
Removing image: 3% complete...
Removing image: 4% complete...
Removing image: 5% complete...
Removing image: 6% complete...
Removing image: 7% complete...
Removing image: 8% complete...
Removing image: 9% complete...
Removing image: 10% complete...
Removing image: 11% complete...
Removing image: 12% complete...
Removing image: 13% complete...
Removing image: 14% complete...
Removing image: 15% complete...
Removing image: 16% complete...
Removing image: 17% complete...
Removing image: 18% complete...
Removing image: 19% complete...
Removing image: 20% complete...
Removing image: 21% complete...
Removing image: 22% complete...
Removing image: 23% complete...
Removing image: 24% complete...
Removing image: 25% complete...
Removing image: 26% complete...
Removing image: 27% complete...
Removing image: 28% complete...
Removing image: 29% complete...
Removing image: 30% complete...
Removing image: 31% complete...
Removing image: 32% complete...
Removing image: 33% complete...
Removing image: 34% complete...
Removing image: 35% complete...
Removing image: 36% complete...
Removing image: 37% complete...
Removing image: 38% complete...
Removing image: 39% complete...
Removing image: 40% complete...
Removing image: 41% complete...
Removing image: 42% complete...
Removing image: 43% complete...
Removing image: 44% complete...
Removing image: 45% complete...
Removing image: 46% complete...
Removing image: 47% complete...
Removing image: 48% complete...
Removing image: 49% complete...
Removing image: 50% complete...
Removing image: 51% complete...
Removing image: 52% complete...
Removing image: 53% complete...
Removing image: 54% complete...
Removing image: 55% complete...
Removing image: 56% complete...
Removing image: 57% complete...
Removing image: 58% complete...
Removing image: 59% complete...
Removing image: 60% complete...
Removing image: 61% complete...
Removing image: 62% complete...
Removing image: 63% complete...
Removing image: 64% complete...
Removing image: 65% complete...
Removing image: 66% complete...
Removing image: 67% complete...
Removing image: 68% complete...
Removing image: 69% complete...
Removing image: 70% complete...
Removing image: 71% complete...
Removing image: 72% complete...
Removing image: 73% complete...
Removing image: 74% complete...
Removing image: 75% complete...
Removing image: 76% complete...
Removing image: 77% complete...
Removing image: 78% complete...
Removing image: 79% complete...
Removing image: 80% complete...
Removing image: 81% complete...
Removing image: 82% complete...
Removing image: 83% complete...
Removing image: 84% complete...
Removing image: 85% complete...
Removing image: 86% complete...
Removing image: 87% complete...
Removing image: 88% complete...
Removing image: 89% complete...
Removing image: 90% complete...
Removing image: 91% complete...
Removing image: 92% complete...
Removing image: 93% complete...
Removing image: 94% complete...
Removing image: 95% complete...
Removing image: 96% complete...
Removing image: 97% complete...
Removing image: 98% complete...
Removing image: 99% complete...
Removing image: 100% complete...done.
TASK ERROR: VM 100 qmp command 'savevm-start' failed - failed to open '/dev/rbd/rbdpool1/vm-100-state-test'

- When creating a snapshot without RAM included, I get:

Task viewer: VM 100 - Snapshot
Output
Status
Stop
TASK OK

When trying to delete the same snapshot, I get:

VM 100 qmp command 'delete-drive-snapshot' failed - this feature or command is not currently supported

I can delete the same snapshot though via 'rbd snap rm rbdpool1/vm-100-disk-1@test' .
Same situation with any Ceph stored virtual machines.
I can create and delete any snapshots on local zfs/glusterfs located vms without issues.

Thank you

 

[spirit]

Hi,
I just have tested here with lat proxmox packages, I don't have problem.

Do you have stop/start the vms since upgrade ? (or live migrate them? ).

 

[acidrop]

I did some further tests:

- Restarted all nodes
- Tried to snapshot when VM is powered off --> Result: success (both in creating and destroying snapshot)
- Tried to snapshot (including RAM) when VM is powered on --> Result: fail (VM 108 qmp command 'savevm-start' failed - failed to open '/dev/rbd/rbdpool1/vm-108-state-test')
- Tried to snapshot (excluding RAM) when VM is powered on --> Result: success when creating snapshot and failed when destroying it. I have to destroy it manually via rbd command and after remove the snapshot entry from vm config file.
- Tried to snapshot a LXC container (stored on Ceph) both in poweron/off --> Result: success

 

[fabian]

are you using KRBD? if so, disable it for the storage used by VMs. Qemu has built-in librbd support, which is superior to the KRBD interface that we have to use for LXC containers. you should setup two storages with separate pools, one for Qemu (no KRBD), one for LXC (with KRBD)

 

[acidrop]

Thanks for the info, yes actually I have a 'shared' rbdpool and KRBD enabled in order to run LXCs.
I will disable KRBD for the existing pool and I will create another pool with KRBD enabled for LXCs.

 

[spirit]

Thanks for the info, yes actually I have a 'shared' rbdpool and KRBD enabled in order to run LXCs.
I will disable KRBD for the existing pool and I will create another pool with KRBD enabled for LXCs.

Note that you can have 1 ceph pool, and define 2 storage on proxmox with this same pool. (1 without krbd for qemu, 1 with krbd for lxc)

 

[acidrop]

Thanks for the clarification. Any ideas how to accomplish this? Can't find it in wiki..

My current Ceph config is:

rbd: cephstor1
    monhost 192.168.149.115;192.168.149.95;192.168.148.65
    pool rbdpool1
    username admin
    content images,rootdir

Tried to add a second entry:

rbd: cephstorlxc2
    monhost 192.168.149.115;192.168.149.95;192.168.148.65
    pool rbdpool1
    username admin
    krbd
    content images,rootdir

...but it doesn't seem to like it.

 

[fabian]

Thanks for the clarification. Any ideas how to accomplish this? Can't find it in wiki..

My current Ceph config is:

rbd: cephstor1
    monhost 192.168.149.115;192.168.149.95;192.168.148.65
    pool rbdpool1
    username admin
    content images,rootdir

Tried to add a second entry:

rbd: cephstorlxc2
    monhost 192.168.149.115;192.168.149.95;192.168.148.65
    pool rbdpool1
    username admin
    krbd
    content images,rootdir

...but it doesn't seem to like it.

you should set the content appropriately (i.e., "images" for the first one and "rootdir" for the second one), but that's probably not your issue. did you put the keyring into /etc/pve/priv/ceph using the correct file name?

 

[acidrop]

- I changed the content to reflect to lxc and images accordingly.
- I copied keyring to the correct file name.

Everything is working perfectly fine now, many thanks to all!
Your help is much appreciated.