Bug/Feature Request - Multipath scsi devices cause snapshots to fail due to existing disk

[jw6677]

Snapshots are failing in the use case where a disk is added to a VM multiple times to increase throughput/parallelism via iothread and multipathd:

[...]
completed saving the VM state in 6m 58s, saved 72.54 GiB
snapshotting 'drive-scsi0' (fast:vm-114-disk-9)
snapshotting 'drive-scsi1' (fast:vm-114-disk-1)
snapshotting 'drive-scsi2' (fast:vm-114-disk-1)
snapshot create failed: starting cleanup
[...]
Removing snap: 100% complete...done.
TASK ERROR: rbd snapshot 'vm-114-disk-1' error: rbd: failed to create snapshot: (17) File exists

Here is a relevant portion of the config

/etc/pve/qemu-server/826.conf:
[...]
scsi0: fast:vm-114-disk-9,backup=1,cache=writeback,discard=on,iothread=1,size=2176G,ssd=1,serial=npmp709009e2,wwn=0x5002538e709007e2
scsi1: fast:vm-114-disk-1,backup=1,cache=writeback,discard=on,iothread=1,serial=npmp709009c9,size=2000G,ssd=1,wwn=0x5002538e709007c9
scsi2: fast:vm-114-disk-1,backup=0,cache=writeback,discard=on,iothread=1,serial=npmp709009c9,size=2000G,ssd=1,wwn=0x5002538e709007c9
scsi3: fast:vm-114-disk-1,backup=0,cache=writeback,discard=on,iothread=1,serial=npmp709009c9,size=2000G,ssd=1,wwn=0x5002538e709007c9
[...]

Backups work well in this case, because the extra disk lines are set to backup=0, however, snapshotting fails as shown above.

This setup, leveraging iothread=1, setting a serial and wwn to allow multipathd to group the disks, and ensuring iothread=1 helped to increase performance on our ceph RBD setup, I believe due to the increased parallelism.

I am hoping:
1) The snapshot process can be given a similar flag as backup=0
2) The snapshot process can check for an existing snapshot and proceed anyways (ignore existing)
3) Someone can help me understand if there is another way to snapshot a VM using this technique

 

[fabian]

yeah, this is not supported. PVE assumes a volume is only used once by a single guest. similar issues also arise when cloning, moving one of those disks, live-migration if the storage is local, ... . I am also not sure whether your setup is not broken in other ways if you are using librbd as opposed to krbd - cache=writeback enables the librbd cache, which has the following caveat:

The cache is in memory on the client, and each RBD image has its own. Since the cache is local to the client, there’s no coherency if there are others accessing the image. Running GFS or OCFS on top of RBD will not work with caching enabled.

not sure whether your VM counts as multiple clients (one per virtual drive) or single (combined client/cache for multiple instances of the same rbd image), although that is probably possible to find out.