Okay, I just tried to replace a QDevice and got the following when running "pvecm qdevice remove":
Host key verification failed.
error during cfs-locked 'file-corosync_conf' operation: command 'ssh -o 'BatchMode=yes' -lroot <ip-address> -- rm -rf /etc/corosync/qdevice' failed: exit code 255...