Help for drbd9 on PVE6: VMs doesn't start

[dendi]

Dear Proxmox staff and users,
I know that drbd9 is not supported by proxmox but I tried drbd9 with linstor on PVE6 and all works except a "stupid" (?) thing: VMs can't start.
As I understand, there is no need to promote a drbd resource to primary. I mount the disk from OS and works.
At the other side, if I manually promote with "drbdadm primary vm-disk-" and I start the VM, it works.
So, what changed from PVE5 to PVE6?
The error is this:

kvm: -drive file=/dev/drbd/by-res/vm-100-disk-1/0,if=none,id=drive-scsi0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on: The device is not writable: Permission denied
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name prononva -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=6996412f-0536-485e-87f2-f37904b68301' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,password -cpu qemu64 -m 512 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'vmgenid,guid=4066aee8-8a9b-4c86-9326-e040c452d0d5' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/100.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:5e763b83f1b' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/drbd/by-res/vm-100-disk-1/0,if=none,id=drive-scsi0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=F2:94:D5:E8:1A:32,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'accel=tcg,type=pc'' failed: exit code 1

I can format, fdisk and mount from host /dev/drbd/by-res/vm-100-disk-1/0 without manual commands

Thank you for your help!

 

[ggzengel]

I got this error while migrating to DRBD:

Aug  4 22:43:22 px4 kernel: [11257.200035] drbd vm-200-disk-1/0 drbd1001 px3: Began resync as SyncSource (will sync 52428800 KB [13107200 bits set]).
Aug  4 22:43:22 px4 pvedaemon[24235]: VM 200 qmp command failed - VM 200 qmp command 'drive-mirror' failed - The device is not writable: Permission denied
Aug  4 22:43:22 px4 pvedaemon[24235]: VM 200 qmp command failed - VM 200 qmp command 'block-job-cancel' failed - Block job 'drive-virtio0' not found

 

[ggzengel]

I found this and it's just work in progress: https://lists.gt.net/drbd/users/30817

 

[Morphushka]

The same issue, waiting for fix.

 

[dendi]

I'm not sure it's a drbd issue, it's doing his work.
maybe qemu4 introduced some check on the storage?

BUT I tried this, when the resource was secondary:

if >> /dev/drbd1001
then
    echo "writable"
else
    echo "write permission denied"
fi

the result is: writable! ... I'm really confused ...

 

[ggzengel]

With linstor-proxmox 4.0.0 it works and there is the new repo:
deb http://packages.linbit.com/proxmox/ proxmox-6 drbd-9.0