[SOLVED] Attach block device as a raw disk to VM

[AHTOH]

Hello,
I'm migrating from KVM to Proxmox VE.
Two small questions about the storage:

1) How to attach a block device as a raw disk to VM? I have a hdd with LUKS encrypted partition with LVM on it. I would like to pass one of the LVs to my VM as a raw disk.

2) Is it possible to attach the disks to the running VM? If I declare the disk from my Question (1) in the VM, the VM will not start if the LUKS partition is not decrypted (since LVM will not be visible). I want my VM to boot without this disk, then I'll open the LUKS partition and attach the LV (raw device) to the running VM manually.

Thank you.

 

[elmarconi]

https://pve.proxmox.com/wiki/Physical_disk_to_kvm

 

[AHTOH]

Thanks elmarconi, I also found this howto, added the line
virtio1: /dev/mapper/gueststorage-intrastorage,size=3500G
in the VM config file and it seems to work.

Now need to find the way to attach it to the running VM

 

[elmarconi]

https://blog.chrishowie.com/2019/09/19/hot-swapping-virtio-disks-on-qemu/

 

[AHTOH]

Thanks elmarconi! Seems to work. Any idea on how to run it in a non-interactive mode? (A workaround would be to run it interactively and redirect stdin from a file, but it's ugly).

For the record:
# qm monitor <vm id>
> drive_add 0 if=none,file=/dev/gueststorage/intrastorage,format=raw,id=drive-virtio-crypto
> device_add virtio-blk-pci,drive=drive-virtio-crypto

 

[elmarconi]

Found this: https://forum.proxmox.com/threads/qm-monitor-and-shell-scripts.6052/#post-34338

 

[AHTOH]

Ok, finally got it working, but I'm open to any simpler solution...

What I did (Please note that I'm new to Proxmox so I can't guarantee that my manipulations are "right", "safe" or "optimal") :

1) Detect the sockets qemu listening on

# qm monitor 101 qm> info chardev

which return something like
qmp: filename=unix:/var/run/qemu-server/101.qmp,server

I have only QMP socket listening, what means that I have to talk JSON to it.

2) Check the configuration created by the VE

Edit the VM config file /etc/pve/qemu-server/101.conf and add the disk:
virtio1: /dev/mapper/gueststorage-intrastorage,size=3500G
Hint: run qm rescan

Run the VM normally, ssh to the Host OS (Proxmos) and check the "footprint" of the running VM:
ps aux | grep qemu

Extract the options related to your drive. In my case it was something like:
-drive file=/dev/mapper/gueststorage-intrastorage,if=none,id=drive-virtio1,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb

3) Stop the VM, remove the drive line from the config, start the VM.

4) Try to attach the disk to the running VM

Run socat on the socket and tell it to take the commands from stdin using "-":
socat -,echo=0,icanon=0 unix-connect:/var/run/qemu-server/101.qmp

you should get it waiting for the commands.

Feed it with JSON. The first command must be {"execute":"qmp_capabilities"} to switch it to the command mode. The others commands are "constructed" from the data found in (2).

Add the backend device (the physical support):
{"execute":"blockdev-add","arguments":{"driver":"raw","node-name":"drive-virtio1","file":{"driver":"file","filename":"/dev/mapper/gueststorage-intrastorage"}}}

Add the frontend device (device visible to the host):
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"drive-virtio1","id":"virtio1","bus":"pci.0","addr":"0xb"}}

At this stage the blk/pci device should become visible in the guest OS. Check it with lsblk, then mount it ...

5) Put everything in a script file to automate the execution

#!/usr/bin/sh

commands='{"execute":"qmp_capabilities"}
{"execute":"blockdev-add","arguments":{"driver":"raw","node-name":"drive-virtio1","file":{"driver":"file","filename":"/dev/mapper/gueststorage-intrastorage"}}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"drive-virtio1","id":"virtio1","bus":"pci.0","addr":"0xb"}}'

echo "$commands" | socat - unix-connect:/var/run/qemu-server/101.qmp

Any suggestions are welcome...

 

[Stoiko Ivanov]

qm set -virtio1 /dev/mapper/gueststorage-intrastorage

should also work

(also consider using virtio-scsi (and attaching the drives as scsi0, scsi1... instead of virtio - it is the default option and thus gets more testing and improvments)

I hope this helps!

 

[AHTOH]

Thank you, Stoiko.
Indeed your procedure is a bit simpler than mine

EDIT: After some testing I discovered that it's not an acceptable solution for me :-(
The problem is that when I do "qm set", the property is added to the VM config file.
It means that when the next time the VM is started, it will be started with this property.
In my case, the VM will not start, since the disk is not available at this moment.
I want the VM to start automatically (if the Host reboots, for example) without the disk, and then I'll attach the disk manually (since I have to decrypt it first with a passphrase).
Your solution works for the first boot only, but not for the consecutive reboots.

Any idea how to tell "qm set" to make a "non persistent" change (only for the current session), without adding it to the conf file?

 

[AHTOH]

After some testing I discovered that it's not an acceptable solution for me :-(
The problem is that when I do "qm set", the property is added to the VM config file.
It means that when the next time the VM is started, it will be started with this property.
In my case, the VM will not start, since the disk is not available at this moment.
I want the VM to start automatically (if the Host reboots, for example) without the disk, and then I'll attach the disk manually (since I have to decrypt it first with a passphrase).
Your solution works for the first boot only, but not for the consecutive reboots.

Any idea how to tell "qm set" to make a "non persistent" change (only for the current session), without adding it to the conf file?

 

[Stoiko Ivanov]

Any idea how to tell "qm set" to make a "non persistent" change (only for the current session), without adding it to the conf file?

no not really

but you could pass through the encrypted block device instead of the one you opened with cryptsetup - and then do the unlocking inside the VM?

 

[AHTOH]

but you could pass through the encrypted block device instead of the one you opened with cryptsetup - and then do the unlocking inside the VM?

I have one big LUKS partition with LVM on top of it, with multiple LVs passed to the different VMs. This allows me to decrypt the whole disk at once, instead of decrypting each LV inside the VMs.
Moreover, when LVM on top of LUKS works well, the opposite (LUKS on top of LVM) has no big sense, since you can't resize the encrypted partitions easily.

 

[AHTOH]

Ok, marking this thread as Solved, but if somebody has any remarks/suggestions, you are welcome.

To resume, I see two solutions:

1) Persistent solution (the drive will be added to the VM config file and automatically attached at the next boot):

qm set 102 -scsi1 file=/crypto/vm/images/102/vm-102-disk-0.qcow2

(the "file=" may point directly to a raw partition).


2) Non-persistent (needs to re-attach the drive after each reboot):

#!/usr/bin/sh

commands='{"execute":"qmp_capabilities"}
{"execute":"blockdev-add","arguments":{"driver":"qcow2","node-name":"drive-scsi1","file":{"driver":"file","filename":"/crypto/vm/images/102/vm-102-disk-0.qcow2"}}}
{"execute":"device_add","arguments":{"driver":"virtio-scsi-pci","id":"scsihw0","bus":"pci.0","addr":"0x5"}}
{"execute":"device_add","arguments":{"driver":"scsi-hd","drive":"drive-scsi1","id":"scsi1","bus":"scsihw0.0","channel":"0","scsi-id":"0","lun":"1"}}'

echo "$commands" | socat - unix-connect:/var/run/qemu-server/102.qmp

Replace "qcow2" by "raw" for a raw partition.

In my case, I don't need the second line "device_add virtio-scsi-pci", since I already have another (boot) drive added via scsi driver, so "scsihw0" is already present in the system.
And I presume one can completely avoid this line by supplying directly "driver":"virtio-scsi-pci","bus":"pci.0.0","addr":"0x5" on the last line (?)