LibVF.I, Support For VM's.

[LexicDarkA7]

LibVF.IO is based on looking glass would allow for hardware acceleration of VM's and also
LibVF.IO automates the creation and management of mediated devices (partitioned commodity GPUs shared by host & guest), identifying NUMA nodes, parsing / managing IOMMU devices, and allocating virtual functions to virtual machines.

I've included both links to the project, considering that spice is no longer supported by red hat i believe this may be a viable alternative to take its place as a replacement display protocol considering that its is based on the looking glass project with is based on spice and is able to copy the gpu frame buffer for a real time video output.

https://github.com/Arc-Compute/LibVF.IO
https://arccompute.com/blog/libvfio-commodity-gpu-multiplexing/

 

[LnxBil]

Is this still live? Commits to the GIT are almost a year old.

 

[LexicDarkA7]

It is mostly the activity is on discord where the maintainer, is usually answering questions about it or helping with troubleshooting, they were discussing proxmox recently, it's worth joining to talk to him.

 

[dcsapak]

But AFAIU this is not a remote display protocol, rather looking glass uses shared memory to copy to the host to display it there.
That is only part of the problem. The more sophisticated part is to develop/implement a remote display protocol that can handle various bandwidth/latency/acceleration situations in a good way.
And besides VNC/SPICE i currently don't know any that is is open (source).


EDIT: btw you can use looking glass with pve already, there is the 'ivshmem' device (only on the cli) that you can configure for vms

 

[LexicDarkA7]

So the question I should ask is can it be used in place of qxl with spice, I mean as an alternative to qxl?

 

[dcsapak]

well it's not that easy, since the spice server sits inside qemu and looking glass works by copying the framebuffer from inside the guest to an application on the host, so not through qemu

one would need to implement basically a 'looking glass' client inside qemu for that to work

most users that want to use a physical gpu in the guest use an in guest solution for remote display, such as remote desktop/parsec/etc... then you don't need the whole 'copy the framebuffer to the host' in the first place

 

[LexicDarkA7]

So quemu would need to have it supported as a display mode?

If you read the links I posted, libvfio does what Microsoft gpu-p does on windows server in regards to GPU partitioning, if I remember correctly libvfio is a fork of looking glass.

 

[dcsapak]

just to clarify, there are several parts to this:

one is to give a vm a (partitioned/virtualized) gpu -> this is done with e.g. nvidia vgpu (see https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE ) or virtio-gl, etc.
another is to get the display of that vm to the user -> this is what spice/vnc/looking glass is doing

partitioning and/or passing through gpus to vms is not a problem with PVE
neither is using VNC/SPICE/looking glass

the problem is when using a passed through gpu (either complete or partitioned) qemu does not have direct access to the framebuffer anymore (since it's in hardware and not virtualized) and thus cannot stream it via VNC/SPICE
the solution here is to use something inside the guest, like parsec/remote desktop etc.

looking glass is a similar solution for a different problem: it's useful if you want to show the display of the guest on the same host, but not remote so it only does the framebuffer copying, not the encoding/decoding streaming via network, etc.

is it maybe more clear now?

i don't see how we currently could integrate looking glass, without having a proper remote desktop protocol to go along with it. in the meantime using something inside the vm is probably better suited for remote access
(for local access via looking glass there isn't anything special needed)

 

[LexicDarkA7]

So to clarify, your saying that libvf.io would have to have its own specific display protocol attached for it to become usable in proxmox?

Looking glass is based on spice, and libvfio is based on looking glass.

If only there was a way to integrate libvf.io with spice itself as therefore to create spice with GPU partitioning support in a sense, it's just a shame there is no way for qxl to have hardware acceleration which would be nice.

Wait so there is no way to hook the frame buffer into qxl some how or to allow qxl to access the hardware frame buffer?

 

[LexicDarkA7]

What if you were to add the qxl device to the quemu parameters?

So that qxl could access the GPU buffer?

 

[LexicDarkA7]

I fully understand if it's too difficult to implement, but I thought I would ask first to confirmI do appreciate your response.

 

[dcsapak]

i think there is still a misunderstanding here:

Libvfi.io is a collection of tools/drivers/etc. to make it easier to create/manager/passthrough (partitioned) gpus to vms and access them *locally*

most of the driver parts are already integrated with proxmox ve (e.g. managing mediated devices for nvidia vgpu, selecting sr-iov devices, etc)
whats not integrated is anything to access that vm *locally* since that is not the main use case for Proxmox VE.
(But nobody is stopping you from using those parts manually e.g. looking glass)

maybe we'll make use of the remaining tools/etc., but probably not by using libvio.io but rather by integrating the underlying tools/programs/apis/etc.

hope that makes it clearer

 

[LexicDarkA7]

I believe you missed the question I asked, I spoke to the libvfio script creator or toolset creator he, told me that the libvf.io tools can be hooked into QXL on spice, and that the
"QXL device would need to be added to the QEMU params."

Which might be able to give qxl access to the frame buffer of the GPU, allowing spice to work, even with a GPU passed through.

 

[dcsapak]

I believe you missed the question I asked, I spoke to the libvfio script creator or toolset creator he, told me that the libvf.io tools can be hooked into QXL on spice, and that the
"QXL device would need to be added to the QEMU params."

Which might be able to give qxl access to the frame buffer of the GPU.

i don't completely understand what is meant here as that makes no sense to me

QXL is virtual gpu and has nothing inherently to do with SPICE. You can use SPICE on any virtual GPU

What exactly was the context of that discussion?

 

[LexicDarkA7]

I

i don't completely understand what is meant here as that makes no sense to me

QXL is virtual gpu and has nothing inherently to do with SPICE. You can use SPICE on any virtual GPU

What exactly was the context of that discussion?

was asking him if it was possible to give the qxl virtual gpu access to the frame buffer of the passed through v GPU, he might not have understood my question my apologies for the confusion.

In the context of how looking glass can copy the GPU frame buffer, I was wondering if something like it could be set up to copy the hardware GPUs frame buffer and our put the copy via the software GPU, because I've been looking for a way to have hardware acceleration via a GPU for spice similar to how Microsoft remote desktop protocol has hardware acceleration.

 

[dcsapak]

yes something like that could work, but it basically does not have anything to do with libvfio but more the underlying technologies (looking glass/qemu/spice) is what i'm saying

 

[LexicDarkA7]

yes something like that could work, but it basically does not have anything to do with libvfio but more the underlying technologies (looking glass/qemu/spice) is what i'm saying

I fully understand what you are saying, if it's doable then that would be perfect, to see something like it added in a future update, where the GPU frame buffer could be output to the virtual GPU via looking glass allowing for spice to still be able have output and qemu having real-time access to the hardware frame buffer.

The reason I'm asking about all of this is that tlx os, or thinking is, currently re writing thier thin client os to add support for the same spice viewer used to access proxmox virtual machines.

I've been in contact with them about it.

 

[LexicDarkA7]

What I'm hoping for is that 3d applications can use the hardware GPU, and the frame buffer of the hardware GPU will be copied to the qxl virtual GPU via looking glass, so that spice is still usable with a thin client terminal, to access the VM, in a sense giving "hardware acceleration" to the qxl GPU in a sense.

 

[dcsapak]

What I'm hoping for is that 3d applications can use the hardware GPU, and the frame buffer of the hardware GPU will be copied to the qxl virtual GPU via looking glass, so that spice is still usable with a thin client terminal, to access the VM, in a sense giving "hardware acceleration" to the qxl GPU in a sense.

i understand, but that's really only part of the whole thing.

what would be optimal that the guest could also simultaneously encode in h264/h265/av1 (hardware accelerated) for that to be given to qemu. there is https://gitlab.freedesktop.org/spice/spice-streaming-agent that could have accomplished this, but it seems development stopped (last commit 1 year ago)

without the encoding/network transfer in a better format than what spice/vnc does it's not really that great e.g. in comparison with a complete in guest solution like RDP/Parsec/etc.

 

[LexicDarkA7]

i understand, but that's really only part of the whole thing.

what would be optimal that the guest could also simultaneously encode in h264/h265/av1 (hardware accelerated) for that to be given to qemu. there is https://gitlab.freedesktop.org/spice/spice-streaming-agent that could have accomplished this, but it seems development stopped (last commit 1 year ago)

without the encoding/network transfer in a better format than what spice/vnc does it's not really that great e.g. in comparison with a complete in guest solution like RDP/Parsec/et

I've been using RDP it's been less than ideal in my experience due to the lack of UDP support on free RDP it's been, painful to say the least though I'm excited for the 3.0 build they have with it included.



True it would be ideal if the vgpu could encode the stream and push it to quemu, similar to what blast extreme does on VMware and Citrix hdx Pro.



Which makes me wonder steam is able to leverage the built in GPU encoder, as nvec and the other amd encoder who's name I can't remember I think its vcn, for thier game streaming protocol, I wonder if something similar could be leveraged as modifications to spice.



I was just reading this have you looked into NVIDIA Frame Buffer Capture I've seen an option to enable that on steam might be if interest.

https://looking-glass.io/docs/B6/usage/#capture-interface