WebRTC desktop console for Proxmox VE — looking for feedback

R

rdmitry0911

Active Member
Apr 11, 2020
13
4
43
65
Hi guys, I've been working on a WebRTC console for Proxmox VE and would like to share it here. I wanted to use a VM desktop directly in the Proxmox web interface, with smoother scrolling, video and sound and copy/paste, without opening a separate remote desktop client. I've managed to get this working as an add-on for PVE 9 called QSM Direct. All dektop graphics runs smoothly. I can even watch youtube clips comfortably with it over the Internet, not only in LAN :)

It takes the VM's display from QEMU over D-Bus and streams it to the browser. The console uses the existing PVE login and VM.Console permission. For VMs, desktop capture and keyboard/mouse input don't need a remote desktop server inside the guest. Clipboard integration needs a guest component.

What works in the add-on:
- H.264 streaming with NVENC, QSV or VA-API, with CPU encoding as a fallback;
- virtio-gl for an accelerated Linux desktop, as well as non-GL display profiles;
- sound, microphone and text clipboard in both directions;
- the console inside the PVE interface or in a separate browser window.

My GPU test setup is PVE 9 with an RTX 3080 and a Linux KDE guest. The stream runs at 1280x800, 60 fps with NVENC. That is the stream rate; the number of different pictures depends on what the guest renders.

I also compared the two consoles on the same Standard VGA VM in a CPU-only lab. A 30 fps 720p clip delivered about 27 different pictures/s through QSM and 7 through noVNC, using about 20 and 120 Mbit/s respectively. The setup, method and results are here

One limitation worth mentioning: with the GL D-Bus display, QEMU cannot offer VNC or SPICE alongside it. A non-GL profile can keep VNC. The current add-on also needs TCP 8007 and UDP access from the browser to the node. Installation details are in the repository

There is also a graphical console for LXC containers with a KDE desktop, supplied as a separate add-on with software inside the container.

I've posted an RFC to pve-devel and prepared patches for qemu-server and pve-manager. This is still an add-on and an upstream proposal. Alexandre Derumier has a related RDP/Kyber series using QEMU's D-Bus display. I've tested its capture path on NVIDIA and prepared a fix; he has now replied and welcomed reworking his patches. I'd like the consoles to share the same D-Bus foundation.

RFC and discussion
Feature request

the prototype was developed with help from an AI coding assistant, and reviewed and tested the code myself.

Does anyone else use the PVE browser console for desktop work? I'd be interested in feedback, especially from people with Intel or AMD GPUs, as my hardware testing has mainly been on NVIDIA. Any feedback would be very much appreciated.

Thank you, Dmitry
 
  • Like
Reactions: dale, UdoB and robertlukan
rdmitry0911 said:
Hi guys, I've been working on a WebRTC console for Proxmox VE and would like to share it here. I wanted to use a VM desktop directly in the Proxmox web interface, with smoother scrolling, video and sound and copy/paste, without opening a separate remote desktop client. I've managed to get this working as an add-on for PVE 9 called QSM Direct. All dektop graphics runs smoothly. I can even watch youtube clips comfortably with it over the Internet, not only in LAN :)

It takes the VM's display from QEMU over D-Bus and streams it to the browser. The console uses the existing PVE login and VM.Console permission. For VMs, desktop capture and keyboard/mouse input don't need a remote desktop server inside the guest. Clipboard integration needs a guest component.

What works in the add-on:
- H.264 streaming with NVENC, QSV or VA-API, with CPU encoding as a fallback;
- virtio-gl for an accelerated Linux desktop, as well as non-GL display profiles;
- sound, microphone and text clipboard in both directions;
- the console inside the PVE interface or in a separate browser window.

My GPU test setup is PVE 9 with an RTX 3080 and a Linux KDE guest. The stream runs at 1280x800, 60 fps with NVENC. That is the stream rate; the number of different pictures depends on what the guest renders.

I also compared the two consoles on the same Standard VGA VM in a CPU-only lab. A 30 fps 720p clip delivered about 27 different pictures/s through QSM and 7 through noVNC, using about 20 and 120 Mbit/s respectively. The setup, method and results are here

One limitation worth mentioning: with the GL D-Bus display, QEMU cannot offer VNC or SPICE alongside it. A non-GL profile can keep VNC. The current add-on also needs TCP 8007 and UDP access from the browser to the node. Installation details are in the repository

There is also a graphical console for LXC containers with a KDE desktop, supplied as a separate add-on with software inside the container.

I've posted an RFC to pve-devel and prepared patches for qemu-server and pve-manager. This is still an add-on and an upstream proposal. Alexandre Derumier has a related RDP/Kyber series using QEMU's D-Bus display. I've tested its capture path on NVIDIA and prepared a fix; he has now replied and welcomed reworking his patches. I'd like the consoles to share the same D-Bus foundation.

RFC and discussion
Feature request

the prototype was developed with help from an AI coding assistant, and reviewed and tested the code myself.

Does anyone else use the PVE browser console for desktop work? I'd be interested in feedback, especially from people with Intel or AMD GPUs, as my hardware testing has mainly been on NVIDIA. Any feedback would be very much appreciated.

Thank you, Dmitry
Click to expand...
Hi guys, a small update. I've now moved the WebRTC console onto the shared
per-VM D-Bus foundation from Alexandre's patches. qemu-server owns the bus
and QSM connects to it as a consumer, so other console implementations can
use the same display too.

The native integration now uses the normal Proxmox API on8006 for both VMs
and LXC containers. TCP8007 and the separate certificate exception are no
longer needed for this version. WebRTC still needs UDP access between the
browser and the node.

I also fixed the Firefox connection and Reconnect issues I ran into during
VPN testing. I've checked VM video/Reconnect and the LXC graphical console
with the old signalling service stopped, including saving the container's
media settings through the PVE API.

The source and matching deb packages are here

This is a preview build with matching patched Proxmox packages; the release
notes list what to install. The shared-base patches are still an upstream
proposal. The older stock-PVE add-on continues to use8007.

If anyone tries this with Intel or AMD encoding, I'd be interested in how
it works on your setup. Thank you for any feedback.
 
Hi guys, a small update on the LXC desktop console.

I've now added the real SDDM login screen with the standard KDE Breeze theme. It has the usual user selection, password prompt and desktop session selector. SDDM/PAM handles authentication using the container's normal user accounts. Logging out brings you back to the login screen in the same browser console.

Different local users can also run desktops on separate console seats at the same time. Keyboard layout switching works on the login screen too; I've checked Russian input with Shift and German AltGr.

I've tested Firefox and Chromium, login/logout and reconnect on an unprivileged Ubuntu 24.04 container with Plasma Wayland. Plasma also uses hardware rendering on the RTX 3080 passed through to my test container. The Breeze theme comes from the normal distribution package; its files are unchanged.

Source and matching deb packages

The native SDDM profile is optional and needs to be enabled explicitly. For now I've qualified Ubuntu 24.04 / Qt 5 / Plasma Wayland. It also needs VKMS with configfs support in the host kernel. Setup, theme selection and rollback instructions

Signalling still uses the normal PVE API on 8006; WebRTC media needs UDP access. This remains a preview on the patched shared-platform setup.

@chrcoluk, thank you for offering to try it. I'd be interested to hear how it works on your setup. If anyone uses LXC for a desktop, feedback on the login and session behaviour would also be very welcome.

Also feedback on accessing the guests from android tablets / ios ipads would help a lot to move forward.

Thank you, Dmitry
 
Hi guys, another small update, this time on mouse responsiveness and video playback.

The LXC console can now draw the guest cursor locally in the browser, including its shape and hotspot. Moving the mouse no longer waits for a video frame to arrive. This made a noticeable difference when using the desktop over VPN. I've also improved input delivery and capture timing. VMs already have a separate cursor path through QEMU's D-Bus display.

While testing Firefox over VPN, I found problems with large video packets and short bursts of traffic, especially around keyframes. The latest build uses smaller packets, spreads video traffic more evenly and bounds NVENC/libx264 bitrate and buffering. H.264 NVENC can now produce a recovery keyframe without restarting the encoder. These video transport and encoder changes apply to both VMs and LXC containers using the native QSM Direct integration.

In a controlled test with Firefox, an LXC desktop and NVENC, I limited the connection to 8 Mbit/s with an 80 ms round-trip delay. The updated version decoded about 60 fps during the moving-picture test without freezes or dropped frames; the previous build repeatedly froze under the same conditions. These measurements cover LXC; the equivalent VM playback test is still pending. I'm also still checking normal YouTube playback over the actual VPN. Full bandwidth adaptation remains work to do; NVENC/libx264 currently use an 8 Mbit/s encoder ceiling.

Source, latest host deb and test results

Local cursor support for LXC needs the updated optional Sway/wlroots package linked from the release notes. This remains a preview on the shared-platform setup.

If anyone tries it, I'd be interested in how mouse movement and video playback feel in your browser, especially over VPN or on a tablet. Please mention whether you're using a VM or LXC and which encoder you're using.

Thank you, Dmitry
 
You must log in or register to reply here.
Top Bottom
Back