WebRTC desktop console for Proxmox VE — looking for feedback

[rdmitry0911]

Hi guys, I've been working on a WebRTC console for Proxmox VE and would like to share it here. I wanted to use a VM desktop directly in the Proxmox web interface, with smoother scrolling, video and sound and copy/paste, without opening a separate remote desktop client. I've managed to get this working as an add-on for PVE 9 called QSM Direct. All dektop graphics runs smoothly. I can even watch youtube clips comfortably with it over the Internet, not only in LAN

It takes the VM's display from QEMU over D-Bus and streams it to the browser. The console uses the existing PVE login and VM.Console permission. For VMs, desktop capture and keyboard/mouse input don't need a remote desktop server inside the guest. Clipboard integration needs a guest component.

What works in the add-on:
- H.264 streaming with NVENC, QSV or VA-API, with CPU encoding as a fallback;
- virtio-gl for an accelerated Linux desktop, as well as non-GL display profiles;
- sound, microphone and text clipboard in both directions;
- the console inside the PVE interface or in a separate browser window.

My GPU test setup is PVE 9 with an RTX 3080 and a Linux KDE guest. The stream runs at 1280x800, 60 fps with NVENC. That is the stream rate; the number of different pictures depends on what the guest renders.

I also compared the two consoles on the same Standard VGA VM in a CPU-only lab. A 30 fps 720p clip delivered about 27 different pictures/s through QSM and 7 through noVNC, using about 20 and 120 Mbit/s respectively. The setup, method and results are here

One limitation worth mentioning: with the GL D-Bus display, QEMU cannot offer VNC or SPICE alongside it. A non-GL profile can keep VNC. The current add-on also needs TCP 8007 and UDP access from the browser to the node. Installation details are in the repository

There is also a graphical console for LXC containers with a KDE desktop, supplied as a separate add-on with software inside the container.

I've posted an RFC to pve-devel and prepared patches for qemu-server and pve-manager. This is still an add-on and an upstream proposal. Alexandre Derumier has a related RDP/Kyber series using QEMU's D-Bus display. I've tested its capture path on NVIDIA and prepared a fix; he has now replied and welcomed reworking his patches. I'd like the consoles to share the same D-Bus foundation.

RFC and discussion
Feature request

the prototype was developed with help from an AI coding assistant, and reviewed and tested the code myself.

Does anyone else use the PVE browser console for desktop work? I'd be interested in feedback, especially from people with Intel or AMD GPUs, as my hardware testing has mainly been on NVIDIA. Any feedback would be very much appreciated.

Thank you, Dmitry

 

[rdmitry0911]

Hi guys, I've been working on a WebRTC console for Proxmox VE and would like to share it here. I wanted to use a VM desktop directly in the Proxmox web interface, with smoother scrolling, video and sound and copy/paste, without opening a separate remote desktop client. I've managed to get this working as an add-on for PVE 9 called QSM Direct. All dektop graphics runs smoothly. I can even watch youtube clips comfortably with it over the Internet, not only in LAN

It takes the VM's display from QEMU over D-Bus and streams it to the browser. The console uses the existing PVE login and VM.Console permission. For VMs, desktop capture and keyboard/mouse input don't need a remote desktop server inside the guest. Clipboard integration needs a guest component.

What works in the add-on:
- H.264 streaming with NVENC, QSV or VA-API, with CPU encoding as a fallback;
- virtio-gl for an accelerated Linux desktop, as well as non-GL display profiles;
- sound, microphone and text clipboard in both directions;
- the console inside the PVE interface or in a separate browser window.

My GPU test setup is PVE 9 with an RTX 3080 and a Linux KDE guest. The stream runs at 1280x800, 60 fps with NVENC. That is the stream rate; the number of different pictures depends on what the guest renders.

I also compared the two consoles on the same Standard VGA VM in a CPU-only lab. A 30 fps 720p clip delivered about 27 different pictures/s through QSM and 7 through noVNC, using about 20 and 120 Mbit/s respectively. The setup, method and results are here

One limitation worth mentioning: with the GL D-Bus display, QEMU cannot offer VNC or SPICE alongside it. A non-GL profile can keep VNC. The current add-on also needs TCP 8007 and UDP access from the browser to the node. Installation details are in the repository

There is also a graphical console for LXC containers with a KDE desktop, supplied as a separate add-on with software inside the container.

I've posted an RFC to pve-devel and prepared patches for qemu-server and pve-manager. This is still an add-on and an upstream proposal. Alexandre Derumier has a related RDP/Kyber series using QEMU's D-Bus display. I've tested its capture path on NVIDIA and prepared a fix; he has now replied and welcomed reworking his patches. I'd like the consoles to share the same D-Bus foundation.

RFC and discussion
Feature request

the prototype was developed with help from an AI coding assistant, and reviewed and tested the code myself.

Does anyone else use the PVE browser console for desktop work? I'd be interested in feedback, especially from people with Intel or AMD GPUs, as my hardware testing has mainly been on NVIDIA. Any feedback would be very much appreciated.

Thank you, Dmitry

Hi guys, a small update. I've now moved the WebRTC console onto the shared
per-VM D-Bus foundation from Alexandre's patches. qemu-server owns the bus
and QSM connects to it as a consumer, so other console implementations can
use the same display too.

The native integration now uses the normal Proxmox API on8006 for both VMs
and LXC containers. TCP8007 and the separate certificate exception are no
longer needed for this version. WebRTC still needs UDP access between the
browser and the node.

I also fixed the Firefox connection and Reconnect issues I ran into during
VPN testing. I've checked VM video/Reconnect and the LXC graphical console
with the old signalling service stopped, including saving the container's
media settings through the PVE API.

The source and matching deb packages are here

This is a preview build with matching patched Proxmox packages; the release
notes list what to install. The shared-base patches are still an upstream
proposal. The older stock-PVE add-on continues to use8007.

If anyone tries this with Intel or AMD encoding, I'd be interested in how
it works on your setup. Thank you for any feedback.

 

[chrcoluk]

The shift to 8006 is good enough for me to try it out. Will try it on my local Proxmox.

 

[rdmitry0911]

Hi guys, a small update on the LXC desktop console.

I've now added the real SDDM login screen with the standard KDE Breeze theme. It has the usual user selection, password prompt and desktop session selector. SDDM/PAM handles authentication using the container's normal user accounts. Logging out brings you back to the login screen in the same browser console.

Different local users can also run desktops on separate console seats at the same time. Keyboard layout switching works on the login screen too; I've checked Russian input with Shift and German AltGr.

I've tested Firefox and Chromium, login/logout and reconnect on an unprivileged Ubuntu 24.04 container with Plasma Wayland. Plasma also uses hardware rendering on the RTX 3080 passed through to my test container. The Breeze theme comes from the normal distribution package; its files are unchanged.

Source and matching deb packages

The native SDDM profile is optional and needs to be enabled explicitly. For now I've qualified Ubuntu 24.04 / Qt 5 / Plasma Wayland. It also needs VKMS with configfs support in the host kernel. Setup, theme selection and rollback instructions

Signalling still uses the normal PVE API on 8006; WebRTC media needs UDP access. This remains a preview on the patched shared-platform setup.

@chrcoluk, thank you for offering to try it. I'd be interested to hear how it works on your setup. If anyone uses LXC for a desktop, feedback on the login and session behaviour would also be very welcome.

Also feedback on accessing the guests from android tablets / ios ipads would help a lot to move forward.

Thank you, Dmitry

 

[rdmitry0911]

Hi guys, another update on mouse responsiveness and video playback.

The LXC console can now draw the guest cursor locally in the browser, including its shape and hotspot. Moving the mouse no longer waits for the next video frame. This made a noticeable difference when using the desktop over VPN. I've also improved input delivery and capture timing.

I've made changes to video packet sizing, pacing and encoder buffering to improve playback over VPN. These video changes apply to both VM and LXC consoles using the native QSM Direct integration.

The maximum video bitrate is now configurable per VM or container in Settings, from 0.5 to 100 Mbit/s, with 8 Mbit/s as the default. With NVENC or software H.264, the working bitrate adapts to network conditions below that limit. Settings shows the current target. QSV and VA-API still use fixed rate control.

During testing I also found a problem where text and details repeatedly became blurry and sharp again, even on a static desktop. Unnecessary bitrate changes and full keyframes every half second both contributed to it. The latest build avoids those repeated changes and sends recovery keyframes when needed. I've now checked the fix in Firefox over the actual VPN connection, and the blur/sharpen pulsing is gone on that setup.

In controlled Firefox tests with an LXC desktop, I reduced the available bandwidth from 20 to 4 Mbit/s and then restored it. Both NVENC and software H.264 lowered their bitrate and returned to the configured 8 Mbit/s limit without reconnecting. The sudden bandwidth drop still caused brief recovery pauses. The test scripts and results are included with the release.

Source, deb packages and test results — shared22

Local cursor support for LXC needs the optional Sway 1.11/wlroots package; setup details are here. This remains a preview on the patched shared-platform setup.

If anyone tries it, I'd be interested in how mouse movement and video playback feel in your browser, especially over VPN or on a tablet. Please mention whether you're using a VM or LXC, your browser and the encoder.

Thank you, Dmitry