[SOLVED] Windbg Remote Kernel debugging and Proxmox not working

D

daubsi

New Member
Jul 17, 2024
6
1
3
Hello,

I have a Windows Server 2022 and Win11 VM running on Proxmox. I'm using Virtio drivers for Network, SATA, etc.
The VMs themselves are running fine.
For testing purposes I need to perform remote kernel debugging on those hosts.

The debugger host is a VM on an ESXi server, and the VMs on Proxmox are in the same VLAN/network like the debuggees.
As long as the debugee is also on ESXi or a physical machine on the same VLAN/network everything works perfectly.

When the debuggee is one of the VMs on Proxmox mentioned above I am not able to get the debuggees to connect to the debugger.

The normal procedure to set this up is:

* Disable SecureBoot on the debugee
* As admin: bcdedit /dbgsettings net hostip:<debugger> port:49152 key:1.2.3.4
* As Admin: bcdedit /debug yes

And reboot the machine

On the debugger I configure the same port and key and "Attach to kernel".

Then during bootup the debuggee should break after few seconds and the debugger should show allow for debugging
This does not happen, the debugee starts up normally.

I tcpdumped on the Proxmox host all interfaces for the port 49152 and see 0 network packets. So it seems the Windows VM does not even try to reach out.

I tried switching from Virtio as the network adapter driver to EE1000, I tried fixed IPs and DHCP address leases.
I verified that the VM and the debugger can communicate without any issues (port probe) with each other once they are up and running, and nothing is filtering there. So it's also not a question of VLAN tags, bridge configuration or anything in that direction I presume.

Is there anything from the Proxmox side which could explain why the debugee is not reaching out? My best guess was the original virtio setup but a switch to EE1000 as the NIC should definitely work? Checked here: https://learn.microsoft.com/en-us/w...cs-for-network-kernel-debugging-in-windows-10

The settings I perform in those VMs are the same as in any other VM, so the commands themselves are correct and work.

Proxmox 8.3.4

CPU(s)​

4 x Intel(R) Celeron(R) J4105 CPU @ 1.50GHz (1 Socket)​
Kernel Version​

Linux 6.8.12-7-pve (2025-01-17T08:18Z)​
Boot Mode​

EFI​
Manager Version​

pve-manager/8.3.4/65224a0f9cd294a3​
Any ideas?
 
Last edited:
Fair question! However, apparently bcdedit /debug on and bcdedit /debug yes actually do the same. When you run bcdedit afterwards, in every case it reads "Debug: yes".

Nevertheless, tried also /debug on, but same result: No "reach out" by the debugee..

That link looks very promising! Will try! Thanks for that!
 
Hm, can you make sense of those suggestions (for proxmox I mean)
I understand for proxmox I am supposed to look at the VM configuration file in /etc/pve/qemu-server?

I manually edited 106.conf to look like this:

Code: 
agent: 1
bios: ovmf
boot: order=virtio0;ide0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-106-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide0: local:iso/virtio-win-0.1.266.iso,media=cdrom,size=707456K
ide2: local:iso/en-us_windows_11_consumer_editions_version_24h2_updated_jan_2025_x64_dvd_7a8e5a29.iso,media=cdrom,size=6388576K
machine: pc-q35-9.0
memory: 4096
meta: creation-qemu=9.0.2,ctime=1738938711
name: Windows11-Playbox
net0: e1000e=BC:24:11:BC:F3:52,bridge=vmbr0,firewall=1
numa: 0
ostype: generic    <-- was win11
scsihw: virtio-scsi-single
smbios1: uuid=2140243c-a953-427a-a9ed-5e86319c5f23
sockets: 1
tpmstate0: local-lvm:vm-106-disk-1,size=4M,version=v2.0
virtio0: local-lvm:vm-106-disk-2,iothread=1,size=32G
vmgenid: fc13b69e-d34e-46dd-b978-9f745a28289c

(changed the ostype value)

But now Windows 11 does not boot up anymore - and that's not because it connected to the debugger host ;-)
There was no "HyperV" entry in the config file.
Am I supposed to ADD this to the config file?
hv-vendor-id: KVMKVMKVM

Or is editing /etc/pve/qemu-server/<id>.conf not the right way in general?
 
Last edited:
I am not sure about right syntax, but a new line in the conf is needed with something like args: -cpu hv-vendor-id=KVMKVMKVM
s. https://www.qemu.org/docs/master/system/i386/hyperv.html and man qm.conf

Update: in my setup this didn't work, but a qm showcmd <vmid> delivered the right options for -cpu. I took them all and added the hv-vendor... just at the end after a comma and at least the vm started afterwards.
 
Last edited:
fba said:
I am not sure about right syntax, but a new line in the conf is needed with something like args: -cpu hv-vendor-id=KVMKVMKVM
s. https://www.qemu.org/docs/master/system/i386/hyperv.html and man qm.conf

Update: in my setup this didn't work, but a qm showcmd <vmid> delivered the right options for -cpu. I took them all and added the hv-vendor... just at the end after a comma and at least the vm started afterwards.
Click to expand...
After setting:

VM Options -> KVM Hardware Virtualization: disabled

The debuggee successfully reaches out to the debugger host and breaks! Success!

Changing the OSType to Other did not make a change (apart from that the OS did no longer boot).

Update:

Alternative:
Keep VM Options->KVM Hardware Virtualization: enabled

But change in /etc/pve/qemu-server/<id>.conf

the line:

cpu: x86-64-v2-AES
to
cpu: x86-64-v2-AES,hv-vendor-id=KVMKVMKVM

This does not seem to be possible via the UI.
When you specify the hv-vendor-id like that it will persist at the right place in the cmdline

Code: 
root@proxmox1:/etc/pve/qemu-server# qm showcmd 106
/usr/bin/kvm -id 106 -name 'Windows11-Playbox,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/106.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/106.pid -daemonize -smbios 'type=1,uuid=2140243c-a953-427a-a9ed-5e86319c5f23' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/pve/vm-106-disk-0,size=540672' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/106.vnc,password=on' -cpu 'qemu64,+aes,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vendor_id=KVMKVMKVM,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3' -m 4096 -object 'iothread,id=iothread-virtio0' -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=fc13b69e-d34e-46dd-b978-9f745a28289c' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -chardev 'socket,id=tpmchar,path=/var/run/qemu-server/106.swtpm' -tpmdev 'emulator,id=tpmdev,chardev=tpmchar' -device 'tpm-tis,tpmdev=tpmdev' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -chardev 'socket,path=/var/run/qemu-server/106.qga,server=on,wait=off,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:7a98b359326' -drive 'file=/var/lib/vz/template/iso/virtio-win-0.1.266.iso,if=none,id=drive-ide0,media=cdrom,format=raw,aio=io_uring' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=101' -drive 'file=/var/lib/vz/template/iso/en-us_windows_11_consumer_editions_version_24h2_updated_jan_2025_x64_dvd_7a8e5a29.iso,if=none,id=drive-ide2,media=cdrom,format=raw,aio=io_uring' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=102' -drive 'file=/dev/pve/vm-106-disk-2,if=none,id=drive-virtio0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap106i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000e,mac=BC:24:11:BC:F3:52,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=103' -rtc 'driftfix=slew,base=localtime' -machine 'hpet=off,type=pc-q35-9.0+pve0' -global 'kvm-pit.lost_tick_policy=discard'

Adding the line: args: -cpu hv-vendor-id=KVMKVMVKM did NOT work, the VM didn't start. Neither with hv_vendor_id (Note that the - are replaced by _ in the cmdline apparently)

So keeping the KVM Hardware virtualization enabled and just "patching" the CPU line is probably the best way?

Also, the NIC needs not be changed from VirtIO Paravirtualized to E1000. It works with VirtIO as well. Just keep in mind that every change in the Proxmox UI will remove the ",hv-vendor-id=KVMKVMKVM" change in the config file again so it needs to be reapplied!
 
Last edited:
Last update ;-)
Installing a new VM (now Windows 10) with OSType=Generic (set on creation, i.e. not after the VM is installed) let's Windbg work properly as well. (Which is clear after knowing that those enlightenments are only enabled when OSType=Windows). No other changes to KVM Virtualization or CPU Parameters are needed in this case. Unfortunately, changing the OSType does not seem to work once the system has been installed. (Tested with Win10 Build 19041/22H2)
 
Last edited:
  • Like
Reactions: fba
You must log in or register to reply here.
Top Bottom