[TUTORIAL] RDP Fix for MAC Users Connecting to Proxmox Window11VM

[thomas461]

Problem: Connecting to Proxmox Win11VM from Mac OS (Sequoia 15.1.1) MacBook Pro M2. The connection would constantly (every 3-5 minutes) timeout and throw error code (Windows App 0x407 Data Encryption Error)

Data: Microsoft is still using a Network stack from 2012 on Win10 and Win11. This is causing issues with modern connections and the stack hasn't been tweaked in a long time. This non-tuned stack is causing havoc on all kinds of applications. To include my above mentioned issue.

• Proxmox PVE-8.3.0 Server
  • Dell 7910 Workstation E5-2897 V4
  • Intel X-540T2 - Dual 10G Copper Nic
    • vmbr0 - LAN Interface (Vlan10)
    • vmbr2 - Video Interface (Vlan40)
  • Windows 11VM
    • Using vmbr2 Interface DHCP
    • RDP Server Running
• Juniper EX3300 48Port switch (Layer2 Only)
  • VLANS
    • LAN - 10
    • WLAN Private - 20
    • WLAN Public - 30
    • VIDEO - 40
  • Interfaces
    • 10G - 2 Proxmox Nics
    • 1G - All other Connections
• PFSense Router/Firewall/DHCP/DNS running on Protectli Hardware
  • igc0 - WAN
  • igc1 - LAN
  • igc2 - WLAN (Both Public/Private) Trunked
  • igc3 - VIDEO
• MacBook Pro - M2
  • MACOS 15.1.1 Sequoia
  • Wired Ethernet 1G - Vlan 10
  • Windows App (RDP Client)
    

Fix: Win10/Win11 Network TCP Stack Optimization (Alexander Fuchs - MysticFoxDE)

There is file on Github that you run at the Administrative Level of Powershell on the Windows VM that fixed this issue for me. MysticFoxDE tweaks the TCP stack to be more optimal. You can read about it here if interested (Deutsch/German). (Alex if you ever come across this Vieln Danke fur Alles)

Steps:

• From Proxomox Console
  • Log into Win11 VM
    • Use browser to navigate to https://github.com/MysticFoxDE/WIND...n/W10ANDW11-NETWORK-TCP-DESUBOPTIMIZATION.ps1
    • Download the .ps1 file from Github website
    • Windows Button
    • Lauch Windows Powershell (Run as administrator)
    • Within PowerShell execute the following commands
      • Type cd ~\Downloads
      • You should now be in the Download Directory
      • Type Dir to verify the file W10ANDW11-NETWORK-TCP-DESUBOPTIMIZATION.ps1 is present
      • Type Set-ExecutionPolicy Unrestricted
      • Type A (Yes to All)
      • Type .\W10ANDW11-NETWORK-TCP-DESUBOPTIMIZATION.ps1
      • Script should run and complete (I had no errors in my script so I cannot address troubleshooting)
    • Reboot Windows11 VM
    • Log back in to the Windows11 VM
    • Verify RDP Is enabled from Settings > System > Remote Desktop > On
• MAC
  • Download Windows App (From App Store)
  • Open Windows App
  • Click + PC
  • Type in PC Name or IP Address
  • Credentials (Local or live) That you use to access your Window11 VM
  • Gateway (Only if this is on another network)
    • Gateway of the MAC NOT the Windows VM
    • Tick Bypass for local addresses (Same Network/Subnet your MAC is on)
  • Click Connect
  • Login with your Windows Credentials

I hope this helps someone as it was quite a long time for me to find a solution that fixed this very annoying problem. Now I am able to access my Win11 VM error free from my Mac. It is solid and error free.