[TUTORIAL] PVE 9.2 arm64 on ASUS GX10 (GB10): SoC throttled to 20 W, affecting performance

[bbgeek17]

Now that the system is "stable" after dealing with the black screen and the watchdog resets, we've started looking at performance. Our first look at storage performance was absolutely terrible. And after some head scratching, learning a bit more about why.

It turns out, the machine was slow at everything. The big cores (Cortex-X925) ran at 1.37 GHz instead of 3.9 GHz, the little cores (A725) sat around 0.36 GHz, and single core memory copy was about 4.6 GB/s when a healthy box does around 24 GB/s. So the big cores were at roughly a third of their clock and memory throughput closer to a fifth. Interestingly, the system has been at full speed since one particular reboot, but I can't cleanly explain that reboot, so read the recovery part before you get your hopes up.

The reason it's easy to misdiagnose is that Linux reports everything as fine. The governor is on performance and scaling_cur_freq says 3.9 GHz. That's the frequency the kernel is asking for, not what the core delivers. To get the real number you have to measure it:

perf stat -e cycles -- taskset -c 19 sh -c 'end=$((SECONDS+2)); while [ $SECONDS -lt $end ]; do :; done'

Divide cycles by seconds. I saw 1.37 GHz there while sysfs claimed 3.9 GHz. That's fiddly, so the faster check I actually used was perf bench mem memcpy -s 1GB: around 5 GB/s means throttled, around 24 GB/s means healthy. It's not the clock, it's memory bandwidth, but the whole SoC gets capped together, so either number tells you which state you're in.

It turns out that there's a firmware power budget register (the SoC's PL1) sitting at 20 watts instead of 140. I found it in the DSDT. The NVIDIA telemetry device (NVDA8800) points at a MediaTek power budget block at 0x1C238000, and PL1 is the u32 at offset 0x160, in milliwatts. I ended up writing a small kernel module to probe the memory, since /dev/mem is locked down under Secure Boot (in hindsight, I should have just disabled Secure Boot). It read 20000 throttled and 140000 healthy. Everything gets squeezed to fit the power budget: core clocks, the memory controllers, and the fabric between the two core clusters. I tried the usual OS knobs (governors, EPP, and the CPPC enable/min/max registers the NVIDIA kernels write) and none of them moved it, which makes sense if the limit lives in firmware rather than the OS.

How it gets set, less sure. The GX10 takes power over USB-C PD, and there's a MediaTek controller that negotiates how much the SoC can draw. My working theory, which fits the behavior and other reports out on the interweb, is that a bad PD negotiation at power-on drops it into a 20-watt safety mode. I never caught the negotiation failing though, so that part is a guess.

Resets don't change the state. A normal reboot keeps whatever you're in (throttled stayed throttled early on, healthy stayed healthy later). Watchdog resets too, I armed the hardware watchdog and kicked it several times and it never dropped, though it was already healthy by then so that doesn't say much. A plain power-button reset kept it throttled. Idle didn't throttle it either: clocks didn't budge through an 8 hour overnight idle session. The one and only time the state ever flipped is the accident I describe next, and I can't fully explain it.

On clearing it, I'll be straight: I don't have a reliable method. Mine was throttled for days. It came back to full speed exactly once, and by accident. I was running a GPU benchmark (to force the system to request more power) and the box hard-froze. The display stayed on, so it wasn't a power drop, just a dead hang, and it was hot to the touch. I reset it by holding the power button, force off... then on. I never unplugged the adapter. That boot came up at 140 watts and full clocks and has stayed there since. The frustrating part: I'd already done the same power-button force-off earlier while it was throttled, and those came back throttled. Mechanically, the reset that worked looked exactly like the ones that didn't.

So I genuinely can't tell you what did it. The things that were arguably different that time:

• It had just run hot. If the EC logged a thermal event, the next power-on might negotiate differently. Weak, but I can't rule it out.
• Maybe the EC power-cycled the SoC on overtemp before I ever touched the button, which would make it a cold cycle in effect. The display could be on a separate rail.
• Maybe the negotiation is non-deterministic and that boot just happened to land at 140 watts.

It wasn't some special reset type either. The screen stayed on so it never fully powered off, and it was the same power-button force-off that had failed before. The other GX10/DGX PD-safety-mode threads recommend a full cold cycle (shut down, unplug the adapter for ~30 s, boot). I never actually did that, so I can't confirm it.

Here's some data on throttled vs healthy on this machine:

	throttled	healthy
X925 clock	1.37 GHz	3.9 GHz
A725 clock	0.36 GHz	2.8 GHz
power cap (PL1)	20 W	140 W
memcpy, 1 GB, one core	4.6 GB/s	24 GB/s
core to core, mean	738 ns	161 ns

The most interesting data is probably the core-to-core latency, which shows the fabric throttling. These are two separate captures on the same box, throttled and healthy, taken with https://github.com/nviennot/core-to-core-latency]core-to-core-latency. Values are nanoseconds between each pair of cores.

Throttled:

         0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18
  1:   582
  2:   580  533
  3:   621  570  571
  4:   688  606  604  605
  5:   529  505  508  502  547
  6:   529  507  506  499  549  418
  7:   553  524  526  573  634  493  493
  8:   536  511  509  556  618  478  477  499
  9:   511  460  462  463  511  404  401  473  458
 10:   959  925  924  946  957  888  905  911  928  856
 11:   987  947  945  952 1000  934  929  944  965  862  573
 12:   980  922  945  965  997  912  927  981  955  837  579  595
 13:  1109 1025 1033 1091 1117  976  978  985  976  938  654  686  630
 14:  1079 1016 1050 1075 1087  934  953 1001  961  929  627  674  616  685
 15:   934  896  913  938  924  891  891  883  861  803  512  528  543  579  566
 16:   939  915  915  929  915  863  890  886  873  796  515  527  543  580  566  452
 17:   908  876  876  865  884  808  799  847  826  726  467  478  490  564  547  431  432
 18:   955  929  928  955  934  841  849  898  833  809  537  546  525  567  547  468  471  452
 19:   917  885  859  895  914  878  871  846  847  787  507  523  518  555  539  446  447  422  442
     min 401 / mean 738 / max 1117

Healthy:

         0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16   17   18
  1:    86
  2:    85   88
  3:    88   90   90
  4:    96  100   98   98
  5:    71   78   79   73   81
  6:    72   79   78   73   81   59
  7:    77   83   83   86   94   71   71
  8:    76   81   81   84   91   69   69   75
  9:    79   81   82   76   85   66   66   78   76
 10:   242  249  250  251  262  242  242  261  257  242
 11:   257  256  253  253  268  246  238  243  252  245   89
 12:   248  248  249  267  255  242  241  253  251  242   88   84
 13:   241  251  269  258  266  243  250  262  257  249   86   89   83
 14:   249  249  252  237  252  245  245  257  254  245   85   87   82   74
 15:   237  234  233  234  241  225  225  230  225  225   67   68   68   63   61
 16:   234  234  233  232  240  225  225  229  225  225   67   68   68   64   61   53
 17:   237  237  238  237  241  228  231  236  232  232   73   73   73   72   70   62   62
 18:   229  231  230  231  239  218  214  230  225  225   68   69   66   61   59   54   54   63
 19:   225  222  222  222  238  214  213  225  222  214   65   67   65   59   57   51   51   59   50
     min 50 / mean 161 / max 269

Cores 0 through 9 are one cluster, 10 through 19 the other, so the big block in the lower left is the cross-cluster hops. Throttled, same-cluster neighbors are 400 ns to 600 ns apart and cross-cluster is 800 ns to 1100 ns. Healthy, it's 55 ns to 100 ns within a cluster and 210 ns to 270 ns across. Others running DGX OS on this silicon report similar healthy numbers, so the fabric was never broken, just throttled. Seriously though, when I measured a microsecond for core-to-core latency, my jaw dropped open

One red herring that cost me some time (but got me thinking about power). This shows up in dmesg on every boot regardless of state:

mlx5_core ...: Detected insufficient power on the PCIe slot (27W).

It turns out, the slot advertises zero watts, so the driver complains. It's not the throttle indicator. Keep in mind that there is no physical "slot."

As for a permanent fix, I don't have one. The obvious thing to try is the v0105 package ASUS posted in August, which bundles new EC and PD firmware (EC 3.3.2.4, PD 5.22). My BIOS reports 0105 but it's a May build, and I never read my EC or PD versions, so I can't tell you whether I'm already on those or whether they help. And since I can't reproduce it, a box that stays healthy after the update isn't a confirmed kill.

Open questions I haven't answered:

• What actually clears it. Mine flipped once after a freeze and a power-button reset. Normal reboots and power-button resets don't do it. Never tried a full unplug.
• What sets it in the first place. I've only ever seen it already throttled.
• Whether it comes back. Hasn't yet, but I don't know the trigger.
• Adapter or cable. Plausible from other threads, didn't test it.
• Whether the August EC/PD firmware stops it. I don't even know what EC/PD I'm on.

If anyone has a repro case to make it throttle, has caught the cap being set, cleared it deliberately (especially with a plain unplug), or has a unit on the same firmware that never does this, I'd like to compare notes.

Lastly, if you think you are running into this problem, the easiest way to verify is perf bench mem memcpy -s 1GB. If you are in the low GB/s, your SoC is throttled.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox