[Docs] (docs/Performance): add a skill for reading Mali GPU counters

2026-11-07 01:53:24 +09:00 · 2026-11-01 19:51:20 +08:00
parent 05dfff9100
commit d3ec9b8179
4 changed files with 268 additions and 0 deletions
@@ -0,0 +1,158 @@
+---
+name: mali-gpu-counters
+description: Capture and compare Mali GPU hardware counters (fragment/vertex cycles, tiles, ZS, bandwidth, tile write elimination) on a rooted Android phone through perfetto and the vendor's libgpudataproducer, to find where one renderer spends more GPU time than another. Use when a GPU-bound workload is slower on Magma (DirectVulkan) than on the device's GLES driver, or slower after a change, and you need to know which part of the GPU pipeline the gap is in.
+---
+
+# Mali GPU performance counters
+
+Use this to answer "where does Magma spend the GPU time that the system GLES driver (or the previous build) doesn't?" on a Mali phone. Two captures, each normalized per frame, show which pipeline stage is responsible: vertex/tiling, fragment shading, depth testing, or memory bandwidth. Do this before guessing at shader or pass-structure fixes.
+
+The workflow was built and verified on a MediaTek MT6893 (Mali-G77 MC9, r32 driver, TrebleDroid). The vendor library path and counter IDs can differ on other devices; step 2 shows how to list them.
+
+## Prerequisites
+
+- A rooted phone (`adb root` or `su`). Check that `/vendor/lib64/libgpudataproducer.so` exists. This is the vendor library behind Android GPU Inspector's counters.
+- The workload must be **GPU-bound**. Check `/sys/kernel/ged/hal/gpu_utilization` on MediaTek (first field ≥ 90%). Otherwise counter differences say nothing about FPS.
+- Fixed clocks, so cycles are comparable across runs. On MT6893:
+  - GPU: `echo 836000 > /proc/gpufreq/gpufreq_opp_freq`.
+  - CPU: `echo 7 1 > /proc/ppm/policy_status`, then `echo <cluster> <kHz>` into `/proc/ppm/policy/userlimit_{min,max}_cpu_freq`.
+  - The sysfs `scaling_*_freq` files are read-only there.
+  - Restore everything afterwards.
+- Host: the NDK, `protoc`, and a venv with `pip install perfetto` (TraceProcessor downloads its shell binary on first use).
+
+## 1. Start the counter producer
+
+`gpuprod.c` dlopens the vendor library and calls its exported `start()`, which registers the perfetto data source `gpu.counters`.
+
+```sh
+$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang -O2 -o gpuprod gpuprod.c
+adb push gpuprod /data/local/tmp/ && adb shell chmod 755 /data/local/tmp/gpuprod
+adb shell 'su -c "getenforce"'          # note the mode to restore later
+adb shell 'su -c "setenforce 0; nohup /data/local/tmp/gpuprod > /data/local/tmp/gpuprod.log 2>&1 &"'
+adb shell perfetto --query | grep gpu.counters   # must list /data/local/tmp/gpuprod
+```
+
+It needs SELinux permissive. **Afterwards**, kill the producer (`su -c "kill $(pidof gpuprod)"`) and restore the mode (`setenforce 1` if it was Enforcing).
+
+## 2. List the counters this GPU offers
+
+A capture without `counter_ids` records nothing, and the IDs are device-specific. Read them from the producer's descriptor:
+
+```sh
+adb exec-out perfetto --query-raw > qraw.bin
+protoc --decode_raw < qraw.bin > qraw.txt
+python3 - <<'P'
+import re
+s = open('qraw.txt').read()
+for i, n in re.findall(r'\n\s*1: (\d+)\n\s*2: "([^"\n]{2,90})"\n\s*3: "', s):
+    print(i, n)
+P
+```
+
+IDs below 65535 are raw hardware counters. IDs 65535 and up are derived expressions (percentages, rates). Prefer raw counters: they add up over time, while ratios don't.
+
+On the Mali-G77 these raw IDs were the useful set:
+
+| ID | Counter | Use |
+|---|---|---|
+| 6 | GPU active cycles | total GPU cost |
+| 196 / 214 | Fragment / non-fragment active cycles | which side the gap is on |
+| 68 | Tiler active cycles | vertex + binning cost |
+| 160 / 175 | Output external read / write beats | DRAM bandwidth |
+| 210 | Tiles | how many tile passes the frame runs (≈ passes × screen tiles) |
+| 211 | Constant tiles killed | tile write-backs eliminated because the tile was unchanged |
+| 254 | Tile buffer write beats to L2 | tile write-back traffic |
+| 203 / 202 | Rasterized / partial rasterized quads | fragment workload size |
+| 204 / 206 | Early ZS tested / killed quads | early depth test working |
+| 207 / 208 | Late ZS tested / killed quads | shaders forced to late ZS (discard, depth writes, clears via shader) |
+| 212 | FPK occluder quads | forward pixel kill |
+| 234 | Texture quads | quads that sample textures |
+| 219 / 220 / 221 / 222 | FMA / CVT / SFU / message instructions | shader instruction mix |
+| 242 / 243 | 32-bit / 16-bit interpolation active | fp32 vs fp16 varyings |
+| 209 | Warps using more than 32 registers | register pressure |
+| 70 / 75 | Triangle / visible primitives | geometry identity check |
+| 85 / 101 | IDVS position / varying shading requests | vertex shading |
+
+`counters.cfg` captures this set: 5 s at a 20 ms sampling period.
+
+## 3. Capture each renderer
+
+Get each renderer into the same steady scene: same camera, frozen world time, same resolution. Then capture while it runs:
+
+```sh
+adb push counters.cfg /data/local/tmp/
+adb shell 'cat /data/local/tmp/counters.cfg | perfetto --txt -c - -o /data/misc/perfetto-traces/cnt.pftrace'
+adb pull /data/misc/perfetto-traces/cnt.pftrace A.pftrace
+```
+
+- Record each run's FPS during the capture (for example, the on-screen FPS counter in a screenshot taken right after).
+- Change only the renderer between captures.
+- Cool the phone between runs and keep the clocks locked.
+
+## 4. Compare per frame
+
+```sh
+python compare.py A.pftrace <fpsA> B.pftrace <fpsB>
+```
+
+It sums each counter over the capture, converts to per second, divides by FPS, and prints A/frame, B/frame and B/A, sorted by A's per-frame cost. Many values come out fractional, because the producer averages per-core counters across shader cores. Only the ratios matter.
+
+## 5. Read the result
+
+Work top-down:
+
+1. **GPU active cycles B/A**: the size of the gap. It should roughly match the FPS ratio when GPU-bound.
+2. **Fragment vs non-fragment active cycles.** Whichever ratio is above 1 holds the gap. Equal vertex counters (non-fragment, tiler, primitives, IDVS) mean the geometry and vertex shading are identical, so look only at fragment work.
+3. **Tiles.** B/A ≈ 1 + k means B renders about k extra full-screen passes per frame. Typical causes:
+   - a standalone clear pass;
+   - a pass split by a mid-frame upload or barrier;
+   - a blit, copy or clear outside a render pass (Mali runs those as fragment jobs too).
+
+   Find them with a per-frame Vulkan command census (see [Census of Magma's Vulkan commands](#census-of-magmas-vulkan-commands)).
+4. **Rasterized quads, late ZS, texture quads.**
+   - With equal tiles: more rasterized quads mean extra full-screen draws inside a pass. For example, `vkCmdClearAttachments` mid-pass costs about one screen of quads each time.
+   - High late-ZS counts mean fragments lost the early depth test: shaders with `discard`, `gl_FragDepth` or `gl_SampleMask` writes, or clears implemented as shader draws.
+   - More early-ZS kills with equal visible primitives often mean the same thing seen from the other side.
+5. **External write beats vs tile buffer write beats.**
+   - If B's extra external writes ≈ extra tile write beats × shader-core count, the difference is tile write-back.
+   - Check `Constant tiles killed`: the GLES driver eliminates write-back of unchanged tiles. A Vulkan path with a much lower count is losing that elimination.
+   - In a static benchmark scene this is a big effect. With a moving camera it is a small one.
+6. **Instruction mix and interpolation.** More FMA or CVT instructions with equal quads point at shader code quality. Much more 32-bit than 16-bit interpolation points at precision: for example, ESSL `mediump` lost when a backend recompiles the shader as desktop GLSL.
+
+## Census of Magma's Vulkan commands
+
+When the tile count says Magma runs extra passes, find out which ones. Use a temporary, never-committed build:
+
+- **Wrappers:** write a header that declares counting wrappers and maps each `vkCmd*` entry point to its wrapper with a function-like macro (`#define vkCmdBeginRenderPass(...) Census_BeginRenderPass(__VA_ARGS__)`). Function-like macros leave function-pointer assignments such as `fns.vkCmdBeginRenderPass = vkCmdBeginRenderPass;` alone.
+  - Include `VK_USE_PLATFORM_ANDROID_KHR` before `<vulkan/vulkan.h>` in that header.
+  - Map at least render pass begin, render pass creation (to record each pass's load/store ops), blit, copy, clear (including `vkCmdClearAttachments`), resolve, dispatch, barrier, submit and present.
+- **Force-include:** apply it only to the DirectVulkan sources:
+
+  ```cmake
+  file(GLOB _census ${CMAKE_CURRENT_SOURCE_DIR}/MobileGL/MG_Backend/DirectVulkan/*.cpp
+                    ${CMAKE_CURRENT_SOURCE_DIR}/MobileGL/MG_Backend/DirectVulkan/Renderer/*.cpp)
+  list(FILTER _census EXCLUDE REGEX "Census")   # the wrapper definitions must call the real entry points
+  set_source_files_properties(${_census} PROPERTIES COMPILE_OPTIONS "-include;<path>/Census.h")
+  ```
+
+- **Recording:** in each wrapper, key the count by kind, the caller's offset in the library (`__builtin_return_address(0)` minus `dladdr`'s `dli_fbase`) and details: render area plus load/store ops for passes, area for clears and blits. In the present wrapper, every few seconds, log the counts divided by the number of frames to logcat.
+- **Symbolize** the offsets against the unstripped library: `llvm-addr2line -f -C -i -e build/intermediates/cxx/RelWithDebInfo/*/obj/arm64-v8a/libMobileGL.so 0x<offset>`. The library under `stripped_native_libs` has no symbols. If a wrapper's caller is a shared helper, log `__builtin_return_address(1..5)` inside that helper; arm64 Android keeps frame pointers.
+
+## Example: Magma vs the system driver
+
+Minecraft 1.12.2 with OptiFine at 2400×1080 on a Mali-G77, clocks locked, SFPEW over each backend:
+
+| Finding | Counters | Cause | Fix |
+|---|---|---|---|
+| Fragment active cycles 1.33×, tiles 1.46× | rasterized quads, late ZS and tiles all up; vertex counters equal | one extra full-screen depth-only pass per frame (census: `vkCmdBeginRenderPass` with only `D24S8` LOAD/STORE, plus a full-screen `vkCmdClearAttachments`) | a window clear with a window pass open now ends the pass and folds into the next one (`VulkanRenderer::DeferDefaultWireClear`) |
+| Extra external writes 1.7× | tile buffer write beats ≈ 23×, constant tiles killed 0.07× | the GLES driver eliminates the write-back of unchanged tiles in a static scene; Magma's attachments get none | still open |
+| Fragment cycles still 1.18× with equal tiles | rasterized quads 1.26×, late ZS 1.44× | the game's two in-pass depth clears per frame, recorded as full-screen `vkCmdClearAttachments` | under investigation |
+
+Separately, SFPEW hands MobileGL ESSL with `precision mediump float;` inside `#ifdef GL_ES`. The source processor rewrites ESSL to desktop GLSL with `GL_ES` undefined, so that guard silently turned the shaders fp32. 16-bit interpolation counters and the RelaxedPrecision decorations in the SPIR-V show this kind of loss.
+
+## Pitfalls
+
+- **Unlocked clocks.** Cycle counts are only comparable at the same GPU frequency, and FPS also depends on the CPU clock when the workload is not fully GPU-bound.
+- **Not checking the build.** Before comparing builds, verify that each run actually ran the build you think it did. `adb install -r` of an APK with a lower versionCode fails with `INSTALL_FAILED_VERSION_DOWNGRADE`, and the run silently uses whatever was installed.
+- **Different scenes.** Counters from different scenes or camera angles are not comparable at all. Freeze world time and camera position between runs.
+- **Leaving the phone modified.** The producer and permissive SELinux stay in place until you undo them. Restore both, and the clock locks, when you are done.
@@ -0,0 +1,34 @@
+"""Per-frame comparison of two Mali counter captures.
+
+usage: python compare.py A.pftrace FPS_A B.pftrace FPS_B
+Needs `pip install perfetto`.
+"""
+import sys
+
+from perfetto.trace_processor import TraceProcessor
+
+
+def per_second(path):
+    tp = TraceProcessor(trace=path)
+    rows = tp.query(
+        "select t.name, sum(c.value) s, (max(c.ts) - min(c.ts)) / 1e9 dur "
+        "from gpu_counter_track t join counter c on c.track_id = t.id group by t.id")
+    out = {r.name: r.s / r.dur for r in rows if r.dur > 0}
+    tp.close()
+    return out
+
+
+def main():
+    if len(sys.argv) != 5:
+        sys.exit(__doc__)
+    a, fps_a, b, fps_b = sys.argv[1], float(sys.argv[2]), sys.argv[3], float(sys.argv[4])
+    A, B = per_second(a), per_second(b)
+    print(f"{'counter':52s} {'A/frame':>14s} {'B/frame':>14s} {'B/A':>6s}")
+    for name in sorted(A, key=lambda k: -A[k] / fps_a):
+        pa, pb = A[name] / fps_a, B.get(name, 0.0) / fps_b
+        ratio = f"{pb / pa:6.2f}" if pa else "   n/a"
+        print(f"{name:52s} {pa:14.0f} {pb:14.0f} {ratio}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,50 @@
+# Mali-G77 raw counters (IDs from step 2 of SKILL.md); 5 s at a 20 ms period.
+buffers { size_kb: 131072 fill_policy: DISCARD }
+data_sources { config { name: "gpu.counters" gpu_counter_config { counter_period_ns: 20000000
+  counter_ids: 6
+  counter_ids: 10
+  counter_ids: 18
+  counter_ids: 196
+  counter_ids: 214
+  counter_ids: 68
+  counter_ids: 160
+  counter_ids: 175
+  counter_ids: 218
+  counter_ids: 219
+  counter_ids: 220
+  counter_ids: 221
+  counter_ids: 222
+  counter_ids: 245
+  counter_ids: 201
+  counter_ids: 216
+  counter_ids: 203
+  counter_ids: 202
+  counter_ids: 204
+  counter_ids: 206
+  counter_ids: 207
+  counter_ids: 208
+  counter_ids: 212
+  counter_ids: 210
+  counter_ids: 211
+  counter_ids: 254
+  counter_ids: 250
+  counter_ids: 251
+  counter_ids: 231
+  counter_ids: 234
+  counter_ids: 241
+  counter_ids: 242
+  counter_ids: 243
+  counter_ids: 247
+  counter_ids: 248
+  counter_ids: 249
+  counter_ids: 70
+  counter_ids: 75
+  counter_ids: 85
+  counter_ids: 101
+  counter_ids: 209
+  counter_ids: 8
+  counter_ids: 16
+  counter_ids: 226
+  counter_ids: 223
+} } }
+duration_ms: 5000
@@ -0,0 +1,26 @@
+/*
+ * Starts the vendor Mali counter producer so perfetto can record the
+ * "gpu.counters" data source. Run as root, with SELinux permissive.
+ *
+ * Build: aarch64-linux-android30-clang -O2 -o gpuprod gpuprod.c
+ */
+#include <dlfcn.h>
+#include <stdio.h>
+#include <unistd.h>
+
+int main(void) {
+    void* lib = dlopen("/vendor/lib64/libgpudataproducer.so", RTLD_NOW);
+    if (!lib) {
+        fprintf(stderr, "dlopen: %s\n", dlerror());
+        return 1;
+    }
+    void (*start)(void) = (void (*)(void))dlsym(lib, "start");
+    if (!start) {
+        fprintf(stderr, "libgpudataproducer.so exports no start()\n");
+        return 1;
+    }
+    fprintf(stderr, "starting producer\n");
+    start();
+    for (;;)
+        pause();
+}