MobileGL/docs/Performance/mali-gpu-counters/SKILL.md

name, description

name

description

mali-gpu-counters

Capture and compare Mali GPU hardware counters (fragment/vertex cycles, tiles, ZS, bandwidth, tile write elimination) on a rooted Android phone through perfetto and the vendor's libgpudataproducer, to find where one renderer spends more GPU time than another. Use when a GPU-bound workload is slower on Magma (DirectVulkan) than on the device's GLES driver, or slower after a change, and you need to know which part of the GPU pipeline the gap is in.

Mali GPU performance counters

Use this to answer "where does Magma spend the GPU time that the system GLES driver (or the previous build) doesn't?" on a Mali phone. Two captures, each normalized per frame, show which pipeline stage is responsible: vertex/tiling, fragment shading, depth testing, or memory bandwidth. Do this before guessing at shader or pass-structure fixes.

The workflow was built and verified on a MediaTek MT6893 (Mali-G77 MC9, r32 driver, TrebleDroid). The vendor library path and counter IDs can differ on other devices; step 2 shows how to list them.

Prerequisites

• A rooted phone (adb root or su). Check that /vendor/lib64/libgpudataproducer.so exists. This is the vendor library behind Android GPU Inspector's counters.
• The workload must be GPU-bound. Check /sys/kernel/ged/hal/gpu_utilization on MediaTek (first field ≥ 90%). Otherwise counter differences say nothing about FPS.
• Fixed clocks, so cycles are comparable across runs. On MT6893:
  • GPU: echo 836000 > /proc/gpufreq/gpufreq_opp_freq.
  • CPU: echo 7 1 > /proc/ppm/policy_status, then echo <cluster> <kHz> into /proc/ppm/policy/userlimit_{min,max}_cpu_freq.
  • The sysfs scaling_*_freq files are read-only there.
  • Restore everything afterwards.
• Host: the NDK, protoc, and a venv with pip install perfetto (TraceProcessor downloads its shell binary on first use).

1. Start the counter producer

gpuprod.c dlopens the vendor library and calls its exported start(), which registers the perfetto data source gpu.counters.

$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang -O2 -o gpuprod gpuprod.c
adb push gpuprod /data/local/tmp/ && adb shell chmod 755 /data/local/tmp/gpuprod
adb shell 'su -c "getenforce"'          # note the mode to restore later
adb shell 'su -c "setenforce 0; nohup /data/local/tmp/gpuprod > /data/local/tmp/gpuprod.log 2>&1 &"'
adb shell perfetto --query | grep gpu.counters   # must list /data/local/tmp/gpuprod

It needs SELinux permissive. Afterwards, kill the producer (su -c "kill $(pidof gpuprod)") and restore the mode (setenforce 1 if it was Enforcing).

2. List the counters this GPU offers

A capture without counter_ids records nothing, and the IDs are device-specific. Read them from the producer's descriptor:

adb exec-out perfetto --query-raw > qraw.bin
protoc --decode_raw < qraw.bin > qraw.txt
python3 - <<'P'
import re
s = open('qraw.txt').read()
for i, n in re.findall(r'\n\s*1: (\d+)\n\s*2: "([^"\n]{2,90})"\n\s*3: "', s):
    print(i, n)
P

IDs below 65535 are raw hardware counters. IDs 65535 and up are derived expressions (percentages, rates). Prefer raw counters: they add up over time, while ratios don't.

On the Mali-G77 these raw IDs were the useful set:

ID	Counter	Use
6	GPU active cycles	total GPU cost
196 / 214	Fragment / non-fragment active cycles	which side the gap is on
68	Tiler active cycles	vertex + binning cost
160 / 175	Output external read / write beats	DRAM bandwidth
210	Tiles	how many tile passes the frame runs (≈ passes × screen tiles)
211	Constant tiles killed	tile write-backs eliminated because the tile was unchanged
254	Tile buffer write beats to L2	tile write-back traffic
203 / 202	Rasterized / partial rasterized quads	fragment workload size
204 / 206	Early ZS tested / killed quads	early depth test working
207 / 208	Late ZS tested / killed quads	shaders forced to late ZS (discard, depth writes, clears via shader)
212	FPK occluder quads	forward pixel kill
234	Texture quads	quads that sample textures
219 / 220 / 221 / 222	FMA / CVT / SFU / message instructions	shader instruction mix
242 / 243	32-bit / 16-bit interpolation active	fp32 vs fp16 varyings
209	Warps using more than 32 registers	register pressure
70 / 75	Triangle / visible primitives	geometry identity check
85 / 101	IDVS position / varying shading requests	vertex shading

counters.cfg captures this set: 5 s at a 20 ms sampling period.

3. Capture each renderer

Get each renderer into the same steady scene: same camera, frozen world time, same resolution. Then capture while it runs:

adb push counters.cfg /data/local/tmp/
adb shell 'cat /data/local/tmp/counters.cfg | perfetto --txt -c - -o /data/misc/perfetto-traces/cnt.pftrace'
adb pull /data/misc/perfetto-traces/cnt.pftrace A.pftrace

• Record each run's FPS during the capture (for example, the on-screen FPS counter in a screenshot taken right after).
• Change only the renderer between captures.
• Cool the phone between runs and keep the clocks locked.

4. Compare per frame

python compare.py A.pftrace <fpsA> B.pftrace <fpsB>

It sums each counter over the capture, converts to per second, divides by FPS, and prints A/frame, B/frame and B/A, sorted by A's per-frame cost. Many values come out fractional, because the producer averages per-core counters across shader cores. Only the ratios matter.

5. Read the result

Work top-down:

1. GPU active cycles B/A: the size of the gap. It should roughly match the FPS ratio when GPU-bound.

2. Fragment vs non-fragment active cycles. Whichever ratio is above 1 holds the gap. Equal vertex counters (non-fragment, tiler, primitives, IDVS) mean the geometry and vertex shading are identical, so look only at fragment work.

3. Tiles. B/A ≈ 1 + k means B renders about k extra full-screen passes per frame. Typical causes:

   • a standalone clear pass;
   • a pass split by a mid-frame upload or barrier;
   • a blit, copy or clear outside a render pass (Mali runs those as fragment jobs too).

   Find them with a per-frame Vulkan command census (see Census of Magma's Vulkan commands).

4. Rasterized quads, late ZS, texture quads.

   • With equal tiles: more rasterized quads mean extra full-screen draws inside a pass. For example, vkCmdClearAttachments mid-pass costs about one screen of quads each time.
   • High late-ZS counts mean fragments lost the early depth test: shaders with discard, gl_FragDepth or gl_SampleMask writes, or clears implemented as shader draws.
   • More early-ZS kills with equal visible primitives often mean the same thing seen from the other side.

5. External write beats vs tile buffer write beats.

   • If B's extra external writes ≈ extra tile write beats × shader-core count, the difference is tile write-back.
   • Check Constant tiles killed: the GLES driver eliminates write-back of unchanged tiles. A Vulkan path with a much lower count is losing that elimination.
   • In a static benchmark scene this is a big effect. With a moving camera it is a small one.

6. Instruction mix and interpolation. More FMA or CVT instructions with equal quads point at shader code quality. Much more 32-bit than 16-bit interpolation points at precision: for example, ESSL mediump lost when a backend recompiles the shader as desktop GLSL.

Census of Magma's Vulkan commands

When the tile count says Magma runs extra passes, find out which ones. Use a temporary, never-committed build:

• Wrappers: write a header that declares counting wrappers and maps each vkCmd* entry point to its wrapper with a function-like macro (#define vkCmdBeginRenderPass(...) Census_BeginRenderPass(__VA_ARGS__)). Function-like macros leave function-pointer assignments such as fns.vkCmdBeginRenderPass = vkCmdBeginRenderPass; alone.

  • Include VK_USE_PLATFORM_ANDROID_KHR before <vulkan/vulkan.h> in that header.
  • Map at least render pass begin, render pass creation (to record each pass's load/store ops), blit, copy, clear (including vkCmdClearAttachments), resolve, dispatch, barrier, submit and present.

• Force-include: apply it only to the DirectVulkan sources:

  file(GLOB _census ${CMAKE_CURRENT_SOURCE_DIR}/MobileGL/MG_Backend/DirectVulkan/*.cpp
                    ${CMAKE_CURRENT_SOURCE_DIR}/MobileGL/MG_Backend/DirectVulkan/Renderer/*.cpp)
  list(FILTER _census EXCLUDE REGEX "Census")   # the wrapper definitions must call the real entry points
  set_source_files_properties(${_census} PROPERTIES COMPILE_OPTIONS "-include;<path>/Census.h")
  

• Recording: in each wrapper, key the count by kind, the caller's offset in the library (__builtin_return_address(0) minus dladdr's dli_fbase) and details: render area plus load/store ops for passes, area for clears and blits. In the present wrapper, every few seconds, log the counts divided by the number of frames to logcat.

• Symbolize the offsets against the unstripped library: llvm-addr2line -f -C -i -e build/intermediates/cxx/RelWithDebInfo/*/obj/arm64-v8a/libMobileGL.so 0x<offset>. The library under stripped_native_libs has no symbols. If a wrapper's caller is a shared helper, log __builtin_return_address(1..5) inside that helper; arm64 Android keeps frame pointers.

Example: Magma vs the system driver

Minecraft 1.12.2 with OptiFine at 2400×1080 on a Mali-G77, clocks locked, SFPEW over each backend:

Finding	Counters	Cause	Fix
Fragment active cycles 1.33×, tiles 1.46×	rasterized quads, late ZS and tiles all up; vertex counters equal	one extra full-screen depth-only pass per frame (census: vkCmdBeginRenderPass with only D24S8 LOAD/STORE, plus a full-screen vkCmdClearAttachments)	a window clear with a window pass open now ends the pass and folds into the next one (VulkanRenderer::DeferDefaultWireClear)
Extra external writes 1.7×	tile buffer write beats ≈ 23×, constant tiles killed 0.07×	the GLES driver eliminates the write-back of unchanged tiles in a static scene; Magma's attachments get none	still open
Fragment cycles still 1.18× with equal tiles	rasterized quads 1.26×, late ZS 1.44×	the game's two in-pass depth clears per frame, recorded as full-screen vkCmdClearAttachments	under investigation

Separately, SFPEW hands MobileGL ESSL with precision mediump float; inside #ifdef GL_ES. The source processor rewrites ESSL to desktop GLSL with GL_ES undefined, so that guard silently turned the shaders fp32. 16-bit interpolation counters and the RelaxedPrecision decorations in the SPIR-V show this kind of loss.

Pitfalls

• Unlocked clocks. Cycle counts are only comparable at the same GPU frequency, and FPS also depends on the CPU clock when the workload is not fully GPU-bound.
• Not checking the build. Before comparing builds, verify that each run actually ran the build you think it did. adb install -r of an APK with a lower versionCode fails with INSTALL_FAILED_VERSION_DOWNGRADE, and the run silently uses whatever was installed.
• Different scenes. Counters from different scenes or camera angles are not comparable at all. Freeze world time and camera position between runs.
• Leaving the phone modified. The producer and permissive SELinux stay in place until you undo them. Restore both, and the clock locks, when you are done.