MobileGL/tools/device_bench/p6/README.md

device_bench/p6 — the device A/B session runner

Two scripts that run a paired performance comparison on a real device and turn its evidence into markdown tables. They exist because the quantity a disaggregation A/B reads on (tenths of a millisecond of client CPU per frame) is smaller than the drift a hand-driven session accumulates between the CPU samples, the pin evidence and the library's own counters — and those three come from three different mechanisms. Collected by hand they are minutes apart; collected by one process they are one window.

file	role
ab_session.py	collects: reboot, pin, launch each arm in order, sample per-thread CPU, pull role logs, verify the arm
render_ab.py	renders: session-summary.json → the markdown tables a report needs

The split is deliberate. The renderer reads only what the runner recorded, so it cannot influence what was collected.

Usage

# One arm is TRANSPORT[:BACKEND[:CASE[:REPEATS]]]; repeat a spec to interleave it.
python tools/device_bench/p6/ab_session.py \
  --tree  /c/Users/me/mobilegl-snapshot \
  --out   /c/Users/me/evidence \
  --apk   /c/Users/me/evidence/trace-debug-signed.apk \
  --package top.mobilegl.plugin.trace \
  --reboot --fan-level 2 --repeats 3 --session my-session \
  --arm monolith:DirectGLES:minecraft-1.21.4-rd12-odinlite-in-world:3 \
  --arm inproc:DirectGLES:minecraft-1.21.4-rd12-odinlite-in-world:3 \
  --arm spawn:DirectGLES:minecraft-1.21.4-rd12-odinlite-in-world:3

python tools/device_bench/p6/render_ab.py /c/Users/me/evidence/my-session/session-summary.json

--tree is the tree the APK and the runner come from. Point it at a snapshot, not the shared tree, when anything else is editing the repository: a build that picks up a half-written source file measures nothing, and the failure is invisible in the result. --apk must be signed with a key the device already trusts for that package — the Gradle release build has no signing config locally (SIGNING_* are CI secrets), so it has to be signed by hand first:

apksigner sign --ks ~/.android/debug.keystore --ks-pass pass:android \
  --ks-key-alias androiddebugkey --key-pass pass:android --out signed.apk unsigned.apk

--package must match the APK's application id. A development build usually needs a suffix (-Pmobilegl.applicationIdSuffix=.mine) so it can sit beside an existing install rather than replacing it: two APKs signed by different keys cannot share an id, and adb install -r fails with INSTALL_FAILED_UPDATE_INCOMPATIBLE. The runner then needs MOBILEGL_TRACE_PACKAGE set to the same id (see run_android_retrace_local.py's note beside BACKENDS), which ab_session.py does itself.

What the protocol guarantees

The whole point of the script is that these are not optional:

• Reboot-clean. --reboot reboots, waits for sys.boot_completed, and compares the boot id before and after. An unchanged id means no reboot happened, which is reported as a failure rather than recorded as a clean window.
• Pinned frequencies, with evidence. Every arm runs pin_device.sh <serial> pin before and ... check after; a non-PINNED exit voids the arm (status: UNPROVEN), it does not degrade it. Un-pinned per-thread CPU is not comparable across arms.
• One thermal window. All arms in one session, interleaved by the order the --arm flags are given, with a cool-down between them so a later arm is not measured hotter than an earlier one.
• Arm proof. Each arm is checked against the library's OWN private log: the transport marker, the split knobs in the Config: IPC line, and for spawn the spawn ARMED - the server role runs in pid N line — which only the client writes after launch, connect and handshake all succeeded. A spawn lane that silently fell back to monolith passes every other check and fails this one.

Evidence layout

<out>/<session>/
  session-summary.json          # session metadata: boot ids, fan, APK hash, per-arm summaries
  <NN>-<case>-<backend>-<transport>-r<N>/
    arm-summary.json            # this arm: status, problems, cpu, stats, benchmark runs
    pin-before.txt pin-after.txt
    mobilegl.client.log mobilegl.server.log   # per role
    benchmark-run1..N.json      # per-frame wall + CPU series per repeat
    benchmark.json result.json logcat.txt runner.stdout.txt runner.stderr.txt

Two asymmetries of the spawn shape that the renderer handles

Both are structural — the client and the server really are different processes — and both produce a zero that looks like a measurement if read carelessly:

1. srv/srvpark live in the server log. The apply thread publishes them into its own process's PipeStats. Under inproc both roles are threads of one process, so the client's summary line carries all four numbers; under spawn the client prints srv=0 srvpark=0, which means "another process", not "never waited". render_ab.py reads the server pair from mobilegl.server.log.

2. The frame count lives in the server log too. frames= is bumped by PipeStats::OnPresent, which the backend calls on Present — and under spawn present is applied in the server process, so the client's frames=0 window=0. That zero is the same shape as srv=0. The per-frame rates therefore divide by whichever log counted the frames, and the rendered table names which one that was.

Building the APK

gradle --no-daemon -p <tree>/android-plugin :app:assembleTraceRelease \
  -Pmobilegl.buildDisaggregated=ON -Pmobilegl.buildDisaggregatedInproc=ON \
  -Pmobilegl.pipePush=ON -Pmobilegl.debuggableRelease=true \
  -Pmobilegl.applicationIdSuffix=.mine -Pmobilegl.apkSuffix=mine

-Pmobilegl.debuggableRelease=true is not optional: without it the release APK is not run-as-able, trace-replay-ci.sh cannot pull the private logs, and every arm produces no library log — which is indistinguishable from a run that produced none.

Pitfalls this tooling works around

• --transport is ignored in --benchmark mode. run_android_retrace_local.py applies --transport only on its run_case path; run_benchmark_case passes args.env alone. So --benchmark --transport spawn runs monolith under a spawn label. This script passes the transport as an explicit --env MOBILEGL_TRANSPORT=… and verifies it from the library's log.
• MOBILEGL_RETRACE_USE_PBUFFER must be in the CI script's environment, not passed as --env — trace-replay-ci.sh reads it itself to decide the surface shape. Without it the spawn arm dies with Fatal{UnmigratedSurface, "AndroidNativeWindow:use-MOBILEGL_IPC_SURFACE=server"}, because an ANativeWindow* means nothing in the server's process (Rule H).
• MOBILEGL_TRACE_SKIP_INSTALL=1 keeps the script's own adb install out of the window. The session installs once itself and waits for dex2oat to exit first: an am_kill … due to installPackageLI landing inside an arm measures the installer.
• A stale benchmark.json is worse than none. The runner's result directory is keyed by case and backend, not by arm, so a failed arm can leave the previous arm's file where read_benchmark will pick it up. The directory is deleted before each arm.

Known limit

Under spawn, the client's windowed fields (frames=, window=, draws/f, wrec/f, bytes/f[…]) are structurally zero: PipeStats::OnPresent runs in the server process (the apply thread applies the present record), so the spawn client never counts frames. Dividing any counter by the client's frames= is undefined. The renderer therefore takes the frame denominator from the log that actually counted frames (normally the server's), and names the source in the table.

What is valid from the client log is the run-total gauges (cli, clipark, maxrec, ringwraps, …): PublishGauge is a store, not an add, so the client's terminal Shutdown() line reports true run totals. Symmetrically, the server log's cli=0 and the client log's srv=0 mean "the other process", not "never waited".

The server's own byte/record/wait classes are available since PipeStats::Init() runs in ServerMain.cpp (P6 closeout; before that the server log carried no summary line at all and srv had to be reported as unavailable).