mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-07 18:13:23 +09:00
632 lines
38 KiB
Markdown
632 lines
38 KiB
Markdown
# MobileGL trace replay
|
||
|
||
This directory builds a Linux command line replay runner for [apitrace](https://github.com/apitrace/apitrace) files. It
|
||
is an integration testing infrastructure of MobileGL.
|
||
|
||
The bundled fixtures cover:
|
||
|
||
- OpenRA: sourced from GL4ES' apitrace corpus.
|
||
|
||
- minecraft-1.21.4-startup: captured from Minecraft 1.21.4's startup screen.
|
||
|
||
- minecraft-1.21.4-main-menu: captured from Minecraft 1.21.4's main menu.
|
||
|
||
- minecraft-1.21.11-main-menu: captured from Minecraft 1.21.11's main menu on a Pixel 8 Pro through FCL MobileGL.
|
||
|
||
- minecraft-1.17-main-menu-854: captured from Minecraft 1.17's 854x480 main menu through FCL MobileGL capture.
|
||
|
||
- minecraft-1.21.4-in-world: captured from Minecraft 1.21.4 after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-rd12-odinlite-in-world: captured on an Android device (FCL MobileGL Magma capture, 854x480) from
|
||
vanilla Minecraft 1.21.4 at render distance 12, drifting down a river valley in a boat. Unlike the other in-world
|
||
fixtures this one is a 251-frame window (gltrim `-f 1094-1343`) rather than a single frame, so it doubles as the
|
||
campaign's benchmark scene: benchmark mode replays the whole window and the tail frames measure steady-state
|
||
in-world frame time. The golden is still the final frame, so it works as an ordinary correctness case too.
|
||
|
||
- minecraft-1.21.4-fabric-sodium-in-world: captured from Minecraft 1.21.4 Fabric with Sodium after entering a
|
||
singleplayer world with Fancy graphics.
|
||
|
||
- improved-transparency-minecraft-26.3: captured from the Minecraft 26.3 improved-transparency scene.
|
||
|
||
- minecraft-1.21.4-fabric-common-mods-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, REI,
|
||
Xaero's Minimap, Xaero's World Map, JourneyMap, and Modern UI, with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-common-mods-inventory: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, REI,
|
||
Xaero's Minimap, Xaero's World Map, JourneyMap, and Modern UI with the creative inventory and REI item list open.
|
||
|
||
- minecraft-1.21.4-fabric-rei-inventory: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and REI, with
|
||
shader packs disabled and the creative inventory and REI item list open.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-minimap-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Xaero's Minimap after entering a singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-world-map-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Xaero's World Map, with shader packs disabled and the world map screen open.
|
||
|
||
- minecraft-1.21.4-fabric-journeymap-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
JourneyMap after entering a singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-modernui-inventory: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and Modern UI,
|
||
with shader packs disabled and the creative inventory open.
|
||
|
||
- minecraft-1.21.4-fabric-rei-inventory-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
REI in a normal singleplayer world, with shader packs disabled and the creative inventory and REI item list open.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-minimap-in-world-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Xaero's Minimap after entering a normal singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-world-map-in-world-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Xaero's World Map in a normal singleplayer world, with shader packs disabled and the world map screen open.
|
||
|
||
- minecraft-1.21.4-fabric-journeymap-in-world-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris,
|
||
and JourneyMap after entering a normal singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-modernui-inventory-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris,
|
||
and Modern UI in a normal singleplayer world, with shader packs disabled and the creative inventory open.
|
||
|
||
- minecraft-1.21.4-fabric-iris-bsl-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and BSL
|
||
Shaders after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-bsl-esc-menu-854: captured on an Android device (Mali-G77, FCL MobileGL capture) from
|
||
Minecraft 1.21.4 Fabric with Sodium, Iris, and BSL Shaders, at the pause menu over a BSL-blurred world. The frame
|
||
pins glyph rendering: every menu label, the menu title and the tutorial toast must be present. Regressions in the
|
||
DirectGLES per-draw texture memo have made the whole text path disappear here while sprites kept rendering, so a
|
||
failure that leaves the buttons but empties them is the signature to look for in the diff.
|
||
|
||
- minecraft-1.21.4-fabric-iris-makeup-ultrafast-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
MakeUP UltraFast after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-super-duper-vanilla-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris,
|
||
and Super Duper Vanilla after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-complementary-reimagined-in-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Complementary Reimagined after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-complementary-unbound-in-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Complementary Unbound after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-mellow-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and Mellow
|
||
after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-nostalgia-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Nostalgia after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-bliss-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and Bliss
|
||
after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-chocapic-v6-lite-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Chocapic V6 Lite after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-iterationt-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
iterationT after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-iterationt-nodsa-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
iterationT after entering a singleplayer world, with Iris' DSA path disabled.
|
||
|
||
- minecraft-1.21.4-fabric-iris-iterationrp-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
iterationRP after entering a singleplayer world, framing the iterationRP name overlay over a lake with far-shore
|
||
tree reflections. iterationRP's temporal auto-exposure makes a single-frame trim overexpose and drop the overlay,
|
||
so the fixture is a prefix trace (all calls up to the target frame) that replays the temporal state. The pack also
|
||
gates an NVIDIA-only shadow path (`subgroupPartitionNV`, `GL_NV_shader_subgroup_partitioned`) on the GL vendor
|
||
string, so the capture reports a masked vendor and the trace carries the portable `subgroupShuffleXor` path that
|
||
non-NVIDIA GPUs take.
|
||
The trace archive and golden are not committed yet (the repository's Git LFS quota rejects new objects with
|
||
`GH009`); the case stays registered and its fixture files are hydrated from the trace fixture mirror.
|
||
- minecraft-1.21.4-fabric-iris-photon-v1.1-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Photon v1.1 after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Photon v1.3b after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-derivative-main-d24.4.14-in-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Derivative Main d24.4.14 after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-sundial-lite-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Sundial Lite after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.1-neoforge-create-indirect-in-world-align1024: **not an original capture** - derived from the retired
|
||
`minecraft-1.21.1-neoforge-create-indirect-in-world` capture (Minecraft 1.21.1 NeoForge with Create, Sodium, and
|
||
Iris (no shader pack) in a world facing Create water wheels and a large cogwheel, with Flywheel's
|
||
`flywheel:indirect` backend: compute-shader culling, glMultiDrawElementsIndirect, persistent-mapped staging). That
|
||
capture came from a desktop driver whose `GL_SHADER_STORAGE_BUFFER_OFFSET_ALIGNMENT` is 16, so Flywheel bound its
|
||
scatter lists at 16-byte offsets, and a driver that requires more (Adreno 830: 64) rejects every one of those binds
|
||
with `GL_INVALID_VALUE`. [`fixture_tools/realign_buffer_ranges.cpp`](fixture_tools/realign_buffer_ranges.cpp)
|
||
`--align 1024` rewrote it: each of the 1,270 range binds off a 1024-byte boundary now binds a 1024-aligned slot of
|
||
an added buffer that is filled with the same bytes right before the dispatch that reads them. The 3,812 inserted
|
||
calls renumber the trace (target call 504631 -> 508443); the golden is the original one (the same frame), and the
|
||
host replay matches it with the original's SSIM to 9 digits on both backends.
|
||
|
||
- minecraft-1.21.1-neoforge-create-instancing-in-world: same world and camera as the indirect case, with Flywheel's
|
||
`flywheel:instancing` backend (texture-buffer instance data, glDrawElementsInstancedBaseVertex).
|
||
|
||
|
||
Build from the MobileGL repository root:
|
||
|
||
```sh
|
||
cmake -S . -B build-test -G Ninja \
|
||
-DMOBILEGL_BUILD_TEST=ON \
|
||
-DMOBILEGL_BUILD_BENCHMARK=OFF \
|
||
-DMOBILEGL_BUILD_TRACE_REPLAY=ON
|
||
cmake --build build-test
|
||
```
|
||
|
||
Run the fixture tests:
|
||
|
||
```sh
|
||
ctest --test-dir build-test -V -R 'MobileGLTraceReplay\.'
|
||
```
|
||
|
||
Run the CLI directly:
|
||
|
||
```sh
|
||
build-test/tools/trace_replay/mobilegl_trace_replay \
|
||
--trace openra.trace \
|
||
--golden openra.0000031249.png \
|
||
--output out/openra \
|
||
--backend DirectGLES \
|
||
--mobilegl-library build-test/libMobileGL.so \
|
||
--target-call 31249 \
|
||
--width 640 \
|
||
--height 480 \
|
||
--crop-x 1 \
|
||
--crop-y 1 \
|
||
--crop-width 638 \
|
||
--crop-height 478 \
|
||
--ssim-threshold 0.99
|
||
```
|
||
|
||
## Dumping framebuffer attachments mid-frame
|
||
|
||
`--target-call` snapshots one framebuffer. To see *inside* a frame - which
|
||
intermediate render target a pass actually produced - pass
|
||
`--dump-fbo-attachments CALL:DIR[:FBO,FBO,...]`, repeatably:
|
||
|
||
```sh
|
||
build-test/tools/trace_replay/mobilegl_trace_replay \
|
||
--trace trace.trace --golden golden.png --output out --target-call 2667619 \
|
||
--dump-fbo-attachments 2666231:out/fbos-before \
|
||
--dump-fbo-attachments 2666232:out/fbos-after
|
||
```
|
||
|
||
At each call boundary it walks every live framebuffer object (or only the named
|
||
ones), reads back every colour attachment and the depth attachment, and writes
|
||
`fbo<N>-att<M>.png` / `fbo<N>-depth.png` plus a `manifest.txt` line per
|
||
attachment recording the attached object, size, internal format, component type
|
||
and per-channel min/max/mean and a content hash. Attachments are read as floats
|
||
whatever their storage, so HDR accumulation buffers stay legible in the
|
||
statistics even though the PNG has to clamp.
|
||
|
||
The manifest is the useful part when comparing two drivers: dump the same call
|
||
on both stacks and `diff`/`paste` the two manifests, and the first attachment
|
||
whose hash differs names the pass that diverged. Read-side and pixel-pack state
|
||
is saved and restored, so the replay continues unperturbed; without the flag
|
||
nothing is installed and the replay is byte-for-byte what it was.
|
||
|
||
Run the macOS native-window DirectVulkan retrace matrix and render the same
|
||
HTML overview shape as CI:
|
||
|
||
```sh
|
||
cmake -S . -B cmake-build-macos-trace-arm64 -G Ninja \
|
||
-DCMAKE_BUILD_TYPE=Release \
|
||
-DCMAKE_OSX_ARCHITECTURES=arm64 \
|
||
-DMOBILEGL_BUILD_TEST=OFF \
|
||
-DMOBILEGL_BUILD_BENCHMARK=OFF \
|
||
-DMOBILEGL_BUILD_TRACE_REPLAY=ON
|
||
cmake --build cmake-build-macos-trace-arm64 --target MobileGL mobilegl_trace_replay
|
||
python3 tools/trace_replay/run_macos_window_retrace_local.py --ci --all
|
||
open .trace-work/macos-window-retrace-summary/mobilegl-macos-window-vulkan-retrace-overview.html
|
||
```
|
||
|
||
The macOS runner hydrates missing fixtures from the trace fixture mirror with
|
||
parallel downloads before falling back to Git LFS. It reuses the
|
||
`cmake-build-macos-trace-arm64` harness by default on Apple Silicon, passes
|
||
`--window-surface`, and defaults to DirectVulkan only. If a native-window replay
|
||
hits a fatal assertion, the runner writes a failure result and stops before
|
||
launching later cases; use `--continue-after-fatal` to collect the full matrix,
|
||
or `--skip-case NAME` for known fatal cases.
|
||
|
||
## Android device replay
|
||
|
||
Build and install the generic trace APK from the repository root. Both
|
||
`DirectGLES` and `DirectVulkan` use the same APK and package; select the
|
||
backend with the intent's `backend` extra.
|
||
|
||
```sh
|
||
gradle --no-daemon -p android-plugin :app:assembleTraceDebug
|
||
TRACE_APK=$(find android-plugin/app/build/outputs/apk/trace/debug -maxdepth 1 -name '*.apk' -print -quit)
|
||
adb install -r "$TRACE_APK"
|
||
```
|
||
|
||
Prepare a fixture and copy it into the app-private directory:
|
||
|
||
```sh
|
||
mkdir -p /tmp/mobilegl-openra
|
||
tar -xzf tools/trace_replay/fixtures/openra.tgz -C /tmp/mobilegl-openra
|
||
adb push /tmp/mobilegl-openra/openra.trace /data/local/tmp/mobilegl-openra.trace
|
||
adb push tools/trace_replay/fixtures/openra.0000031249.png /data/local/tmp/mobilegl-openra.golden.png
|
||
|
||
PKG=top.mobilegl.plugin.trace
|
||
APP_DIR=/data/user/0/$PKG/files/trace-replay
|
||
adb shell run-as $PKG rm -rf files/trace-replay
|
||
adb shell run-as $PKG mkdir -p files/trace-replay/input files/trace-replay/output
|
||
adb shell run-as $PKG cp /data/local/tmp/mobilegl-openra.trace files/trace-replay/input/openra.trace
|
||
adb shell run-as $PKG cp /data/local/tmp/mobilegl-openra.golden.png files/trace-replay/input/openra.golden.png
|
||
```
|
||
|
||
Launch the standalone trace runner Activity:
|
||
|
||
```sh
|
||
adb shell am force-stop $PKG
|
||
adb shell am start -W -a top.mobilegl.plugin.TRACE_REPLAY \
|
||
-n $PKG/top.mobilegl.plugin.trace.TraceReplayActivity \
|
||
--es trace_path $APP_DIR/input/openra.trace \
|
||
--es golden_path $APP_DIR/input/openra.golden.png \
|
||
--es output_dir $APP_DIR/output \
|
||
--es diff_path $APP_DIR/output/openra-diff.png \
|
||
--es backend DirectGLES \
|
||
--el target_call 31249 \
|
||
--ei width 640 \
|
||
--ei height 480 \
|
||
--ei crop_x 1 \
|
||
--ei crop_y 1 \
|
||
--ei crop_width 638 \
|
||
--ei crop_height 478 \
|
||
--es ssim_threshold 0.99
|
||
```
|
||
|
||
Read back the result and images:
|
||
|
||
```sh
|
||
adb shell run-as $PKG cat files/trace-replay/output/result.json
|
||
adb exec-out run-as $PKG cat files/trace-replay/output/actual.png > openra-actual.png
|
||
adb exec-out run-as $PKG cat files/trace-replay/output/openra-diff.png > openra-diff.png
|
||
```
|
||
|
||
For Vulkan replay, keep the same APK and `$PKG`, then pass
|
||
`--es backend DirectVulkan`. DirectGLES also renders to the Activity surface by
|
||
default; pass `--ez use_pbuffer true` to use the offscreen pbuffer path. Always
|
||
`adb shell am force-stop $PKG` before another replay: apitrace snapshot state is
|
||
process-local. For cases registered with `coherent_as_flush` (Flywheel-style
|
||
unflushed persistent maps, e.g. the Create fixtures), pass
|
||
`--ez coherent_as_flush true` so the replay runs with
|
||
`MOBILEGL_COHERENT_AS_FLUSH=1`. For cases registered with
|
||
`avoid_angle_llvmpipe_explicit_lod_bias` (DirectGLES on ANGLE llvmpipe, e.g. the
|
||
sundial-lite fixture), pass `--ez avoid_angle_llvmpipe_explicit_lod_bias true` so
|
||
the replay runs with `MOBILEGL_ESPRYT_AVOID_EXPLICIT_LOD_BIAS=1`.
|
||
|
||
### Repeats, the CI split subset, and keeping every repeat's evidence
|
||
|
||
`run_android_retrace_local.py` drives the above through `trace-replay-ci.sh`. Three
|
||
options exist for the P7-7 device diagnostics, where the question is not "did this
|
||
case pass" but "which cases diverge, and is the divergence reproducible":
|
||
|
||
| option | what it does |
|
||
| --- | --- |
|
||
| `--matrix` | run the **CI split subset** (the set `retrace-split` runs; 39 cases today) instead of the whole manifest. `--all` additionally includes the non-CI workloads such as rd12, which ID-P7-4 excludes from the exit-gate-3 denominator. Each case runs only the backends its own `ci_backends` names - and a case NAMED with `--case` that does not run the requested backend is an error, not a silent skip. |
|
||
| `--repeat N` | replay each case/backend N times. Repeats after the first pass `--reuse-fixture`, so they measure the replay and not the push. |
|
||
| `--archive-dir DIR` | copy each repeat's `result.json`, actual PNG, both role logs, `transport-proof.json` and `logcat.txt` to `DIR/<case>-<backend>/repeat-NN/` **before the next run overwrites them**, plus `DIR/run.json` recording the APK SHA-256, the arm and the environment. |
|
||
|
||
The archive exists because `.trace-work/android-retrace-result/<case>-<backend>/` is
|
||
keyed by case and backend and by nothing else: without `--archive-dir` the only
|
||
artefact that survives a three-repeat run is the last one, and "were the three
|
||
repeats the same picture?" becomes unanswerable after the fact.
|
||
|
||
```sh
|
||
python tools/trace_replay/run_android_retrace_local.py \
|
||
--matrix --backend DirectVulkan --use-pbuffer --transport inproc --repeat 3 \
|
||
--archive-dir .trace-work/p7w1/E0-inproc
|
||
python3 tools/trace_replay/compare_actuals.py summary .trace-work/p7w1/E0-inproc \
|
||
--json E0-inproc-summary.json
|
||
```
|
||
|
||
`compare_actuals.py` computes the pair `result.json` cannot: SSIM between two actual
|
||
PNGs. It is the retrace gate's OWN SSIM - a transcription of `ComputeChannelSsim` /
|
||
`ComputeRgbSsim` (`android-plugin/app/src/trace/cpp/trace_replay_core.cpp:630` and
|
||
`:671`), one global window per channel, C1 = (0.01*255)², C2 = (0.03*255)², alpha
|
||
ignored - and the crop comes from each repeat's own `result.json`, so the
|
||
`ssim_vs_first` column describes the same rectangle as the recorded `ssim_vs_golden`.
|
||
`summary` prints `case × repeat × ssim_vs_golden × ssim_vs_first × bit-identical`;
|
||
`compare A.png B.png` scores one pair; `--self-test` checks the transcription against
|
||
closed-form values. Unit tests: `python3 tools/trace_replay/test_compare_actuals.py`.
|
||
|
||
## Benchmark mode (frame timing)
|
||
|
||
Benchmark mode reuses the same fixtures as a performance harness instead of a
|
||
correctness one: it replays the trace from the first call to the last, takes a
|
||
wall-clock timestamp at every frame boundary, and takes no snapshot and runs no
|
||
SSIM comparison. `passed` then only means the replay reached the end of the trace
|
||
without an error.
|
||
|
||
Frame times include GPU completion by default, because a swap boundary on a tiled
|
||
mobile GPU returns long before the tiler is done and would otherwise time CPU
|
||
submission alone. That is what `benchmark_finish` / `--benchmark-finish` controls:
|
||
on (the default) it issues a `glFinish` through the replayed context at every
|
||
frame boundary, which serializes CPU/GPU overlap - pessimistic against a real
|
||
running game, but deterministic and comparable between backends and revisions.
|
||
Turn it off to measure CPU-side submission only.
|
||
|
||
The mean/median/p95 are computed over the last `benchmark_tail_frames` frames
|
||
(default 200, clamped to the frames actually recorded); the head of a trace is
|
||
dominated by shader compiles and first-use uploads. The full per-frame array is in
|
||
the timing JSON, next to the headline numbers, which also appear in `result.json`.
|
||
|
||
Whole device runs, one line per run plus the best of the repeats:
|
||
|
||
```sh
|
||
python tools/trace_replay/run_android_retrace_local.py --benchmark \
|
||
--case minecraft-1.21.4-fabric-iris-photon-in-world --backend DirectVulkan \
|
||
--benchmark-repeats 3
|
||
```
|
||
|
||
`--benchmark-tail-frames N`, `--benchmark-no-finish` and
|
||
`--benchmark-timeout-seconds N` are available; only the first repeat installs the
|
||
APK and pushes the trace. Each run's `benchmark.json` is kept next to the case
|
||
result as `benchmark-run<N>.json`.
|
||
|
||
The Activity takes the same settings directly:
|
||
|
||
```sh
|
||
adb shell am start -a top.mobilegl.plugin.TRACE_REPLAY \
|
||
-n $PKG/top.mobilegl.plugin.trace.TraceReplayActivity \
|
||
--es trace_path $APP_DIR/input/openra.trace \
|
||
--es output_dir $APP_DIR/output \
|
||
--es backend DirectGLES \
|
||
--ei width 640 --ei height 480 \
|
||
--ez benchmark true \
|
||
--ei benchmark_tail_frames 200 \
|
||
--ez benchmark_finish true \
|
||
--es benchmark_result_path $APP_DIR/output/benchmark.json
|
||
adb exec-out run-as $PKG cat files/trace-replay/output/benchmark.json > benchmark.json
|
||
```
|
||
|
||
`golden_path` and `target_call` are not needed in benchmark mode. The Linux CLI
|
||
takes the same options:
|
||
|
||
```sh
|
||
./mobilegl_trace_replay --trace openra.trace --output out --backend DirectGLES \
|
||
--benchmark --benchmark-tail-frames=200 --benchmark-finish=1 \
|
||
--benchmark-result=out/benchmark.json
|
||
```
|
||
|
||
Two caveats when reading the numbers. Frame times are taken at `eglSwapBuffers`,
|
||
so a trace that ends frames with `glFrameTerminatorGREMEDY` instead is not timed.
|
||
And on a window surface the swap can block on the compositor, which pins frame
|
||
times to the display refresh; replay against the pbuffer surface
|
||
(`--ez use_pbuffer true`) to measure the renderer rather than the presentation
|
||
path.
|
||
|
||
## Reproducing the Android DirectGLES lane on Linux (ANGLE on lavapipe)
|
||
|
||
The APK workflow's DirectGLES lane is not the same stack as the Linux one, which
|
||
is why a case can be green here and red there:
|
||
|
||
| lane | stack |
|
||
| --- | --- |
|
||
| Linux `Test` retrace, DirectGLES | Espryt -> Mesa GLES -> llvmpipe |
|
||
| Android `APK` retrace, DirectGLES | Espryt -> **ANGLE** -> Mesa Vulkan (lavapipe) |
|
||
| Android `APK` retrace, DirectVulkan | Magma -> lavapipe (no ANGLE) |
|
||
|
||
Only the Android DirectGLES lane puts ANGLE in the middle, so an ANGLE
|
||
translation difference shows up in exactly one of the six combinations. That
|
||
stack can be reproduced on Linux without an emulator, which is far faster to
|
||
iterate on than a CI round trip. The Android emulator SDK ships a glibc ANGLE:
|
||
|
||
```sh
|
||
ANGLE=$ANDROID_SDK_ROOT/emulator/lib64/gles_angle
|
||
mkdir -p ~/angle-farm && cd ~/angle-farm
|
||
# MobileGL dlopens these two names; ANGLE's own libEGL then dlopens the
|
||
# unsuffixed libGLESv2.so from the same directory - without that symlink it
|
||
# loads a truncated entry-point table and dies on a missing EGL function.
|
||
ln -sf $ANGLE/libEGL.so libEGL_angle.so
|
||
ln -sf $ANGLE/libGLESv2.so libGLESv2_angle.so
|
||
ln -sf $ANGLE/libEGL.so libEGL.so
|
||
ln -sf $ANGLE/libGLESv2.so libGLESv2.so
|
||
ln -sf $ANGLE/libvulkan.so.1 libvulkan.so.1 # else eglInitialize fails
|
||
|
||
MOBILEGL_ESPRYT_USE_ANGLE=1 \
|
||
LD_LIBRARY_PATH=~/angle-farm:/path/to/build/ \
|
||
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json \
|
||
ANGLE_DEFAULT_PLATFORM=vulkan \
|
||
./mobilegl_trace_replay --trace trace.trace --golden golden.png \
|
||
--target-call N --width 854 --height 480 --backend DirectGLES \
|
||
--output outdir --pbuffer-surface
|
||
```
|
||
|
||
`ANGLE_DEFAULT_PLATFORM=vulkan` is required: ANGLE otherwise picks its OpenGL
|
||
backend and you get `ANGLE (Mesa, llvmpipe ..., OpenGL 4.6 (Core Profile))`
|
||
instead of the CI-shaped `ANGLE (Mesa, Vulkan 1.x (llvmpipe ...))`. Check
|
||
`MOBILEGL_TRACE_GL_RENDERER` in `outdir/retrace.log` before trusting a result.
|
||
Run the binary directly rather than through `ctest`, whose `ENVIRONMENT`
|
||
property overrides these variables. Build with clang, not gcc: gcc rejects
|
||
`GLXImpl.cpp` under `-Wchanges-meaning`.
|
||
|
||
One more caveat before attributing anything: the emulator SDK's ANGLE is not
|
||
the ANGLE the Android lane runs. The CI lane uses a pinned build
|
||
(`MOBILEGL_TRACE_ANGLE_VARIANT`, default `ec889e6ea831`) whose version and
|
||
extension set differ from the SDK copy (`GL_EXT_texture_buffer` support, ES 3.2
|
||
entry points). Compare `GL_RENDERER` and the relevant extension lists on both
|
||
stacks before treating a local result as a statement about CI.
|
||
|
||
|
||
## TCP matrix driver (Linux / WSL)
|
||
|
||
`run_tcp_matrix.py` uses a CTest JSON catalog plus the **current** `trace_cases.json`.
|
||
It does not build, install an APK, or launch the remote server. For the formal
|
||
P6.5 run-ahead matrix, use `--require-run-ahead`. It forces `MOBILEGL_IPC_RUN_AHEAD=1`
|
||
and `MOBILEGL_IPC_VERB_BARRIER=1`, then requires `run-ahead ARMED` in each private
|
||
client log and rejects any `running lockstep` / `DISARMED` marker. The checkpoint
|
||
records `requested_arm` and `actual_arm`; strict resume cannot reuse an earlier
|
||
lockstep image pass. Start the intended server first, then run serially:
|
||
|
||
```bash
|
||
ctest --test-dir /path/to/retrace-build --show-only=json-v1 > /tmp/trace-catalog.json
|
||
python3 tools/trace_replay/run_tcp_matrix.py \
|
||
--catalog /tmp/trace-catalog.json \
|
||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken \
|
||
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix
|
||
# Continue the same frozen library/runner/peer:
|
||
python3 tools/trace_replay/run_tcp_matrix.py \
|
||
--catalog /tmp/trace-catalog.json \
|
||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken \
|
||
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix --resume
|
||
```
|
||
|
||
`--build-dir` can replace `--catalog` to obtain the JSON without building.
|
||
`--runner` overrides the catalog runner; fixture paths default to the catalog's
|
||
fixture directory and can be overridden with `--fixtures`. Repeat `--case` or
|
||
`--backend` to select a subset. The default is the manifest's CI split subset
|
||
(currently 39 cases), honoring each case's backend list. Explicit `--case
|
||
minecraft-1.21.4-rd12-odinlite-in-world` permits that non-CI workload separately;
|
||
an explicit `split:false` always remains excluded.
|
||
|
||
### Both backends, one driver
|
||
|
||
`--backend DirectVulkan` runs P7 exit gate 3's device matrix with the same
|
||
run-ahead proof, resume and watchdog the DirectGLES matrix uses. Three things are
|
||
per backend and the rest is deliberately shared:
|
||
|
||
- **selection** - a case is planned for a backend only if its own `ci_backends`
|
||
names it, so `minecraft-1.21.4-fabric-iris-iterationrp-in-world` appears in a
|
||
DirectVulkan sweep (39 cases) and not in a DirectGLES one (38). A `--case` named
|
||
explicitly that does not run a requested backend is a refusal that names the
|
||
case, not one plan fewer and no message;
|
||
- **golden and threshold** - from `backend_overrides.<backend>` in
|
||
`trace_cases.json` when the case declares one. A per-backend `golden` REPLACES
|
||
the shared `golden`/`alternate_golden` pair for that backend rather than adding
|
||
to it, and a run that matched an image another backend declares is refused by
|
||
name. (`alternate_golden` alone cannot express this: it is an OR across
|
||
backends, so a DirectVulkan run matching the DirectGLES picture passes.) The
|
||
block may restate only `golden`, `alternate_golden` and `ssim_threshold` -
|
||
restating `target_call` or a crop would stop the two backends retracing the same
|
||
frame through the same window, which is what makes the two numbers comparable;
|
||
- **result naming and checkpoint key** - `<backend>/credit-<n>/<case>`, plus the
|
||
golden, alternate golden and threshold recorded in each `results.json` row.
|
||
|
||
A case with no `backend_overrides` is resolved to itself, so DirectGLES plans,
|
||
identities and checkpoint keys are unchanged by this option's existence - and the
|
||
key is dropped from the fingerprint even for the backend that overrides nothing,
|
||
so giving DirectVulkan its own golden does not invalidate the DirectGLES
|
||
checkpoints. **The CTest catalog emitter and the APK matrix refuse a case that
|
||
declares `backend_overrides`**: both carry one golden path for both backends, so
|
||
emitting such a case through them would register one backend's arm against the
|
||
other's image. Teach them the key in the same commit that adds an override.
|
||
|
||
The watchdog defaults to **300 seconds without log progress** and a separate
|
||
**7200-second absolute ceiling** (`--idle-seconds`, `--max-seconds`). It observes
|
||
runner output and trace/client/server log changes. It deliberately ignores the
|
||
old local CTest/manifest `timeout_seconds`: TCP main-menu kept progressing beyond
|
||
its old local 180-second budget. A growing log does not bypass the explicit absolute ceiling.
|
||
On timeout the driver terminates and reaps the entire case process group,
|
||
including a child that ignores TERM after CMake exits, and stops the matrix to
|
||
avoid cascading Busy failures. Check the peer before resuming. Ctrl-C/TERM/HUP
|
||
also clean up the group and save a cancelled checkpoint.
|
||
|
||
Each attempt has a new directory. `checkpoint.json` is written atomically, and
|
||
only a zero exit plus current-attempt successful result, matching backend/call/
|
||
golden, adequate SSIM, an actual image, and the requested TCP arm marker count
|
||
as passed. Resume also checks request/artifact identity and stored evidence
|
||
hashes. Failed, timed-out, interrupted, changed, or missing evidence is rerun;
|
||
old scratch `results.json` files are retained but never silently promoted into
|
||
successful checkpoints. Replacing the remote deployment requires a new output
|
||
root unless it is the same frozen peer. `results.json` summarizes the currently
|
||
selected runs and marks reused evidence with `resumed:true`.
|
||
|
||
`--wake-adb-serial 2f7cbe2e` explicitly opts into `adb shell input keyevent
|
||
KEYCODE_WAKEUP` every 15 seconds during a case. It does not install anything or
|
||
change global power/idle settings. Its own adb log is excluded from the progress
|
||
watchdog, so wakeup messages cannot conceal a stalled trace.
|
||
|
||
Tool regression tests use short process stubs, never a GPU/device/trace. They run
|
||
in CI beside the other executable negative controls (`test.yml`, the R-16 step):
|
||
|
||
```bash
|
||
python3 tools/trace_replay/test_run_tcp_matrix.py
|
||
```
|
||
|
||
## Device servers: offscreen or on the phone's screen
|
||
|
||
`tcp_device_server.py` starts one of the trace APK's two TCP servers; only one is active
|
||
at a time (whichever starts last stops the other). `stop` force-stops the package.
|
||
|
||
```bash
|
||
# Offscreen (default): MobileGLServerService exec's the supervisor, one child per session.
|
||
python3 tools/trace_replay/tcp_device_server.py start --serial 2f7cbe2e \
|
||
--listen tcp://127.0.0.1:40613 --token devtoken-of-16-bytes --forward
|
||
# On-screen: MobileGLDisplayActivity (process :mglwin) serves in its own process and a
|
||
# client's window surface renders on its SurfaceView. The screen must be on and unlocked.
|
||
python3 tools/trace_replay/tcp_device_server.py start --surface window [--backend DirectVulkan] \
|
||
--serial 2f7cbe2e --listen tcp://127.0.0.1:40613 --token devtoken-of-16-bytes --forward
|
||
```
|
||
|
||
A headless client replays onto the phone's screen with `--window-surface` and
|
||
`MOBILEGL_IPC_SURFACE=server` (plus the usual `MOBILEGL_TRANSPORT=spawn`,
|
||
`MOBILEGL_IPC_CONTROL=tcp://127.0.0.1:40613`, `MOBILEGL_IPC_DATA=stream`,
|
||
`MOBILEGL_IPC_TOKEN`). The server sizes its window to the trace's EGL_WIDTH/EGL_HEIGHT
|
||
(letterboxed on screen) and the client's `eglQuerySurface` answers that size. Without
|
||
`--backend` the on-screen server pins the first session's backend for its process; restart
|
||
it (`stop`, then `start`) to switch. The same knob against the offscreen service is refused
|
||
by name (`NoServerDisplay`) and that service goes on serving.
|
||
|
||
## TCP credit and Stage measurements
|
||
|
||
For a paired frame-time breakdown, enable `MOBILEGL_PIPE_STATS=1` in both roles
|
||
and keep their logs. The following reducer matches client and server present
|
||
serials, drops the first (unpaired) server interval and one warmup frame, and
|
||
reports avg/p50/p99 in milliseconds plus percentages of client present time:
|
||
|
||
```bash
|
||
python3 tools/trace_replay/frame_stats.py \
|
||
--client-log mobilegl.client.log --server-log mobilegl.server.log \
|
||
--warmup-frames 1 --output frame-stats.json
|
||
```
|
||
|
||
`transport_wait_ms` measures wall time in the client producer's transport park
|
||
(flush, spin and sleep). It includes waiting for replies, present credit and
|
||
ring space; it is not socket latency. Client and server CPU can overlap with
|
||
each other and with a transport wait, so each percentage uses the client
|
||
present interval as its denominator and the three percentages need not sum to
|
||
100%. A missing frame is rejected. Percentiles use nearest rank over per-frame
|
||
samples.
|
||
|
||
`benchmark_tcp_credits.py` runs the complete OpenRA and rd12 traces at credits
|
||
1, 2, and 3, then selects matching client/server tail frame IDs (100 and 200
|
||
frames respectively). Extract the unchanged trace files as
|
||
`inputs/OpenRA/openra.trace` and `inputs/rd12/trace.trace`. Keep the server and
|
||
library frozen, enable server `MOBILEGL_PIPE_STATS=1` and
|
||
`MOBILEGL_PIPE_STATS_PERIOD=1`, and leave the device connection free:
|
||
|
||
```bash
|
||
python3 tools/trace_replay/benchmark_tcp_credits.py \
|
||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||
--inputs /path/to/inputs --out /tmp/tcp-credits \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken --serial 2f7cbe2e
|
||
python3 tools/trace_replay/measure_tcp_stage_burst.py \
|
||
--library /path/to/libMobileGL.so --out /tmp/tcp-stage-burst \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken
|
||
```
|
||
|
||
The credit runner serializes all six runs, sends the same explicit awake key as
|
||
the matrix driver, and preserves complete benchmark and role logs. It requires
|
||
the requested credit and continuous run-ahead ARMED proof in the client log,
|
||
matching complete tail frames, and positive per-thread CPU observations.
|
||
`--case` and `--credit` select a subset; `--resume` reuses only matching identity
|
||
and intact evidence. FPS, `kWaitReply` counts, bucketed RTT distribution, Stage
|
||
bytes, and the two role CPU clocks come from the selected steady window.
|
||
Completing this benchmark does not perform or replace a golden image comparison.
|
||
|
||
The Stage tool uploads exactly 64 MiB after one warmup frame, consumes the dirty
|
||
buffer in a draw, and waits for apply. It requires actual TCP run-ahead and at
|
||
least 64 MiB of observed Stage bytes in frame 2, preserves artifact hashes, and
|
||
refuses to overwrite an existing output directory. Its throughput includes
|
||
client copies and backend work; it is not raw Wi-Fi bandwidth. Divide steady
|
||
frame Stage traffic by this separately measured throughput only with that
|
||
interpretation. Device matrices, benchmarks, and the burst must run one at a time.
|
||
|
||
Offline evidence controls (no device):
|
||
|
||
```bash
|
||
python3 tools/trace_replay/test_benchmark_tcp_credits.py
|
||
```
|