mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-13 04:53:23 +09:00
swung0x48
3ba150d021
wave 0 item 5. Three instruments P7's device window needs, and the CI steps that
can make each of them red.
run_tcp_matrix.py --backend DirectVulkan end to end. --backend already existed and
already filtered by each case's ci_backends; what did not exist is a per-backend
GOLDEN. The two backends shared `golden` plus an optional `alternate_golden`, and
alternate_golden is an OR across backends, not a per-backend key - so a
DirectVulkan run that matched the DirectGLES picture passed, which is the "went
green on the wrong arm" shape the exit gate exists to catch. trace_cases.json
gains `backend_overrides.<backend>` ({golden, alternate_golden, ssim_threshold}
only: restating target_call or a crop would stop the two backends retracing the
same frame through the same window). A per-backend golden REPLACES the shared
pair, matching another backend's declared image is refused BY NAME, and a --case
named explicitly that does not run a requested backend is an error instead of one
plan fewer and no message. The CMake emitter and the APK matrix refuse a case that
declares the key rather than silently emitting one golden for both arms.
DirectGLES is unchanged and that is asserted, not asserted-in-prose: a case with no
override resolves to itself, and the key is dropped from the resolved case even for
the backend that overrides nothing - so giving DirectVulkan its own golden does not
move the DirectGLES identity fingerprint and does not invalidate its checkpoints.
Each results.json row now records the golden, alternate golden and threshold it was
scored against.
run_android_retrace_local.py gets --matrix (the CI split subset, 39 cases, each
running only its own ci_backends - not --all, which adds the rd12 workload ID-P7-4
excludes), --repeat N, and --archive-dir. The archive is the point: the result root
is keyed by case and backend and by NOTHING else, so repeat 2 overwrites repeat 1
and the only artefact that survives a three-repeat run is the last one, which makes
"were the three repeats the same picture?" unanswerable after the fact - in the one
place where re-running to find out is expensive. It also refuses --benchmark with a
non-monolith --transport, which has always produced a monolith run under a spawn
label (run_benchmark_case passes args.env alone; p6/README.md records the pitfall).
compare_actuals.py computes the pair result.json cannot: SSIM between two actual
PNGs, plus a case x repeat x ssim-vs-golden x ssim-vs-first-repeat table over an
archive. It is the retrace gate's OWN SSIM - a transcription of ComputeChannelSsim
/ ComputeRgbSsim (trace_replay_core.cpp:630/:671), the only SSIM in the tree - and
the crop comes from each repeat's own result.json, so the two columns describe the
same rectangle. PNG decode is stdlib zlib, because an unavailable Pillow in the
device window would turn "the repeats diverged" into "the tool did not run".
TraceReplayActivity: resolveSpawnServerPath's two decisions were String.contains()
on the whole joined K=V;K=V list and neither was entry-aware. The one that bit:
`--env MOBILEGL_IPC_SERVER_PATH` with no '=' is the documented UNSET spelling
(trace_env_overrides_test.cpp:92) and the only way to ask for "spawn with no image"
and watch LaunchServer refuse by name - it did not match "MOBILEGL_IPC_SERVER_PATH="
so a resolved path was appended AFTER it and, because ApplyEnvOverrides applies in
order with setenv(...,1), the deliberate unset was silently undone. That is exactly
the "silently replacing it would make that knob untestable" the method's own comment
forbids. Two more: the key inside another entry's VALUE flipped both decisions the
wrong way, and the transport match was case-sensitive while ConfigLoader lower-cases
the value before matching (ConfigLoader.cpp:322). The parse moves to
SpawnServerPath, a class with no android.* import, so test_android_lifecycle.py
compiles and runs it with plain javac - the parse had no host-side coverage at all,
and its on-device failure mode is "the knob had no effect".
Only spawn gets a path, deliberately: it is the only transport that execs an image.
Red-once, all four: a stub case whose result names the DirectGLES golden fails a
--backend DirectVulkan run with "matched a golden another backend declares"; a
synthetic PNG pair with a closed-form SSIM (identical = 1.0 exactly, constant
black vs white = C1/(255^2+C1), inverted halves = (C2-2v)/(2v+C2)); the pre-fix
resolveSpawnServerPath body fails eight of SpawnServerPathTest's checks; and the
DirectVulkan TCP proof battery is the DirectGLES one under the other prefix,
because an empty prefix selection raises nothing and reads as a clean pass.
CI: test_run_tcp_matrix.py and test_compare_actuals.py join the R-16 step. The
matrix driver's stub regressions had NO CI step - they were run by hand and
recorded in a note - and they guard the tool that decides whether a device golden
run passed.
Counts in WSL: test_run_tcp_matrix 12 -> 18, tcp_lane_tools 4 -> 6,
test_compare_actuals 12 new, test_android_lifecycle green with the new Java unit.
581 lines
35 KiB
Markdown
581 lines
35 KiB
Markdown
# MobileGL trace replay
|
||
|
||
This directory builds a Linux command line replay runner for [apitrace](https://github.com/apitrace/apitrace) files. It
|
||
is an integration testing infrastructure of MobileGL.
|
||
|
||
The bundled fixtures cover:
|
||
|
||
- OpenRA: sourced from GL4ES' apitrace corpus.
|
||
|
||
- minecraft-1.21.4-startup: captured from Minecraft 1.21.4's startup screen.
|
||
|
||
- minecraft-1.21.4-main-menu: captured from Minecraft 1.21.4's main menu.
|
||
|
||
- minecraft-1.21.11-main-menu: captured from Minecraft 1.21.11's main menu on a Pixel 8 Pro through FCL MobileGL.
|
||
|
||
- minecraft-1.17-main-menu-854: captured from Minecraft 1.17's 854x480 main menu through FCL MobileGL capture.
|
||
|
||
- minecraft-1.21.4-in-world: captured from Minecraft 1.21.4 after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-rd12-odinlite-in-world: captured on an Android device (FCL MobileGL Magma capture, 854x480) from
|
||
vanilla Minecraft 1.21.4 at render distance 12, drifting down a river valley in a boat. Unlike the other in-world
|
||
fixtures this one is a 251-frame window (gltrim `-f 1094-1343`) rather than a single frame, so it doubles as the
|
||
campaign's benchmark scene: benchmark mode replays the whole window and the tail frames measure steady-state
|
||
in-world frame time. The golden is still the final frame, so it works as an ordinary correctness case too.
|
||
|
||
- minecraft-1.21.4-fabric-sodium-in-world: captured from Minecraft 1.21.4 Fabric with Sodium after entering a
|
||
singleplayer world with Fancy graphics.
|
||
|
||
- improved-transparency-minecraft-26.3: captured from the Minecraft 26.3 improved-transparency scene.
|
||
|
||
- minecraft-1.21.4-fabric-common-mods-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, REI,
|
||
Xaero's Minimap, Xaero's World Map, JourneyMap, and Modern UI, with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-common-mods-inventory: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, REI,
|
||
Xaero's Minimap, Xaero's World Map, JourneyMap, and Modern UI with the creative inventory and REI item list open.
|
||
|
||
- minecraft-1.21.4-fabric-rei-inventory: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and REI, with
|
||
shader packs disabled and the creative inventory and REI item list open.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-minimap-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Xaero's Minimap after entering a singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-world-map-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Xaero's World Map, with shader packs disabled and the world map screen open.
|
||
|
||
- minecraft-1.21.4-fabric-journeymap-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
JourneyMap after entering a singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-modernui-inventory: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and Modern UI,
|
||
with shader packs disabled and the creative inventory open.
|
||
|
||
- minecraft-1.21.4-fabric-rei-inventory-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
REI in a normal singleplayer world, with shader packs disabled and the creative inventory and REI item list open.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-minimap-in-world-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Xaero's Minimap after entering a normal singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-xaero-world-map-in-world-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Xaero's World Map in a normal singleplayer world, with shader packs disabled and the world map screen open.
|
||
|
||
- minecraft-1.21.4-fabric-journeymap-in-world-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris,
|
||
and JourneyMap after entering a normal singleplayer world with shader packs disabled.
|
||
|
||
- minecraft-1.21.4-fabric-modernui-inventory-normal-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris,
|
||
and Modern UI in a normal singleplayer world, with shader packs disabled and the creative inventory open.
|
||
|
||
- minecraft-1.21.4-fabric-iris-bsl-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and BSL
|
||
Shaders after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-bsl-esc-menu-854: captured on an Android device (Mali-G77, FCL MobileGL capture) from
|
||
Minecraft 1.21.4 Fabric with Sodium, Iris, and BSL Shaders, at the pause menu over a BSL-blurred world. The frame
|
||
pins glyph rendering: every menu label, the menu title and the tutorial toast must be present. Regressions in the
|
||
DirectGLES per-draw texture memo have made the whole text path disappear here while sprites kept rendering, so a
|
||
failure that leaves the buttons but empties them is the signature to look for in the diff.
|
||
|
||
- minecraft-1.21.4-fabric-iris-makeup-ultrafast-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
MakeUP UltraFast after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-super-duper-vanilla-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris,
|
||
and Super Duper Vanilla after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-complementary-reimagined-in-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Complementary Reimagined after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-complementary-unbound-in-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Complementary Unbound after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-mellow-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and Mellow
|
||
after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-nostalgia-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Nostalgia after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-bliss-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and Bliss
|
||
after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-chocapic-v6-lite-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Chocapic V6 Lite after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-iterationt-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
iterationT after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-iterationt-nodsa-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
iterationT after entering a singleplayer world, with Iris' DSA path disabled.
|
||
|
||
- minecraft-1.21.4-fabric-iris-iterationrp-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
iterationRP after entering a singleplayer world, framing the iterationRP name overlay over a lake with far-shore
|
||
tree reflections. iterationRP's temporal auto-exposure makes a single-frame trim overexpose and drop the overlay,
|
||
so the fixture is a prefix trace (all calls up to the target frame) that replays the temporal state. The pack also
|
||
gates an NVIDIA-only shadow path (`subgroupPartitionNV`, `GL_NV_shader_subgroup_partitioned`) on the GL vendor
|
||
string, so the capture reports a masked vendor and the trace carries the portable `subgroupShuffleXor` path that
|
||
non-NVIDIA GPUs take.
|
||
The trace archive and golden are not committed yet (the repository's Git LFS quota rejects new objects with
|
||
`GH009`); the case stays registered and its fixture files are hydrated from the trace fixture mirror.
|
||
- minecraft-1.21.4-fabric-iris-photon-v1.1-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Photon v1.1 after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Photon v1.3b after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-derivative-main-d24.4.14-in-world: captured from Minecraft 1.21.4 Fabric with Sodium,
|
||
Iris, and Derivative Main d24.4.14 after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.4-fabric-iris-sundial-lite-in-world: captured from Minecraft 1.21.4 Fabric with Sodium, Iris, and
|
||
Sundial Lite after entering a singleplayer world.
|
||
|
||
- minecraft-1.21.1-neoforge-create-indirect-in-world: captured from Minecraft 1.21.1 NeoForge with Create, Sodium,
|
||
and Iris (no shader pack) in a world facing Create water wheels and a large cogwheel, with Flywheel's
|
||
`flywheel:indirect` backend (compute-shader culling, glMultiDrawElementsIndirect, persistent-mapped staging).
|
||
|
||
- minecraft-1.21.1-neoforge-create-instancing-in-world: same world and camera as the indirect case, with Flywheel's
|
||
`flywheel:instancing` backend (texture-buffer instance data, glDrawElementsInstancedBaseVertex).
|
||
|
||
|
||
Build from the MobileGL repository root:
|
||
|
||
```sh
|
||
cmake -S . -B build-test -G Ninja \
|
||
-DMOBILEGL_BUILD_TEST=ON \
|
||
-DMOBILEGL_BUILD_BENCHMARK=OFF \
|
||
-DMOBILEGL_BUILD_TRACE_REPLAY=ON
|
||
cmake --build build-test
|
||
```
|
||
|
||
Run the fixture tests:
|
||
|
||
```sh
|
||
ctest --test-dir build-test -V -R 'MobileGLTraceReplay\.'
|
||
```
|
||
|
||
Run the CLI directly:
|
||
|
||
```sh
|
||
build-test/tools/trace_replay/mobilegl_trace_replay \
|
||
--trace openra.trace \
|
||
--golden openra.0000031249.png \
|
||
--output out/openra \
|
||
--backend DirectGLES \
|
||
--mobilegl-library build-test/libMobileGL.so \
|
||
--target-call 31249 \
|
||
--width 640 \
|
||
--height 480 \
|
||
--crop-x 1 \
|
||
--crop-y 1 \
|
||
--crop-width 638 \
|
||
--crop-height 478 \
|
||
--ssim-threshold 0.99
|
||
```
|
||
|
||
## Dumping framebuffer attachments mid-frame
|
||
|
||
`--target-call` snapshots one framebuffer. To see *inside* a frame - which
|
||
intermediate render target a pass actually produced - pass
|
||
`--dump-fbo-attachments CALL:DIR[:FBO,FBO,...]`, repeatably:
|
||
|
||
```sh
|
||
build-test/tools/trace_replay/mobilegl_trace_replay \
|
||
--trace trace.trace --golden golden.png --output out --target-call 2667619 \
|
||
--dump-fbo-attachments 2666231:out/fbos-before \
|
||
--dump-fbo-attachments 2666232:out/fbos-after
|
||
```
|
||
|
||
At each call boundary it walks every live framebuffer object (or only the named
|
||
ones), reads back every colour attachment and the depth attachment, and writes
|
||
`fbo<N>-att<M>.png` / `fbo<N>-depth.png` plus a `manifest.txt` line per
|
||
attachment recording the attached object, size, internal format, component type
|
||
and per-channel min/max/mean and a content hash. Attachments are read as floats
|
||
whatever their storage, so HDR accumulation buffers stay legible in the
|
||
statistics even though the PNG has to clamp.
|
||
|
||
The manifest is the useful part when comparing two drivers: dump the same call
|
||
on both stacks and `diff`/`paste` the two manifests, and the first attachment
|
||
whose hash differs names the pass that diverged. Read-side and pixel-pack state
|
||
is saved and restored, so the replay continues unperturbed; without the flag
|
||
nothing is installed and the replay is byte-for-byte what it was.
|
||
|
||
Run the macOS native-window DirectVulkan retrace matrix and render the same
|
||
HTML overview shape as CI:
|
||
|
||
```sh
|
||
cmake -S . -B cmake-build-macos-trace-arm64 -G Ninja \
|
||
-DCMAKE_BUILD_TYPE=Release \
|
||
-DCMAKE_OSX_ARCHITECTURES=arm64 \
|
||
-DMOBILEGL_BUILD_TEST=OFF \
|
||
-DMOBILEGL_BUILD_BENCHMARK=OFF \
|
||
-DMOBILEGL_BUILD_TRACE_REPLAY=ON
|
||
cmake --build cmake-build-macos-trace-arm64 --target MobileGL mobilegl_trace_replay
|
||
python3 tools/trace_replay/run_macos_window_retrace_local.py --ci --all
|
||
open .trace-work/macos-window-retrace-summary/mobilegl-macos-window-vulkan-retrace-overview.html
|
||
```
|
||
|
||
The macOS runner hydrates missing fixtures from the trace fixture mirror with
|
||
parallel downloads before falling back to Git LFS. It reuses the
|
||
`cmake-build-macos-trace-arm64` harness by default on Apple Silicon, passes
|
||
`--window-surface`, and defaults to DirectVulkan only. If a native-window replay
|
||
hits a fatal assertion, the runner writes a failure result and stops before
|
||
launching later cases; use `--continue-after-fatal` to collect the full matrix,
|
||
or `--skip-case NAME` for known fatal cases.
|
||
|
||
## Android device replay
|
||
|
||
Build and install the generic trace APK from the repository root. Both
|
||
`DirectGLES` and `DirectVulkan` use the same APK and package; select the
|
||
backend with the intent's `backend` extra.
|
||
|
||
```sh
|
||
gradle --no-daemon -p android-plugin :app:assembleTraceDebug
|
||
TRACE_APK=$(find android-plugin/app/build/outputs/apk/trace/debug -maxdepth 1 -name '*.apk' -print -quit)
|
||
adb install -r "$TRACE_APK"
|
||
```
|
||
|
||
Prepare a fixture and copy it into the app-private directory:
|
||
|
||
```sh
|
||
mkdir -p /tmp/mobilegl-openra
|
||
tar -xzf tools/trace_replay/fixtures/openra.tgz -C /tmp/mobilegl-openra
|
||
adb push /tmp/mobilegl-openra/openra.trace /data/local/tmp/mobilegl-openra.trace
|
||
adb push tools/trace_replay/fixtures/openra.0000031249.png /data/local/tmp/mobilegl-openra.golden.png
|
||
|
||
PKG=top.mobilegl.plugin.trace
|
||
APP_DIR=/data/user/0/$PKG/files/trace-replay
|
||
adb shell run-as $PKG rm -rf files/trace-replay
|
||
adb shell run-as $PKG mkdir -p files/trace-replay/input files/trace-replay/output
|
||
adb shell run-as $PKG cp /data/local/tmp/mobilegl-openra.trace files/trace-replay/input/openra.trace
|
||
adb shell run-as $PKG cp /data/local/tmp/mobilegl-openra.golden.png files/trace-replay/input/openra.golden.png
|
||
```
|
||
|
||
Launch the standalone trace runner Activity:
|
||
|
||
```sh
|
||
adb shell am force-stop $PKG
|
||
adb shell am start -W -a top.mobilegl.plugin.TRACE_REPLAY \
|
||
-n $PKG/top.mobilegl.plugin.trace.TraceReplayActivity \
|
||
--es trace_path $APP_DIR/input/openra.trace \
|
||
--es golden_path $APP_DIR/input/openra.golden.png \
|
||
--es output_dir $APP_DIR/output \
|
||
--es diff_path $APP_DIR/output/openra-diff.png \
|
||
--es backend DirectGLES \
|
||
--el target_call 31249 \
|
||
--ei width 640 \
|
||
--ei height 480 \
|
||
--ei crop_x 1 \
|
||
--ei crop_y 1 \
|
||
--ei crop_width 638 \
|
||
--ei crop_height 478 \
|
||
--es ssim_threshold 0.99
|
||
```
|
||
|
||
Read back the result and images:
|
||
|
||
```sh
|
||
adb shell run-as $PKG cat files/trace-replay/output/result.json
|
||
adb exec-out run-as $PKG cat files/trace-replay/output/actual.png > openra-actual.png
|
||
adb exec-out run-as $PKG cat files/trace-replay/output/openra-diff.png > openra-diff.png
|
||
```
|
||
|
||
For Vulkan replay, keep the same APK and `$PKG`, then pass
|
||
`--es backend DirectVulkan`. DirectGLES also renders to the Activity surface by
|
||
default; pass `--ez use_pbuffer true` to use the offscreen pbuffer path. Always
|
||
`adb shell am force-stop $PKG` before another replay: apitrace snapshot state is
|
||
process-local. For cases registered with `coherent_as_flush` (Flywheel-style
|
||
unflushed persistent maps, e.g. the Create fixtures), pass
|
||
`--ez coherent_as_flush true` so the replay runs with
|
||
`MOBILEGL_COHERENT_AS_FLUSH=1`. For cases registered with
|
||
`avoid_angle_llvmpipe_explicit_lod_bias` (DirectGLES on ANGLE llvmpipe, e.g. the
|
||
sundial-lite fixture), pass `--ez avoid_angle_llvmpipe_explicit_lod_bias true` so
|
||
the replay runs with `MOBILEGL_ESPRYT_AVOID_EXPLICIT_LOD_BIAS=1`.
|
||
|
||
### Repeats, the CI split subset, and keeping every repeat's evidence
|
||
|
||
`run_android_retrace_local.py` drives the above through `trace-replay-ci.sh`. Three
|
||
options exist for the P7-7 device diagnostics, where the question is not "did this
|
||
case pass" but "which cases diverge, and is the divergence reproducible":
|
||
|
||
| option | what it does |
|
||
| --- | --- |
|
||
| `--matrix` | run the **CI split subset** (the set `retrace-split` runs; 39 cases today) instead of the whole manifest. `--all` additionally includes the non-CI workloads such as rd12, which ID-P7-4 excludes from the exit-gate-3 denominator. Each case runs only the backends its own `ci_backends` names - and a case NAMED with `--case` that does not run the requested backend is an error, not a silent skip. |
|
||
| `--repeat N` | replay each case/backend N times. Repeats after the first pass `--reuse-fixture`, so they measure the replay and not the push. |
|
||
| `--archive-dir DIR` | copy each repeat's `result.json`, actual PNG, both role logs, `transport-proof.json` and `logcat.txt` to `DIR/<case>-<backend>/repeat-NN/` **before the next run overwrites them**, plus `DIR/run.json` recording the APK SHA-256, the arm and the environment. |
|
||
|
||
The archive exists because `.trace-work/android-retrace-result/<case>-<backend>/` is
|
||
keyed by case and backend and by nothing else: without `--archive-dir` the only
|
||
artefact that survives a three-repeat run is the last one, and "were the three
|
||
repeats the same picture?" becomes unanswerable after the fact.
|
||
|
||
```sh
|
||
python tools/trace_replay/run_android_retrace_local.py \
|
||
--matrix --backend DirectVulkan --use-pbuffer --transport inproc --repeat 3 \
|
||
--archive-dir .trace-work/p7w1/E0-inproc
|
||
python3 tools/trace_replay/compare_actuals.py summary .trace-work/p7w1/E0-inproc \
|
||
--json E0-inproc-summary.json
|
||
```
|
||
|
||
`compare_actuals.py` computes the pair `result.json` cannot: SSIM between two actual
|
||
PNGs. It is the retrace gate's OWN SSIM - a transcription of `ComputeChannelSsim` /
|
||
`ComputeRgbSsim` (`android-plugin/app/src/trace/cpp/trace_replay_core.cpp:630` and
|
||
`:671`), one global window per channel, C1 = (0.01*255)², C2 = (0.03*255)², alpha
|
||
ignored - and the crop comes from each repeat's own `result.json`, so the
|
||
`ssim_vs_first` column describes the same rectangle as the recorded `ssim_vs_golden`.
|
||
`summary` prints `case × repeat × ssim_vs_golden × ssim_vs_first × bit-identical`;
|
||
`compare A.png B.png` scores one pair; `--self-test` checks the transcription against
|
||
closed-form values. Unit tests: `python3 tools/trace_replay/test_compare_actuals.py`.
|
||
|
||
## Benchmark mode (frame timing)
|
||
|
||
Benchmark mode reuses the same fixtures as a performance harness instead of a
|
||
correctness one: it replays the trace from the first call to the last, takes a
|
||
wall-clock timestamp at every frame boundary, and takes no snapshot and runs no
|
||
SSIM comparison. `passed` then only means the replay reached the end of the trace
|
||
without an error.
|
||
|
||
Frame times include GPU completion by default, because a swap boundary on a tiled
|
||
mobile GPU returns long before the tiler is done and would otherwise time CPU
|
||
submission alone. That is what `benchmark_finish` / `--benchmark-finish` controls:
|
||
on (the default) it issues a `glFinish` through the replayed context at every
|
||
frame boundary, which serializes CPU/GPU overlap - pessimistic against a real
|
||
running game, but deterministic and comparable between backends and revisions.
|
||
Turn it off to measure CPU-side submission only.
|
||
|
||
The mean/median/p95 are computed over the last `benchmark_tail_frames` frames
|
||
(default 200, clamped to the frames actually recorded); the head of a trace is
|
||
dominated by shader compiles and first-use uploads. The full per-frame array is in
|
||
the timing JSON, next to the headline numbers, which also appear in `result.json`.
|
||
|
||
Whole device runs, one line per run plus the best of the repeats:
|
||
|
||
```sh
|
||
python tools/trace_replay/run_android_retrace_local.py --benchmark \
|
||
--case minecraft-1.21.4-fabric-iris-photon-in-world --backend DirectVulkan \
|
||
--benchmark-repeats 3
|
||
```
|
||
|
||
`--benchmark-tail-frames N`, `--benchmark-no-finish` and
|
||
`--benchmark-timeout-seconds N` are available; only the first repeat installs the
|
||
APK and pushes the trace. Each run's `benchmark.json` is kept next to the case
|
||
result as `benchmark-run<N>.json`.
|
||
|
||
The Activity takes the same settings directly:
|
||
|
||
```sh
|
||
adb shell am start -a top.mobilegl.plugin.TRACE_REPLAY \
|
||
-n $PKG/top.mobilegl.plugin.trace.TraceReplayActivity \
|
||
--es trace_path $APP_DIR/input/openra.trace \
|
||
--es output_dir $APP_DIR/output \
|
||
--es backend DirectGLES \
|
||
--ei width 640 --ei height 480 \
|
||
--ez benchmark true \
|
||
--ei benchmark_tail_frames 200 \
|
||
--ez benchmark_finish true \
|
||
--es benchmark_result_path $APP_DIR/output/benchmark.json
|
||
adb exec-out run-as $PKG cat files/trace-replay/output/benchmark.json > benchmark.json
|
||
```
|
||
|
||
`golden_path` and `target_call` are not needed in benchmark mode. The Linux CLI
|
||
takes the same options:
|
||
|
||
```sh
|
||
./mobilegl_trace_replay --trace openra.trace --output out --backend DirectGLES \
|
||
--benchmark --benchmark-tail-frames=200 --benchmark-finish=1 \
|
||
--benchmark-result=out/benchmark.json
|
||
```
|
||
|
||
Two caveats when reading the numbers. Frame times are taken at `eglSwapBuffers`,
|
||
so a trace that ends frames with `glFrameTerminatorGREMEDY` instead is not timed.
|
||
And on a window surface the swap can block on the compositor, which pins frame
|
||
times to the display refresh; replay against the pbuffer surface
|
||
(`--ez use_pbuffer true`) to measure the renderer rather than the presentation
|
||
path.
|
||
|
||
## Reproducing the Android DirectGLES lane on Linux (ANGLE on lavapipe)
|
||
|
||
The APK workflow's DirectGLES lane is not the same stack as the Linux one, which
|
||
is why a case can be green here and red there:
|
||
|
||
| lane | stack |
|
||
| --- | --- |
|
||
| Linux `Test` retrace, DirectGLES | Espryt -> Mesa GLES -> llvmpipe |
|
||
| Android `APK` retrace, DirectGLES | Espryt -> **ANGLE** -> Mesa Vulkan (lavapipe) |
|
||
| Android `APK` retrace, DirectVulkan | Magma -> lavapipe (no ANGLE) |
|
||
|
||
Only the Android DirectGLES lane puts ANGLE in the middle, so an ANGLE
|
||
translation difference shows up in exactly one of the six combinations. That
|
||
stack can be reproduced on Linux without an emulator, which is far faster to
|
||
iterate on than a CI round trip. The Android emulator SDK ships a glibc ANGLE:
|
||
|
||
```sh
|
||
ANGLE=$ANDROID_SDK_ROOT/emulator/lib64/gles_angle
|
||
mkdir -p ~/angle-farm && cd ~/angle-farm
|
||
# MobileGL dlopens these two names; ANGLE's own libEGL then dlopens the
|
||
# unsuffixed libGLESv2.so from the same directory - without that symlink it
|
||
# loads a truncated entry-point table and dies on a missing EGL function.
|
||
ln -sf $ANGLE/libEGL.so libEGL_angle.so
|
||
ln -sf $ANGLE/libGLESv2.so libGLESv2_angle.so
|
||
ln -sf $ANGLE/libEGL.so libEGL.so
|
||
ln -sf $ANGLE/libGLESv2.so libGLESv2.so
|
||
ln -sf $ANGLE/libvulkan.so.1 libvulkan.so.1 # else eglInitialize fails
|
||
|
||
MOBILEGL_ESPRYT_USE_ANGLE=1 \
|
||
LD_LIBRARY_PATH=~/angle-farm:/path/to/build/ \
|
||
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json \
|
||
ANGLE_DEFAULT_PLATFORM=vulkan \
|
||
./mobilegl_trace_replay --trace trace.trace --golden golden.png \
|
||
--target-call N --width 854 --height 480 --backend DirectGLES \
|
||
--output outdir --pbuffer-surface
|
||
```
|
||
|
||
`ANGLE_DEFAULT_PLATFORM=vulkan` is required: ANGLE otherwise picks its OpenGL
|
||
backend and you get `ANGLE (Mesa, llvmpipe ..., OpenGL 4.6 (Core Profile))`
|
||
instead of the CI-shaped `ANGLE (Mesa, Vulkan 1.x (llvmpipe ...))`. Check
|
||
`MOBILEGL_TRACE_GL_RENDERER` in `outdir/retrace.log` before trusting a result.
|
||
Run the binary directly rather than through `ctest`, whose `ENVIRONMENT`
|
||
property overrides these variables. Build with clang, not gcc: gcc rejects
|
||
`GLXImpl.cpp` under `-Wchanges-meaning`.
|
||
|
||
One more caveat before attributing anything: the emulator SDK's ANGLE is not
|
||
the ANGLE the Android lane runs. The CI lane uses a pinned build
|
||
(`MOBILEGL_TRACE_ANGLE_VARIANT`, default `ec889e6ea831`) whose version and
|
||
extension set differ from the SDK copy (`GL_EXT_texture_buffer` support, ES 3.2
|
||
entry points). Compare `GL_RENDERER` and the relevant extension lists on both
|
||
stacks before treating a local result as a statement about CI.
|
||
|
||
|
||
## TCP matrix driver (Linux / WSL)
|
||
|
||
`run_tcp_matrix.py` uses a CTest JSON catalog plus the **current** `trace_cases.json`.
|
||
It does not build, install an APK, or launch the remote server. For the formal
|
||
P6.5 run-ahead matrix, use `--require-run-ahead`. It forces `MOBILEGL_IPC_RUN_AHEAD=1`
|
||
and `MOBILEGL_IPC_VERB_BARRIER=1`, then requires `run-ahead ARMED` in each private
|
||
client log and rejects any `running lockstep` / `DISARMED` marker. The checkpoint
|
||
records `requested_arm` and `actual_arm`; strict resume cannot reuse an earlier
|
||
lockstep image pass. Start the intended server first, then run serially:
|
||
|
||
```bash
|
||
ctest --test-dir /path/to/retrace-build --show-only=json-v1 > /tmp/trace-catalog.json
|
||
python3 tools/trace_replay/run_tcp_matrix.py \
|
||
--catalog /tmp/trace-catalog.json \
|
||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken \
|
||
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix
|
||
# Continue the same frozen library/runner/peer:
|
||
python3 tools/trace_replay/run_tcp_matrix.py \
|
||
--catalog /tmp/trace-catalog.json \
|
||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken \
|
||
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix --resume
|
||
```
|
||
|
||
`--build-dir` can replace `--catalog` to obtain the JSON without building.
|
||
`--runner` overrides the catalog runner; fixture paths default to the catalog's
|
||
fixture directory and can be overridden with `--fixtures`. Repeat `--case` or
|
||
`--backend` to select a subset. The default is the manifest's CI split subset
|
||
(currently 39 cases), honoring each case's backend list. Explicit `--case
|
||
minecraft-1.21.4-rd12-odinlite-in-world` permits that non-CI workload separately;
|
||
an explicit `split:false` always remains excluded.
|
||
|
||
### Both backends, one driver
|
||
|
||
`--backend DirectVulkan` runs P7 exit gate 3's device matrix with the same
|
||
run-ahead proof, resume and watchdog the DirectGLES matrix uses. Three things are
|
||
per backend and the rest is deliberately shared:
|
||
|
||
- **selection** - a case is planned for a backend only if its own `ci_backends`
|
||
names it, so `minecraft-1.21.4-fabric-iris-iterationrp-in-world` appears in a
|
||
DirectVulkan sweep (39 cases) and not in a DirectGLES one (38). A `--case` named
|
||
explicitly that does not run a requested backend is a refusal that names the
|
||
case, not one plan fewer and no message;
|
||
- **golden and threshold** - from `backend_overrides.<backend>` in
|
||
`trace_cases.json` when the case declares one. A per-backend `golden` REPLACES
|
||
the shared `golden`/`alternate_golden` pair for that backend rather than adding
|
||
to it, and a run that matched an image another backend declares is refused by
|
||
name. (`alternate_golden` alone cannot express this: it is an OR across
|
||
backends, so a DirectVulkan run matching the DirectGLES picture passes.) The
|
||
block may restate only `golden`, `alternate_golden` and `ssim_threshold` -
|
||
restating `target_call` or a crop would stop the two backends retracing the same
|
||
frame through the same window, which is what makes the two numbers comparable;
|
||
- **result naming and checkpoint key** - `<backend>/credit-<n>/<case>`, plus the
|
||
golden, alternate golden and threshold recorded in each `results.json` row.
|
||
|
||
A case with no `backend_overrides` is resolved to itself, so DirectGLES plans,
|
||
identities and checkpoint keys are unchanged by this option's existence - and the
|
||
key is dropped from the fingerprint even for the backend that overrides nothing,
|
||
so giving DirectVulkan its own golden does not invalidate the DirectGLES
|
||
checkpoints. **The CTest catalog emitter and the APK matrix refuse a case that
|
||
declares `backend_overrides`**: both carry one golden path for both backends, so
|
||
emitting such a case through them would register one backend's arm against the
|
||
other's image. Teach them the key in the same commit that adds an override.
|
||
|
||
The watchdog defaults to **300 seconds without log progress** and a separate
|
||
**7200-second absolute ceiling** (`--idle-seconds`, `--max-seconds`). It observes
|
||
runner output and trace/client/server log changes. It deliberately ignores the
|
||
old local CTest/manifest `timeout_seconds`: TCP main-menu kept progressing beyond
|
||
its old local 180-second budget. A growing log does not bypass the explicit absolute ceiling.
|
||
On timeout the driver terminates and reaps the entire case process group,
|
||
including a child that ignores TERM after CMake exits, and stops the matrix to
|
||
avoid cascading Busy failures. Check the peer before resuming. Ctrl-C/TERM/HUP
|
||
also clean up the group and save a cancelled checkpoint.
|
||
|
||
Each attempt has a new directory. `checkpoint.json` is written atomically, and
|
||
only a zero exit plus current-attempt successful result, matching backend/call/
|
||
golden, adequate SSIM, an actual image, and the requested TCP arm marker count
|
||
as passed. Resume also checks request/artifact identity and stored evidence
|
||
hashes. Failed, timed-out, interrupted, changed, or missing evidence is rerun;
|
||
old scratch `results.json` files are retained but never silently promoted into
|
||
successful checkpoints. Replacing the remote deployment requires a new output
|
||
root unless it is the same frozen peer. `results.json` summarizes the currently
|
||
selected runs and marks reused evidence with `resumed:true`.
|
||
|
||
`--wake-adb-serial 2f7cbe2e` explicitly opts into `adb shell input keyevent
|
||
KEYCODE_WAKEUP` every 15 seconds during a case. It does not install anything or
|
||
change global power/idle settings. Its own adb log is excluded from the progress
|
||
watchdog, so wakeup messages cannot conceal a stalled trace.
|
||
|
||
Tool regression tests use short process stubs, never a GPU/device/trace. They run
|
||
in CI beside the other executable negative controls (`test.yml`, the R-16 step):
|
||
|
||
```bash
|
||
python3 tools/trace_replay/test_run_tcp_matrix.py
|
||
```
|
||
|
||
## TCP credit and Stage measurements
|
||
|
||
`benchmark_tcp_credits.py` runs the complete OpenRA and rd12 traces at credits
|
||
1, 2, and 3, then selects matching client/server tail frame IDs (100 and 200
|
||
frames respectively). Extract the unchanged trace files as
|
||
`inputs/OpenRA/openra.trace` and `inputs/rd12/trace.trace`. Keep the server and
|
||
library frozen, enable server `MOBILEGL_PIPE_STATS=1` and
|
||
`MOBILEGL_PIPE_STATS_PERIOD=1`, and leave the device connection free:
|
||
|
||
```bash
|
||
python3 tools/trace_replay/benchmark_tcp_credits.py \
|
||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||
--inputs /path/to/inputs --out /tmp/tcp-credits \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken --serial 2f7cbe2e
|
||
python3 tools/trace_replay/measure_tcp_stage_burst.py \
|
||
--library /path/to/libMobileGL.so --out /tmp/tcp-stage-burst \
|
||
--endpoint tcp://192.168.21.181:40613 --token devtoken
|
||
```
|
||
|
||
The credit runner serializes all six runs, sends the same explicit awake key as
|
||
the matrix driver, and preserves complete benchmark and role logs. It requires
|
||
the requested credit and continuous run-ahead ARMED proof in the client log,
|
||
matching complete tail frames, and positive per-thread CPU observations.
|
||
`--case` and `--credit` select a subset; `--resume` reuses only matching identity
|
||
and intact evidence. FPS, `kWaitReply` counts, bucketed RTT distribution, Stage
|
||
bytes, and the two role CPU clocks come from the selected steady window.
|
||
Completing this benchmark does not perform or replace a golden image comparison.
|
||
|
||
The Stage tool uploads exactly 64 MiB after one warmup frame, consumes the dirty
|
||
buffer in a draw, and waits for apply. It requires actual TCP run-ahead and at
|
||
least 64 MiB of observed Stage bytes in frame 2, preserves artifact hashes, and
|
||
refuses to overwrite an existing output directory. Its throughput includes
|
||
client copies and backend work; it is not raw Wi-Fi bandwidth. Divide steady
|
||
frame Stage traffic by this separately measured throughput only with that
|
||
interpretation. Device matrices, benchmarks, and the burst must run one at a time.
|
||
|
||
Offline evidence controls (no device):
|
||
|
||
```bash
|
||
python3 tools/trace_replay/test_benchmark_tcp_credits.py
|
||
```
|