mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-21 00:41:57 +09:00
[tools, android-plugin, CI] (P7): the device matrix runner selects DirectVulkan, repeats are archived, and the spawn env parse is entry-aware
wave 0 item 5. Three instruments P7's device window needs, and the CI steps that
can make each of them red.
run_tcp_matrix.py --backend DirectVulkan end to end. --backend already existed and
already filtered by each case's ci_backends; what did not exist is a per-backend
GOLDEN. The two backends shared `golden` plus an optional `alternate_golden`, and
alternate_golden is an OR across backends, not a per-backend key - so a
DirectVulkan run that matched the DirectGLES picture passed, which is the "went
green on the wrong arm" shape the exit gate exists to catch. trace_cases.json
gains `backend_overrides.<backend>` ({golden, alternate_golden, ssim_threshold}
only: restating target_call or a crop would stop the two backends retracing the
same frame through the same window). A per-backend golden REPLACES the shared
pair, matching another backend's declared image is refused BY NAME, and a --case
named explicitly that does not run a requested backend is an error instead of one
plan fewer and no message. The CMake emitter and the APK matrix refuse a case that
declares the key rather than silently emitting one golden for both arms.
DirectGLES is unchanged and that is asserted, not asserted-in-prose: a case with no
override resolves to itself, and the key is dropped from the resolved case even for
the backend that overrides nothing - so giving DirectVulkan its own golden does not
move the DirectGLES identity fingerprint and does not invalidate its checkpoints.
Each results.json row now records the golden, alternate golden and threshold it was
scored against.
run_android_retrace_local.py gets --matrix (the CI split subset, 39 cases, each
running only its own ci_backends - not --all, which adds the rd12 workload ID-P7-4
excludes), --repeat N, and --archive-dir. The archive is the point: the result root
is keyed by case and backend and by NOTHING else, so repeat 2 overwrites repeat 1
and the only artefact that survives a three-repeat run is the last one, which makes
"were the three repeats the same picture?" unanswerable after the fact - in the one
place where re-running to find out is expensive. It also refuses --benchmark with a
non-monolith --transport, which has always produced a monolith run under a spawn
label (run_benchmark_case passes args.env alone; p6/README.md records the pitfall).
compare_actuals.py computes the pair result.json cannot: SSIM between two actual
PNGs, plus a case x repeat x ssim-vs-golden x ssim-vs-first-repeat table over an
archive. It is the retrace gate's OWN SSIM - a transcription of ComputeChannelSsim
/ ComputeRgbSsim (trace_replay_core.cpp:630/:671), the only SSIM in the tree - and
the crop comes from each repeat's own result.json, so the two columns describe the
same rectangle. PNG decode is stdlib zlib, because an unavailable Pillow in the
device window would turn "the repeats diverged" into "the tool did not run".
TraceReplayActivity: resolveSpawnServerPath's two decisions were String.contains()
on the whole joined K=V;K=V list and neither was entry-aware. The one that bit:
`--env MOBILEGL_IPC_SERVER_PATH` with no '=' is the documented UNSET spelling
(trace_env_overrides_test.cpp:92) and the only way to ask for "spawn with no image"
and watch LaunchServer refuse by name - it did not match "MOBILEGL_IPC_SERVER_PATH="
so a resolved path was appended AFTER it and, because ApplyEnvOverrides applies in
order with setenv(...,1), the deliberate unset was silently undone. That is exactly
the "silently replacing it would make that knob untestable" the method's own comment
forbids. Two more: the key inside another entry's VALUE flipped both decisions the
wrong way, and the transport match was case-sensitive while ConfigLoader lower-cases
the value before matching (ConfigLoader.cpp:322). The parse moves to
SpawnServerPath, a class with no android.* import, so test_android_lifecycle.py
compiles and runs it with plain javac - the parse had no host-side coverage at all,
and its on-device failure mode is "the knob had no effect".
Only spawn gets a path, deliberately: it is the only transport that execs an image.
Red-once, all four: a stub case whose result names the DirectGLES golden fails a
--backend DirectVulkan run with "matched a golden another backend declares"; a
synthetic PNG pair with a closed-form SSIM (identical = 1.0 exactly, constant
black vs white = C1/(255^2+C1), inverted halves = (C2-2v)/(2v+C2)); the pre-fix
resolveSpawnServerPath body fails eight of SpawnServerPathTest's checks; and the
DirectVulkan TCP proof battery is the DirectGLES one under the other prefix,
because an empty prefix selection raises nothing and reads as a clean pass.
CI: test_run_tcp_matrix.py and test_compare_actuals.py join the R-16 step. The
matrix driver's stub regressions had NO CI step - they were run by hand and
recorded in a note - and they guard the tool that decides whether a device golden
run passed.
Counts in WSL: test_run_tcp_matrix 12 -> 18, tcp_lane_tools 4 -> 6,
test_compare_actuals 12 new, test_android_lifecycle green with the new Java unit.
This commit is contained in:
@@ -286,6 +286,41 @@ unflushed persistent maps, e.g. the Create fixtures), pass
|
||||
sundial-lite fixture), pass `--ez avoid_angle_llvmpipe_explicit_lod_bias true` so
|
||||
the replay runs with `MOBILEGL_ESPRYT_AVOID_EXPLICIT_LOD_BIAS=1`.
|
||||
|
||||
### Repeats, the CI split subset, and keeping every repeat's evidence
|
||||
|
||||
`run_android_retrace_local.py` drives the above through `trace-replay-ci.sh`. Three
|
||||
options exist for the P7-7 device diagnostics, where the question is not "did this
|
||||
case pass" but "which cases diverge, and is the divergence reproducible":
|
||||
|
||||
| option | what it does |
|
||||
| --- | --- |
|
||||
| `--matrix` | run the **CI split subset** (the set `retrace-split` runs; 39 cases today) instead of the whole manifest. `--all` additionally includes the non-CI workloads such as rd12, which ID-P7-4 excludes from the exit-gate-3 denominator. Each case runs only the backends its own `ci_backends` names - and a case NAMED with `--case` that does not run the requested backend is an error, not a silent skip. |
|
||||
| `--repeat N` | replay each case/backend N times. Repeats after the first pass `--reuse-fixture`, so they measure the replay and not the push. |
|
||||
| `--archive-dir DIR` | copy each repeat's `result.json`, actual PNG, both role logs, `transport-proof.json` and `logcat.txt` to `DIR/<case>-<backend>/repeat-NN/` **before the next run overwrites them**, plus `DIR/run.json` recording the APK SHA-256, the arm and the environment. |
|
||||
|
||||
The archive exists because `.trace-work/android-retrace-result/<case>-<backend>/` is
|
||||
keyed by case and backend and by nothing else: without `--archive-dir` the only
|
||||
artefact that survives a three-repeat run is the last one, and "were the three
|
||||
repeats the same picture?" becomes unanswerable after the fact.
|
||||
|
||||
```sh
|
||||
python tools/trace_replay/run_android_retrace_local.py \
|
||||
--matrix --backend DirectVulkan --use-pbuffer --transport inproc --repeat 3 \
|
||||
--archive-dir .trace-work/p7w1/E0-inproc
|
||||
python3 tools/trace_replay/compare_actuals.py summary .trace-work/p7w1/E0-inproc \
|
||||
--json E0-inproc-summary.json
|
||||
```
|
||||
|
||||
`compare_actuals.py` computes the pair `result.json` cannot: SSIM between two actual
|
||||
PNGs. It is the retrace gate's OWN SSIM - a transcription of `ComputeChannelSsim` /
|
||||
`ComputeRgbSsim` (`android-plugin/app/src/trace/cpp/trace_replay_core.cpp:630` and
|
||||
`:671`), one global window per channel, C1 = (0.01*255)², C2 = (0.03*255)², alpha
|
||||
ignored - and the crop comes from each repeat's own `result.json`, so the
|
||||
`ssim_vs_first` column describes the same rectangle as the recorded `ssim_vs_golden`.
|
||||
`summary` prints `case × repeat × ssim_vs_golden × ssim_vs_first × bit-identical`;
|
||||
`compare A.png B.png` scores one pair; `--self-test` checks the transcription against
|
||||
closed-form values. Unit tests: `python3 tools/trace_replay/test_compare_actuals.py`.
|
||||
|
||||
## Benchmark mode (frame timing)
|
||||
|
||||
Benchmark mode reuses the same fixtures as a performance harness instead of a
|
||||
@@ -438,6 +473,38 @@ fixture directory and can be overridden with `--fixtures`. Repeat `--case` or
|
||||
minecraft-1.21.4-rd12-odinlite-in-world` permits that non-CI workload separately;
|
||||
an explicit `split:false` always remains excluded.
|
||||
|
||||
### Both backends, one driver
|
||||
|
||||
`--backend DirectVulkan` runs P7 exit gate 3's device matrix with the same
|
||||
run-ahead proof, resume and watchdog the DirectGLES matrix uses. Three things are
|
||||
per backend and the rest is deliberately shared:
|
||||
|
||||
- **selection** - a case is planned for a backend only if its own `ci_backends`
|
||||
names it, so `minecraft-1.21.4-fabric-iris-iterationrp-in-world` appears in a
|
||||
DirectVulkan sweep (39 cases) and not in a DirectGLES one (38). A `--case` named
|
||||
explicitly that does not run a requested backend is a refusal that names the
|
||||
case, not one plan fewer and no message;
|
||||
- **golden and threshold** - from `backend_overrides.<backend>` in
|
||||
`trace_cases.json` when the case declares one. A per-backend `golden` REPLACES
|
||||
the shared `golden`/`alternate_golden` pair for that backend rather than adding
|
||||
to it, and a run that matched an image another backend declares is refused by
|
||||
name. (`alternate_golden` alone cannot express this: it is an OR across
|
||||
backends, so a DirectVulkan run matching the DirectGLES picture passes.) The
|
||||
block may restate only `golden`, `alternate_golden` and `ssim_threshold` -
|
||||
restating `target_call` or a crop would stop the two backends retracing the same
|
||||
frame through the same window, which is what makes the two numbers comparable;
|
||||
- **result naming and checkpoint key** - `<backend>/credit-<n>/<case>`, plus the
|
||||
golden, alternate golden and threshold recorded in each `results.json` row.
|
||||
|
||||
A case with no `backend_overrides` is resolved to itself, so DirectGLES plans,
|
||||
identities and checkpoint keys are unchanged by this option's existence - and the
|
||||
key is dropped from the fingerprint even for the backend that overrides nothing,
|
||||
so giving DirectVulkan its own golden does not invalidate the DirectGLES
|
||||
checkpoints. **The CTest catalog emitter and the APK matrix refuse a case that
|
||||
declares `backend_overrides`**: both carry one golden path for both backends, so
|
||||
emitting such a case through them would register one backend's arm against the
|
||||
other's image. Teach them the key in the same commit that adds an override.
|
||||
|
||||
The watchdog defaults to **300 seconds without log progress** and a separate
|
||||
**7200-second absolute ceiling** (`--idle-seconds`, `--max-seconds`). It observes
|
||||
runner output and trace/client/server log changes. It deliberately ignores the
|
||||
@@ -463,7 +530,8 @@ KEYCODE_WAKEUP` every 15 seconds during a case. It does not install anything or
|
||||
change global power/idle settings. Its own adb log is excluded from the progress
|
||||
watchdog, so wakeup messages cannot conceal a stalled trace.
|
||||
|
||||
Tool regression tests use short process stubs, never a GPU/device/trace:
|
||||
Tool regression tests use short process stubs, never a GPU/device/trace. They run
|
||||
in CI beside the other executable negative controls (`test.yml`, the R-16 step):
|
||||
|
||||
```bash
|
||||
python3 tools/trace_replay/test_run_tcp_matrix.py
|
||||
|
||||
Reference in New Issue
Block a user