wave 0 item 5. Three instruments P7's device window needs, and the CI steps that
can make each of them red.
run_tcp_matrix.py --backend DirectVulkan end to end. --backend already existed and
already filtered by each case's ci_backends; what did not exist is a per-backend
GOLDEN. The two backends shared `golden` plus an optional `alternate_golden`, and
alternate_golden is an OR across backends, not a per-backend key - so a
DirectVulkan run that matched the DirectGLES picture passed, which is the "went
green on the wrong arm" shape the exit gate exists to catch. trace_cases.json
gains `backend_overrides.<backend>` ({golden, alternate_golden, ssim_threshold}
only: restating target_call or a crop would stop the two backends retracing the
same frame through the same window). A per-backend golden REPLACES the shared
pair, matching another backend's declared image is refused BY NAME, and a --case
named explicitly that does not run a requested backend is an error instead of one
plan fewer and no message. The CMake emitter and the APK matrix refuse a case that
declares the key rather than silently emitting one golden for both arms.
DirectGLES is unchanged and that is asserted, not asserted-in-prose: a case with no
override resolves to itself, and the key is dropped from the resolved case even for
the backend that overrides nothing - so giving DirectVulkan its own golden does not
move the DirectGLES identity fingerprint and does not invalidate its checkpoints.
Each results.json row now records the golden, alternate golden and threshold it was
scored against.
run_android_retrace_local.py gets --matrix (the CI split subset, 39 cases, each
running only its own ci_backends - not --all, which adds the rd12 workload ID-P7-4
excludes), --repeat N, and --archive-dir. The archive is the point: the result root
is keyed by case and backend and by NOTHING else, so repeat 2 overwrites repeat 1
and the only artefact that survives a three-repeat run is the last one, which makes
"were the three repeats the same picture?" unanswerable after the fact - in the one
place where re-running to find out is expensive. It also refuses --benchmark with a
non-monolith --transport, which has always produced a monolith run under a spawn
label (run_benchmark_case passes args.env alone; p6/README.md records the pitfall).
compare_actuals.py computes the pair result.json cannot: SSIM between two actual
PNGs, plus a case x repeat x ssim-vs-golden x ssim-vs-first-repeat table over an
archive. It is the retrace gate's OWN SSIM - a transcription of ComputeChannelSsim
/ ComputeRgbSsim (trace_replay_core.cpp:630/:671), the only SSIM in the tree - and
the crop comes from each repeat's own result.json, so the two columns describe the
same rectangle. PNG decode is stdlib zlib, because an unavailable Pillow in the
device window would turn "the repeats diverged" into "the tool did not run".
TraceReplayActivity: resolveSpawnServerPath's two decisions were String.contains()
on the whole joined K=V;K=V list and neither was entry-aware. The one that bit:
`--env MOBILEGL_IPC_SERVER_PATH` with no '=' is the documented UNSET spelling
(trace_env_overrides_test.cpp:92) and the only way to ask for "spawn with no image"
and watch LaunchServer refuse by name - it did not match "MOBILEGL_IPC_SERVER_PATH="
so a resolved path was appended AFTER it and, because ApplyEnvOverrides applies in
order with setenv(...,1), the deliberate unset was silently undone. That is exactly
the "silently replacing it would make that knob untestable" the method's own comment
forbids. Two more: the key inside another entry's VALUE flipped both decisions the
wrong way, and the transport match was case-sensitive while ConfigLoader lower-cases
the value before matching (ConfigLoader.cpp:322). The parse moves to
SpawnServerPath, a class with no android.* import, so test_android_lifecycle.py
compiles and runs it with plain javac - the parse had no host-side coverage at all,
and its on-device failure mode is "the knob had no effect".
Only spawn gets a path, deliberately: it is the only transport that execs an image.
Red-once, all four: a stub case whose result names the DirectGLES golden fails a
--backend DirectVulkan run with "matched a golden another backend declares"; a
synthetic PNG pair with a closed-form SSIM (identical = 1.0 exactly, constant
black vs white = C1/(255^2+C1), inverted halves = (C2-2v)/(2v+C2)); the pre-fix
resolveSpawnServerPath body fails eight of SpawnServerPathTest's checks; and the
DirectVulkan TCP proof battery is the DirectGLES one under the other prefix,
because an empty prefix selection raises nothing and reads as a clean pass.
CI: test_run_tcp_matrix.py and test_compare_actuals.py join the R-16 step. The
matrix driver's stub regressions had NO CI step - they were run by hand and
recorded in a note - and they guard the tool that decides whether a device golden
run passed.
Counts in WSL: test_run_tcp_matrix 12 -> 18, tcp_lane_tools 4 -> 6,
test_compare_actuals 12 new, test_android_lifecycle green with the new Java unit.
The Android retrace matrix gains a third lane, spawn-acceptance, beside
monolith-control and inproc-acceptance. Until now nothing in CI had ever
started a second MobileGL process on a device at all.
The apk.yml retrace job already carried a `transport` matrix dimension and
already branched on it, so the lane itself is three edits: a row in
github_transport_matrix, an elif in the retrace step, and a summary to render.
What needed thought was the gate.
THE SPAWN LANE IS APK-ONLY, and that is a statement about where evidence is
missing rather than about what spawn can do. Linux already has
test.yml's `retrace-split`, which is {backend, case, transport} over inproc AND
spawn with the two negative controls that make it mean anything. A spawn row in
the Linux transport matrix would pay for a second full sweep proving nothing -
and test.yml's `retrace` job stages no server image, so the lane would not
merely be redundant, it would be red. self_test_transport_matrices now asserts
the absence in both directions, because an accidental removal of the `if apk`
has to be caught in the self-test rather than in CI.
--require-spawn REQUIRES TWO SENTENCES, and the second is the one that matters.
ConfigLoader's marker proves the VALUE RESOLVED - it exists only in the Spawn
arm, which exists only under MOBILEGL_BUILD_DISAGGREGATED - but that is a
statement about a parser. `spawn ARMED - the server role runs in pid N` is
written only after the launch, the connect and the handshake have all
succeeded, and the pid in it is the SERVER's. It is the only line in the log a
same-process run cannot produce.
Both halves red once, measured:
* a monolith run demanded to prove spawn: REFUSED, transport_resolution_marker
false, server_pid null - while the retrace itself scored ssim=1.000000,
which is exactly the trap. OpenRA matches its golden under monolith too, so
the picture has never been the evidence.
* the same run's real spawn log with the one ARMED line deleted: REFUSED,
transport_resolution_marker TRUE and server_pid null. That is the
"resolved the transport and then never left the process" case, isolated.
THE PROOF USES `any` WHERE THE inproc GATE USES `all`, and the difference is
load-bearing rather than a relaxation. Under spawn two processes append to one
MOBILEGL_LOG_FILE_PATH, and the launcher deliberately scrubs every
MOBILEGL_IPC_* from the child's environment (anti-recursion catch (b)) - so the
server's own `Config: IPC` line reads strict=0 role-split-state=0 BY
CONSTRUCTION. Demanding every line match would red every spawn run for the one
thing the design requires.
--use-pbuffer is not optional on this lane: until P12 a spawned server cannot be
handed a window, because an ANativeWindow* is a pointer into the CLIENT's
process (Rule H). MOBILEGL_IPC_SERVER_PATH is deliberately NOT passed -
nativeLibraryDir carries an install-time hash that nothing off the device can
spell, and the Activity resolves it.
The local runner now passes the same knobs and the same --require-* flag CI
does, so "it works locally" can no longer mean "it ran monolith locally". That
change immediately caught its own omission: the first local run with the gate
armed failed on client_ipc_line_present=false because the runner had been
sending MOBILEGL_TRANSPORT alone.
Verified on device 2f7cbe2e: OpenRA.DirectGLES spawn, ssim=1.000000,
mismatchPixels=0, "real spawn transport, server role in pid 9885, and zero
Fatal records confirmed".
It was one case on one backend under one transport: OpenRA, DirectGLES,
inproc. It is now every CI case, both backends, inproc AND spawn - 154 CI
entries, 156 local ctest entries - because every defect the spawn work turned
up was a question about the SERVER's process and not about any one trace or
either backend.
THE DirectVulkan ARM. tools/trace_replay/CMakeLists.txt carried "a
DirectVulkan arm would be measuring a server nobody has written yet". True
when P5 wrote it; false since P5f/P7 landed Magma's server consumers and its
run-ahead readiness gate. Leaving it would have shipped a split shape whose
second backend nothing ever ran - the same class of hole as a lane that runs
monolith under a name that says split. Magma now brings up a server process
of its own and OpenRA retraces across it at ssim=1.000000, which is only
possible because this branch also taught the child to load its config: before
that, a DirectVulkan session brought up a DirectGLES server and the two
halves disagreed about which backend the capability mask describes.
`split` BECOMES AN OPT-OUT. It began as an opt-in naming one case, because
one case was the work. Marking 39 cases `"split": true` would carry no
information and would reinstate the hole the key was written to close, only
mirrored: a case added later would get the monolith arms and be silently
absent from the split ones, and absent is the one state nothing reds on. A CI
case is now in the subset unless it says `"split": false`, which is a
reviewable claim that a trace cannot cross a role boundary.
THE CI MATRIX GAINS A THIRD DIMENSION. The job took {backend, case} and
hard-coded MOBILEGL_TRANSPORT=inproc in its own run block, so `spawn` had no
way into CI at all; the .SPAWN ctest entries existed only for a local build.
It is {backend, case, transport} now, the backends are the case's own
ci_backends rather than a hard-coded DirectGLES, and the artifact names carry
the transport or the two arms of one {backend, case} would overwrite each
other's uploads.
inproc STAYS AS THE CONTROL. Same library, same codec, same applier, same
picture; the only difference is which process the server role runs in.
Running only spawn would leave "did the transport break it?" and "did the
phase break it?" answerable only together - which is the question every
failure in this phase has turned on, including the SIGSEGV that D1 explained.
BOTH NEGATIVE CONTROLS FOLLOW THE ARM. retrace_pull_library_control.sh and
retrace_drop_draw_control.sh each exported MOBILEGL_TRANSPORT=inproc
themselves, so on a spawn job they would have gone on proving something about
the other arm while spawn ran unguarded. They read SPLIT_TRANSPORT now, and
default to inproc so a caller that sets nothing behaves exactly as before.
scripts/ci/control_smoke_test.sh: 22 passed, 0 failed against the stubbed
ctest, including the two modes that matter most - a red whose SSIM never fell
and a knob that armed and dropped nothing.
The spawn arm needs a second binary and the inproc arm needs none, so the job
places libMobileGLServer.so only on the spawn arm and a missing one is a
named failure there rather than a LaunchServer refusal 78 jobs deep.
Measured locally as this lands: 39 of the 78 SPAWN entries complete, all
green. The rest are still running.
- A retrace that exports MOBILEGL_PIPE_VERIFY=1 at a library configured without
-DMOBILEGL_PIPE_VERIFY=ON is a no-op: the frames still match their goldens and the
case reports green having compared nothing. run_trace_case.cmake now demands the
evidence whenever the variable is set to anything but 0/false - mobilegl.log must
exist, must carry "MGPipe: verify armed", and must carry neither
Fatal{PipeVerifyDiffer nor Fatal{UnmigratedPipeInput. With the variable unset the
script is byte-for-byte the old one.
- The Fatal scan is not redundant with the replay's exit status:
MOBILEGL_PIPE_VERIFY_FATAL=0 is the supported triage configuration, and there a
divergence is logged and counted rather than aborted, so the run would finish 0
with its own report sitting unread in the log.
- LABELS retrace on both registrations: the lane was selectable only by regex, so
`ctest -L retrace --no-tests=error` - the spelling that reds a lane which
registered nothing - could not be written at all.
- "verify": true on eight cases (the five 180s cases plus minecraft-1.21.4-in-world,
minecraft-1.21.4-fabric-sodium-in-world and improved-transparency-minecraft-26.3),
and --format github-verify-matrix over that subset: 16 entries, against the full
lane's 77. The verify build compares at every verb boundary and again at every
accessor read, which the design budgets at 5-10x, so the per-push job runs the
subset and the full sweep is a phase-exit / workflow_dispatch run.
- The flag is validated in the manifest loader, not at the matrix, so a non-boolean
or a verify case excluded from CI is a loud error in every consumer instead of a
subset that is quietly one case short.
swung0x48