run_matrix.sh drives interleaved monolith/inproc/spawn+shm/spawn+tcp
benchmark rounds on an adb device with per-run arm-marker verification;
profile_matrix.sh layers simpleperf, ftrace wakeup capture and a /proc
sampler on top; make_report.py aggregates runs into REPORT.md. The wifi
kit carries the cross-host control-plane hang repro and phone-side
strace/fd-reconcile helpers. .trace-work/ (run output) is now ignored.
MOBILEGL_BUILD_TRACE_REPLAY now also produces mobilegl_trace_replay on
ANDROID: the same CLI as desktop, linked against the APK runner's
glws/glproc glue. On the device it dials a render server with no
Activity at all - MOBILEGL_IPC_CONTROL=unix:@name with shm data to the
offscreen supervisor (a cross-uid abstract connect needs root or an
SELinux waiver; loopback TCP works from plain adb shell), or
tcp:// with MOBILEGL_IPC_SURFACE=server to the on-screen display server.
glws_android grows the desktop glws_egl's server-owned null-window
createWindowSurface, gated on MOBILEGL_TRACE_SURFACE=window so the APK's
pbuffer mode keeps its old answer, and the config carries EGL_WINDOW_BIT
for it. The control screen refuses unix endpoints in on-screen mode:
the in-process display server serves tcp:// only.
Verified on-device with an OpenRA retrace, ssim=1.0 both ways:
offscreen service on @mgl-cli with the CLI as root, and the display
Activity on tcp://127.0.0.1:40613 with the replay's frames on the
phone's screen.
transport_wait_ns measures wall time in Ring::Wait's flush/spin/sleep, so
the client's parked time is read directly instead of inferred by
subtracting CPU clocks that run concurrently with the server.
frame_stats.py matches client and server present serials into per-frame
avg/p50/p99 splits; the README documents the MOBILEGL_PIPE_STATS=1
workflow.
- README, ARCHITECTURE, ROADMAP, MEASUREMENTS and CURRENT_STAGE_PROGRESS are rewritten as indexes
(84 KB -> 15 KB): what the project is, where it stands, and where each detail lives. ROADMAP is one
plain sentence per phase; MEASUREMENTS is the five conclusions that matter plus one line per phase;
CURRENT_STAGE_PROGRESS is the P12 goal, a checklist and the next steps.
- ARCHITECTURE's body moves unchanged into ten design/ files, keeping §1-§17 and appendices A/B, so
`ARCHITECTURE.md §N` references in code resolve through the table in ARCHITECTURE.md.
- guide/ holds build-and-run, code map, glossary and the engineering discipline (from README and
ROADMAP), and the device pinning record that used to sit in devices/.
- The open-debt table moves to notes/DEBTS.md; the detailed current-stage tables (with decision IDs,
the open line items and evidence locations) move into notes/p12/README.md. notes/README.md maps
every moved path and section.
- The documentation citation lint in test.yml also covers design/ and guide/, which describe the
current tree as the top-level files did.
- ROADMAP, MEASUREMENTS and ARCHITECTURE are rewritten as summaries (333 KB of index text -> 84 KB).
ROADMAP's phase table is one line per phase plus a link; MEASUREMENTS keeps headline numbers under
its old §1-§13 numbering (plus §14-§16 for P6.5, P7 and P12); ARCHITECTURE keeps every decided rule
under the same §1-§17 / appendix numbering, so the `ARCHITECTURE.md §N` references in code still
land, and §17 becomes an index of the phase landing shapes.
- CURRENT_STAGE_PROGRESS now describes the current stage (the P12 on-screen-server subset) and absorbs
CURRENT_STAGE_STATUS, which is removed. The P7 status tables, the P6 closing snapshot and the P6.5
first-wave snapshot move to notes/p7, notes/p6 and notes/p65.
- Each phase gets notes/<phase>/README.md: a short summary, then the verbatim material that left
the index documents (its ROADMAP row, its MEASUREMENTS section with subsection numbers kept, its
ARCHITECTURE §17 part, milestones) and a file index. P8-P11 and P13 get their planned scope;
the full open-question list moves to notes/OPEN-QUESTIONS.md.
- The phase reports that sat at the top level move into their phase folders with unchanged names:
P5D-INPROC-PERFORMANCE, P5E-RUNAHEAD, P5F-WIRE-COMPLETENESS, P6-SPAWN-PLAN, P6-CONTRACT-DRAFT,
P6-ENDSTATE-REVIEW. Their relative links, CONTRACT-P6.md's links to them and one tool docstring
are updated; notes/README.md maps every old path and section number to its new place.
- 71 files under notes/recovered/ that were byte-identical copies of phase-folder files are removed.
- Corrections made while condensing: the server process reads Transport as Spawn (CONTRACT-P6 §3.1
amended the old "child forces monolith" rule); both backends now publish kCapRunAheadApply;
MOBILEGL_IPC_RESPAWN is a named refusal and MOBILEGL_IPC_IDLE_EXIT_S is not parsed; the Redmi
pin row is in tools/device_bench/pin_device.sh.
check_doc_citations over docs/Disaggregated/*.md: 0 problems. No relative link is newly broken.
- the round-2 merge folded "TCP session has no forwarded server log" into the one server-log
check for every transport, and its message named only MOBILEGL_TRANSPORT - which is `spawn`
for the tcp arm too (tcp = spawn + MOBILEGL_IPC_CONTROL=tcp://), so the tcp arm's missing
forwarded log read as a plain spawn red
- the message carries MOBILEGL_IPC_CONTROL='...' after the transport (empty for inproc and
plain spawn); message only, no check moved
- the spawn server-log fatal from the previous round sat BEFORE the marker check and the
`spawn ARMED ... pid` check, so a launch or handshake failure - no server process, no ARMED
line - reported "the run wrote no server log" instead of the more specific "resolved, but
carries no spawn ARMED", and sent the reader to the wrong process. It now sits just before
the census; the tcp arm's "no forwarded server log" check, which had the same placement,
moves with it
- the inproc row demands the server log too. Under inproc every server-role line is routed
by thread role to the `.server.log` sink, opened lazily on the first such line, and at the
INFO level this lane demands ServerLoop writes one when mgl-srv-apply starts - so a clean
inproc run always has the file, and "no file" read as "0 lines" was a green Fatal{ and
MGWIRE-FLOOR census of the role that emits both. The two census reads are unconditional
now: the fatal above them is what makes the file exist
- fail-closed demonstrations, OpenRA DirectVulkan, the replay wrapped to damage one log after
a real run: inproc with the server log deleted went rc 0 on the previous runner (both
censuses printed 0) and is now a fatal at the census; spawn with the server log deleted is
the same fatal; spawn with the server log deleted AND the ARMED line removed reports the
ARMED message, where the previous runner reported the server-log one; spawn with only the
ARMED line removed reports the ARMED message. Undamaged, OpenRA DirectVulkan and DirectGLES
SPLIT and SPAWN stay green, 0 unsound lines
Fable review of the B3 rework, should-fix: run_trace_case.cmake read the
server log only `if(EXISTS ...)`. Under spawn the server process writes its
own file, and the lines the runner counts out of it - the applier Fatal{
lines and the MGWIRE-FLOOR unsound-serial-complete line, which only the
server's VulkanRenderer::OnSubmitsCompletedUpTo emits - are nowhere else,
so a server that died before its first line read as "0 lines" and went
green. The client log was already fail-closed, and so was the tcp arm's
forwarded server log; plain spawn was not.
The plain-spawn branch of the transport block now FATALs when
mobilegl.server.log does not exist, beside the tcp arm's identical check.
Demonstration: the SPAWN ctest command line with TRACE_REPLAY_EXE replaced
by a wrapper that runs the real replay and then deletes
output/mobilegl.server.log (541 lines):
runner at aab01275 -> rc 0, "Fatal{ lines across ... : 0",
"MGWIRE-FLOOR unsound-serial-complete lines: 0" (green, half unseen)
this runner -> rc 1:
CMake Error at run_trace_case.cmake:321 (message):
OpenRA DirectVulkan: MOBILEGL_TRANSPORT=spawn but the run wrote no
.../output/mobilegl.server.log, so the server role's Fatal{ and
MGWIRE-FLOOR lines cannot be counted. A spawn retrace with no
server log cannot be counted as a clean one.
MobileGLTraceReplay.OpenRA.DirectVulkan.SPLIT / .SPAWN unmodified: both
pass, ssim 1.000000, 0 px, 0 MGWIRE-FLOOR lines.
Review ID-P7-34, item 2. The primary B3 fix - the clamp in
VulkanRenderer::OnSubmitsCompletedUpTo that raises the completed-frame-
serial floor only to a serial no in-flight submission still carries - had
nothing behind it: deleting the clamp loop left every lane green and the
OpenRA picture gold on lavapipe (the host's barriers hide the tear). Its
only observable was the per-event MGLOG_W "MGWIRE-FLOOR
unsound-serial-complete".
run_trace_case.cmake now reads that line out of both roles' logs on every
split run, exactly as it already reads Fatal{: the count is printed pass
or fail and any nonzero count is a FATAL_ERROR. The R-16 counter (26 -> 0
on the OpenRA retrace) is thereby a gate.
Red/green on MobileGLTraceReplay.OpenRA.DirectVulkan.SPLIT / .SPAWN:
clamp loop deleted -> both FAIL at ssim 1.000000 / 0 px:
MGPipe split: OpenRA DirectVulkan transport=inproc, MGWIRE-FLOOR
unsound-serial-complete lines: 26
[mgl-srv-apply/WARN]: MGWIRE-FLOOR unsound-serial-complete #1: advancing
the completed frame-serial floor to 3, which submission 3 (serial 3)
still carries
(spawn: identical, 26)
clamp restored -> both PASS, 0 lines.
run_cts.py silently tolerates a missing --skip-file, so the known device-killers from
the skill (clip_distance.functional, the multisampled_to_singlesampled blit family,
the rgb565 repeat-mode case) would each have cost a reboot in the gate-5 window.
KHR-GL46 has no uniform_buffer_object group; the 90 UBO cases are
GTF-GL46.gtf31.GL3Tests.uniform_buffer_object*, present only when the deployed glcts
was built with GTF - the device probe decides. KHR-GL46.shader_storage_buffer_object
(124) is the block that exercises what cluster A actually refuses today: unaligned
and over-range SSBO bindings, not only UBO byte tails.
Cut from VK-GL-CTS 0b04c470ed78569b0b778ece1262d88455ab2b00 (main; the deployed glcts build's source) mustpass
gl46-main.txt and gl33-main.txt: texture_*, shader_image*, packed_pixels,
direct_state_access, uniform_buffer*. The gl33 variants are the fallback the runbook
names for an Android runner that cannot open a 4.6 core context; whichever version the
probe selects, BASE and AFTER read the same list. A 0.5 pp gate needs a committed
denominator, not a glob over someone's checkout.
wave 0 item 5. Three instruments P7's device window needs, and the CI steps that
can make each of them red.
run_tcp_matrix.py --backend DirectVulkan end to end. --backend already existed and
already filtered by each case's ci_backends; what did not exist is a per-backend
GOLDEN. The two backends shared `golden` plus an optional `alternate_golden`, and
alternate_golden is an OR across backends, not a per-backend key - so a
DirectVulkan run that matched the DirectGLES picture passed, which is the "went
green on the wrong arm" shape the exit gate exists to catch. trace_cases.json
gains `backend_overrides.<backend>` ({golden, alternate_golden, ssim_threshold}
only: restating target_call or a crop would stop the two backends retracing the
same frame through the same window). A per-backend golden REPLACES the shared
pair, matching another backend's declared image is refused BY NAME, and a --case
named explicitly that does not run a requested backend is an error instead of one
plan fewer and no message. The CMake emitter and the APK matrix refuse a case that
declares the key rather than silently emitting one golden for both arms.
DirectGLES is unchanged and that is asserted, not asserted-in-prose: a case with no
override resolves to itself, and the key is dropped from the resolved case even for
the backend that overrides nothing - so giving DirectVulkan its own golden does not
move the DirectGLES identity fingerprint and does not invalidate its checkpoints.
Each results.json row now records the golden, alternate golden and threshold it was
scored against.
run_android_retrace_local.py gets --matrix (the CI split subset, 39 cases, each
running only its own ci_backends - not --all, which adds the rd12 workload ID-P7-4
excludes), --repeat N, and --archive-dir. The archive is the point: the result root
is keyed by case and backend and by NOTHING else, so repeat 2 overwrites repeat 1
and the only artefact that survives a three-repeat run is the last one, which makes
"were the three repeats the same picture?" unanswerable after the fact - in the one
place where re-running to find out is expensive. It also refuses --benchmark with a
non-monolith --transport, which has always produced a monolith run under a spawn
label (run_benchmark_case passes args.env alone; p6/README.md records the pitfall).
compare_actuals.py computes the pair result.json cannot: SSIM between two actual
PNGs, plus a case x repeat x ssim-vs-golden x ssim-vs-first-repeat table over an
archive. It is the retrace gate's OWN SSIM - a transcription of ComputeChannelSsim
/ ComputeRgbSsim (trace_replay_core.cpp:630/:671), the only SSIM in the tree - and
the crop comes from each repeat's own result.json, so the two columns describe the
same rectangle. PNG decode is stdlib zlib, because an unavailable Pillow in the
device window would turn "the repeats diverged" into "the tool did not run".
TraceReplayActivity: resolveSpawnServerPath's two decisions were String.contains()
on the whole joined K=V;K=V list and neither was entry-aware. The one that bit:
`--env MOBILEGL_IPC_SERVER_PATH` with no '=' is the documented UNSET spelling
(trace_env_overrides_test.cpp:92) and the only way to ask for "spawn with no image"
and watch LaunchServer refuse by name - it did not match "MOBILEGL_IPC_SERVER_PATH="
so a resolved path was appended AFTER it and, because ApplyEnvOverrides applies in
order with setenv(...,1), the deliberate unset was silently undone. That is exactly
the "silently replacing it would make that knob untestable" the method's own comment
forbids. Two more: the key inside another entry's VALUE flipped both decisions the
wrong way, and the transport match was case-sensitive while ConfigLoader lower-cases
the value before matching (ConfigLoader.cpp:322). The parse moves to
SpawnServerPath, a class with no android.* import, so test_android_lifecycle.py
compiles and runs it with plain javac - the parse had no host-side coverage at all,
and its on-device failure mode is "the knob had no effect".
Only spawn gets a path, deliberately: it is the only transport that execs an image.
Red-once, all four: a stub case whose result names the DirectGLES golden fails a
--backend DirectVulkan run with "matched a golden another backend declares"; a
synthetic PNG pair with a closed-form SSIM (identical = 1.0 exactly, constant
black vs white = C1/(255^2+C1), inverted halves = (C2-2v)/(2v+C2)); the pre-fix
resolveSpawnServerPath body fails eight of SpawnServerPathTest's checks; and the
DirectVulkan TCP proof battery is the DirectGLES one under the other prefix,
because an empty prefix selection raises nothing and reads as a clean pass.
CI: test_run_tcp_matrix.py and test_compare_actuals.py join the R-16 step. The
matrix driver's stub regressions had NO CI step - they were run by hand and
recorded in a note - and they guard the tool that decides whether a device golden
run passed.
Counts in WSL: test_run_tcp_matrix 12 -> 18, tcp_lane_tools 4 -> 6,
test_compare_actuals 12 new, test_android_lifecycle green with the new Java unit.
Gate 8's last two numbers (SEG_STAGE bytes/frame, records/frame post-
chunking) had no producer at all: the summary line's byte classes are
backend-to-driver bytes and maxrec is a max, not a per-frame. Two push-
guarded counters answer them at their single choke points - ByteClass::
StageSegmentBytes at PipeWireCodec's StageAllocate (the only allocation
behind StageBytes, so every buffer walk, texture slab and CSO archive
lands there and emitters cannot double-count) and CallClass::WireRecords
on EncodeRecord's committed path (post-chunking an emitter no longer
knows how many records its one call produced). A staged-blob histogram
rides along. G1 holds: symbols 0/0/0/0, .text byte-identical, pull build
carries no MG_Remote symbols.
The spawn server never counted anything: PipeStats::Init() is a step of
MobileGL::Initialize(), which ServerMain never reaches, so every Enabled()
site in that process - including ServerLoop's own ServerWaits/ServerParks
publish - stayed false and the server log carried no summary line at all
(a zero that means 'another process' printed in the shape of one that
means 'never waited' is the milder misreading; the line was simply
absent). ServerMain now arms and closes PipeStats at its role boundary -
after MG_ConfigLoader::Init (Init latches from the parsed config), after
the server-role latch (the log sink's role), and before ::_exit(0).
Red-once: 0 server summary lines before, 29 after, SSIM 1.0 unchanged.
MOBILEGL_PIPE_STATS_FILE now derives per role through the log sink's
exported RoleLogPath rule - run.client.json / run.server.json, the bare
name resolves to no file - because both roles reach FormatJson() and ofstream
trunc made the second writer replace the first. G1 caught three separate
leaks on the way (a string temporary in the monolith body, one resized
Shutdown() from unequal #if arms, two rewritten MGLOG_W literals in
.rodata); the pull arm is statement-for-statement the original.
Device tooling for the paired A/B lands with it: pin_device.sh gains a
measured 2f7cbe2e entry (this board clamps the GPU at 1050 MHz -
thermal_pwrlevel writes 0 and reads back 1, max_gpuclk is read-only;
absolute times are not comparable with the 1100 MHz-era rows), and
tools/device_bench/p6/ carries the session collector and renderer beside
run_android_retrace_local.py's MOBILEGL_TRACE_PACKAGE override.
Unit 2367/2367 (PipeStats 25/25), spawn integration 102/102.
The spawn work had the two roles append to one MOBILEGL_LOG_FILE_PATH under
O_APPEND. It was correct - two processes at one offset would corrupt the file,
O_APPEND made each write atomic - but it put two sessions' lines in one place
and made every per-side assertion a search. This splits the sink: the client
role writes <base>.client.log and the server role writes <base>.server.log.
THE BASE NAME NAMES NO FILE ANY MORE, AND THAT IS THE POINT. Leaving the
client at the old unsuffixed path would have been the compatible choice and the
wrong one: a reader that was not updated would go on finding a file, go on
passing, and go on seeing only half the session - and a refusal census that
reports a confident zero for the half it cannot see is worse than one that
fails. With both names moved, an un-updated reader gets "file not found" and
says so. There were 34 such readers; every one is moved here.
THE ROLE IS PER-THREAD, not per-process, because under inproc both roles are
threads of one process. Log.cpp keys the sink on a thread-local role: the apply
thread declares itself server (ServerLoop.cpp), a spawned image is server for
its whole life (MOBILEGL_IPC_ROLE=server, read once), and everything else is
the client. The naming rule (RoleLogPath) and the two whole-log operations
(ReadRoleLogs / TruncateRoleLogs) are EXPORTED from the library, so no reader
reimplements the derivation - a second copy is the copy that falls behind.
Readers, by how they read:
- unit mains set MOBILEGL_LOG_FILE_PATH and read a role file: they keep the
base in the env and derive the client path from the library helper.
- the death-test child runners (ReadLog) now concatenate BOTH roles: a death
test asserts the CHILD said something, and the applier's refusals - the very
diagnostics these check - are written under the server role.
- the offset-based Emit readers derive the client path once after setenv.
- PipeStatsWindow reads both, because draws/vbs increment in the backend's
PrepareForDraw - the apply thread, the server role - so the summary line is
in the server file even under inproc.
- run_trace_case.cmake's split census reads both files; the client keeps the
transport markers.
- split_log_paths.py (and its self-test, and the control stub) derive both
role files from the declared base.
- the Android script pulls both role files off the device and concatenates
them, so every downstream check is unchanged - and it judges a role file by
CONTENT, because `run-as test -f` through `exec-out` returned 0 for a missing
file on the device and `exec-out` folds cat's own not-found line into stdout.
The monolith arm legitimately has no server file, and that is a silent skip
rather than a one-line server log.
Guarded so a pull build is untouched: RoleLogPath and the role sink are wholly
inside MOBILEGL_BUILD_DISAGGREGATED; a non-disaggregated build writes the base
path exactly as before. G1 re-measured - 0 symbol diff, .text unchanged at
0xa52203.
Verified on device 2f7cbe2e: monolith writes only the client log (no phantom
server log), inproc and spawn each write both and pass their transport proofs
(spawn: server role in pid 16125). Unit 2353 (one pre-existing environmental
failure), integration-split 186/186, integration-spawn 102/102, retrace 6/6,
control smoke 12/12, split-log-paths self-test 9/9.
swung0x48