MOBILEGL_BUILD_TRACE_REPLAY now also produces mobilegl_trace_replay on
ANDROID: the same CLI as desktop, linked against the APK runner's
glws/glproc glue. On the device it dials a render server with no
Activity at all - MOBILEGL_IPC_CONTROL=unix:@name with shm data to the
offscreen supervisor (a cross-uid abstract connect needs root or an
SELinux waiver; loopback TCP works from plain adb shell), or
tcp:// with MOBILEGL_IPC_SURFACE=server to the on-screen display server.
glws_android grows the desktop glws_egl's server-owned null-window
createWindowSurface, gated on MOBILEGL_TRACE_SURFACE=window so the APK's
pbuffer mode keeps its old answer, and the config carries EGL_WINDOW_BIT
for it. The control screen refuses unix endpoints in on-screen mode:
the in-process display server serves tcp:// only.
Verified on-device with an OpenRA retrace, ssim=1.0 both ways:
offscreen service on @mgl-cli with the CLI as root, and the display
Activity on tcp://127.0.0.1:40613 with the replay's frames on the
phone's screen.
transport_wait_ns measures wall time in Ring::Wait's flush/spin/sleep, so
the client's parked time is read directly instead of inferred by
subtracting CPU clocks that run concurrently with the server.
frame_stats.py matches client and server present serials into per-frame
avg/p50/p99 splits; the README documents the MOBILEGL_PIPE_STATS=1
workflow.
- the round-2 merge folded "TCP session has no forwarded server log" into the one server-log
check for every transport, and its message named only MOBILEGL_TRANSPORT - which is `spawn`
for the tcp arm too (tcp = spawn + MOBILEGL_IPC_CONTROL=tcp://), so the tcp arm's missing
forwarded log read as a plain spawn red
- the message carries MOBILEGL_IPC_CONTROL='...' after the transport (empty for inproc and
plain spawn); message only, no check moved
- the spawn server-log fatal from the previous round sat BEFORE the marker check and the
`spawn ARMED ... pid` check, so a launch or handshake failure - no server process, no ARMED
line - reported "the run wrote no server log" instead of the more specific "resolved, but
carries no spawn ARMED", and sent the reader to the wrong process. It now sits just before
the census; the tcp arm's "no forwarded server log" check, which had the same placement,
moves with it
- the inproc row demands the server log too. Under inproc every server-role line is routed
by thread role to the `.server.log` sink, opened lazily on the first such line, and at the
INFO level this lane demands ServerLoop writes one when mgl-srv-apply starts - so a clean
inproc run always has the file, and "no file" read as "0 lines" was a green Fatal{ and
MGWIRE-FLOOR census of the role that emits both. The two census reads are unconditional
now: the fatal above them is what makes the file exist
- fail-closed demonstrations, OpenRA DirectVulkan, the replay wrapped to damage one log after
a real run: inproc with the server log deleted went rc 0 on the previous runner (both
censuses printed 0) and is now a fatal at the census; spawn with the server log deleted is
the same fatal; spawn with the server log deleted AND the ARMED line removed reports the
ARMED message, where the previous runner reported the server-log one; spawn with only the
ARMED line removed reports the ARMED message. Undamaged, OpenRA DirectVulkan and DirectGLES
SPLIT and SPAWN stay green, 0 unsound lines
Fable review of the B3 rework, should-fix: run_trace_case.cmake read the
server log only `if(EXISTS ...)`. Under spawn the server process writes its
own file, and the lines the runner counts out of it - the applier Fatal{
lines and the MGWIRE-FLOOR unsound-serial-complete line, which only the
server's VulkanRenderer::OnSubmitsCompletedUpTo emits - are nowhere else,
so a server that died before its first line read as "0 lines" and went
green. The client log was already fail-closed, and so was the tcp arm's
forwarded server log; plain spawn was not.
The plain-spawn branch of the transport block now FATALs when
mobilegl.server.log does not exist, beside the tcp arm's identical check.
Demonstration: the SPAWN ctest command line with TRACE_REPLAY_EXE replaced
by a wrapper that runs the real replay and then deletes
output/mobilegl.server.log (541 lines):
runner at aab01275 -> rc 0, "Fatal{ lines across ... : 0",
"MGWIRE-FLOOR unsound-serial-complete lines: 0" (green, half unseen)
this runner -> rc 1:
CMake Error at run_trace_case.cmake:321 (message):
OpenRA DirectVulkan: MOBILEGL_TRANSPORT=spawn but the run wrote no
.../output/mobilegl.server.log, so the server role's Fatal{ and
MGWIRE-FLOOR lines cannot be counted. A spawn retrace with no
server log cannot be counted as a clean one.
MobileGLTraceReplay.OpenRA.DirectVulkan.SPLIT / .SPAWN unmodified: both
pass, ssim 1.000000, 0 px, 0 MGWIRE-FLOOR lines.
Review ID-P7-34, item 2. The primary B3 fix - the clamp in
VulkanRenderer::OnSubmitsCompletedUpTo that raises the completed-frame-
serial floor only to a serial no in-flight submission still carries - had
nothing behind it: deleting the clamp loop left every lane green and the
OpenRA picture gold on lavapipe (the host's barriers hide the tear). Its
only observable was the per-event MGLOG_W "MGWIRE-FLOOR
unsound-serial-complete".
run_trace_case.cmake now reads that line out of both roles' logs on every
split run, exactly as it already reads Fatal{: the count is printed pass
or fail and any nonzero count is a FATAL_ERROR. The R-16 counter (26 -> 0
on the OpenRA retrace) is thereby a gate.
Red/green on MobileGLTraceReplay.OpenRA.DirectVulkan.SPLIT / .SPAWN:
clamp loop deleted -> both FAIL at ssim 1.000000 / 0 px:
MGPipe split: OpenRA DirectVulkan transport=inproc, MGWIRE-FLOOR
unsound-serial-complete lines: 26
[mgl-srv-apply/WARN]: MGWIRE-FLOOR unsound-serial-complete #1: advancing
the completed frame-serial floor to 3, which submission 3 (serial 3)
still carries
(spawn: identical, 26)
clamp restored -> both PASS, 0 lines.
wave 0 item 5. Three instruments P7's device window needs, and the CI steps that
can make each of them red.
run_tcp_matrix.py --backend DirectVulkan end to end. --backend already existed and
already filtered by each case's ci_backends; what did not exist is a per-backend
GOLDEN. The two backends shared `golden` plus an optional `alternate_golden`, and
alternate_golden is an OR across backends, not a per-backend key - so a
DirectVulkan run that matched the DirectGLES picture passed, which is the "went
green on the wrong arm" shape the exit gate exists to catch. trace_cases.json
gains `backend_overrides.<backend>` ({golden, alternate_golden, ssim_threshold}
only: restating target_call or a crop would stop the two backends retracing the
same frame through the same window). A per-backend golden REPLACES the shared
pair, matching another backend's declared image is refused BY NAME, and a --case
named explicitly that does not run a requested backend is an error instead of one
plan fewer and no message. The CMake emitter and the APK matrix refuse a case that
declares the key rather than silently emitting one golden for both arms.
DirectGLES is unchanged and that is asserted, not asserted-in-prose: a case with no
override resolves to itself, and the key is dropped from the resolved case even for
the backend that overrides nothing - so giving DirectVulkan its own golden does not
move the DirectGLES identity fingerprint and does not invalidate its checkpoints.
Each results.json row now records the golden, alternate golden and threshold it was
scored against.
run_android_retrace_local.py gets --matrix (the CI split subset, 39 cases, each
running only its own ci_backends - not --all, which adds the rd12 workload ID-P7-4
excludes), --repeat N, and --archive-dir. The archive is the point: the result root
is keyed by case and backend and by NOTHING else, so repeat 2 overwrites repeat 1
and the only artefact that survives a three-repeat run is the last one, which makes
"were the three repeats the same picture?" unanswerable after the fact - in the one
place where re-running to find out is expensive. It also refuses --benchmark with a
non-monolith --transport, which has always produced a monolith run under a spawn
label (run_benchmark_case passes args.env alone; p6/README.md records the pitfall).
compare_actuals.py computes the pair result.json cannot: SSIM between two actual
PNGs, plus a case x repeat x ssim-vs-golden x ssim-vs-first-repeat table over an
archive. It is the retrace gate's OWN SSIM - a transcription of ComputeChannelSsim
/ ComputeRgbSsim (trace_replay_core.cpp:630/:671), the only SSIM in the tree - and
the crop comes from each repeat's own result.json, so the two columns describe the
same rectangle. PNG decode is stdlib zlib, because an unavailable Pillow in the
device window would turn "the repeats diverged" into "the tool did not run".
TraceReplayActivity: resolveSpawnServerPath's two decisions were String.contains()
on the whole joined K=V;K=V list and neither was entry-aware. The one that bit:
`--env MOBILEGL_IPC_SERVER_PATH` with no '=' is the documented UNSET spelling
(trace_env_overrides_test.cpp:92) and the only way to ask for "spawn with no image"
and watch LaunchServer refuse by name - it did not match "MOBILEGL_IPC_SERVER_PATH="
so a resolved path was appended AFTER it and, because ApplyEnvOverrides applies in
order with setenv(...,1), the deliberate unset was silently undone. That is exactly
the "silently replacing it would make that knob untestable" the method's own comment
forbids. Two more: the key inside another entry's VALUE flipped both decisions the
wrong way, and the transport match was case-sensitive while ConfigLoader lower-cases
the value before matching (ConfigLoader.cpp:322). The parse moves to
SpawnServerPath, a class with no android.* import, so test_android_lifecycle.py
compiles and runs it with plain javac - the parse had no host-side coverage at all,
and its on-device failure mode is "the knob had no effect".
Only spawn gets a path, deliberately: it is the only transport that execs an image.
Red-once, all four: a stub case whose result names the DirectGLES golden fails a
--backend DirectVulkan run with "matched a golden another backend declares"; a
synthetic PNG pair with a closed-form SSIM (identical = 1.0 exactly, constant
black vs white = C1/(255^2+C1), inverted halves = (C2-2v)/(2v+C2)); the pre-fix
resolveSpawnServerPath body fails eight of SpawnServerPathTest's checks; and the
DirectVulkan TCP proof battery is the DirectGLES one under the other prefix,
because an empty prefix selection raises nothing and reads as a clean pass.
CI: test_run_tcp_matrix.py and test_compare_actuals.py join the R-16 step. The
matrix driver's stub regressions had NO CI step - they were run by hand and
recorded in a note - and they guard the tool that decides whether a device golden
run passed.
Counts in WSL: test_run_tcp_matrix 12 -> 18, tcp_lane_tools 4 -> 6,
test_compare_actuals 12 new, test_android_lifecycle green with the new Java unit.
Gate 8's last two numbers (SEG_STAGE bytes/frame, records/frame post-
chunking) had no producer at all: the summary line's byte classes are
backend-to-driver bytes and maxrec is a max, not a per-frame. Two push-
guarded counters answer them at their single choke points - ByteClass::
StageSegmentBytes at PipeWireCodec's StageAllocate (the only allocation
behind StageBytes, so every buffer walk, texture slab and CSO archive
lands there and emitters cannot double-count) and CallClass::WireRecords
on EncodeRecord's committed path (post-chunking an emitter no longer
knows how many records its one call produced). A staged-blob histogram
rides along. G1 holds: symbols 0/0/0/0, .text byte-identical, pull build
carries no MG_Remote symbols.
The spawn server never counted anything: PipeStats::Init() is a step of
MobileGL::Initialize(), which ServerMain never reaches, so every Enabled()
site in that process - including ServerLoop's own ServerWaits/ServerParks
publish - stayed false and the server log carried no summary line at all
(a zero that means 'another process' printed in the shape of one that
means 'never waited' is the milder misreading; the line was simply
absent). ServerMain now arms and closes PipeStats at its role boundary -
after MG_ConfigLoader::Init (Init latches from the parsed config), after
the server-role latch (the log sink's role), and before ::_exit(0).
Red-once: 0 server summary lines before, 29 after, SSIM 1.0 unchanged.
MOBILEGL_PIPE_STATS_FILE now derives per role through the log sink's
exported RoleLogPath rule - run.client.json / run.server.json, the bare
name resolves to no file - because both roles reach FormatJson() and ofstream
trunc made the second writer replace the first. G1 caught three separate
leaks on the way (a string temporary in the monolith body, one resized
Shutdown() from unequal #if arms, two rewritten MGLOG_W literals in
.rodata); the pull arm is statement-for-statement the original.
Device tooling for the paired A/B lands with it: pin_device.sh gains a
measured 2f7cbe2e entry (this board clamps the GPU at 1050 MHz -
thermal_pwrlevel writes 0 and reads back 1, max_gpuclk is read-only;
absolute times are not comparable with the 1100 MHz-era rows), and
tools/device_bench/p6/ carries the session collector and renderer beside
run_android_retrace_local.py's MOBILEGL_TRACE_PACKAGE override.
Unit 2367/2367 (PipeStats 25/25), spawn integration 102/102.
The spawn work had the two roles append to one MOBILEGL_LOG_FILE_PATH under
O_APPEND. It was correct - two processes at one offset would corrupt the file,
O_APPEND made each write atomic - but it put two sessions' lines in one place
and made every per-side assertion a search. This splits the sink: the client
role writes <base>.client.log and the server role writes <base>.server.log.
THE BASE NAME NAMES NO FILE ANY MORE, AND THAT IS THE POINT. Leaving the
client at the old unsuffixed path would have been the compatible choice and the
wrong one: a reader that was not updated would go on finding a file, go on
passing, and go on seeing only half the session - and a refusal census that
reports a confident zero for the half it cannot see is worse than one that
fails. With both names moved, an un-updated reader gets "file not found" and
says so. There were 34 such readers; every one is moved here.
THE ROLE IS PER-THREAD, not per-process, because under inproc both roles are
threads of one process. Log.cpp keys the sink on a thread-local role: the apply
thread declares itself server (ServerLoop.cpp), a spawned image is server for
its whole life (MOBILEGL_IPC_ROLE=server, read once), and everything else is
the client. The naming rule (RoleLogPath) and the two whole-log operations
(ReadRoleLogs / TruncateRoleLogs) are EXPORTED from the library, so no reader
reimplements the derivation - a second copy is the copy that falls behind.
Readers, by how they read:
- unit mains set MOBILEGL_LOG_FILE_PATH and read a role file: they keep the
base in the env and derive the client path from the library helper.
- the death-test child runners (ReadLog) now concatenate BOTH roles: a death
test asserts the CHILD said something, and the applier's refusals - the very
diagnostics these check - are written under the server role.
- the offset-based Emit readers derive the client path once after setenv.
- PipeStatsWindow reads both, because draws/vbs increment in the backend's
PrepareForDraw - the apply thread, the server role - so the summary line is
in the server file even under inproc.
- run_trace_case.cmake's split census reads both files; the client keeps the
transport markers.
- split_log_paths.py (and its self-test, and the control stub) derive both
role files from the declared base.
- the Android script pulls both role files off the device and concatenates
them, so every downstream check is unchanged - and it judges a role file by
CONTENT, because `run-as test -f` through `exec-out` returned 0 for a missing
file on the device and `exec-out` folds cat's own not-found line into stdout.
The monolith arm legitimately has no server file, and that is a silent skip
rather than a one-line server log.
Guarded so a pull build is untouched: RoleLogPath and the role sink are wholly
inside MOBILEGL_BUILD_DISAGGREGATED; a non-disaggregated build writes the base
path exactly as before. G1 re-measured - 0 symbol diff, .text unchanged at
0xa52203.
Verified on device 2f7cbe2e: monolith writes only the client log (no phantom
server log), inproc and spawn each write both and pass their transport proofs
(spawn: server role in pid 16125). Unit 2353 (one pre-existing environmental
failure), integration-split 186/186, integration-spawn 102/102, retrace 6/6,
control smoke 12/12, split-log-paths self-test 9/9.
The Android retrace matrix gains a third lane, spawn-acceptance, beside
monolith-control and inproc-acceptance. Until now nothing in CI had ever
started a second MobileGL process on a device at all.
The apk.yml retrace job already carried a `transport` matrix dimension and
already branched on it, so the lane itself is three edits: a row in
github_transport_matrix, an elif in the retrace step, and a summary to render.
What needed thought was the gate.
THE SPAWN LANE IS APK-ONLY, and that is a statement about where evidence is
missing rather than about what spawn can do. Linux already has
test.yml's `retrace-split`, which is {backend, case, transport} over inproc AND
spawn with the two negative controls that make it mean anything. A spawn row in
the Linux transport matrix would pay for a second full sweep proving nothing -
and test.yml's `retrace` job stages no server image, so the lane would not
merely be redundant, it would be red. self_test_transport_matrices now asserts
the absence in both directions, because an accidental removal of the `if apk`
has to be caught in the self-test rather than in CI.
--require-spawn REQUIRES TWO SENTENCES, and the second is the one that matters.
ConfigLoader's marker proves the VALUE RESOLVED - it exists only in the Spawn
arm, which exists only under MOBILEGL_BUILD_DISAGGREGATED - but that is a
statement about a parser. `spawn ARMED - the server role runs in pid N` is
written only after the launch, the connect and the handshake have all
succeeded, and the pid in it is the SERVER's. It is the only line in the log a
same-process run cannot produce.
Both halves red once, measured:
* a monolith run demanded to prove spawn: REFUSED, transport_resolution_marker
false, server_pid null - while the retrace itself scored ssim=1.000000,
which is exactly the trap. OpenRA matches its golden under monolith too, so
the picture has never been the evidence.
* the same run's real spawn log with the one ARMED line deleted: REFUSED,
transport_resolution_marker TRUE and server_pid null. That is the
"resolved the transport and then never left the process" case, isolated.
THE PROOF USES `any` WHERE THE inproc GATE USES `all`, and the difference is
load-bearing rather than a relaxation. Under spawn two processes append to one
MOBILEGL_LOG_FILE_PATH, and the launcher deliberately scrubs every
MOBILEGL_IPC_* from the child's environment (anti-recursion catch (b)) - so the
server's own `Config: IPC` line reads strict=0 role-split-state=0 BY
CONSTRUCTION. Demanding every line match would red every spawn run for the one
thing the design requires.
--use-pbuffer is not optional on this lane: until P12 a spawned server cannot be
handed a window, because an ANativeWindow* is a pointer into the CLIENT's
process (Rule H). MOBILEGL_IPC_SERVER_PATH is deliberately NOT passed -
nativeLibraryDir carries an install-time hash that nothing off the device can
spell, and the Activity resolves it.
The local runner now passes the same knobs and the same --require-* flag CI
does, so "it works locally" can no longer mean "it ran monolith locally". That
change immediately caught its own omission: the first local run with the gate
armed failed on client_ipc_line_present=false because the runner had been
sending MOBILEGL_TRANSPORT alone.
Verified on device 2f7cbe2e: OpenRA.DirectGLES spawn, ssim=1.000000,
mismatchPixels=0, "real spawn transport, server role in pid 9885, and zero
Fatal records confirmed".
OpenRA.DirectGLES, transport=spawn, on a Redmi: ssim=1.000000,
mismatchPixels=0, zero Fatal{ lines, three runs for three. The server is a
process the client launched out of the APK's lib/arm64/, and the client's own
log names its pid.
Four things stood between the desktop green and the device green, and every
one of them was invisible on desktop for the same reason: a build tree is not
a deployment, and Linux is not Android.
1. THE SERVER COULD NOT LINK. It links libMobileGL.so and is EXEC'd, not
dlopen'd, so it gets a fresh linker namespace with the DEFAULT search path
- and an APK's lib/<abi>/ is not on it. `CANNOT LINK EXECUTABLE ...:
library "libMobileGL.so" not found: needed by main executable`, before a
single line of MobileGL ran. On desktop CMake had been giving the target an
absolute build-tree RPATH, so the WSL server resolved its dependency
through a path no phone has. $ORIGIN is asked for via BUILD_RPATH (a raw
-Wl,-rpath,$ORIGIN reaches the generator unquoted and the SHELL eats it -
the device linker said `normalize_path - invalid input: "$"`), and because
the NDK drops RPATH from the link anyway, the LAUNCHER states
LD_LIBRARY_PATH: it is the one component that has already resolved where
the image lives, and a pair that is not co-located is not a pair.
2. THE RENDEZVOUS NEEDED A WRITABLE DIRECTORY. An Android app has no /tmp, so
bind() in the child had nowhere to put the node and the client refused by
name 20 s later. AF_UNIX endpoints now accept the ABSTRACT namespace with a
leading '@' - the systemd/D-Bus convention - which has no filesystem
presence at all: no directory, no mode bits, and no stale node to unlink
after a crash, which is the bind() failure Listen() has been working
around. Its reach is the network namespace, which for an app is the app.
StartSpawned mints '@' names; an explicit path still works everywhere, and
that is what P12's "connect to something somebody else is listening on"
will use.
3. PBUFFER WAS A DirectGLES-ONLY KNOB. Twice over: trace-replay-ci.sh gated
`--ez use_pbuffer true` on the backend name, and the JNI demanded a native
window for DirectVulkan unconditionally. A surface SHAPE is not a backend
property, and the coupling mattered because the Android GLWS makes a window
surface whenever gNativeWindow is non-null - so handing Magma a window was
the same as asking for one. Until P12 a spawned server cannot be given a
window at all: an ANativeWindow* is a pointer into the CLIENT's process and
SetWindowHandle is refused on the way across with Fatal{UnmigratedSurface,
"AndroidNativeWindow@P12"} (Rule H). Which is exactly what happened, one op
after the server had come up green.
4. THE RUNNER LOOKED IN ONE OF THE TWO OUTPUT DIRECTORIES. assembleTraceRelease
writes apk/trace/release; CI has staged the same artifact under
apk/trace/debug. A tree that has built locally has both, and picking the
wrong one retraces last month's APK and says nothing about it. Newest of
either, now.
MEASURED AND NOT FIXED: DirectVulkan on this device diverges under BOTH split
transports and does so nondeterministically - inproc scored ssim=0.988998 /
9836 px, spawn scored 0.976494 / 14658 px and then 0.988998 / 9836 px on a
repeat, while monolith with the same pbuffer surface scores 1.000000 / 0 px.
The two transports draw from the same set of outcomes, so this is the split
path on Magma against a real GPU, not the process boundary: P6 neither caused
it nor can fix it. DirectVulkan spawn IS green on desktop (lavapipe, 1.000000).
G14: one test name added, AnAbstractEndpointNeedsNoWritableDirectoryAndLeaves-
NoFile, which pins both halves - the handshake crosses an abstract name, and
nothing is left on the filesystem. A '@' mistaken for a path would have bound
a file of that literal name, passed here, and failed on the one platform this
exists for.
Two small pieces, so an Android spawn arm is a flag rather than a project.
THE RUNNER GAINS --transport. It already had a generic env passthrough - one
intent extra carries `K=V;K=V` all the way to setenv before the library is
dlopened - so the transport itself costs one entry and no new plumbing in the
five files between here and the device. monolith passes NOTHING rather than
MOBILEGL_TRANSPORT=monolith, so the default arm's environment stays
byte-identical to what it was before this option existed: a control that
changed when the option landed would not be one.
THE SERVER PATH CANNOT COME FROM THE HOST. Every other knob can, but this one
names a file that must be EXEC-ABLE, and on Android there is exactly one such
place: an APK's lib/<abi>/. W^X has forbidden the app's own data dirs since
API 29, and the packager only puts a file in lib/<abi>/ if it is named
lib*.so - which is why the server executable wears that name, and what P0
spike A proved on two devices. nativeLibraryDir contains an install-time
hash, so no host-side script can spell the path; the process about to launch
the server is the only one that can. TraceReplayActivity resolves it, and
only when the caller actually asked for spawn.
AN EXPLICIT MOBILEGL_IPC_SERVER_PATH STILL WINS. A caller that set one is
pointing somewhere deliberately - at a pushed build, say - and silently
replacing it would make that knob untestable on the one platform where it
matters most.
The APK must be built with -Pmobilegl.buildDisaggregated=ON or none of this
is reachable: without it ConfigLoader has no transport parser, accepts the
value, ignores it (CONTRACT-P5 rule 5), and the run is monolith under a name
that says spawn. The --transport help says so where somebody will read it.
It was one case on one backend under one transport: OpenRA, DirectGLES,
inproc. It is now every CI case, both backends, inproc AND spawn - 154 CI
entries, 156 local ctest entries - because every defect the spawn work turned
up was a question about the SERVER's process and not about any one trace or
either backend.
THE DirectVulkan ARM. tools/trace_replay/CMakeLists.txt carried "a
DirectVulkan arm would be measuring a server nobody has written yet". True
when P5 wrote it; false since P5f/P7 landed Magma's server consumers and its
run-ahead readiness gate. Leaving it would have shipped a split shape whose
second backend nothing ever ran - the same class of hole as a lane that runs
monolith under a name that says split. Magma now brings up a server process
of its own and OpenRA retraces across it at ssim=1.000000, which is only
possible because this branch also taught the child to load its config: before
that, a DirectVulkan session brought up a DirectGLES server and the two
halves disagreed about which backend the capability mask describes.
`split` BECOMES AN OPT-OUT. It began as an opt-in naming one case, because
one case was the work. Marking 39 cases `"split": true` would carry no
information and would reinstate the hole the key was written to close, only
mirrored: a case added later would get the monolith arms and be silently
absent from the split ones, and absent is the one state nothing reds on. A CI
case is now in the subset unless it says `"split": false`, which is a
reviewable claim that a trace cannot cross a role boundary.
THE CI MATRIX GAINS A THIRD DIMENSION. The job took {backend, case} and
hard-coded MOBILEGL_TRANSPORT=inproc in its own run block, so `spawn` had no
way into CI at all; the .SPAWN ctest entries existed only for a local build.
It is {backend, case, transport} now, the backends are the case's own
ci_backends rather than a hard-coded DirectGLES, and the artifact names carry
the transport or the two arms of one {backend, case} would overwrite each
other's uploads.
inproc STAYS AS THE CONTROL. Same library, same codec, same applier, same
picture; the only difference is which process the server role runs in.
Running only spawn would leave "did the transport break it?" and "did the
phase break it?" answerable only together - which is the question every
failure in this phase has turned on, including the SIGSEGV that D1 explained.
BOTH NEGATIVE CONTROLS FOLLOW THE ARM. retrace_pull_library_control.sh and
retrace_drop_draw_control.sh each exported MOBILEGL_TRANSPORT=inproc
themselves, so on a spawn job they would have gone on proving something about
the other arm while spawn ran unguarded. They read SPLIT_TRANSPORT now, and
default to inproc so a caller that sets nothing behaves exactly as before.
scripts/ci/control_smoke_test.sh: 22 passed, 0 failed against the stubbed
ctest, including the two modes that matter most - a red whose SSIM never fell
and a knob that armed and dropped nothing.
The spawn arm needs a second binary and the inproc arm needs none, so the job
places libMobileGLServer.so only on the spawn arm and a missing one is a
named failure there rather than a LaunchServer refusal 78 jobs deep.
Measured locally as this lands: 39 of the 78 SPAWN entries complete, all
green. The rest are still running.
OpenRA.DirectGLES.SPAWN is green: ssim=1.000000, mismatchPixels=0, zero
Fatal{ lines, against a server in a process the client started by name. The
inproc SPLIT arm stays beside it as the control, so "did the transport break
it" and "did the phase break it" remain separately answerable.
Five defects stood between the handshake and that picture. Each one was
silent in its own way, and none of them looked like what it was.
1. THE DOORBELL COULD NOT WAKE ITSELF. SocketDoorbell's one-fd form sends on
the end it polls, so a write is delivered to the PEER - correct for waking
the other process, useless for waking a thread in this one. The server's
control pump posts into the apply thread's mailbox and must ring the bell
that thread parks on. Symptom: every eglInitialize crossed, landed in the
mailbox, and timed out at 5 s. The two-fd form (park on [0], notify on [1])
is restored here, and the bell handover goes from two descriptors to THREE
- the client needs the same split for the same reason. Seven offers now.
2. InitCapabilities HAD NO WIRE KIND. CONTRACT-P6 §2 predicted this exactly
("the request has a live client caller and the codec refuses to encode
it"). SurfaceOpKind gains 11, append-only; the decode range's upper bound
moves with it, or the new tag reads as out-of-range and surfaces as
Fatal{ProtocolCorruption} rather than the missing row it is. The op's name
loses its InprocOnly suffix, which stopped being true here.
3. THE CHILD LOADED NO CONFIG AT ALL. mobilegl_server_main calls
InitServerRoleForSpawn directly, and MG_ConfigLoader::Init is a step of
MobileGL::Initialize - so every MG_Config value in the server was its
static default. MOBILEGL_BACKEND_TYPE survives the launcher's scrub on
purpose and was read by nobody: a DirectVulkan session would have brought
up a DirectGLES server and then disagreed with its client about which
backend the capability mask describes.
4. AND THEREFORE Transport READ AS Monolith, which CONTRACT-P6 §3.1 (D1)
forbids in as many words. a6 measured the cost: 226 live lines test
`Transport != Monolith` to select the SERVER arm, 148 in MG_Backend.
Monolith flips all of them to frontend glue in the one process with no
frontend. Observed precisely as predicted - UnitTexturesByHandle() answered
false, Clear took the legacy arm that dereferences borrowed FRONTEND slot
pointers, and the apply thread took a SIGSEGV inside SyncNeccessaryTextures
on the first Clear record of the trace. It is pinned in-process rather than
through the environment because the environment is where it must NOT be:
LooksLikeAClientEnvironment exits 65 on a surviving MOBILEGL_TRANSPORT, and
that check is anti-recursion catch (b). Dial, not Transport, is that axis.
5. THE LOG WAS CORRUPT, NOT MERELY INTERLEAVED. Two processes opened the same
MOBILEGL_LOG_FILE_PATH with "w" and each carried its OWN file offset, so
they overwrote each other's bytes - half-lines like `ote client: CapsMirror
generation 1 adopted`, and the server's whole startup block vanishing under
the client's later writes. This cost real diagnosis time twice, because a
missing line read as a code path not taken. O_APPEND now; the client still
truncates once so a fresh run is not a pile of old ones, and the server
never does - it would delete the very startup lines a failed session needs.
run_trace_case.cmake's Fatal{ census reads exactly this file, so it was
also reading a lie.
The retrace runner's split assertions become transport-aware: one row per
transport it admits, and a transport with no row is a FATAL rather than a
fall-through, because the fall-through is the accident the block exists to
prevent. The spawn row carries a SECOND sentence and it is the stronger half
- ConfigLoader's marker proves the value RESOLVED, while `spawn ARMED - the
server role runs in pid N` is written only after the launch, the connect and
the handshake all succeeded, and carries a pid that is not this process's.
inproc resolves its own marker while running the server role on a thread
here; nothing on that path can write this one.
G14: one test name added (TheEleventhFramedOpIsInitCapabilitiesAndItCrosses-
TheWire), none removed. The ten-op table keeps its name and its scope rather
than being renamed to eleven.
Two unit failures predate this change and are environmental in this tree:
PipeCatalogue.FrontendNeverTakesAnApplierAddress (the frontend source root
resolves empty) and SessionHandshakeTest.TheAbiFingerprintChangesWhenAnyOf-
ItsInputsDoes (GIT_COMMIT_HASH_SHORT is empty here, which a6 already recorded
as red in this configuration).
Stage updates to adopted GPU-resident buffers on the command timeline instead of overwriting memory that earlier draws still read. Wait for submitted copies as well as pending commands before CPU readback.
Commit coherent mapped writes before Android trace buffer copies, and retire buffer shadows on deletion so copies see current vertices and name reuse cannot leave dangling dirty-shadow entries.
Validated draw ordering, submitted-copy readback, traced buffer copies, and mapped-buffer deletion/name reuse on Redmi Adreno 830. Minecraft 26.3-rc-3 Magma passed movement, HUD, and menu checks and exited normally; the new trace parsed all 64,410 frames without warnings.
- format_benchmark printed a p50 taken with the nearest-rank rule beside a medianFrameCpuMs the
device computes as the average of the two middle frames, and documented the two as one rule; on
an even window they differ (the pre-flight printed p50=8.261ms next to medianCpuMs=271.766).
p50 now goes through series_median, which is SummarizeSeries' rule transcribed; p95 and p99 stay
nearest rank, which is the device's rule for p95 and the honest extension of it for the p99 the
device does not compute at all
- require_verified_profile treated a profile that simply omits PROFILE_VERIFIED as verified, which
is the fail-open default a profile written by copying another one inherits - exactly the case the
guard exists for. It defaults to unverified now, odinlite.env carries PROFILE_VERIFIED=1
explicitly (it is the one profile that earned it), and the refusal says "says 0, or says nothing"
- the two new profiles claimed profile.sh refuses an unverified profile; it has no such check and
needs none - it records a simpleperf profile and pins nothing. The claim is corrected in both
profiles and in the README rather than a guard added where there is nothing to guard
- the handle-ABA / CSO control step in test.yml set only MOBILEGL_ITEST_REQUIRE_GPU while its
sibling verify step sets the three MOBILEGL_MAGMA_* fixes and arms core dumps. It runs the same
DirectVulkan binary on the same runner, so a crash there left no core; it now carries both
- The metric the disaggregation GO/NO-GO hangs on is per-thread CPU p50/p99, and the tree had no
first-party collector for it: no CLOCK_THREAD_CPUTIME_ID, no getrusage, no RUSAGE_THREAD, no
/proc/self/task anywhere under MobileGL/, tools/, android-plugin/ or scripts/.
- The retrace loop is the cheapest honest place to take it. Retrace runs --singlethread and
trace_benchmark states that Begin/OnFrameBoundary/End are only ever reached from that one
thread, so that thread's CPU time IS the client-side CPU cost: one extra clock_gettime per
frame, no root, no profiler, no debuggable build, no sampling - and no instrumentation
committed to a hot path inside the library.
- The CPU reading is taken before the wall reading at each boundary, so the syscall lands in the
wall delta rather than hiding inside the CPU delta. An inflated wall number is visible; a
deflated CPU number would not be.
- Report::frameCpuMs is the same length as frameMs or it is EMPTY. A clock that started failing
mid-run would otherwise be silently re-indexed and put frame N next to frame N+k, and an empty
series and a series of zeroes are different claims about the platform.
- SummarizeBenchmark is split into SummarizeSeries and reused verbatim for the CPU series rather
than duplicated: same tail window, same even-count median rule, same nearest-rank p95, or the
delta between the two series stops meaning anything.
- benchmark.json gains meanFrameCpuMs / medianFrameCpuMs / p95FrameCpuMs and the WHOLE
frameCpuTimesMs[] array beside frameTimesMs[]; result.json and the completion line gain the
three headline numbers. p99 therefore needs no device change - it is a host-side reduction over
an artefact that already exists, and run_android_retrace_local.py prints p50/p95/p99 off the
same trailing window the device summarised.
- Pre-flighted on the desktop CLI, which shares the same core: 2-frame run, cpu series aligned
with the wall series and strictly below it (wall 1290.787/20.225 ms, cpu 535.271/8.261 ms).
- the block looked for Fatal{PipeVerifyDiffer and Fatal{UnmigratedPipeInput only. A
misspelt MOBILEGL_PIPE_VERIFY_CORRUPT / MOBILEGL_PIPE_POISON_OMIT reports
Fatal{PipeVerifyBadKnob, and it was caught only because D2 makes that one abort the
process - which is precisely what MOBILEGL_PIPE_VERIFY_FATAL=0, the supported triage
configuration, takes away. A typo'd knob would have left a negative-control run
looking healthy
- the FATAL_ERROR text now says what each of the three means and where the two
vocabularies live, because that message is the whole diagnosis a `cmake -P` step gets
- A retrace that exports MOBILEGL_PIPE_VERIFY=1 at a library configured without
-DMOBILEGL_PIPE_VERIFY=ON is a no-op: the frames still match their goldens and the
case reports green having compared nothing. run_trace_case.cmake now demands the
evidence whenever the variable is set to anything but 0/false - mobilegl.log must
exist, must carry "MGPipe: verify armed", and must carry neither
Fatal{PipeVerifyDiffer nor Fatal{UnmigratedPipeInput. With the variable unset the
script is byte-for-byte the old one.
- The Fatal scan is not redundant with the replay's exit status:
MOBILEGL_PIPE_VERIFY_FATAL=0 is the supported triage configuration, and there a
divergence is logged and counted rather than aborted, so the run would finish 0
with its own report sitting unread in the log.
- LABELS retrace on both registrations: the lane was selectable only by regex, so
`ctest -L retrace --no-tests=error` - the spelling that reds a lane which
registered nothing - could not be written at all.
- "verify": true on eight cases (the five 180s cases plus minecraft-1.21.4-in-world,
minecraft-1.21.4-fabric-sodium-in-world and improved-transparency-minecraft-26.3),
and --format github-verify-matrix over that subset: 16 entries, against the full
lane's 77. The verify build compares at every verb boundary and again at every
accessor read, which the design budgets at 5-10x, so the per-push job runs the
subset and the full sweep is a phase-exit / workflow_dispatch run.
- The flag is validated in the manifest loader, not at the matrix, so a non-boolean
or a verify case excluded from CI is a loud error in every consumer instead of a
subset that is quietly one case short.
- PLAN-B.md §8.2 and appendix B add a batch of new runtime switches
(MOBILEGL_PIPE_PUSH / _VERIFY / _STATS / _LEGACY_MEMOS / _TEXEL_RETAIN_MB /
_INDEX_MIRROR_MB, plus MOBILEGL_IPC_* later), and §11 P0 wants them parsed beside
the existing ones. Today every knob that has to reach an Android replay costs an
edit in five files - run_android_retrace_local.py, trace-replay-ci.sh,
TraceReplayActivity's request record, the JNI marshalling, and the setenv block in
trace_replay_core.cpp. That per-knob tax is what this replaces: one extra,
`--es mobilegl_env "K=V;K=V"`, carries all of them.
- Applied last, immediately before dlopen(libMobileGL.so), so it can also override
the dedicated fields above it - MobileGL's config is read during the load, and an
escape hatch that cannot beat the defaults is not one. An entry with no '=' unsets
the variable, which is the only way to clear a default the marshalling sets.
- The existing per-knob flags stay: they carry semantics beyond a setenv (use_angle
also selects a variant, the dump lists are joined, DirectVulkan forces the
R11G11B10F fallback), and rewriting them as env strings would move that logic into
the callers.
- Surface: --env / MOBILEGL_TRACE_ENV in trace-replay-ci.sh, repeatable --env
KEY=VALUE in run_android_retrace_local.py, `mobilegl_env` intent extra,
Request::envOverrides.
- The two-level parse now lives in trace_env_overrides.hpp, beside the semicolon
splitter it shares with the texture and FBO dump lists, and
tools/trace_replay/trace_env_overrides_test.cpp pins it: the empty entries a
trailing ';' leaves behind must not become unsetenv(""), `K=` must stay a Set of
the empty string rather than an Unset (a knob read with getenv() != nullptr sees
those as opposite answers), and only the FIRST '=' may separate, or a value
carrying '=' is truncated without a word of warning. The whole MOBILEGL_PIPE_*
batch rides on this parse, and the only lane that exercised it end to end was an
on-device retrace, which would have reported a splitting bug as "the knob had no
effect".
- The check is built and RUN at build time and mobilegl_trace_replay depends on it,
so `cmake --build ... --target mobilegl_trace_replay` - the exact command of
test.yml's "Build trace replay" job, which never invokes ctest - runs it. It is
assert-free on purpose: that lane configures Release, and <cassert> under NDEBUG
would compile every check into a green run that checked nothing. Negative control:
swapping find('=') for rfind('=') fails 2 checks, and keeping the splitter's empty
entries fails 2 more.
Initialize the desktop replay Request from the three iterationRP environment flags before loading MobileGL. Without this, the Request defaults caused the replay core to unset CI's exported flags, leaving all repairs disabled despite the workflow configuration.
iterationRP's Program 203 declares shared vec2 prefixSumCache[32] for a
512-invocation workgroup indexed by gl_SubgroupID; any device narrower
than 16 lanes partitions into more than 32 subgroups and the pack writes
shared memory out of bounds (heap corruption on lavapipe's CPU
rasterizer, ssim 0.028 on the CI retrace). Fix it where the fault lies -
in the fixture - and keep the GL contract sound everywhere else:
- FixIterationRPSubgroupScratchPass: fingerprint-gated SPIR-V pass that
grows exactly that array to ceil(invocations/width) entries on sub-16-lane devices; every other module passes through byte-identical.
- DeriveNumSubgroupsPass stays default-on for the Adreno topology bug
and is made spec-sound: pipelines request REQUIRE_FULL_SUBGROUPS
whenever the workgroup shape makes the flag legal (computeFullSubgroups
enabled, local_size_x a multiple of the native width, subgroup count
within maxComputeWorkgroupSubgroups).
- EmulateSubgroupsPass: 32-lane virtual-subgroup lowering kept in-tree
as a last resort, enabled only by MOBILEGL_MAGMA_EMULATE_SUBGROUP=1 on
devices with no native subgroup support; fails closed on extended
subgroup instructions and on modules whose added scratch would exceed
maxComputeSharedMemorySize.
- IterationRPFirstReductionScenario skips gracefully outside the pack's
16..256-lane source domain; the new IterationRPScratchFixScenario runs
the fixture-shaped reduction on any width and asserts the exact
width-independent total. DriverPost keeps reporting FAIL on
out-of-domain devices.
- Program203 -> IterationRP rename throughout; the per-trace
num_subgroups_quirk plumbing is removed from the trace replayer, JNI
chain, and CI workflows.
swung0x48
