[MG_Remote, CI] (P6): the split retrace lane becomes the whole CI matrix, both backends, both transports

It was one case on one backend under one transport: OpenRA, DirectGLES,
inproc. It is now every CI case, both backends, inproc AND spawn - 154 CI
entries, 156 local ctest entries - because every defect the spawn work turned
up was a question about the SERVER's process and not about any one trace or
either backend.

THE DirectVulkan ARM. tools/trace_replay/CMakeLists.txt carried "a
DirectVulkan arm would be measuring a server nobody has written yet". True
when P5 wrote it; false since P5f/P7 landed Magma's server consumers and its
run-ahead readiness gate. Leaving it would have shipped a split shape whose
second backend nothing ever ran - the same class of hole as a lane that runs
monolith under a name that says split. Magma now brings up a server process
of its own and OpenRA retraces across it at ssim=1.000000, which is only
possible because this branch also taught the child to load its config: before
that, a DirectVulkan session brought up a DirectGLES server and the two
halves disagreed about which backend the capability mask describes.

`split` BECOMES AN OPT-OUT. It began as an opt-in naming one case, because
one case was the work. Marking 39 cases `"split": true` would carry no
information and would reinstate the hole the key was written to close, only
mirrored: a case added later would get the monolith arms and be silently
absent from the split ones, and absent is the one state nothing reds on. A CI
case is now in the subset unless it says `"split": false`, which is a
reviewable claim that a trace cannot cross a role boundary.

THE CI MATRIX GAINS A THIRD DIMENSION. The job took {backend, case} and
hard-coded MOBILEGL_TRANSPORT=inproc in its own run block, so `spawn` had no
way into CI at all; the .SPAWN ctest entries existed only for a local build.
It is {backend, case, transport} now, the backends are the case's own
ci_backends rather than a hard-coded DirectGLES, and the artifact names carry
the transport or the two arms of one {backend, case} would overwrite each
other's uploads.

inproc STAYS AS THE CONTROL. Same library, same codec, same applier, same
picture; the only difference is which process the server role runs in.
Running only spawn would leave "did the transport break it?" and "did the
phase break it?" answerable only together - which is the question every
failure in this phase has turned on, including the SIGSEGV that D1 explained.

BOTH NEGATIVE CONTROLS FOLLOW THE ARM. retrace_pull_library_control.sh and
retrace_drop_draw_control.sh each exported MOBILEGL_TRANSPORT=inproc
themselves, so on a spawn job they would have gone on proving something about
the other arm while spawn ran unguarded. They read SPLIT_TRANSPORT now, and
default to inproc so a caller that sets nothing behaves exactly as before.
scripts/ci/control_smoke_test.sh: 22 passed, 0 failed against the stubbed
ctest, including the two modes that matter most - a red whose SSIM never fell
and a knob that armed and dropped nothing.

The spawn arm needs a second binary and the inproc arm needs none, so the job
places libMobileGLServer.so only on the spawn arm and a missing one is a
named failure there rather than a LaunchServer refusal 78 jobs deep.

Measured locally as this lands: 39 of the 78 SPAWN entries complete, all
green. The rest are still running.
This commit is contained in:
2026-09-21 12:31:49 -04:00
parent 7e244b3775
commit dc5bd935de
5 changed files with 146 additions and 59 deletions
+50 -14
View File
@@ -1995,17 +1995,19 @@ jobs:
# The subset the verify build retraces ("verify": true in trace_cases.json). It is a
# SUBSET of the matrix above, so retrace-verify needs no fixtures of its own.
echo "verify-matrix=$(python3 tools/trace_replay/trace_cases.py --ci --format github-verify-matrix)" >> "$GITHUB_OUTPUT"
# P5's split subset ("split": true, DirectGLES only). Also a SUBSET of the matrix above,
# so retrace-split needs no fixtures of its own either. It is one case today - OpenRA,
# which is what the phase gate names - and trace_cases.py refuses a `split` case that is
# not in CI or does not run DirectGLES, so the subset cannot silently become empty.
# The split subset: {backend, case, transport} over inproc AND spawn, both backends, for
# every CI case that has not said `"split": false`. Still a SUBSET of the matrix above,
# so retrace-split needs no fixtures of its own. P5 shipped it as one case (OpenRA, which
# the phase gate names); P6 widened it to the whole CI matrix, because every bug the
# spawn work found was a question about the SERVER's process rather than about one trace.
# trace_cases.py refuses a `split` case that is not in CI, so the subset stays a subset.
SPLIT_MATRIX=$(python3 tools/trace_replay/trace_cases.py --ci --format github-split-matrix)
# AND IT MUST NOT BE EMPTY. An empty `include` is not an error to GitHub - it skips the
# whole retrace-split job with no red anywhere - so the one way this subset can vanish
# silently is guarded here. trace_cases.py now also rejects an unknown manifest key, which
# was the hole: `"splitt": true` loaded clean and emptied the subset (review N-4).
if [ "$(printf '%s' "${SPLIT_MATRIX}" | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["include"]))')" -lt 1 ]; then
echo "::error::the split retrace subset is EMPTY. No case in trace_cases.json carries \"split\": true, so retrace-split would be skipped with no red. Exit gate E2 names OpenRA."
echo "::error::the split retrace subset is EMPTY. `split` is an OPT-OUT as of P6 - a CI case is in this subset unless it carries \"split\": false - so an empty subset means either every case opted out or the CI matrix itself is empty. Either way retrace-split would be skipped with no red, and exit gate E2 names OpenRA."
exit 1
fi
echo "split-matrix=${SPLIT_MATRIX}" >> "$GITHUB_OUTPUT"
@@ -2415,10 +2417,15 @@ jobs:
build-retrace/tools/trace_replay/${{ matrix.case }}/${{ matrix.backend }}/output/**
if-no-files-found: warn
# P5's retrace arm: retrace-verify with MOBILEGL_PIPE_VERIFY=1 swapped for
# MOBILEGL_TRANSPORT=inproc and the comparator's symbol swapped for an MG_Remote one. Exit gate
# E2 is one row - OpenRA at SSIM >= 0.99 - and trace_cases.json's `split: true` is where that
# subset lives.
# The retrace arm of the split shape: retrace-verify with MOBILEGL_PIPE_VERIFY=1 swapped for
# MOBILEGL_TRANSPORT and the comparator's symbol swapped for an MG_Remote one. Exit gate E2 is
# one row - OpenRA at SSIM >= 0.99 - but the lane is no longer one row: as of P6 it is
# {backend, case, transport} over the whole CI matrix, both backends, inproc AND spawn.
#
# inproc IS THE CONTROL AND SPAWN IS THE CLAIM. Same library, same codec, same applier, same
# picture; the only difference is which PROCESS the server role runs in. Running only spawn
# would leave "did the transport break it?" and "did the phase break it?" answerable only
# together, which is the question every failure in this phase turned out to hinge on.
#
# IT RUNS THE UNCHANGED CTEST NAMES with the transport exported in the JOB environment, rather
# than the SPLIT-suffixed variant entries. build-retrace configures without
@@ -2434,7 +2441,7 @@ jobs:
# SSIM is neither of them: the nm check below (this library carries MG_Remote) and the
# transport-resolution line in the library's own log (this PROCESS resolved inproc).
retrace-split:
name: retrace split (${{ matrix.backend }}, ${{ matrix.case }})
name: retrace split (${{ matrix.backend }} / ${{ matrix.transport }}, ${{ matrix.case }})
runs-on: ubuntu-latest
timeout-minutes: 240
needs:
@@ -2522,7 +2529,24 @@ jobs:
fi
echo "the library at build-linux/libMobileGL.so is the split build"
- name: Retrace and validate under MOBILEGL_TRANSPORT=inproc
# THE SPAWN ARM NEEDS A SECOND BINARY and the inproc arm needs none, which is the whole
# difference between the two at this level: inproc's server role is a thread in the replay
# process, spawn's is a PROCESS the client launches from this image. It is packaged by
# build-linux-split's `find -name '*.so'` sweep - libMobileGLServer.so is an executable that
# wears a .so name so an APK will install it somewhere exec-able - and a missing one must be
# a NAMED failure here rather than a LaunchServer refusal 78 jobs deep.
- name: Place the server image (spawn arm only)
if: ${{ matrix.transport == 'spawn' }}
run: |
if [ ! -f build-split/libMobileGLServer.so ]; then
echo "::error::the split runtime carries no libMobileGLServer.so, so transport=spawn has no server to launch. ConfigLoader would resolve Spawn, LaunchServer would refuse BY NAME, and this job would red on a message about an unresolvable image rather than about a packaging gap."
exit 1
fi
cp build-split/libMobileGLServer.so build-linux/libMobileGLServer.so
chmod +x build-linux/libMobileGLServer.so
echo "server image: $(readlink -f build-linux/libMobileGLServer.so)"
- name: Retrace and validate under MOBILEGL_TRANSPORT=${{ matrix.transport }}
working-directory: build-retrace/tools/trace_replay
# run_trace_case.cmake turns MOBILEGL_TRANSPORT into assertions of its own - the library
# must have RESOLVED the transport in this process, and its log must carry no Fatal{ - so a
@@ -2532,7 +2556,11 @@ jobs:
run: |
ulimit -c unlimited
sudo sysctl -w kernel.core_pattern='/tmp/core.%e.%p'
export MOBILEGL_TRANSPORT=inproc
export MOBILEGL_TRANSPORT='${{ matrix.transport }}'
# Absolute, because the replay's working directory is not this one and LaunchServer
# resolves the image relative to nothing. Harmless on the inproc arm, which reads it
# only to decide it does not need it.
export MOBILEGL_IPC_SERVER_PATH="${GITHUB_WORKSPACE}/build-linux/libMobileGLServer.so"
ctest -V --no-tests=error --timeout 10800 \
-R '^MobileGLTraceReplay\.${{ matrix.case }}\.${{ matrix.backend }}$'
@@ -2572,6 +2600,10 @@ jobs:
CONTROL_TMPDIR: ${{ runner.temp }}
PULL_LIBRARY: ${{ github.workspace }}/pull-runtime/build-monolith-control/libMobileGL.so
FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so
# The control has to run the arm it is a control FOR. Hard-coded inproc, it would go on
# proving something about the other arm while spawn ran unguarded.
SPLIT_TRANSPORT: ${{ matrix.transport }}
MOBILEGL_IPC_SERVER_PATH: ${{ github.workspace }}/build-linux/libMobileGLServer.so
run: >-
bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_pull_library_control.sh"
'${{ matrix.case }}' '${{ matrix.backend }}'
@@ -2599,6 +2631,10 @@ jobs:
CONTROL_TMPDIR: ${{ runner.temp }}
LIBRARY_LOG: ${{ matrix.case }}/${{ matrix.backend }}/output/mobilegl.log
FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so
# The control has to run the arm it is a control FOR. Hard-coded inproc, it would go on
# proving something about the other arm while spawn ran unguarded.
SPLIT_TRANSPORT: ${{ matrix.transport }}
MOBILEGL_IPC_SERVER_PATH: ${{ github.workspace }}/build-linux/libMobileGLServer.so
run: >-
bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_drop_draw_control.sh"
'${{ matrix.case }}' '${{ matrix.backend }}'
@@ -2624,7 +2660,7 @@ jobs:
if: failure()
uses: actions/upload-artifact@v7
with:
name: retrace-split-core-dumps-${{ matrix.backend }}-${{ matrix.case }}
name: retrace-split-core-dumps-${{ matrix.transport }}-${{ matrix.backend }}-${{ matrix.case }}
path: /tmp/core.*
if-no-files-found: ignore
@@ -2632,7 +2668,7 @@ jobs:
if: always()
uses: actions/upload-artifact@v7
with:
name: retrace-split-result-${{ matrix.backend }}-${{ matrix.case }}
name: retrace-split-result-${{ matrix.transport }}-${{ matrix.backend }}-${{ matrix.case }}
path: |
build-retrace/tools/trace_replay/${{ matrix.case }}/actual-images/**
build-retrace/tools/trace_replay/${{ matrix.case }}/${{ matrix.backend }}/output/**
+12 -5
View File
@@ -84,14 +84,21 @@ if [ "${matched}" -lt 1 ]; then
exit 1
fi
# A PREVIOUS RUN'S LINE MUST NEVER ARM THIS ONE. The library opens its log fopen(path, "w"), so
# the replay truncates it - but only if the replay gets that far, and a run that died in the
# loader would leave the baseline's log in place with a perfectly good "control armed" line in
# it. Removing it first is the same rule split_negative_controls.sh's `reset` step follows.
# A PREVIOUS RUN'S LINE MUST NEVER ARM THIS ONE. The CLIENT truncates this log once at open and
# then appends (P6 made the sink O_APPEND, because under spawn a second process writes the same
# file and two truncating handles overwrite each other's bytes) - but it only truncates if the
# replay gets that far, and a run that died in the loader would leave the baseline's log in place
# with a perfectly good "control armed" line in it. Removing it first is the same rule
# split_negative_controls.sh's `reset` step follows.
rm -f "${LIBRARY_LOG}"
out="${CONTROL_TMPDIR}/retrace-drop-draw-output.txt"
export MOBILEGL_TRANSPORT=inproc
# THE TRANSPORT COMES FROM THE JOB, not from this file. retrace-split is a matrix over
# {backend, case, transport} as of P6, and a control that hard-coded `inproc` would have gone on
# proving something about the OTHER arm while the spawn arm ran unguarded. Default inproc so a
# caller that sets nothing behaves exactly as before.
transport="${SPLIT_TRANSPORT:-inproc}"
export MOBILEGL_TRANSPORT="${transport}"
export MOBILEGL_IPC_E2_DROP_DRAW=1
"${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1
control_rc=$?
+6 -1
View File
@@ -105,7 +105,12 @@ if [ "${remote_count}" -ne 0 ]; then
fi
out="${CONTROL_TMPDIR}/retrace-control-output.txt"
export MOBILEGL_TRANSPORT=inproc
# THE TRANSPORT COMES FROM THE JOB, not from this file. retrace-split is a matrix over
# {backend, case, transport} as of P6, and a control that hard-coded `inproc` would have gone on
# proving something about the OTHER arm while the spawn arm ran unguarded. Default inproc so a
# caller that sets nothing behaves exactly as before.
transport="${SPLIT_TRANSPORT:-inproc}"
export MOBILEGL_TRANSPORT="${transport}"
"${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1
control_rc=$?
cat "${out}"
+31 -20
View File
@@ -449,10 +449,14 @@ endfunction()
# --- P5: the SPLIT arm of the retrace matrix -------------------------------------------------
#
# One variant entry per case that trace_cases.json marks `split: true` - today that is OpenRA
# alone, which is the only target the phase gate names and (measurably) the only fixture hydrated
# locally. DirectGLES only: the split shape's first backend is Espryt, and a DirectVulkan arm
# would be measuring a server nobody has written yet.
# One variant entry per case that trace_cases.json marks `split: true`.
#
# BOTH BACKENDS, as of P6. The DirectGLES-only note this comment used to carry -
# "a DirectVulkan arm would be measuring a server nobody has written yet" - was
# true when P5 wrote it and stopped being true at P5f/P7, which landed Magma's
# server consumers and its run-ahead readiness gate. Leaving it would have meant
# shipping a split shape whose second backend nothing ever ran, which is the
# same class of hole as a lane that runs monolith under a name that says split.
#
# REGISTERED ONLY UNDER MOBILEGL_BUILD_DISAGGREGATED, because the arm is a statement about a
# library that compiled MG_Remote. A build without the option accepts MOBILEGL_TRANSPORT=inproc
@@ -463,24 +467,31 @@ function(add_trace_replay_split_test CASE_NAME)
if(NOT MOBILEGL_BUILD_DISAGGREGATED)
return()
endif()
add_trace_replay_test(${CASE_NAME} DirectGLES VARIANT SPLIT TRANSPORT inproc
IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN})
# --- P6: the SPAWN arm, beside the inproc one rather than instead of it ------------------
# --- P6: TWO TRANSPORTS x TWO BACKENDS, four entries per case ---------------------------
#
# TWO ARMS, NOT ONE RENAMED. The inproc arm is the CONTROL: it is the same
# library, the same codec, the same applier and the same picture, with the
# only difference being which process the server role runs in. A spawn arm
# that had replaced it would have left "did the transport break it?" and
# "did the phase break it?" answerable only together.
# TWO TRANSPORTS, NOT ONE RENAMED. The inproc arm is the CONTROL: same
# library, same codec, same applier, same picture, with the only difference
# being which PROCESS the server role runs in. A spawn arm that had replaced
# it would leave "did the transport break it?" and "did the phase break it?"
# answerable only together.
#
# G14 is satisfied by construction - a name is added, none is removed - and
# the VARIANT gives it its own output and artifact directories, which is the
# half the comment above says is easy to forget: both arms write
# output/mobilegl.log, and that log is the ONLY valid refusal census because
# the console sink is compiled out of these configurations.
add_trace_replay_test(${CASE_NAME} DirectGLES VARIANT SPAWN TRANSPORT spawn
IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN})
# TWO BACKENDS, because every bug this phase found so far was a question
# about the SERVER's process rather than about Espryt - the child loading no
# config, Transport reading as Monolith there, a doorbell that could not wake
# its own thread. None of those is backend-shaped, and all of them would have
# been just as invisible on a Magma server nobody ran.
#
# G14 is satisfied by construction - names are added, none removed - and the
# VARIANT gives each entry its own output and artifact directories, which is
# the half that is easy to forget: all four write output/mobilegl.log, and
# that log is the ONLY valid refusal census because the console sink is
# compiled out of these configurations.
foreach(backend DirectGLES DirectVulkan)
add_trace_replay_test(${CASE_NAME} ${backend} VARIANT SPLIT TRANSPORT inproc
IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN})
add_trace_replay_test(${CASE_NAME} ${backend} VARIANT SPAWN TRANSPORT spawn
IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN})
endforeach()
endfunction()
set(MOBILEGL_TRACE_CASES_CMAKE "${CMAKE_CURRENT_BINARY_DIR}/trace_cases.cmake")
+47 -19
View File
@@ -12,8 +12,10 @@ CI_BACKENDS = ("DirectGLES", "DirectVulkan")
# silent no-op, which is review finding N-4: `"split": true` mistyped as `"splitt": true` loaded
# clean, the split subset became [], the GitHub matrix became {"include":[]}, and `retrace-split`
# was skipped with no red anywhere. Every other way of getting `split` wrong already raised
# (`"ci": false`, a backend list without DirectGLES, a non-bool value) - the typo was the one hole,
# and it is the shape that makes a whole CI job quietly stop existing.
# (`"ci": false`, a non-bool value) - the typo was the one hole, and it is the shape that makes a
# whole CI job quietly stop existing. P6 inverted the key's default (see load_trace_case_manifest),
# which closes the mirror-image hole: a case added later is IN the split subset unless it says
# otherwise, so forgetting the key can no longer drop a case out of the lane.
#
# Adding a key means adding it here, deliberately, in the same commit. That is the point.
KNOWN_CASE_KEYS = frozenset({
@@ -80,23 +82,32 @@ def load_trace_case_manifest(path=TRACE_CASES_JSON):
raise ValueError(
f"{name} is marked verify but excluded from CI, so the verify matrix would drop it"
)
# "split" opts a case into the MOBILEGL_TRANSPORT=inproc retrace subset, the same shape
# and the same reason as "verify" above: a typo has to be a loud manifest error in every
# consumer, not a subset that is quietly one case short. The split arm additionally runs
# DirectGLES ONLY - the server the arm exercises is Espryt's - so a case that excluded
# DirectGLES from CI would leave the split matrix with nothing to run.
split = merged.get("split", False)
# "split" IS AN OPT-OUT, NOT AN OPT-IN, and P6 inverted it deliberately.
#
# It began as an opt-in listing one case, because the split arm was one
# case's worth of work. Now that the whole CI matrix runs split, 40
# identical `"split": true` lines would carry no information AND would
# reinstate the exact hole the key was written to close: a case added
# later gets the monolith arms by default and is silently absent from
# the split ones. Absent is the one state nothing reds on.
#
# So: a CI case is in the split subset unless it says `"split": false`,
# and saying so is a reviewable claim that this trace cannot cross a
# role boundary. `ci: false` removes a case from CI entirely and takes
# its split arms with it - the subset is a SUBSET of the CI matrix, and
# a case CI never runs cannot be one CI runs split.
#
# The arm runs the case's OWN ci_backends, both of them by default. The
# DirectGLES-only requirement that used to live here is gone with the
# DirectGLES-only arm (tools/trace_replay/CMakeLists.txt says why).
split = merged.get("split", merged.get("ci", True))
if not isinstance(split, bool):
raise ValueError(f"split must be true or false for {name}")
if split and not merged.get("ci", True):
raise ValueError(
f"{name} is marked split but excluded from CI, so the split matrix would drop it"
)
if split and "DirectGLES" not in ci_backends(merged):
raise ValueError(
f"{name} is marked split but does not run DirectGLES in CI; the split arm is "
f"DirectGLES-only, so the entry would be registered with no backend"
)
merged["split"] = split
cases.append(merged)
return {"defaults": defaults, "cases": cases}
@@ -154,11 +165,12 @@ def verify_trace_cases(cases):
def split_trace_cases(cases):
"""The subset the split (MOBILEGL_TRANSPORT=inproc) CI mode retraces.
"""The subset the split CI modes retrace - inproc AND spawn, both backends.
P5's phase gate names exactly one: OpenRA, at SSIM >= 0.99. It is also the only fixture that
is hydrated locally, so keeping the subset explicit in the manifest is what stops a later
phase from widening the arm into an LFS fetch by accident.
P5's phase gate named exactly one case, OpenRA at SSIM >= 0.99, because one case was the
work. P6 runs the whole CI matrix across a process boundary, so the subset is now every CI
case that has not explicitly opted out. load_cases() has already resolved the default, so
"split" is present and boolean on every case by the time this reads it.
"""
return [case for case in cases if case.get("split", False)]
@@ -193,12 +205,28 @@ def github_verify_matrix(cases):
return github_test_matrix(verify_trace_cases(cases))
# The two transports the split retrace lane runs. inproc is the CONTROL - same library, same
# codec, same picture, server role on a thread HERE - and spawn is the claim. Both, always: a
# spawn failure that inproc also shows is a phase bug, and one inproc does not show is a
# transport bug, and collapsing the two makes that distinction unaskable.
SPLIT_TRANSPORTS = ("inproc", "spawn")
def github_split_matrix(cases):
"""{backend, case} for the split arm. DirectGLES only - see split_trace_cases."""
"""{backend, case, transport} for the split arm.
THREE DIMENSIONS, NOT TWO. The job used to take {backend, case} and hard-code
MOBILEGL_TRANSPORT=inproc in its own run block, so `spawn` had no way into CI at all. The
backend list is the case's own ci_backends - the same list the monolith matrix uses - so a
case restricted to one backend stays restricted here instead of being registered with a
backend it does not run.
"""
return {
"include": [
{"backend": "DirectGLES", "case": case["name"]}
{"backend": backend, "case": case["name"], "transport": transport}
for case in split_trace_cases(cases)
for backend in ci_backends(case)
for transport in SPLIT_TRANSPORTS
]
}