From dc5bd935de24f6eff01f5e62bdf52a557ec5b258 Mon Sep 17 00:00:00 2001 From: Swung0x48 Date: Mon, 21 Sep 2026 12:31:49 -0400 Subject: [PATCH] [MG_Remote, CI] (P6): the split retrace lane becomes the whole CI matrix, both backends, both transports It was one case on one backend under one transport: OpenRA, DirectGLES, inproc. It is now every CI case, both backends, inproc AND spawn - 154 CI entries, 156 local ctest entries - because every defect the spawn work turned up was a question about the SERVER's process and not about any one trace or either backend. THE DirectVulkan ARM. tools/trace_replay/CMakeLists.txt carried "a DirectVulkan arm would be measuring a server nobody has written yet". True when P5 wrote it; false since P5f/P7 landed Magma's server consumers and its run-ahead readiness gate. Leaving it would have shipped a split shape whose second backend nothing ever ran - the same class of hole as a lane that runs monolith under a name that says split. Magma now brings up a server process of its own and OpenRA retraces across it at ssim=1.000000, which is only possible because this branch also taught the child to load its config: before that, a DirectVulkan session brought up a DirectGLES server and the two halves disagreed about which backend the capability mask describes. `split` BECOMES AN OPT-OUT. It began as an opt-in naming one case, because one case was the work. Marking 39 cases `"split": true` would carry no information and would reinstate the hole the key was written to close, only mirrored: a case added later would get the monolith arms and be silently absent from the split ones, and absent is the one state nothing reds on. A CI case is now in the subset unless it says `"split": false`, which is a reviewable claim that a trace cannot cross a role boundary. THE CI MATRIX GAINS A THIRD DIMENSION. The job took {backend, case} and hard-coded MOBILEGL_TRANSPORT=inproc in its own run block, so `spawn` had no way into CI at all; the .SPAWN ctest entries existed only for a local build. It is {backend, case, transport} now, the backends are the case's own ci_backends rather than a hard-coded DirectGLES, and the artifact names carry the transport or the two arms of one {backend, case} would overwrite each other's uploads. inproc STAYS AS THE CONTROL. Same library, same codec, same applier, same picture; the only difference is which process the server role runs in. Running only spawn would leave "did the transport break it?" and "did the phase break it?" answerable only together - which is the question every failure in this phase has turned on, including the SIGSEGV that D1 explained. BOTH NEGATIVE CONTROLS FOLLOW THE ARM. retrace_pull_library_control.sh and retrace_drop_draw_control.sh each exported MOBILEGL_TRANSPORT=inproc themselves, so on a spawn job they would have gone on proving something about the other arm while spawn ran unguarded. They read SPLIT_TRANSPORT now, and default to inproc so a caller that sets nothing behaves exactly as before. scripts/ci/control_smoke_test.sh: 22 passed, 0 failed against the stubbed ctest, including the two modes that matter most - a red whose SSIM never fell and a knob that armed and dropped nothing. The spawn arm needs a second binary and the inproc arm needs none, so the job places libMobileGLServer.so only on the spawn arm and a missing one is a named failure there rather than a LaunchServer refusal 78 jobs deep. Measured locally as this lands: 39 of the 78 SPAWN entries complete, all green. The rest are still running. --- .github/workflows/test.yml | 64 ++++++++++++++++----- scripts/ci/retrace_drop_draw_control.sh | 17 ++++-- scripts/ci/retrace_pull_library_control.sh | 7 ++- tools/trace_replay/CMakeLists.txt | 51 ++++++++++------- tools/trace_replay/trace_cases.py | 66 +++++++++++++++------- 5 files changed, 146 insertions(+), 59 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a735df4d..319ecdbd 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -1995,17 +1995,19 @@ jobs: # The subset the verify build retraces ("verify": true in trace_cases.json). It is a # SUBSET of the matrix above, so retrace-verify needs no fixtures of its own. echo "verify-matrix=$(python3 tools/trace_replay/trace_cases.py --ci --format github-verify-matrix)" >> "$GITHUB_OUTPUT" - # P5's split subset ("split": true, DirectGLES only). Also a SUBSET of the matrix above, - # so retrace-split needs no fixtures of its own either. It is one case today - OpenRA, - # which is what the phase gate names - and trace_cases.py refuses a `split` case that is - # not in CI or does not run DirectGLES, so the subset cannot silently become empty. + # The split subset: {backend, case, transport} over inproc AND spawn, both backends, for + # every CI case that has not said `"split": false`. Still a SUBSET of the matrix above, + # so retrace-split needs no fixtures of its own. P5 shipped it as one case (OpenRA, which + # the phase gate names); P6 widened it to the whole CI matrix, because every bug the + # spawn work found was a question about the SERVER's process rather than about one trace. + # trace_cases.py refuses a `split` case that is not in CI, so the subset stays a subset. SPLIT_MATRIX=$(python3 tools/trace_replay/trace_cases.py --ci --format github-split-matrix) # AND IT MUST NOT BE EMPTY. An empty `include` is not an error to GitHub - it skips the # whole retrace-split job with no red anywhere - so the one way this subset can vanish # silently is guarded here. trace_cases.py now also rejects an unknown manifest key, which # was the hole: `"splitt": true` loaded clean and emptied the subset (review N-4). if [ "$(printf '%s' "${SPLIT_MATRIX}" | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["include"]))')" -lt 1 ]; then - echo "::error::the split retrace subset is EMPTY. No case in trace_cases.json carries \"split\": true, so retrace-split would be skipped with no red. Exit gate E2 names OpenRA." + echo "::error::the split retrace subset is EMPTY. `split` is an OPT-OUT as of P6 - a CI case is in this subset unless it carries \"split\": false - so an empty subset means either every case opted out or the CI matrix itself is empty. Either way retrace-split would be skipped with no red, and exit gate E2 names OpenRA." exit 1 fi echo "split-matrix=${SPLIT_MATRIX}" >> "$GITHUB_OUTPUT" @@ -2415,10 +2417,15 @@ jobs: build-retrace/tools/trace_replay/${{ matrix.case }}/${{ matrix.backend }}/output/** if-no-files-found: warn - # P5's retrace arm: retrace-verify with MOBILEGL_PIPE_VERIFY=1 swapped for - # MOBILEGL_TRANSPORT=inproc and the comparator's symbol swapped for an MG_Remote one. Exit gate - # E2 is one row - OpenRA at SSIM >= 0.99 - and trace_cases.json's `split: true` is where that - # subset lives. + # The retrace arm of the split shape: retrace-verify with MOBILEGL_PIPE_VERIFY=1 swapped for + # MOBILEGL_TRANSPORT and the comparator's symbol swapped for an MG_Remote one. Exit gate E2 is + # one row - OpenRA at SSIM >= 0.99 - but the lane is no longer one row: as of P6 it is + # {backend, case, transport} over the whole CI matrix, both backends, inproc AND spawn. + # + # inproc IS THE CONTROL AND SPAWN IS THE CLAIM. Same library, same codec, same applier, same + # picture; the only difference is which PROCESS the server role runs in. Running only spawn + # would leave "did the transport break it?" and "did the phase break it?" answerable only + # together, which is the question every failure in this phase turned out to hinge on. # # IT RUNS THE UNCHANGED CTEST NAMES with the transport exported in the JOB environment, rather # than the SPLIT-suffixed variant entries. build-retrace configures without @@ -2434,7 +2441,7 @@ jobs: # SSIM is neither of them: the nm check below (this library carries MG_Remote) and the # transport-resolution line in the library's own log (this PROCESS resolved inproc). retrace-split: - name: retrace split (${{ matrix.backend }}, ${{ matrix.case }}) + name: retrace split (${{ matrix.backend }} / ${{ matrix.transport }}, ${{ matrix.case }}) runs-on: ubuntu-latest timeout-minutes: 240 needs: @@ -2522,7 +2529,24 @@ jobs: fi echo "the library at build-linux/libMobileGL.so is the split build" - - name: Retrace and validate under MOBILEGL_TRANSPORT=inproc + # THE SPAWN ARM NEEDS A SECOND BINARY and the inproc arm needs none, which is the whole + # difference between the two at this level: inproc's server role is a thread in the replay + # process, spawn's is a PROCESS the client launches from this image. It is packaged by + # build-linux-split's `find -name '*.so'` sweep - libMobileGLServer.so is an executable that + # wears a .so name so an APK will install it somewhere exec-able - and a missing one must be + # a NAMED failure here rather than a LaunchServer refusal 78 jobs deep. + - name: Place the server image (spawn arm only) + if: ${{ matrix.transport == 'spawn' }} + run: | + if [ ! -f build-split/libMobileGLServer.so ]; then + echo "::error::the split runtime carries no libMobileGLServer.so, so transport=spawn has no server to launch. ConfigLoader would resolve Spawn, LaunchServer would refuse BY NAME, and this job would red on a message about an unresolvable image rather than about a packaging gap." + exit 1 + fi + cp build-split/libMobileGLServer.so build-linux/libMobileGLServer.so + chmod +x build-linux/libMobileGLServer.so + echo "server image: $(readlink -f build-linux/libMobileGLServer.so)" + + - name: Retrace and validate under MOBILEGL_TRANSPORT=${{ matrix.transport }} working-directory: build-retrace/tools/trace_replay # run_trace_case.cmake turns MOBILEGL_TRANSPORT into assertions of its own - the library # must have RESOLVED the transport in this process, and its log must carry no Fatal{ - so a @@ -2532,7 +2556,11 @@ jobs: run: | ulimit -c unlimited sudo sysctl -w kernel.core_pattern='/tmp/core.%e.%p' - export MOBILEGL_TRANSPORT=inproc + export MOBILEGL_TRANSPORT='${{ matrix.transport }}' + # Absolute, because the replay's working directory is not this one and LaunchServer + # resolves the image relative to nothing. Harmless on the inproc arm, which reads it + # only to decide it does not need it. + export MOBILEGL_IPC_SERVER_PATH="${GITHUB_WORKSPACE}/build-linux/libMobileGLServer.so" ctest -V --no-tests=error --timeout 10800 \ -R '^MobileGLTraceReplay\.${{ matrix.case }}\.${{ matrix.backend }}$' @@ -2572,6 +2600,10 @@ jobs: CONTROL_TMPDIR: ${{ runner.temp }} PULL_LIBRARY: ${{ github.workspace }}/pull-runtime/build-monolith-control/libMobileGL.so FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so + # The control has to run the arm it is a control FOR. Hard-coded inproc, it would go on + # proving something about the other arm while spawn ran unguarded. + SPLIT_TRANSPORT: ${{ matrix.transport }} + MOBILEGL_IPC_SERVER_PATH: ${{ github.workspace }}/build-linux/libMobileGLServer.so run: >- bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_pull_library_control.sh" '${{ matrix.case }}' '${{ matrix.backend }}' @@ -2599,6 +2631,10 @@ jobs: CONTROL_TMPDIR: ${{ runner.temp }} LIBRARY_LOG: ${{ matrix.case }}/${{ matrix.backend }}/output/mobilegl.log FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so + # The control has to run the arm it is a control FOR. Hard-coded inproc, it would go on + # proving something about the other arm while spawn ran unguarded. + SPLIT_TRANSPORT: ${{ matrix.transport }} + MOBILEGL_IPC_SERVER_PATH: ${{ github.workspace }}/build-linux/libMobileGLServer.so run: >- bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_drop_draw_control.sh" '${{ matrix.case }}' '${{ matrix.backend }}' @@ -2624,7 +2660,7 @@ jobs: if: failure() uses: actions/upload-artifact@v7 with: - name: retrace-split-core-dumps-${{ matrix.backend }}-${{ matrix.case }} + name: retrace-split-core-dumps-${{ matrix.transport }}-${{ matrix.backend }}-${{ matrix.case }} path: /tmp/core.* if-no-files-found: ignore @@ -2632,7 +2668,7 @@ jobs: if: always() uses: actions/upload-artifact@v7 with: - name: retrace-split-result-${{ matrix.backend }}-${{ matrix.case }} + name: retrace-split-result-${{ matrix.transport }}-${{ matrix.backend }}-${{ matrix.case }} path: | build-retrace/tools/trace_replay/${{ matrix.case }}/actual-images/** build-retrace/tools/trace_replay/${{ matrix.case }}/${{ matrix.backend }}/output/** diff --git a/scripts/ci/retrace_drop_draw_control.sh b/scripts/ci/retrace_drop_draw_control.sh index 5198b1f6..b064abc6 100644 --- a/scripts/ci/retrace_drop_draw_control.sh +++ b/scripts/ci/retrace_drop_draw_control.sh @@ -84,14 +84,21 @@ if [ "${matched}" -lt 1 ]; then exit 1 fi -# A PREVIOUS RUN'S LINE MUST NEVER ARM THIS ONE. The library opens its log fopen(path, "w"), so -# the replay truncates it - but only if the replay gets that far, and a run that died in the -# loader would leave the baseline's log in place with a perfectly good "control armed" line in -# it. Removing it first is the same rule split_negative_controls.sh's `reset` step follows. +# A PREVIOUS RUN'S LINE MUST NEVER ARM THIS ONE. The CLIENT truncates this log once at open and +# then appends (P6 made the sink O_APPEND, because under spawn a second process writes the same +# file and two truncating handles overwrite each other's bytes) - but it only truncates if the +# replay gets that far, and a run that died in the loader would leave the baseline's log in place +# with a perfectly good "control armed" line in it. Removing it first is the same rule +# split_negative_controls.sh's `reset` step follows. rm -f "${LIBRARY_LOG}" out="${CONTROL_TMPDIR}/retrace-drop-draw-output.txt" -export MOBILEGL_TRANSPORT=inproc +# THE TRANSPORT COMES FROM THE JOB, not from this file. retrace-split is a matrix over +# {backend, case, transport} as of P6, and a control that hard-coded `inproc` would have gone on +# proving something about the OTHER arm while the spawn arm ran unguarded. Default inproc so a +# caller that sets nothing behaves exactly as before. +transport="${SPLIT_TRANSPORT:-inproc}" +export MOBILEGL_TRANSPORT="${transport}" export MOBILEGL_IPC_E2_DROP_DRAW=1 "${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1 control_rc=$? diff --git a/scripts/ci/retrace_pull_library_control.sh b/scripts/ci/retrace_pull_library_control.sh index 89ce9036..16ef9821 100644 --- a/scripts/ci/retrace_pull_library_control.sh +++ b/scripts/ci/retrace_pull_library_control.sh @@ -105,7 +105,12 @@ if [ "${remote_count}" -ne 0 ]; then fi out="${CONTROL_TMPDIR}/retrace-control-output.txt" -export MOBILEGL_TRANSPORT=inproc +# THE TRANSPORT COMES FROM THE JOB, not from this file. retrace-split is a matrix over +# {backend, case, transport} as of P6, and a control that hard-coded `inproc` would have gone on +# proving something about the OTHER arm while the spawn arm ran unguarded. Default inproc so a +# caller that sets nothing behaves exactly as before. +transport="${SPLIT_TRANSPORT:-inproc}" +export MOBILEGL_TRANSPORT="${transport}" "${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1 control_rc=$? cat "${out}" diff --git a/tools/trace_replay/CMakeLists.txt b/tools/trace_replay/CMakeLists.txt index d5a6fae0..12c3a7b1 100644 --- a/tools/trace_replay/CMakeLists.txt +++ b/tools/trace_replay/CMakeLists.txt @@ -449,10 +449,14 @@ endfunction() # --- P5: the SPLIT arm of the retrace matrix ------------------------------------------------- # -# One variant entry per case that trace_cases.json marks `split: true` - today that is OpenRA -# alone, which is the only target the phase gate names and (measurably) the only fixture hydrated -# locally. DirectGLES only: the split shape's first backend is Espryt, and a DirectVulkan arm -# would be measuring a server nobody has written yet. +# One variant entry per case that trace_cases.json marks `split: true`. +# +# BOTH BACKENDS, as of P6. The DirectGLES-only note this comment used to carry - +# "a DirectVulkan arm would be measuring a server nobody has written yet" - was +# true when P5 wrote it and stopped being true at P5f/P7, which landed Magma's +# server consumers and its run-ahead readiness gate. Leaving it would have meant +# shipping a split shape whose second backend nothing ever ran, which is the +# same class of hole as a lane that runs monolith under a name that says split. # # REGISTERED ONLY UNDER MOBILEGL_BUILD_DISAGGREGATED, because the arm is a statement about a # library that compiled MG_Remote. A build without the option accepts MOBILEGL_TRANSPORT=inproc @@ -463,24 +467,31 @@ function(add_trace_replay_split_test CASE_NAME) if(NOT MOBILEGL_BUILD_DISAGGREGATED) return() endif() - add_trace_replay_test(${CASE_NAME} DirectGLES VARIANT SPLIT TRANSPORT inproc - IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN}) - - # --- P6: the SPAWN arm, beside the inproc one rather than instead of it ------------------ + # --- P6: TWO TRANSPORTS x TWO BACKENDS, four entries per case --------------------------- # - # TWO ARMS, NOT ONE RENAMED. The inproc arm is the CONTROL: it is the same - # library, the same codec, the same applier and the same picture, with the - # only difference being which process the server role runs in. A spawn arm - # that had replaced it would have left "did the transport break it?" and - # "did the phase break it?" answerable only together. + # TWO TRANSPORTS, NOT ONE RENAMED. The inproc arm is the CONTROL: same + # library, same codec, same applier, same picture, with the only difference + # being which PROCESS the server role runs in. A spawn arm that had replaced + # it would leave "did the transport break it?" and "did the phase break it?" + # answerable only together. # - # G14 is satisfied by construction - a name is added, none is removed - and - # the VARIANT gives it its own output and artifact directories, which is the - # half the comment above says is easy to forget: both arms write - # output/mobilegl.log, and that log is the ONLY valid refusal census because - # the console sink is compiled out of these configurations. - add_trace_replay_test(${CASE_NAME} DirectGLES VARIANT SPAWN TRANSPORT spawn - IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN}) + # TWO BACKENDS, because every bug this phase found so far was a question + # about the SERVER's process rather than about Espryt - the child loading no + # config, Transport reading as Monolith there, a doorbell that could not wake + # its own thread. None of those is backend-shaped, and all of them would have + # been just as invisible on a Magma server nobody ran. + # + # G14 is satisfied by construction - names are added, none removed - and the + # VARIANT gives each entry its own output and artifact directories, which is + # the half that is easy to forget: all four write output/mobilegl.log, and + # that log is the ONLY valid refusal census because the console sink is + # compiled out of these configurations. + foreach(backend DirectGLES DirectVulkan) + add_trace_replay_test(${CASE_NAME} ${backend} VARIANT SPLIT TRANSPORT inproc + IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN}) + add_trace_replay_test(${CASE_NAME} ${backend} VARIANT SPAWN TRANSPORT spawn + IPC_SERVER_PATH "${CMAKE_BINARY_DIR}/libMobileGLServer.so" ${ARGN}) + endforeach() endfunction() set(MOBILEGL_TRACE_CASES_CMAKE "${CMAKE_CURRENT_BINARY_DIR}/trace_cases.cmake") diff --git a/tools/trace_replay/trace_cases.py b/tools/trace_replay/trace_cases.py index e2c8e105..4d1b2db0 100644 --- a/tools/trace_replay/trace_cases.py +++ b/tools/trace_replay/trace_cases.py @@ -12,8 +12,10 @@ CI_BACKENDS = ("DirectGLES", "DirectVulkan") # silent no-op, which is review finding N-4: `"split": true` mistyped as `"splitt": true` loaded # clean, the split subset became [], the GitHub matrix became {"include":[]}, and `retrace-split` # was skipped with no red anywhere. Every other way of getting `split` wrong already raised -# (`"ci": false`, a backend list without DirectGLES, a non-bool value) - the typo was the one hole, -# and it is the shape that makes a whole CI job quietly stop existing. +# (`"ci": false`, a non-bool value) - the typo was the one hole, and it is the shape that makes a +# whole CI job quietly stop existing. P6 inverted the key's default (see load_trace_case_manifest), +# which closes the mirror-image hole: a case added later is IN the split subset unless it says +# otherwise, so forgetting the key can no longer drop a case out of the lane. # # Adding a key means adding it here, deliberately, in the same commit. That is the point. KNOWN_CASE_KEYS = frozenset({ @@ -80,23 +82,32 @@ def load_trace_case_manifest(path=TRACE_CASES_JSON): raise ValueError( f"{name} is marked verify but excluded from CI, so the verify matrix would drop it" ) - # "split" opts a case into the MOBILEGL_TRANSPORT=inproc retrace subset, the same shape - # and the same reason as "verify" above: a typo has to be a loud manifest error in every - # consumer, not a subset that is quietly one case short. The split arm additionally runs - # DirectGLES ONLY - the server the arm exercises is Espryt's - so a case that excluded - # DirectGLES from CI would leave the split matrix with nothing to run. - split = merged.get("split", False) + # "split" IS AN OPT-OUT, NOT AN OPT-IN, and P6 inverted it deliberately. + # + # It began as an opt-in listing one case, because the split arm was one + # case's worth of work. Now that the whole CI matrix runs split, 40 + # identical `"split": true` lines would carry no information AND would + # reinstate the exact hole the key was written to close: a case added + # later gets the monolith arms by default and is silently absent from + # the split ones. Absent is the one state nothing reds on. + # + # So: a CI case is in the split subset unless it says `"split": false`, + # and saying so is a reviewable claim that this trace cannot cross a + # role boundary. `ci: false` removes a case from CI entirely and takes + # its split arms with it - the subset is a SUBSET of the CI matrix, and + # a case CI never runs cannot be one CI runs split. + # + # The arm runs the case's OWN ci_backends, both of them by default. The + # DirectGLES-only requirement that used to live here is gone with the + # DirectGLES-only arm (tools/trace_replay/CMakeLists.txt says why). + split = merged.get("split", merged.get("ci", True)) if not isinstance(split, bool): raise ValueError(f"split must be true or false for {name}") if split and not merged.get("ci", True): raise ValueError( f"{name} is marked split but excluded from CI, so the split matrix would drop it" ) - if split and "DirectGLES" not in ci_backends(merged): - raise ValueError( - f"{name} is marked split but does not run DirectGLES in CI; the split arm is " - f"DirectGLES-only, so the entry would be registered with no backend" - ) + merged["split"] = split cases.append(merged) return {"defaults": defaults, "cases": cases} @@ -154,11 +165,12 @@ def verify_trace_cases(cases): def split_trace_cases(cases): - """The subset the split (MOBILEGL_TRANSPORT=inproc) CI mode retraces. + """The subset the split CI modes retrace - inproc AND spawn, both backends. - P5's phase gate names exactly one: OpenRA, at SSIM >= 0.99. It is also the only fixture that - is hydrated locally, so keeping the subset explicit in the manifest is what stops a later - phase from widening the arm into an LFS fetch by accident. + P5's phase gate named exactly one case, OpenRA at SSIM >= 0.99, because one case was the + work. P6 runs the whole CI matrix across a process boundary, so the subset is now every CI + case that has not explicitly opted out. load_cases() has already resolved the default, so + "split" is present and boolean on every case by the time this reads it. """ return [case for case in cases if case.get("split", False)] @@ -193,12 +205,28 @@ def github_verify_matrix(cases): return github_test_matrix(verify_trace_cases(cases)) +# The two transports the split retrace lane runs. inproc is the CONTROL - same library, same +# codec, same picture, server role on a thread HERE - and spawn is the claim. Both, always: a +# spawn failure that inproc also shows is a phase bug, and one inproc does not show is a +# transport bug, and collapsing the two makes that distinction unaskable. +SPLIT_TRANSPORTS = ("inproc", "spawn") + + def github_split_matrix(cases): - """{backend, case} for the split arm. DirectGLES only - see split_trace_cases.""" + """{backend, case, transport} for the split arm. + + THREE DIMENSIONS, NOT TWO. The job used to take {backend, case} and hard-code + MOBILEGL_TRANSPORT=inproc in its own run block, so `spawn` had no way into CI at all. The + backend list is the case's own ci_backends - the same list the monolith matrix uses - so a + case restricted to one backend stays restricted here instead of being registered with a + backend it does not run. + """ return { "include": [ - {"backend": "DirectGLES", "case": case["name"]} + {"backend": backend, "case": case["name"], "transport": transport} for case in split_trace_cases(cases) + for backend in ci_backends(case) + for transport in SPLIT_TRANSPORTS ] }