From 75edebdab7148ee7aeba46b81e9b8cab801dcdee Mon Sep 17 00:00:00 2001 From: Swung0x48 Date: Sat, 17 Oct 2026 12:23:53 -0400 Subject: [PATCH] [CI] (.github/workflows, docs/ci, scripts/ci, tools/trace_replay): drop the retrace-split reruns - retrace-split loses its two negative controls, which re-ran every leg (~68 runner-min per run), and its dependency on the monolith-control artifact; the in-test evidence check from the previous commit replaces them - build-linux-monolith-control keeps its zero-MG_Remote proof and its upload; on the disaggregation branches the new retrace-shipped job replays OpenRA on both backends through that library (elsewhere the retrace job's library has that shape), so the shipped shape is still replayed on every branch - delete retrace_transport_control.sh, retrace_drop_draw_control.sh, their smoke cases and stub modes, and the uncalled redcheck_control_smoke_test.sh - control-runtime/build-push-control leaves the eleven metadata loops; remove-artifact-clutter waits for retrace-shipped --- .github/workflows/test.yml | 249 ++++++++++++++-------- docs/ci/ci-cleanup-log.md | 3 + scripts/ci/control_smoke_test.sh | 96 +-------- scripts/ci/redcheck_control_smoke_test.sh | 174 --------------- scripts/ci/retrace_drop_draw_control.sh | 159 -------------- scripts/ci/retrace_transport_control.sh | 179 ---------------- scripts/ci/testdata/stub_ctest.sh | 106 --------- tools/trace_replay/trace_cases.py | 10 +- 8 files changed, 171 insertions(+), 805 deletions(-) delete mode 100755 scripts/ci/redcheck_control_smoke_test.sh delete mode 100644 scripts/ci/retrace_drop_draw_control.sh delete mode 100644 scripts/ci/retrace_transport_control.sh diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 536daf6e..7304df67 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -220,8 +220,9 @@ jobs: path: ci-artifacts/mobilegl-linux-lib.tgz if-no-files-found: error - # The pull-library negative control must not consume the primary artifact: - # on disaggregation branches that artifact now contains the real transport. + # The shipped shape: a library with no MG_Remote, which FCL embeds. On the disaggregation + # branches the primary artifact carries the transport, so this is the only build of that shape + # there; retrace-shipped replays it. build-linux-monolith-control: name: build push monolith control (no MG_Remote) runs-on: ubuntu-26.04 @@ -282,11 +283,9 @@ jobs: run: | sudo apt-get update sudo apt-get install -y ccache clang-20 clang++-20 lld-20 libc++-20-dev libc++abi-20-dev libvulkan-dev libegl1-mesa-dev libgles2-mesa-dev libgl1-mesa-dri mesa-vulkan-drivers ninja-build - # THE PUSH MONOLITH WITHOUT MG_Remote - the shape FCL embeds (PLAN-P13 §4.1, §4.3; ID-P13-1). - # Two jobs in one: it is the transport-negative control retrace-split swaps in (P13 W2 retired - # the pull library that used to play that part), and its runtime_mode_proof is the - # zero-MG_Remote-symbol check, one of G1's successors (ID-P13-4): a push library that links - # any MG_Remote symbol is red here. + # The push monolith without MG_Remote, the shape FCL embeds. Its runtime_mode_proof is the + # zero-MG_Remote-symbol check: a push library that links any MG_Remote symbol is red here. + # Its library is uploaded for retrace-shipped, a positive replay of that shape. - name: Configure push control without MG_Remote run: | cmake -S . -B "${BUILD_DIR}" -G Ninja \ @@ -357,7 +356,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -447,7 +446,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -789,7 +788,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -1034,7 +1033,7 @@ jobs: # otherwise also match this label and the monolith one would match both. - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -1507,7 +1506,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -1745,7 +1744,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -2370,7 +2369,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -2471,7 +2470,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -2746,7 +2745,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -2970,7 +2969,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -3054,23 +3053,18 @@ jobs: # process differs, so a spawn-only failure is a transport fact. It runs the unchanged ctest # names with MOBILEGL_TRANSPORT exported around ctest, since build-retrace does not register the # split variant entries. SSIM cannot catch a monolith run (OpenRA scores 1.000 either way); the - # guards are the nm check for MG_Remote and run_trace_case.cmake's transport-resolution check. + # guards are the nm check for MG_Remote, run_trace_case.cmake's transport assertions (and, on + # OpenRA, its wire-volume assertion), and the per-leg check of the evidence file they leave. retrace-split: name: retrace split (${{ matrix.backend }} / ${{ matrix.transport }}, ${{ matrix.case }}) runs-on: ubuntu-26.04 timeout-minutes: 240 # The setup action selects lavapipe on GitHub-hosted runners. needs: - - build-linux-monolith-control - build-linux-split - build-retrace - trace-cases - trace-fixtures - # The monolith-control artifact is for the negative controls ONLY: its libraries are what the - # controls swap in to prove the transport-resolution assertion is load-bearing. P13 W1: the - # legs no longer require that build to have SUCCEEDED - gating on it skipped the whole split - # matrix (grey, not red) whenever the control build failed (PLAN-P13 §0-7). The control steps - # below download its artifact, and a missing one is a red step on every leg instead. if: ${{ !cancelled() && needs.build-linux-split.result == 'success' && needs.build-retrace.result == 'success' && needs.trace-cases.result == 'success' }} strategy: fail-fast: false @@ -3130,19 +3124,6 @@ jobs: name: mobilegl-trace-replay path: . - # The push monolith without MG_Remote, for the negative control at the end of this job and for - # nothing else. - - name: Download control runtime (negative control) - uses: actions/download-artifact@v8 - with: - name: mobilegl-linux-monolith-control - path: control-runtime - - - name: Unpack the control runtime (negative control) - run: | - tar -xzf control-runtime/mobilegl-linux-monolith-control.tgz -C control-runtime - test -f control-runtime/build-push-control/libMobileGL.so - - name: Unpack split runtime (library under test) # build-retrace's CTestTestfile.cmake has the absolute path # /build-linux/libMobileGL.so frozen into every case, so the swap happens here @@ -3162,7 +3143,7 @@ jobs: - name: Prepare downloaded test metadata run: | - for root in build-linux build-verify build-split build-verify-split build-retrace control-runtime/build-push-control; do + for root in build-linux build-verify build-split build-verify-split build-retrace; do if [ -d "$root" ]; then python3 scripts/ci/prepare_test_runtime.py "$root"; fi done @@ -3183,12 +3164,8 @@ jobs: chmod +x build-linux/libMobileGLServer.so echo "server image: $(readlink -f build-linux/libMobileGLServer.so)" - # THE MAGMA KNOBS GO TO $GITHUB_ENV, NOT TO ONE STEP'S SHELL: the replay below AND both - # negative controls after it must run the same repair set. Exported inside "Retrace and - # validate" only, they never reached the pull-library control, which replayed - # iterationrp x DirectVulkan unrepaired (ssim 0.028 < 0.98), reddened at - # run_trace_case.cmake:173 before the transport assertion, and failed as "the failure never - # says the library did not resolve the transport" - a red that is not the control's own. + # The Magma knobs go to $GITHUB_ENV rather than to one step's shell, so every later step of + # this leg runs the same repair set as the replay. - name: Magma repair knobs (all later steps) run: | # P7 package L: THE SAME MAGMA KNOBS THE MONOLITH RETRACE JOB EXPORTS, ALL OF THEM. @@ -3247,47 +3224,6 @@ jobs: --transport '${{ matrix.transport }}' --case '${{ matrix.case }}' \ --backend '${{ matrix.backend }}' "${wire[@]}" - # Negative control: copy a library without MG_Remote (the push monolith FCL embeds) over the - # frozen path and require the same replay to fail. OpenRA scores ssim 1.000 either way, so - # the picture cannot catch the swap; this proves that run_trace_case.cmake's - # transport-resolution check still does. The body is scripts/ci/retrace_transport_control.sh, - # which control_smoke_test.sh runs off a runner. It counts the selection first, requires the - # red to carry run_trace_case.cmake's own sentence, and restores the split library and the - # good run's images, since "Upload actual image" runs `if: always()`. - - name: Negative control (library without a transport) - working-directory: build-retrace/tools/trace_replay - env: - CONTROL_TMPDIR: ${{ runner.temp }} - CONTROL_LIBRARY: ${{ github.workspace }}/control-runtime/build-push-control/libMobileGL.so - FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so - # The control has to run the arm it is a control FOR. Hard-coded inproc, it would go on - # proving something about the other arm while spawn ran unguarded. - SPLIT_TRANSPORT: ${{ matrix.transport }} - MOBILEGL_IPC_SERVER_PATH: ${{ github.workspace }}/build-linux/libMobileGLServer.so - run: >- - bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_transport_control.sh" - '${{ matrix.case }}' '${{ matrix.backend }}' - - # Negative control for the picture: the step above cannot show that the image came through - # the wire, because OpenRA also scores ssim 1.000 on a monolith library. This drops the - # DrawVbo records and requires the SSIM to fall. Dropping clears does not work: OpenRA - # overdraws every pixel it clears (retrace_drop_draw_control.sh's header has the census). - # The body is a script so control_smoke_test.sh can run it against a stubbed ctest, including - # a red whose SSIM never fell and a knob that armed but dropped nothing. - - name: Negative control (dropped draws) - working-directory: build-retrace/tools/trace_replay - env: - CONTROL_TMPDIR: ${{ runner.temp }} - LIBRARY_LOG: ${{ matrix.case }}/${{ matrix.backend }}/output/mobilegl.client.log - FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so - # The control has to run the arm it is a control FOR. Hard-coded inproc, it would go on - # proving something about the other arm while spawn ran unguarded. - SPLIT_TRANSPORT: ${{ matrix.transport }} - MOBILEGL_IPC_SERVER_PATH: ${{ github.workspace }}/build-linux/libMobileGLServer.so - run: >- - bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_drop_draw_control.sh" - '${{ matrix.case }}' '${{ matrix.backend }}' - # The refusal census, recorded rather than gated. run_trace_case.cmake already REDS the case # on any Fatal{, so reaching here means the count is zero - but the number and the distinct # slot names are what MEASUREMENTS wants from every split run, and reading them out of the @@ -3333,14 +3269,141 @@ jobs: build-retrace/tools/trace_replay/${{ matrix.case }}/${{ matrix.backend }}/output/** if-no-files-found: warn + # The shipped library, replayed. build-linux-monolith-control's library has no MG_Remote, the + # shape FCL embeds; on the disaggregation branches nothing else replays that shape, so it runs + # there only (elsewhere the retrace job's library is that shape). One short case on both + # backends, with no transport set, must match its golden like any monolith row. + retrace-shipped: + name: retrace shipped library (${{ matrix.backend }}, ${{ matrix.case }}) + runs-on: ubuntu-26.04 + timeout-minutes: 60 + needs: + - build-linux-monolith-control + - build-retrace + - trace-fixtures + if: ${{ !cancelled() && (github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && needs.build-linux-monolith-control.result == 'success' && needs.build-retrace.result == 'success' && needs.trace-fixtures.result != 'skipped' }} + strategy: + fail-fast: false + matrix: + backend: [DirectGLES, DirectVulkan] + case: [OpenRA] + + steps: + - name: Set swap space + if: runner.environment != 'self-hosted' + uses: pierotofy/set-swap-space@v1.0 + with: + swap-size-gb: 16 + + - name: Checkout repo + uses: actions/checkout@v6 + + - name: Configure runner environment + uses: ./.github/actions/setup-test-runner + with: + gpu: "true" + + - name: Download trace fixture + uses: actions/download-artifact@v8 + with: + name: trace-fixture-${{ matrix.case }} + path: trace-fixture-download + + - name: Install trace fixture + run: | + mkdir -p tools/trace_replay/fixtures + find trace-fixture-download -type f -exec cp {} tools/trace_replay/fixtures/ \; + + - name: Get CMake + timeout-minutes: 15 + uses: lukka/get-cmake@v4.3.3 + with: + cmakeVersion: 3.31.10 + + - name: Install runtime dependencies + timeout-minutes: 15 + if: env.MOBILEGL_CI_NATIVE_GPU != '1' + run: | + sudo apt-get update + sudo apt-get install -y libvulkan1 libegl1-mesa-dev libgles2-mesa-dev libgl1-mesa-dri mesa-vulkan-drivers + test -e /usr/lib/x86_64-linux-gnu/libEGL.so + test -e /usr/lib/x86_64-linux-gnu/libGLESv2.so + + - name: Download shipped library + uses: actions/download-artifact@v8 + with: + name: mobilegl-linux-monolith-control + path: . + + - name: Download trace replay + uses: actions/download-artifact@v8 + with: + name: mobilegl-trace-replay + path: . + + # build-retrace froze /build-linux/libMobileGL.so into every case, so the library + # is put where that path points, as retrace-verify does. `nm`, not `nm -D`: the MG_Remote + # symbols are hidden in a Release build and only the static table lists them. + - name: Unpack shipped library (library under test) + run: | + tar -xzf mobilegl-linux-monolith-control.tgz + tar -xzf mobilegl-trace-replay.tgz + test -f build-push-control/libMobileGL.so + test -f build-retrace/tools/trace_replay/mobilegl_trace_replay + mkdir -p build-linux + cp build-push-control/libMobileGL.so build-linux/libMobileGL.so + defined=$(nm --defined-only build-linux/libMobileGL.so | wc -l) + remote=$(nm --defined-only build-linux/libMobileGL.so | grep -c -i MG_Remote || true) + if [ "${defined}" -lt 1000 ] || [ "${remote}" -ne 0 ]; then + echo "::error::build-linux/libMobileGL.so defines ${defined} symbols and ${remote} MG_Remote symbol(s): it is not the shipped shape (no MG_Remote, not stripped)" + exit 1 + fi + echo "the library at build-linux/libMobileGL.so is the shipped shape (${defined} defined symbols, no MG_Remote)" + + - name: Prepare downloaded test metadata + run: python3 scripts/ci/prepare_test_runtime.py build-retrace + + - name: Retrace and validate + working-directory: build-retrace/tools/trace_replay + run: | + ulimit -c unlimited + if [ '${{ matrix.backend }}' = 'DirectVulkan' ]; then + export MOBILEGL_MAGMA_R11G11B10F_FALLBACK=1 + fi + ctest -V --no-tests=error -R '^MobileGLTraceReplay\.${{ matrix.case }}\.${{ matrix.backend }}$' \ + --output-junit "${RUNNER_TEMP}/retrace.xml" + + - name: Annotate failed entries + if: failure() + run: python3 scripts/ci/junit_annotate.py --excerpts 1 "${RUNNER_TEMP}" + + - name: Upload core dumps + if: failure() + uses: actions/upload-artifact@v7 + with: + name: retrace-shipped-core-dumps-${{ matrix.backend }}-${{ matrix.case }} + path: /tmp/core.* + if-no-files-found: ignore + + - name: Upload actual image + if: ${{ always() }} + uses: actions/upload-artifact@v7 + with: + name: retrace-shipped-result-${{ matrix.backend }}-${{ matrix.case }} + path: | + build-retrace/tools/trace_replay/${{ matrix.case }}/actual-images/** + build-retrace/tools/trace_replay/${{ matrix.case }}/${{ matrix.backend }}/output/** + if-no-files-found: warn + remove-artifact-clutter: name: remove artifact clutter runs-on: ubuntu-26.04 - # Every retrace arm downloads the shared fixtures, including split controls. + # Every retrace job downloads the shared fixtures, so this waits for all of them. needs: - retrace-summary - retrace-verify - retrace-split + - retrace-shipped if: ${{ always() }} permissions: actions: write @@ -3359,14 +3422,15 @@ jobs: # an unchecked process substitution. # 50 rows per page: the jobs endpoint answers HTTP 502 for 90+ rows on a re-run attempt of a run this size. gh api --paginate "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/jobs?per_page=50" \ - --jq '.jobs[] | select((.name | startswith("retrace (")) or (.name | startswith("retrace verify (")) or (.name | startswith("retrace split ("))) | select(.conclusion == "failure" or .conclusion == "cancelled" or .conclusion == "timed_out" or .conclusion == "action_required") | .name' \ + --jq '.jobs[] | select((.name | startswith("retrace (")) or (.name | startswith("retrace verify (")) or (.name | startswith("retrace split (")) or (.name | startswith("retrace shipped library ("))) | select(.conclusion == "failure" or .conclusion == "cancelled" or .conclusion == "timed_out" or .conclusion == "action_required") | .name' \ > "${RUNNER_TEMP}/cleanup-jobs.txt" gh api --paginate "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/artifacts?per_page=100" \ - --jq '.artifacts[] | select(.name | startswith("trace-fixture-") or startswith("retrace-result-") or startswith("retrace-split-result-")) | [.id, .name] | @tsv' \ + --jq '.artifacts[] | select(.name | startswith("trace-fixture-") or startswith("retrace-result-") or startswith("retrace-split-result-") or startswith("retrace-shipped-result-")) | [.id, .name] | @tsv' \ > "${RUNNER_TEMP}/cleanup-artifacts.txt" declare -A failed_cases=() while IFS= read -r job_name; do case_name="${job_name#retrace split (*, }" + case_name="${case_name#retrace shipped library (*, }" case_name="${case_name#retrace verify (*, }" case_name="${case_name#retrace (*, }" case_name="${case_name%)}" @@ -3515,10 +3579,11 @@ jobs: python3 scripts/gen_pipe_field_ownership.py --self-test # Negative controls must themselves be able to go red (ARCHITECTURE.md §13.2). - # control_smoke_test.sh runs the split and retrace control scripts against a stubbed ctest: - # an unrelated failure or an empty selection must make each report FAILED, and the - # scenarios' real diagnostics must make it report PASSED. The Python tests run the trace and - # TCP lane tooling on stubs and synthetic data. No branch guard; no build, GPU or device. + # control_smoke_test.sh runs the split control script against a stubbed ctest: an unrelated + # failure or an empty selection must make it report FAILED, and the scenarios' real + # diagnostics must make it report PASSED. The Python tests run the trace and TCP lane tooling + # on stubs and synthetic data; test_run_trace_case.py feeds run_trace_case.cmake canned logs. + # No branch guard; no build, GPU or device. - name: Control and trace-tool smoke tests (R-16) run: | bash scripts/ci/control_smoke_test.sh diff --git a/docs/ci/ci-cleanup-log.md b/docs/ci/ci-cleanup-log.md index 1c4bbcdc..1672da43 100644 --- a/docs/ci/ci-cleanup-log.md +++ b/docs/ci/ci-cleanup-log.md @@ -36,6 +36,9 @@ them wrong. | 2026-10-06 | unit `DirectGLESSlotTable.{ANegativeLookupIsNotCachedAcrossAnotherHoldersAcquire, GetOrCreateToleratesANullStateObject}` | Test / unit | P13 (`# P13` ratchet gate): they pinned the slot table's frontend-keyed half - `HandleOf`'s lifetime-id memo and the minting `GetOrCreate(StatePtr)`'s null-object parking - which minted handles off frontend lifetime ids from inside MG_Backend; that half is deleted (the table only receives client-minted handles now). The other ten `DirectGLESSlotTable` mechanics cases were RETARGETED at the handle entries (`GetOrCreate(handle)` / `FindByHandle` / `ReleaseTwinByHandle`), not removed | the retargeted `DirectGLESSlotTable` cases, `SlotAllocatorTest`, and the link ratchet's P13 exit gate (no server object references the client allocator) | | 2026-10-06 | unit `RemoteGuards.{SlotTableHandleOfFromTheApplyThreadIsFatalByName, SlotTableMintingGetOrCreateFromTheApplyThreadIsFatalByName, AnAllocatorProbeInsideAnExemptionScopeFromTheApplyThreadIsFatalEvenAtTheOldDebtSites, BarrieredFrontendRegistryMembersRefuseBothLegacyScopes, LegacyRegistryWrapperCannotExposeFrontendKeysOnApply, AnExemptionScopeHeldOnTheGLThreadDoesNotExemptTheApplyThread}` | Test / unit (split) | P13 (`# P13` ratchet gate): they drove the apply-thread refusals hung on Espryt's frontend-keyed registry entries (`HandleOf`, `GetOrCreate(StatePtr)`, `NoteStateForHandle`, `StateForHandle`, `Find(object)`, `begin()`) and the two no-op marker scopes; the entries, the registry refusal and both scopes are deleted, so there is no apply-thread surface left to refuse | `RemoteGuards.AnAllocatorProbeFromTheApplyThreadOutsideEveryScopeIsFatalByName` / `BarrieredLegacyScopesCannotExemptAllocator` (the allocator's own guard, kept) and the link ratchet's P13 exit gate | | 2026-10-06 | unit `ServerLoopEglTest.FrontendFramebufferDeathDeletesOnTheContextOwner` | Test / unit (split) | P13 (`# P13` ratchet gate): its pin was the refusal of the apply-thread `GetOrCreate(StatePtr)` mint of a framebuffer twin; that overload is deleted | `CtWireScenario`'s object_death cases (framebuffer death by handle on the server) | +| 2026-10-17 | `Negative control (library without a transport)` (with its `Download/Unpack control runtime` steps) | Test / `retrace-split` (every leg) | a full replay rerun on all 154 legs (~60 runner-min) that, since the per-role logs, only ever proved "no client log was written"; a broken control build once reddened every leg at the download and hid every real split replay | in-test: `run_trace_case.cmake`'s transport assertions now leave `output/split-evidence.txt`, which `retrace_split_evidence.py` checks on every leg; `test_run_trace_case.py` (pipe-gates) proves each assertion red on canned logs, including the no-client-log and env-dump-line shapes this control used to rely on | +| 2026-10-17 | `Negative control (dropped draws)` | Test / `retrace-split` (every leg) | a rerun of every leg with `MOBILEGL_IPC_E2_DROP_DRAW`, a production hook; its knob's own early return produced the DirectVulkan `Fatal{RoleViolation, "gPipeInputs"}` reds of 10-02 to 10-05 | in-test on the OpenRA legs (both backends, inproc and spawn): `MGTRACE_WIRE_EVIDENCE` turns on the library's counters and `run_trace_case.cmake` requires DrawVbo records on the wire and draws on the server role (758 each locally); canned reds in `test_run_trace_case.py` | +| 2026-10-17 | `scripts/ci/retrace_transport_control.sh`, `scripts/ci/retrace_drop_draw_control.sh`, the retrace and drop-draw halves of `control_smoke_test.sh` and `stub_ctest.sh`, and `scripts/ci/redcheck_control_smoke_test.sh` | Test / pipe-gates (R-16) | the bodies of the two controls above and their smoke cases; the redcheck script had no caller and mutated the deleted scripts | `test_run_trace_case.py` | ## Kept after investigation diff --git a/scripts/ci/control_smoke_test.sh b/scripts/ci/control_smoke_test.sh index 71c40a5e..2c60ba8b 100644 --- a/scripts/ci/control_smoke_test.sh +++ b/scripts/ci/control_smoke_test.sh @@ -7,21 +7,15 @@ # SPDX-License-Identifier: LGPL-3.0-only # End of Source File Header -# R-16 FOR THE CI NEGATIVE CONTROLS THEMSELVES: a control-run smoke test. +# A smoke test for the CI negative controls themselves. # -# BRIEF-P5 13 (R-16) says a negative control must assert its own failure reason and that every gate -# carries a line saying "I made it red once, by doing X". The two controls this file exercises ARE -# gates, and until ID-46 finding 8 nobody could make either of them red, because a workflow `run:` -# block only executes on a runner. The wave-1 verification agent had to hand-copy the blocks into -# throwaway harnesses to show they were broken (wave1-codex-verify.md 8). This file is that -# experiment, kept: it runs the REAL control scripts - the same files .github/workflows/test.yml -# invokes, not copies of them - against a stubbed ctest, and checks that each one passes exactly -# when it should. +# A negative control is a gate, and a workflow `run:` block only executes on a runner, so this +# file runs the real control script - the same file .github/workflows/test.yml invokes, not a copy +# of it - against a stubbed ctest, and checks that it passes exactly when it should. # -# The case that matters is the first one. A stubbed ctest reports a NON-EMPTY selection and then +# The case that matters is the first one. A stubbed ctest reports a non-empty selection and then # fails with UNRELATED_CONTROL_FAILURE: a reason that has nothing to do with the knob the control -# turns. Before ID-48's fix both controls printed their success message and the step exited 0. They -# must now report FAILED. +# turns. The control must report FAILED on it, and PASSED only on its scenarios' own diagnostic. # # usage: control_smoke_test.sh set -u @@ -61,29 +55,6 @@ run_split() { # $1 = STUB_MODE bash "${HERE}/split_negative_controls.sh" } -run_retrace() { # $1 = STUB_MODE - cd "${WORK}" || return 127 - mkdir -p "${WORK}/OpenRA" - local rc=0 - env -i PATH="${STUB_DIR}:/usr/bin:/bin" STUB_MODE="$1" \ - CTEST=ctest CONTROL_TMPDIR="${WORK}/tmp-$1" \ - CONTROL_LIBRARY="${WORK}/pull.so" FROZEN_LIBRARY="${WORK}/frozen.so" \ - bash "${HERE}/retrace_transport_control.sh" OpenRA DirectGLES || rc=$? - cmp -s "${WORK}/frozen.so" "${WORK}/split.so" || { - echo 'F6 FAILED: pull control did not restore the split library'; return 1; - } - return "${rc}" -} - -run_drop_draw() { # $1 = STUB_MODE - cd "${WORK}" || return 127 - mkdir -p "${WORK}/OpenRA" - env -i PATH="${STUB_DIR}:/usr/bin:/bin" STUB_MODE="$1" \ - CTEST=ctest CONTROL_TMPDIR="${WORK}/tmp-$1" \ - FROZEN_LIBRARY="${WORK}/frozen.so" LIBRARY_LOG="${WORK}/tmp-$1/mobilegl.log" \ - bash "${HERE}/retrace_drop_draw_control.sh" OpenRA DirectGLES -} - echo "=== the split lane's E1 / E3(a) controls (scripts/ci/split_negative_controls.sh)" # THE FINDING, REPRODUCED. Non-empty selection, green baseline, and a red that is not the knob's. expect FAILED "unrelated failure with a non-empty selection" -- run_split unrelated @@ -102,61 +73,6 @@ expect FAILED "E1 record never reaches the peer" -- run_split e1-no- expect FAILED "E1 observed no real emission" -- run_split e1-no-emit expect FAILED "E1 incorrectly waits every record" -- run_split e1-waitall -echo -echo "=== the retrace lane's pull-library control (scripts/ci/retrace_transport_control.sh)" -# A pull-shaped library the nm identity check accepts: a real ELF .so defining no MG_Remote symbol. -if command -v cc > /dev/null 2>&1; then - printf '%s\n' 'int mobilegl_pull_only(void) { return 1; }' > "${WORK}/pull.c" - cc -shared -fPIC -o "${WORK}/pull.so" "${WORK}/pull.c" || { echo "cannot build the stand-in library"; exit 1; } -else - echo "no cc available; the retrace half of this smoke test needs one" >&2 - exit 1 -fi -printf '%s\n' 'int MG_Remote_stub(void) { return 1; }' > "${WORK}/split.c" -cc -shared -fPIC -o "${WORK}/frozen.so" "${WORK}/split.c" || exit 1 -cp "${WORK}/frozen.so" "${WORK}/split.so" - -# THE FINDING, part (b): a regex matching no tests. --no-tests=error exits non-zero and the old -# control read that as "the pull library turned it red". -expect FAILED "empty selection (--no-tests=error exit)" -- run_retrace retrace-noselect -# A red that never names the transport: a fixture failure, a loader failure, a timeout. -expect FAILED "red without the transport-resolution message" -- run_retrace retrace-unrelated -# The real thing - and note the stub emits it CMake-wrapped across two lines, which a line-oriented -# grep for the literal sentence would miss. -expect PASSED "run_trace_case.cmake's own sentence, wrapped" -- run_retrace retrace-evidence -# The sentence a pull library ACTUALLY gets since P6's log rename: it wrote the unsuffixed -# mobilegl.log, the runner reads mobilegl.client.log, and the no-log check fires before the marker -# search. The control accepted only the marker sentence until B3's fix round 2, and was red on -# every real pull-library run (measured on the B3 package tree). -expect PASSED "the no-client-log sentence a pull library gets" -- run_retrace retrace-evidence-nolog -# Both sentences again as `ctest -V` really prints them, with "1: " on every line: folded -# naively that is "never 1: reported resolving it", and the control used to red on it. -expect PASSED "the marker sentence under ctest -V's N: prefix" -- run_retrace retrace-evidence-prefixed -expect PASSED "the no-log sentence under ctest -V's N: prefix" -- run_retrace retrace-evidence-nolog-prefixed -# The pull library replaying green is the failure this control exists to catch. -expect FAILED "a pull library passed the split retrace" -- run_retrace retrace-green - -echo -echo "=== the retrace lane's draw-drop control (scripts/ci/retrace_drop_draw_control.sh)" -# Exit gate E2's picture control. The joint gate's finding is the reason it exists at all: the -# CLEAR-drop knob was armed, was read, dropped all 29 of OpenRA's clears - and the retrace still -# scored ssim 1.000000, because OpenRA overdraws every pixel it clears. So this control's three -# guards are each a different way for "the picture went red" to be someone else's red. -expect FAILED "empty selection (--no-tests=error exit)" -- run_drop_draw dropdraw-noselect -# The failure the gate exists to catch: the golden survives the loss of every draw. -expect FAILED "the retrace passed with every draw dropped" -- run_drop_draw dropdraw-green -# A red with no comparator output at all - a loader failure, a missing fixture, a timeout. -expect FAILED "red with no ssim summary in the output" -- run_drop_draw dropdraw-nossim -# A red whose SSIM is FINE: something else (a Fatal{, a transport assertion) reddened the case. -expect FAILED "red but the ssim is above the threshold" -- run_drop_draw dropdraw-ssimhigh -# A red picture with no evidence the knob was ever read by the process that produced it. -expect FAILED "no 'E2 control armed' line in the library log" -- run_drop_draw dropdraw-nolog -# The R-16 case the dropped-record COUNT exists for: the knob armed and dropped nothing, so the -# wrong picture came from somewhere else. -expect FAILED "the knob armed but dropped zero records" -- run_drop_draw dropdraw-zero -# The real thing: a red picture, a fallen SSIM, and N > 0 records the library says it dropped. -expect PASSED "ssim below threshold and N records dropped" -- run_drop_draw dropdraw-evidence - echo echo "smoke test: ${passes} passed, ${failures} failed" # Keep the private-file cases on the entry point used by CI and the local gate. diff --git a/scripts/ci/redcheck_control_smoke_test.sh b/scripts/ci/redcheck_control_smoke_test.sh deleted file mode 100755 index ca0bb6d9..00000000 --- a/scripts/ci/redcheck_control_smoke_test.sh +++ /dev/null @@ -1,174 +0,0 @@ -#!/bin/bash -# MobileGL - scripts/ci/redcheck_control_smoke_test.sh -# Copyright (c) 2025-2026 MobileGL-Dev -# Licensed under the GNU Lesser General Public License v3.0: -# https://www.gnu.org/licenses/gpl-3.0.txt -# https://www.gnu.org/licenses/lgpl-3.0.txt -# SPDX-License-Identifier: LGPL-3.0-only -# End of Source File Header - -# R-16's "I made it red once, by doing X" for scripts/ci/control_smoke_test.sh, mechanised so the -# claim can be re-checked rather than believed. -# -# X = revert the message check in each control, i.e. put the controls back in the state ID-46 -# finding 8 found them in: a non-zero ctest exit is accepted whatever the failure was. -# -# The smoke test must then FAIL on the cases that exist for this defect - -# "unrelated failure with a non-empty selection" and "red without the transport-resolution -# message", plus the missing private-file Fatal - and not merely somewhere. A smoke test that goes -# red for any other reason when the evidence check is removed would not be pinning the evidence check. -set -u -HERE="$(cd "$(dirname "$0")" && pwd)" -WORK="$(mktemp -d)" || exit 1 -HELPER="${HERE}/../../MobileGL/MG_IntegrationTest/Harness/split_log_paths.py" -WAIT_HELPER="${HERE}/wait_boundary_negative_control.py" -trap 'cp "${WORK}/split.orig" "${HERE}/split_negative_controls.sh"; cp "${WORK}/retrace.orig" "${HERE}/retrace_transport_control.sh"; cp "${WORK}/dropdraw.orig" "${HERE}/retrace_drop_draw_control.sh"; cp "${WORK}/helper.orig" "${HELPER}"; cp "${WORK}/wait.orig" "${WAIT_HELPER}"; rm -rf "${WORK}"' EXIT - -cp "${HERE}/split_negative_controls.sh" "${WORK}/split.orig" -cp "${HERE}/retrace_transport_control.sh" "${WORK}/retrace.orig" -cp "${HERE}/retrace_drop_draw_control.sh" "${WORK}/dropdraw.orig" -cp "${HELPER}" "${WORK}/helper.orig" -cp "${WAIT_HELPER}" "${WORK}/wait.orig" - -echo "=== baseline: the smoke test must be GREEN before anything is perturbed" -if ! bash "${HERE}/control_smoke_test.sh" > "${WORK}/before.log" 2>&1; then - echo "the smoke test is ALREADY RED; the red-check below would prove nothing" - cat "${WORK}/before.log" - exit 1 -fi -tail -1 "${WORK}/before.log" - -echo -echo "=== perturbation: remove the evidence check from both controls" -python3 - "${HERE}/split_negative_controls.sh" "${HERE}/retrace_transport_control.sh" \ - "${HERE}/retrace_drop_draw_control.sh" "${WAIT_HELPER}" <<'PY' || exit 1 -import sys -split, retrace, dropdraw, wait = sys.argv[1:] -# (file, [(needle, the prefix the line must start with)]). The draw-drop control has THREE -# evidence checks rather than one, because "the picture went red" has three different ways of -# being somebody else's red: the SSIM never fell, the library never said the knob armed, and the -# knob armed and dropped nothing. All three are reverted together, and the smoke cases that pin -# each of them must go red below. -# -# THE PREFIXES ARE A TUPLE, AND THAT IS A FIX, not a generalisation for its own sake. The first -# version matched only lines starting with `if ! `, and the split control's evidence check has -# been `elif ! tr -s ... | grep -qE "${evidence}"` since it was written - so this red-check found -# ZERO checks in it, the SystemExit below was not checked by the caller, and the run went on to -# smoke-test the UNPERTURBED controls and report "RED-CHECK FAILED: the controls accept an -# unrelated failure again and the smoke test still passed". It failed safe rather than green, so -# nothing was ever silently proved - but the one control it was most about was never perturbed. -# Reproduced on p5/joint@e61d0012 before this line changed. -# -# AND THE SPLIT CONTROL HAS THREE EVIDENCE CHECKS, NOT ONE. E1 checks each boundary -# probe's own observation and assertion; E3(a) checks its assertion and private log. -# All three are reverted: perturbing only one leaves the -# other two catching the smoke test's "unrelated failure" case, and the case never flips - which -# is what this script measured the first time the perturbation actually applied. -rules = [ - (split, [('LINE', '"${log_helper}" assertion', ('if ! ', 'elif ! ')), - ('SUBST', '"${private_evidence}" "${name}" || exit 1', - '"${private_evidence}" "${name}" || true')]), - (wait, [('SUBST', 'if not own_negative(text, observation, diagnostic):', - 'if False: # E1 evidence check removed by red-check')]), - (retrace, [('LINE', 'grep -qF "${EVIDENCE}"', ('if ! ',))]), - (dropdraw, [('LINE', 'awk -v a="${ssim}"', ('if ! ',)), - ('LINE', '[ -z "${armed_line}" ]', ('if ',)), - ('LINE', '[ "${dropped}" -lt 1 ]', ('if ',))]), -] -for path, checks in rules: - text = open(path).read() - for rule in checks: - if rule[0] == 'SUBST': - _, old, new = rule - if text.count(old) != 1: - raise SystemExit(f"expected exactly one {old!r} in {path}, found {text.count(old)}") - text = text.replace(old, new) - continue - _, needle, prefixes = rule - out, hit = [], 0 - for line in text.splitlines(keepends=True): - if needle in line and line.lstrip().startswith(prefixes): - indent = line[:len(line) - len(line.lstrip())] - # KEEP THE KEYWORD. Turning an `elif` into an `if` splits the chain in two and - # leaves the second half's `fi` dangling - a syntax error, which the smoke test - # then reports as rc=2 on every case and which looks nothing like "the control - # accepted an unrelated failure". - keyword = 'elif' if line.lstrip().startswith('elif') else 'if' - out.append(f"{indent}{keyword} false; then\n") - hit += 1 - else: - out.append(line) - if hit != 1: - raise SystemExit(f"expected exactly one {needle!r} check in {path}, found {hit}") - text = ''.join(out) - open(path, 'w').write(text) -print("all seven evidence checks reverted to 'any non-zero ctest exit is accepted'") -PY -# AND THE PERTURBATION'S OWN FAILURE IS FATAL. Without this, a SystemExit above left the controls -# UNTOUCHED and the run continued to smoke-test them, printing "RED-CHECK FAILED: the controls -# accept an unrelated failure again and the smoke test still passed" - a true statement about a -# perturbation that never happened, and the wrong diagnosis to hand whoever reads it. -if [ $? -ne 0 ]; then - echo "RED-CHECK ABORTED: the perturbation did not apply, so nothing below would be measuring it." - exit 1 -fi - -echo -echo "=== the smoke test on the reverted controls (it MUST be red on the pinned cases)" -bash "${HERE}/control_smoke_test.sh" > "${WORK}/after.log" 2>&1 -rc=$? -cat "${WORK}/after.log" - -if [ "${rc}" -eq 0 ]; then - echo - echo "RED-CHECK FAILED: the controls accept an unrelated failure again and the smoke test still passed." - exit 1 -fi - -missed=0 -grep -q "NOT OK unrelated failure with a non-empty selection" "${WORK}/after.log" || missed=1 -grep -q "NOT OK red without the transport-resolution message" "${WORK}/after.log" || missed=1 -grep -qF "NOT OK missing-fatal: control must report FAILED for its own reason" "${WORK}/after.log" || missed=1 -for case in \ - "red but the ssim is above the threshold" \ - "no 'E2 control armed' line in the library log" \ - "the knob armed but dropped zero records"; do - grep -q "NOT OK ${case}" "${WORK}/after.log" || { echo "still green: ${case}"; missed=1; } -done -if [ "${missed}" -ne 0 ]; then - echo - echo "RED-CHECK FAILED: the smoke test did not red every named split, retrace, draw-drop and private-log evidence case." - exit 1 -fi - -echo -cp "${WORK}/split.orig" "${HERE}/split_negative_controls.sh" -cp "${WORK}/retrace.orig" "${HERE}/retrace_transport_control.sh" -cp "${WORK}/dropdraw.orig" "${HERE}/retrace_drop_draw_control.sh" -cp "${WORK}/wait.orig" "${WAIT_HELPER}" -echo "=== ID-62 perturbation: remove only the skip check" -python3 - "${HELPER}" "${WAIT_HELPER}" <<'PY' || exit 1 -from pathlib import Path -import sys -path = Path(sys.argv[1]) -text = path.read_text() -needle = ' if skipped:\n' -if text.count(needle) != 1: - raise SystemExit('expected exactly one skip check') -path.write_text(text.replace(needle, ' if False: # skip check removed by red-check\n')) -path = Path(sys.argv[2]) -text = path.read_text() -needle = ' if case.find("skipped") is not None or case.get("status") in {"notrun", "disabled"}:\n' -if text.count(needle) != 1: - raise SystemExit('expected exactly one E1 skip check') -path.write_text(text.replace(needle, ' if False: # E1 skip check removed by red-check\n')) -PY -bash "${HERE}/control_smoke_test.sh" > "${WORK}/skips.log" 2>&1 -rc=$? -cat "${WORK}/skips.log" -if [ "${rc}" -eq 0 ] || ! grep -qFx 'NOT OK skipped-selection: control must report FAILED for its own reason' "${WORK}/skips.log"; then - echo 'RED-CHECK FAILED: removing the skip check must red the skipped-selection message assertion' - exit 1 -fi -echo 'ID-62 red-once: removing only the skip check made skipped-selection red (smoke rc=1)' -echo "P5_T1_CONTROL_SMOKE_REDCHECK_OK - evidence and skipped-selection checks each made every named smoke case red" diff --git a/scripts/ci/retrace_drop_draw_control.sh b/scripts/ci/retrace_drop_draw_control.sh deleted file mode 100644 index 2ad07e5d..00000000 --- a/scripts/ci/retrace_drop_draw_control.sh +++ /dev/null @@ -1,159 +0,0 @@ -#!/bin/bash -# MobileGL - scripts/ci/retrace_drop_draw_control.sh -# Copyright (c) 2025-2026 MobileGL-Dev -# Licensed under the GNU Lesser General Public License v3.0: -# https://www.gnu.org/licenses/gpl-3.0.txt -# https://www.gnu.org/licenses/lgpl-3.0.txt -# SPDX-License-Identifier: LGPL-3.0-only -# End of Source File Header - -# EXIT GATE E2's PICTURE CONTROL: dropping the draws on the wire must redden this retrace, and -# it must be THIS control's red. -# -# WHY THERE IS A SECOND CONTROL ON THIS LANE, and why it is the draws and not the clears. -# -# The retrace-split job already carries scripts/ci/retrace_transport_control.sh, which swaps a -# PULL library in and requires run_trace_case.cmake's transport-resolution assertion to catch it. -# That control proves the lane is running a SPLIT library. It does not prove that the PICTURE came -# through the wire, and it cannot: OpenRA scores ssim 1.000000 against a monolith library too, so -# the transport assertion is what reds it and the comparator is never consulted. BRIEF-P5 7 E2 -# names a second, sharper statement - "patch an emitter to drop a record and the SSIM must fall -# below the threshold" - and until now nothing executed it. -# -# c1 shipped MOBILEGL_IPC_E2_DROP_CLEAR for exactly that, and the joint gate ran it and it DID NOT -# WORK: with the knob armed and its WARN in the library's own log, the OpenRA retrace under inproc -# still scored ssim=1.000000 with mismatchPixels=0 (joint-v1.md 3, "E2 clear-drop control - NOT -# VERIFIED"). That is not a broken knob. `apitrace dump` over the 31249 replayed calls counts 30 -# glClear, 30 glXSwapBuffers and 788 glDrawArrays, and the final frame issues its clear at call -# 30197 and then covers the surface four times over with a terrain layer before the snapshot at -# 31249. A frame that overdraws every pixel it clears has a picture that does not depend on the -# clear. Measured here with the dropped-record count published: 29 Clear records dropped, ssim -# still 1.000000. The knob worked; the observable was invisible. -# -# So this control drops every DrawVbo record instead (MOBILEGL_IPC_E2_DROP_DRAW=1): the surface -# can then only carry the clear colour, and the golden is made of the geometry. Measured on the -# same head: 758 DrawVbo records dropped, ssim=0.000036, mismatchPixels=295296. -# -# WHAT THE CONTROL ASSERTS, and none of the three is the process exit code alone (R-16): -# 1. a NON-EMPTY selection, counted before the run - `--no-tests=error` turns an empty selection -# into a non-zero exit, which is how ID-46 finding 8(b) got a control to congratulate itself; -# 2. the SSIM ACTUALLY FELL: the numbers are parsed out of the retrace's own summary and -# compared, rather than "ctest was non-zero". A loader failure, a missing fixture, a timeout -# and a Fatal{ all exit non-zero and none of them is this control's red; -# 3. the LIBRARY SAID IT DROPPED SOMETHING: its own "E2 control armed ... N records dropped on -# the wire" line, with N > 0, from the log file the replay wrote. Without this, a knob that -# was never read - a stale library, a variable that did not reach the process, an emit table -# that fell through to the driver - would redden the picture for some other reason and pass. -# -# Usage: retrace_drop_draw_control.sh -# CTEST ctest binary (default: ctest) -# CONTROL_TMPDIR scratch dir (default: ${RUNNER_TEMP:-/tmp}) -# LIBRARY_LOG the replay's CLIENT library log (default: //output/mobilegl.client.log) -set -u - -CASE="${1:?usage: retrace_drop_draw_control.sh }" -BACKEND="${2:?usage: retrace_drop_draw_control.sh }" - -CTEST="${CTEST:-ctest}" -CONTROL_TMPDIR="${CONTROL_TMPDIR:-${RUNNER_TEMP:-/tmp}}" -# P6: BOTH ROLES HAVE THEIR OWN LOG and neither keeps the old name, so an un-updated default -# here would be a file that does not exist rather than half a session read as a whole one. This -# control's marker (`E2 control armed`) is a CLIENT-side line. -LIBRARY_LOG="${LIBRARY_LOG:-${CASE}/${BACKEND}/output/mobilegl.client.log}" -mkdir -p "${CONTROL_TMPDIR}" -FROZEN_LIBRARY="${FROZEN_LIBRARY:?FROZEN_LIBRARY must name the split library the replay loads}" -symbols=$(nm --defined-only "${FROZEN_LIBRARY}") || exit 1 -remote_count=$(printf '%s\n' "${symbols}" | grep -ic MG_Remote || true) -echo "draw-drop control library: ${FROZEN_LIBRARY}: MG_Remote=${remote_count}" -if [ "${remote_count}" -lt 1 ]; then - echo '::error::draw-drop control requires a split library: MG_Remote=0' - exit 1 -fi - -selector="^MobileGLTraceReplay\.${CASE}\.${BACKEND}$" - -# The rerun replays into the same case directory, so the good run's images are put aside and -# restored whichever way the control goes; "Upload actual image" runs `if: always()` and would -# otherwise ship the deliberately-wrong run's output under the good run's name. -GOOD_OUTPUT="${CONTROL_TMPDIR}/drop-draw-verified-output" -rm -rf "${GOOD_OUTPUT}" -if [ -d "${CASE}" ]; then cp -a "${CASE}" "${GOOD_OUTPUT}"; fi - -restore_good_output() { - if [ -d "${GOOD_OUTPUT}" ]; then - rm -rf "${CASE}"; mv "${GOOD_OUTPUT}" "${CASE}" - echo "restored the verified run's output over the control's" - fi -} -trap restore_good_output EXIT -trap 'exit 130' INT TERM - -matched=$("${CTEST}" -N -R "${selector}" | grep -cE '^ *Test *#[0-9]+:') -if [ "${matched}" -lt 1 ]; then - restore_good_output - echo "::error::the control selected ${matched} tests with -R '${selector}', so there is nothing for the dropped draws to redden. --no-tests=error would have exited non-zero on the empty selection and a control without this guard reads that as success (ID-46 finding 8b)." - exit 1 -fi - -# A PREVIOUS RUN'S LINE MUST NEVER ARM THIS ONE. The CLIENT truncates this log once at open and -# then appends (P6 made the sink O_APPEND, because under spawn a second process writes the same -# file and two truncating handles overwrite each other's bytes) - but it only truncates if the -# replay gets that far, and a run that died in the loader would leave the baseline's log in place -# with a perfectly good "control armed" line in it. Removing it first is the same rule -# split_negative_controls.sh's `reset` step follows. -rm -f "${LIBRARY_LOG}" - -out="${CONTROL_TMPDIR}/retrace-drop-draw-output.txt" -# THE TRANSPORT COMES FROM THE JOB, not from this file. retrace-split is a matrix over -# {backend, case, transport} as of P6, and a control that hard-coded `inproc` would have gone on -# proving something about the OTHER arm while the spawn arm ran unguarded. Default inproc so a -# caller that sets nothing behaves exactly as before. -transport="${SPLIT_TRANSPORT:-inproc}" -export MOBILEGL_TRANSPORT="${transport}" -export MOBILEGL_IPC_E2_DROP_DRAW=1 -"${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1 -control_rc=$? -unset MOBILEGL_IPC_E2_DROP_DRAW -cat "${out}" - -# Read the library's evidence BEFORE the good output is restored over it. -armed_line="" -dropped=0 -if [ -f "${LIBRARY_LOG}" ]; then - armed_line=$(grep 'MGPipe: E2 control armed' "${LIBRARY_LOG}" | tail -1) - dropped=$(printf '%s' "${armed_line}" | sed -n 's/.*armed[^,]*, \([0-9][0-9]*\) records dropped.*/\1/p') - dropped="${dropped:-0}" -fi -cp -f "${LIBRARY_LOG}" "${CONTROL_TMPDIR}/drop-draw-library.log" 2>/dev/null - -restore_good_output - -if [ "${control_rc}" -eq 0 ]; then - echo "::error::the split retrace PASSED with every DrawVbo record dropped on the wire. The golden is made of that geometry, so a green here means the picture did not come from the wire: the emit table fell through to the driver, the library under test is not the one the lane thinks it is, or MOBILEGL_IPC_E2_DROP_DRAW never reached the process. Exit gate E2 is exactly this statement and nothing weaker - MOBILEGL_IPC_E2_DROP_CLEAR is NOT a substitute (measured: 29 clears dropped, ssim still 1.000000, because OpenRA overdraws every pixel it clears)." - exit 1 -fi - -# 2. THE SSIM ACTUALLY FELL. Parsed, not inferred from the exit code. -ssim_line=$(grep -o 'ssim=[0-9.]*, ssimThreshold=[0-9.]*' "${out}" | tail -1) -if [ -z "${ssim_line}" ]; then - echo "::error::the split retrace went red (ctest exit ${control_rc}) with the draws dropped, but its output carries no 'ssim=..., ssimThreshold=...' summary at all, so the comparator never ran. A loader failure, a missing fixture, a timeout or a Fatal{ all land here and none of them is this control's red." - exit 1 -fi -ssim=${ssim_line#ssim=}; ssim=${ssim%%,*} -threshold=${ssim_line##*ssimThreshold=} -if ! awk -v a="${ssim}" -v b="${threshold}" 'BEGIN { exit !(a + 0 < b + 0) }'; then - echo "::error::the split retrace went red (ctest exit ${control_rc}) but its ${ssim_line} is NOT below the threshold, so the picture is not what reddened it. This control's whole claim is that the golden is made of the dropped geometry." - exit 1 -fi - -# 3. THE LIBRARY SAID IT DROPPED SOMETHING. -if [ -z "${armed_line}" ]; then - echo "::error::the split retrace went red with ssim ${ssim} < ${threshold}, but ${LIBRARY_LOG} carries no 'MGPipe: E2 control armed' line, so there is no evidence the knob was ever read by the process that produced the picture. A library that is not the one under test, an emit table that fell through to the driver, or an environment that did not reach the replay all produce a wrong picture for a reason that has nothing to do with this control." - exit 1 -fi -if [ "${dropped}" -lt 1 ]; then - echo "::error::the split retrace went red with ssim ${ssim} < ${threshold} and the knob announced itself - '${armed_line}' - but it reports ZERO records dropped. The emitter was never reached, so whatever changed the picture was not this control. This is the R-16 case the dropped-record COUNT exists for: the arming message alone proves only that the knob was read." - exit 1 -fi - -echo "the dropped draws turned the split retrace red for their own reason (ctest exit ${control_rc}): ${matched} selected case(s), ssim ${ssim} < ${threshold}, and the library dropped ${dropped} record(s) on the wire" diff --git a/scripts/ci/retrace_transport_control.sh b/scripts/ci/retrace_transport_control.sh deleted file mode 100644 index ad593c61..00000000 --- a/scripts/ci/retrace_transport_control.sh +++ /dev/null @@ -1,179 +0,0 @@ -#!/bin/bash -# MobileGL - scripts/ci/retrace_transport_control.sh -# Copyright (c) 2025-2026 MobileGL-Dev -# Licensed under the GNU Lesser General Public License v3.0: -# https://www.gnu.org/licenses/gpl-3.0.txt -# https://www.gnu.org/licenses/lgpl-3.0.txt -# SPDX-License-Identifier: LGPL-3.0-only -# End of Source File Header - -# THE RETRACE-SPLIT LANE'S NEGATIVE CONTROL: a library WITHOUT A TRANSPORT must red this split retrace. -# -# P13 W2: the library is the push monolith without MG_Remote (the shape FCL embeds). Until W2 it was -# the pull build, and this file was retrace_pull_library_control.sh; both libraries were run side by -# side for one wave and both went red here for the same sentence. "pull" below is history: what the -# control needs is a library with no MG_Remote, and the push monolith is one. -# -# This file is the body of .github/workflows/test.yml's "Negative control (library without a transport)" step in -# the retrace-split job, extracted for the reason given at the top of -# scripts/ci/split_negative_controls.sh: a `run:` block is unreviewable and untestable off a -# runner, and scripts/ci/control_smoke_test.sh now runs THIS file rather than a hand-made copy. -# -# WHAT THE CONTROL IS FOR. The retrace-split job replays a trace against the SPLIT runtime under -# MOBILEGL_TRANSPORT=inproc. OpenRA scores ssim 1.000000 against a MONOLITH library too - measured - -# so the picture is not and cannot be this lane's gate. What stands between the job and a green that -# ran monolith end to end is run_trace_case.cmake's transport-resolution assertion -# (run_trace_case.cmake:265-289): the library must have logged -# "MOBILEGL_TRANSPORT=inproc - the MGPipe record stream", which exists only in -# ConfigLoader::InitTransport's InProcess arm, which exists only under MOBILEGL_BUILD_DISAGGREGATED. -# This control swaps the pull library over the frozen path and requires the same replay to fail FOR -# THAT REASON. -# -# WHAT THE REVIEW FOUND (ID-46 finding 8 part (b), CONFIRMED by execution against the REAL ctest in -# a REAL build tree; ID-48 assigns it here). Two holes, both of which let the control pass while -# asserting nothing: -# -# 1. NO SELECTION GUARD AT ALL - unlike the split lane's run_control, which has had one since -# review finding M-4. With a case/backend regex matching no tests, `--no-tests=error` exits 8, -# and the old `if [ "${control_rc}" -eq 0 ]` accepted 8 as "the pull library turned it red". -# The verifier measured exactly that: "real ctest exit for a regex matching NO tests: 8", -# HARNESS_EXIT=0. An empty selection was the one thing this control could not tell apart from -# a working transport-identity assertion. -# 2. ONLY "non-zero ctest" WAS CHECKED after the nm identity check. The nm check establishes that -# the library IS a pull build; it says nothing about why the replay failed. A loader failure, a -# missing fixture or a timeout all passed the control. -# -# Both are closed below: the selection is counted before the run, and the red must carry -# run_trace_case.cmake's own words. -# -# Usage: retrace_transport_control.sh -# CTEST ctest binary (default: ctest) -# CONTROL_TMPDIR scratch dir (default: ${RUNNER_TEMP:-/tmp}) -# CONTROL_LIBRARY the libMobileGL.so without MG_Remote to swap in -# FROZEN_LIBRARY the path every case has baked in, which CONTROL_LIBRARY is copied over -set -u - -CASE="${1:?usage: retrace_transport_control.sh }" -BACKEND="${2:?usage: retrace_transport_control.sh }" - -CTEST="${CTEST:-ctest}" -CONTROL_TMPDIR="${CONTROL_TMPDIR:-${RUNNER_TEMP:-/tmp}}" -CONTROL_LIBRARY="${CONTROL_LIBRARY:?CONTROL_LIBRARY must name a libMobileGL.so built without MG_Remote}" -FROZEN_LIBRARY="${FROZEN_LIBRARY:?FROZEN_LIBRARY must name the path the cases have baked in}" -mkdir -p "${CONTROL_TMPDIR}" - -# run_trace_case.cmake's own sentences for "this library never resolved the transport", one per -# line (grep -F takes its patterns newline-separated). Anchored on the distinctive clause rather -# than on the whole paragraph, which carries substituted paths. -# -# TWO SENTENCES SINCE P6's LOG RENAME, and the second is the one a pull library actually gets. -# A pull build has one log role and writes MOBILEGL_LOG_FILE_PATH unchanged - output/mobilegl.log -# (MG_Util/Debug/Log.h, the pull arm of RoleLogPath) - while the disaggregated build's readers, -# run_trace_case.cmake among them, read output/mobilegl.client.log. So the pull library never -# reaches the marker search that says "never reported resolving it": the runner stops one check -# earlier, at "the run wrote no , so there is no evidence the transport ever -# resolved". Both are the transport-identity assertion refusing the same library for the same -# reason (a pull build has no split log for the marker to be in); neither is a loader failure, a -# missing fixture, a timeout or an SSIM drop, which are the reds hole 2 below exists to refuse. -# MEASURED on the B3 package tree with the real ctest and the real pull library: with only the -# first sentence here, the control exited 1 through hole 2 on every pull-library run - the step -# had been red for no product reason since the rename. scripts/ci/testdata/stub_ctest.sh's -# retrace-evidence-nolog mode reproduces the shape. -EVIDENCE='never reported resolving it -no evidence the transport ever resolved' -# EVERY LINE OF EVIDENCE IS A PATTERN, SO AN EMPTY LINE IS A PATTERN THAT MATCHES EVERYTHING. A -# trailing newline, or a blank line between the two sentences, would turn hole 2's check below -# into "any red will do". The smoke test's "red without the transport-resolution message" case -# would catch that (22/23), but only once somebody ran it; this refuses it here, before the -# selection is counted or the library is swapped. -case "${EVIDENCE}" in - *$'\n\n'*|*$'\n'|$'\n'*) - echo "::error::EVIDENCE carries an empty grep -F pattern (a blank line or a leading/trailing newline), which would match every failure" - exit 1;; -esac - -selector="^MobileGLTraceReplay\.${CASE}\.${BACKEND}$" - -# The rerun replays into the same case directory, so the good run's images are put aside and -# restored whichever way the control goes; "Upload actual image" runs `if: always()` and would -# otherwise ship the deliberately-wrong run's output under the good run's name. -GOOD_OUTPUT="${CONTROL_TMPDIR}/split-verified-output" -rm -rf "${GOOD_OUTPUT}" -if [ -d "${CASE}" ]; then cp -a "${CASE}" "${GOOD_OUTPUT}"; fi - -restore_good_output() { - if [ -d "${GOOD_OUTPUT}" ]; then - rm -rf "${CASE}"; mv "${GOOD_OUTPUT}" "${CASE}" - echo "restored the verified run's output over the control's" - fi -} - -# Restore the exact split library on success, failure, and interruption. The following -# draw-drop control uses this same frozen path. -saved_library=$(mktemp "${CONTROL_TMPDIR}/split-library.XXXXXX") || exit 1 -cp -p "${FROZEN_LIBRARY}" "${saved_library}" || exit 1 -restore_control() { - cp -p "${saved_library}" "${FROZEN_LIBRARY}" - rm -f "${saved_library}" - restore_good_output -} -trap restore_control EXIT -trap 'exit 130' INT TERM - -# HOLE 1: COUNT THE SELECTION FIRST. `--no-tests=error` turns an empty selection into a non-zero -# exit, which is indistinguishable from a working control unless the selection is counted. -matched=$("${CTEST}" -N -R "${selector}" | grep -cE '^ *Test *#[0-9]+:') -if [ "${matched}" -lt 1 ]; then - restore_good_output - echo "::error::the control selected ${matched} tests with -R '${selector}', so there is nothing for the transport-less library to red. --no-tests=error would have exited non-zero on the empty selection and this control used to read that as success (ID-46 finding 8b, measured: ctest exit 8, step green)." - exit 1 -fi - -# The pull library, unpacked from build-linux's artifact, over the frozen path every case has baked -# in. It defines no MG_Remote symbol, so ConfigLoader has no transport parser and -# MOBILEGL_TRANSPORT=inproc is accepted and ignored - the exact shape of "the split lane ran -# monolith". -cp "${CONTROL_LIBRARY}" "${FROZEN_LIBRARY}" || exit 1 -symbols=$(nm --defined-only "${FROZEN_LIBRARY}") || exit 1 -remote_count=$(printf '%s\n' "${symbols}" | grep -ic MG_Remote || true) -echo "transport control library: ${FROZEN_LIBRARY}: MG_Remote=${remote_count}" -if [ "${remote_count}" -ne 0 ]; then - restore_good_output - echo "::error::the control's own library defines MG_Remote symbols, so it has a transport and this control would prove nothing" - exit 1 -fi - -out="${CONTROL_TMPDIR}/retrace-control-output.txt" -# THE TRANSPORT COMES FROM THE JOB, not from this file. retrace-split is a matrix over -# {backend, case, transport} as of P6, and a control that hard-coded `inproc` would have gone on -# proving something about the OTHER arm while the spawn arm ran unguarded. Default inproc so a -# caller that sets nothing behaves exactly as before. -transport="${SPLIT_TRANSPORT:-inproc}" -export MOBILEGL_TRANSPORT="${transport}" -"${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1 -control_rc=$? -cat "${out}" - -restore_good_output - -if [ "${control_rc}" -eq 0 ]; then - echo "::error::a library WITHOUT A TRANSPORT passed the split retrace. OpenRA scores ssim 1.000000 under a monolith library too (measured), so the picture is not and cannot be this lane's gate - run_trace_case.cmake's transport-resolution assertion is, and it has stopped working. Every green in this job is then a monolith run under a name that says split." - exit 1 -fi - -# HOLE 2: THE RED MUST BE THIS CONTROL'S RED. -# -# Whitespace is normalised across the WHOLE file before the match, newlines included, because the -# sentence is emitted by CMake's message(FATAL_ERROR ...) and CMake re-wraps that text to its own -# width: "never reported resolving it" arrives split over two lines with a two-space continuation -# indent, and a line-oriented grep for the literal finds nothing. That is not hypothetical - it is -# the shape the stub reproduces in scripts/ci/testdata/stub_ctest.sh. -# `ctest -V` ALSO PREFIXES EVERY OUTPUT LINE WITH ": ", continuation lines included, -# so without stripping it first the folded text reads "never 1: reported resolving it" and the -# control reds on the very sentence it is looking for (the *-prefixed stub modes). -if ! sed -E 's/^[0-9]+: //' "${out}" | tr -s '[:space:]' ' ' | grep -qF "${EVIDENCE}"; then - echo "::error::the split retrace went red (ctest exit ${control_rc}) with the transport-less library in place, but the failure never says the library did not resolve the transport - neither of run_trace_case.cmake's sentences (\"${EVIDENCE//$'\n'/\" / \"}\") is in the output. A loader failure, a missing fixture, a timeout or an SSIM drop all land here, and none of them establishes that the transport-identity assertion is what caught the pull library. Only 'non-zero ctest' used to be checked (ID-46 finding 8b)." - exit 1 -fi - -echo "the transport-less library turned the split retrace red for its own reason (ctest exit ${control_rc}): ${matched} selected case(s) named the transport, not the picture" diff --git a/scripts/ci/testdata/stub_ctest.sh b/scripts/ci/testdata/stub_ctest.sh index 6dfc2d0f..bf9e4db6 100755 --- a/scripts/ci/testdata/stub_ctest.sh +++ b/scripts/ci/testdata/stub_ctest.sh @@ -33,28 +33,6 @@ # green baseline green; the control's own run PASSES (the knob is not load-bearing) # red-baseline the baseline itself has a failed entry # all-skipped the baseline is entirely skipped (lost implementation, a hard failure) -# retrace-noselect `ctest -N` matches nothing; the run exits 8 the way --no-tests=error does -# retrace-unrelated one match; the run fails without naming the transport -# retrace-evidence one match; the run fails with run_trace_case.cmake's own sentence -# retrace-evidence-nolog -# one match; the run fails one check earlier, with the runner's "wrote no -# ... no evidence the transport ever resolved" - the shape a -# PULL library produces since P6's per-role log rename -# retrace-evidence-prefixed, retrace-evidence-nolog-prefixed -# the two above with `ctest -V`'s "1: " prefix on every line -# retrace-green one match; the run PASSES -# -# EXIT GATE E2's DRAW-DROP CONTROL (scripts/ci/retrace_drop_draw_control.sh). The library's own -# log is modelled as a separate sink from ctest stdout, exactly as it is for the split lane: the -# console sink is compiled out of the configurations these lanes run, so the dropped-record count -# can only ever arrive through ${LIBRARY_LOG}. -# dropdraw-noselect `ctest -N` matches nothing -# dropdraw-green the retrace PASSES with the draws dropped -# dropdraw-nossim red, but no ssim summary at all (loader failure / timeout shape) -# dropdraw-ssimhigh red, and the ssim is ABOVE the threshold: something else reddened it -# dropdraw-nolog red with a low ssim, but the library never said the knob armed -# dropdraw-zero red with a low ssim, the knob armed, and it dropped NOTHING -# dropdraw-evidence red with a low ssim and N > 0 records dropped: the real thing set -u mode="${STUB_MODE:?STUB_MODE must be set}" @@ -142,10 +120,6 @@ fi emit_listing() { echo "Test project /stub" - if [ "${mode}" = "retrace-noselect" ] || [ "${mode}" = "dropdraw-noselect" ]; then - echo "Total Tests: 0" - return - fi echo " Test #1: DirectGLES.Split.ClearThenReadPixelsScenario.ClearWithNoDrawIsVisibleToDefaultFramebufferReadPixels" if [ "${mode}" = partial-fatal ]; then echo " Test #2: DirectGLES.Split.TriangleScenario.SecondEntry" @@ -288,86 +262,6 @@ case "${mode}" in echo "100% tests passed, 0 tests failed out of 1" exit 0 ;; - retrace-noselect) - echo "No tests were found!!!" - exit 8 - ;; - retrace-unrelated) - echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed" - echo "CMake Error: the fixture could not be unpacked" - exit 8 - ;; - retrace-evidence) - echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed" - echo "CMake Error at run_trace_case.cmake:279 (message):" - echo " MOBILEGL_TRANSPORT=inproc is set for OpenRA DirectGLES and the library never" - echo " reported resolving it: mobilegl.log carries no" - echo ' "MOBILEGL_TRANSPORT=inproc - the MGPipe record stream".' - exit 8 - ;; - retrace-evidence-nolog) - # What a pull library gets since P6: it wrote output/mobilegl.log (one role, no suffix), the - # runner reads output/mobilegl.client.log and stops at the no-log check BEFORE the marker - # search. CMake-wrapped like the mode above, with the clause split across lines. - echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed" - echo "CMake Error at run_trace_case.cmake:270 (message):" - echo " MOBILEGL_TRANSPORT=inproc is set for OpenRA DirectGLES but the run wrote no" - echo " OpenRA/DirectGLES/output/mobilegl.client.log, so there is no evidence the" - echo " transport ever resolved. A split retrace with no library log cannot be" - echo " counted as a split retrace." - exit 8 - ;; - retrace-evidence-prefixed|retrace-evidence-nolog-prefixed) - # The two modes above as `ctest -V` really prints them: EVERY line of a test's output carries - # a ": " prefix, continuation lines included, so after whitespace folding the - # wrapped sentence reads "never 1: reported resolving it" / "no evidence the 1: transport ever - # resolved". A short case directory breaks the sentence at exactly those words; CI's long - # paths happened not to (retrace-split, run 35671704873: 5 false reds from this shape). - STUB_MODE="${mode%-prefixed}" bash "$0" "$@" | sed 's/^/1: /' - exit "${PIPESTATUS[0]}" - ;; - retrace-green) - echo "100% tests passed, 0 tests failed out of 1" - exit 0 - ;; - dropdraw-*) - # The library's own log, written by the replay the way the real one is. The control removes - # it before the run, so anything here is this run's. - armed="" - case "${mode}" in - dropdraw-zero) - armed='MGPipe: E2 control armed - drop-draw=1 drop-clear=0, 0 records dropped on the wire (draw=0 clear=0), frame 29' ;; - dropdraw-nolog) armed="" ;; - *) - armed='MGPipe: E2 control armed - drop-draw=1 drop-clear=0, 758 records dropped on the wire (draw=758 clear=0), frame 29' ;; - esac - if [ -n "${LIBRARY_LOG:-}" ] && [ -n "${armed}" ]; then - mkdir -p "$(dirname "${LIBRARY_LOG}")" - printf '%s\n' "[10:38:30] [Linux mobilegl_trace_/WARN]: ${armed}" > "${LIBRARY_LOG}" - fi - case "${mode}" in - dropdraw-green) - echo "100% tests passed, 0 tests failed out of 1" - exit 0 - ;; - dropdraw-nossim) - echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed" - echo "CMake Error: the replay could not load the library" - exit 8 - ;; - dropdraw-ssimhigh) - echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed" - echo "-- retrace completed; ssim=1.000000, ssimThreshold=0.990000, mismatchPixels=0" - echo "CMake Error at run_trace_case.cmake:301 (message): 3 MGPipe Fatal(s)" - exit 8 - ;; - *) - echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed" - echo "-- retrace completed; ssim=0.000036, ssimThreshold=0.990000, mismatchPixels=295296" - exit 8 - ;; - esac - ;; *) echo "stub_ctest: unknown STUB_MODE '${mode}'" >&2 exit 127 diff --git a/tools/trace_replay/trace_cases.py b/tools/trace_replay/trace_cases.py index 8e334fa8..20ef49cb 100644 --- a/tools/trace_replay/trace_cases.py +++ b/tools/trace_replay/trace_cases.py @@ -394,11 +394,11 @@ def github_transport_matrix(cases, *, apk=False): THE SPAWN LANE IS APK-ONLY, and that is a statement about where the evidence is missing rather than about what spawn can do. On Linux the spawn arm already has a job of its own - test.yml's `retrace-split`, which is {backend, case, transport} over inproc AND spawn and - carries the two negative controls that make it mean anything (the pull library must red it, - dropping the draws must red it). Adding spawn to THIS matrix as well would pay for a second - full Linux sweep that proves nothing the first one does not, and test.yml's `retrace` job - has neither the server image staged nor MOBILEGL_IPC_SERVER_PATH set - so the lane would - not merely be redundant, it would be red. + whose legs prove in-test that the transport resolved and, on OpenRA, that draw records + crossed the wire (run_trace_case.cmake's evidence). Adding spawn to THIS matrix would pay + for a second full Linux sweep that proves nothing the first one does not, and test.yml's + `retrace` job has neither the server image staged nor MOBILEGL_IPC_SERVER_PATH set - so the + lane would not merely be redundant, it would be red. On Android there is no such job. Until this lane existed, nothing in CI ever started a second MobileGL process on a device.