Files
MobileGL/docs/Disaggregated/notes/recovered/wf4/scout-test-infra.md
T
swung0x48 aba329cc53 [Docs] (Disaggregated): P6's plan and contract draft; the working notes move from a local directory into the repository
Two things land together: the plan and contract draft P6 needs before it starts, and the
working notes that until now existed on exactly one machine.

1. P6's plan and contract draft (neither started; not one line of code)

     docs/Disaggregated/P6-SPAWN-PLAN.md      the package plan
     docs/Disaggregated/P6-CONTRACT-DRAFT.md  the contract draft (moves to
                                              MG_Remote/CONTRACT-P6.md when c6 lands)

   The three findings that mattered, all written into the documents:

   a. P6 is SMALLER than its roadmap cell suggests. The transport primitives are already in
      the tree and already tested: SocketDoorbell including hangup detection and the two
      lost-wakeup fences (Doorbell.cpp:211-360, four cases in FdPassingTest), SCM_RIGHTS fd
      passing, ShmSegment::Adopt, and the SurfaceOp / WindowKind schema already defined in
      protocol.fbs. inproc uses a real second shm mapping rather than an alias, and
      SessionRings.h:22-28 says that was done so today is not the first time it runs. P6 is
      assembly plus a control plane plus a process, not "write a socket transport".

   b. The evidence for "P6 is only a transport swap" has EXPIRED. It was bought by P5c, whose
      audit found 59 direct memory accesses in a tree that already looked clean. P5e changed
      the tree again. So the first package, a6, is a read-only audit that writes no code, and
      every contract row that depends on it is marked PENDING a6 rather than decided in
      advance.

   c. P6 does not get a game running on the phone. ANativeWindow* is meaningless in the child,
      and protocol.fbs:186-196's own comment says Android transfers the window out of band -
      that band is P12. P6 lands pbuffer / surfaceless / offscreen, which is what HeadlessGL,
      the split-equivalent lane and trace replay need, and which is exactly what P6's own exit
      gate measures. A real window reaching the server is a named refusal. This gets its own
      section because "P6 is done" would otherwise read as "the game runs now".

   The contract adds rule G: a role may not name a process-local handle. It extends P5c's
   rule E (a role may not name the other role's memory) from memory to identity - EGLDisplay /
   EGLSurface / EGLContext / ANativeWindow* / fd number / pid / mapped address either cross as
   a token this contract defines or they do not cross, and a descriptor crosses only through
   SCM_RIGHTS, where the kernel renumbers it.

   What P5e left for P6 gets its own section. The latch semantics are already designed
   (ARCHITECTURE.md:504) and are copied unchanged. What is new is that after run-ahead the
   client no longer waits while the peer process can DIE: the client holds records that were
   published and will never be applied, and Android's freezer suspends the child without
   closing anything. So slow and dead must be distinguishable WITHOUT a timeout threshold.
   That is P6's only new design; everything else is carriage. It shares a predicate with E1's
   outstanding re-specification (ID-122), and the two should be designed together.

   Two gates carried over from P5e's lessons: G2 requires integration-spawn's test-name set to
   match integration-split's name for name, because a differing set means the lane is quietly
   running less; and the arm-proof gate requires every entry to record the child pid and
   transport=spawn in its OWN private log - ID-124 established that this kind of false green
   really happens, and a spawn lane that actually ran monolith would pass every other gate.
   S2's shape must be settled before it is written, and the document says why: E1 is the
   cautionary case.

2. The working notes move into the repository (docs/Disaggregated/notes/, 415 .md files)

   ROADMAP / CURRENT_STAGE_PROGRESS / MEASUREMENTS cite ruling numbers, audit line numbers and
   the provenance of gate figures from ~/w7/notes/, a directory that exists on one machine and
   is under no version control. A record cited hundreds of times that lives only locally is not
   a record, so it moves in whole, with its directory structure intact.

   Only .md moves. Scripts, run logs, census and gate result JSON, patches, APKs and nm dumps
   stay where they are: they are evidence DATA rather than documents - large, regenerated per
   run, and mostly carrying absolute paths and device serials that mean something only on the
   machine that produced them. The mapping and this split are written in
   docs/Disaggregated/notes/README.md.

   Two exceptions are recorded there too: P6's BRIEF has been promoted to a first-class
   document and keeps no copy under notes/, so the two cannot drift apart; and recovered/ holds
   material salvaged from failed workflow runs, which may duplicate the like-named phase
   directories and is kept because a few of those were the only complete version at the time.

   CI's citation lint takes docs/Disaggregated/*.md only (non-recursive), and notes/ is
   deliberately outside it: those notes' file:line citations were written against their own
   code heads, and resolving them against today's HEAD would fail wholesale - which is not an
   error, it is their correct state as a historical record. To check one, use the revision it
   declares (--rev).

3. Inconsistencies fixed in passing

   - README.md's status line still said P5e was in progress while the stage table already said
     closed.
   - CURRENT_STAGE_PROGRESS's next-step item 1 still listed the exit gate as pending, though
     (a) and (b) were run in the previous commit; rewritten to closed plus the one item that
     actually remains (E1).
   - ROADMAP's P6 row still had the pre-phase plan in three of its cells; it now carries the
     package order, the scope boundary and the two new gates.

Checked with check_doc_citations.py: 125 citations across docs/Disaggregated/*.md, 0 problems.
2026-09-18 23:31:39 -04:00

68 KiB
Raw Blame History

P1 verification infrastructure — scout report

Worktree: C:/Users/geekerwan/AndroidStudioProjects/FoldCraftLauncher/MobileGL-disagg, branch feat/disaggregated. Every claim below carries a path:line I opened. Read-only pass; nothing in the tree was modified.

Reading order: Part A is the inventory (what exists, exactly where). Part B is the proposal for the three things the task asks for. Part C is the trap list — read it before writing any CMake, because three of the traps have already cost this project a shipped bug or a silently-green lane.


Part A — inventory

A1. .github/workflows/test.yml — the whole job graph

Triggers: push to dev, Feat/Backend-Direct-GLES, Feat/Backend-Direct-Vulkan, plus workflow_dispatch (.github/workflows/test.yml:3-9). feat/disaggregated is not a trigger branch — P1 work does not run this workflow on push unless the branch list is extended or workflow_dispatch is used. There is no pull_request trigger and no timeout-minutes on any job in this file (grep timeout-minutes finds only apk.yml:331), so every job inherits GitHub's 6 h default.

Nine jobs:

build-linux (test.yml:12-147)

  • runs-on: ubuntu-latest, 32 GB swap (:26-29), submodules: recursive (:31-34).
  • ccache via job env (:17-23): BUILD_DIR=build-linux, CCACHE_DIR=${{ github.workspace }}/.ccache, key ${{ runner.os }}-test-${{ github.job }}-ccache-v1 (:39-45), saved only on the default branch (:109-122).
  • Toolchain: clang-20 / libc++-20 / lld-20 / ninja / mesa + lavapipe + Vulkan loader (:58-61), Vulkan SDK 1.4.304.1 headers+loader (:47-52), python update_glslang_sources.py (:54-56).
  • Configure (:71-93) — the flags P1 has to mirror:
    -DCMAKE_BUILD_TYPE=Release            (Debug only when ACTIONS_STEP_DEBUG=true, :73-77)
    -DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO
    -DMOBILEGL_BUILD_TEST=ON
    -DMOBILEGL_BUILD_BENCHMARK=ON
    -DMOBILEGL_BUILD_INTEGRATION_TEST=ON
    -DMOBILEGL_ITEST_VK_ICD=/usr/share/vulkan/icd.d/lvp_icd.json
    -DMOBILEGL_BUILD_TRACE_REPLAY=OFF
    -DBENCHMARK_DOWNLOAD_DEPENDENCIES=ON -DBENCHMARK_ENABLE_TESTING=OFF
    -DCMAKE_POLICY_VERSION_MINIMUM=3.5
    
    Note what is absent: MOBILEGL_BUILD_DISAGGREGATED is never passed, so it stays OFF (CMakeLists.txt:23), and MOBILEGL_ITEST_REQUIRE_GPU is never passed at configure time (it arrives as job env instead — see integration below). MOBILEGL_ENABLE_LTO defaults OFF (CMakeLists.txt:108).
  • Packaging (:124-147): ci-artifacts/mobilegl-linux-runtime.tgz containing the root build-linux/CTestTestfile.cmake, the three test/benchmark/integration subtrees and every *.so found under the build dir. Uploaded as artifact mobilegl-linux-runtime.

test (test.yml:149-203) — the unit lane

  • needs: build-linux; installs runtime deps only (:160-163); downloads + unpacks the runtime tgz.
  • "Normalize CTest command paths" (:174-184): a python inline script rewriting "…/cmake-…/bin/cmake" → "cmake" in every CTestTestfile.cmake under build-linux. Repeated verbatim in integration (:234-244), benchmark (:380-390) and build-retrace (:526-536). Any new job that unpacks the runtime artifact and calls ctest must repeat this step.
  • Run (:186-195): cores armed (ulimit -c unlimited, kernel.core_pattern=/tmp/core.%e.%p), then ctest --output-on-failure -L unit --no-tests=error (-V under ACTIONS_STEP_DEBUG).
  • Failure-only core-dump artifact unit-core-dumps (:197-203).

integration (test.yml:205-296) — the GPU-scenario lane, the one P1 doubles

  • needs: build-linux. Extra runtime dep libegl-mesa0, with the reason spelled out (:216-223).
  • Job env (:266-270):
    MOBILEGL_ITEST_REQUIRE_GPU: "1"
    MOBILEGL_MAGMA_FIX_ITERATIONRP_SUBGROUP_SCRATCH: "1"
    MOBILEGL_MAGMA_DERIVE_NUM_SUBGROUPS: "1"
    MOBILEGL_MAGMA_ITERATIONRP_FIX_BARRIER: "1"
    
  • The ENVIRONMENT-property warning lives here in full (:252-259): the lavapipe ICD pin is passed at configure (-DMOBILEGL_ITEST_VK_ICD) because the configure bakes it into each test's ctest ENVIRONMENT property, "and a property entry OVERRIDES the job environment — a VK_ICD_FILENAMES exported here would be silently ignored while looking like it works."
  • The counter-example is in the same job (:274-288): the second, filtered pass exports MOBILEGL_ESPRYT_DISABLE_INVALIDATE_FLUSH=1 inline with the comment "The flag is NOT baked into the ctest ENVIRONMENT properties, so an inline env reaches the test processes (unlike the ICD pin above)." This is the decisive evidence for P1: a job-env variable that no ENVIRONMENT property names does reach the test process. MOBILEGL_PIPE_VERIFY / MOBILEGL_PIPE_PUSH are named by no property today.
  • Two ctest invocations:
    ctest --output-on-failure -L integration-gpu --no-tests=error
    MOBILEGL_ESPRYT_DISABLE_INVALIDATE_FLUSH=1 ctest --output-on-failure -L integration-gpu \
        -R 'Buffer|Readback|Atomic|Ssbo|Arena' --no-tests=error
    
  • integration-core-dumps on failure (:290-296).

flatc-check (test.yml:298-323)

Independent of build-linux; regenerates protocol_generated.h with scripts/gen_protocol.py and git diff --exit-code. The model for a source-level gate that must not be hidden by a broken build.

include-graph-check (test.yml:332-349) — the P0.5 purity gate, and the template for P1's controls

  • No needs, three header submodules only (:340-343), clang-20 (:345-346), then: python3 scripts/check_include_closure.py --mode both --compiler clang++-20 --self-test --require-all.
  • The job comment states the doctrine P1 must satisfy: "The script's own --self-test is always on: a negative control that stopped tripping fails the job, because a gate that cannot go red is not a gate" (test.yml:330-331, quoting ROADMAP.md:7).

benchmark (test.yml:351-405)

needs: build-linux; ctest -V -C Release -L benchmark --no-tests=error.

build-retrace (test.yml:407-550)

  • needs: build-linux, test, benchmark, integration — i.e. the retrace lane only starts after the three ctest lanes are green.
  • Job env MOBILEGL_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so (:424); the runtime artifact is unpacked and that path asserted to exist (:478-481).
  • Second configure (:483-502), separate build dir build-retrace, separate ccache key:
    -DMOBILEGL_BUILD_TEST=OFF -DMOBILEGL_BUILD_BENCHMARK=OFF
    -DMOBILEGL_BUILD_TRACE_REPLAY=ON
    -DMOBILEGL_TRACE_REPLAY_MOBILEGL_LIBRARY="${MOBILEGL_LIBRARY}"
    
  • Builds only --target mobilegl_trace_replay (:504-505), normalizes CTest paths (:526-536), and packages exactly two files (:538-543): build-retrace/tools/trace_replay/mobilegl_trace_replay and build-retrace/tools/trace_replay/CTestTestfile.cmake. Artifact mobilegl-trace-replay.
  • Consequence for P1: the absolute paths of the trace case commands (replay exe, MOBILEGL_LIBRARY, fixture paths) are frozen into that CTestTestfile.cmake at configure time. A later job re-creates the same paths by unpacking into the same workspace layout, it does not reconfigure.

trace-cases / trace-fixtures / retrace / retrace-summary / remove-artifact-clutter

  • trace-cases (:552-570) emits two outputs from tools/trace_replay/trace_cases.py --ci: matrix (--format github-test-matrix) and names (--format names).
  • trace-fixtures (:572-635) is a per-case matrix (names only): derive cache key (.github/scripts/trace-fixture-cache.sh key), restore, verify, else fetch-trace-fixture-lfs.sh, stage the files listed by trace_cases.py --format fixture-files --case <name>, upload as trace-fixture-<case>.
  • retrace (:637-741) is the per-(backend, case) matrix, fail-fast: false, max-parallel: 4. It downloads trace-fixture-<case> into tools/trace_replay/fixtures/ (:660-669), unpacks both runtime tarballs (:693-698), then in build-retrace/tools/trace_replay:
    ulimit -c unlimited; sysctl kernel.core_pattern=/tmp/core.%e.%p
    export MOBILEGL_MAGMA_R11G11B10F_FALLBACK=1              # DirectVulkan
    export MOBILEGL_MAGMA_{FIX_ITERATIONRP_SUBGROUP_SCRATCH,DERIVE_NUM_SUBGROUPS,ITERATIONRP_FIX_BARRIER}=1
                                                              # DirectVulkan + iterationrp case
    export MOBILEGL_MAGMA_DISABLE_BLENDED_DEPTH_WRITE=1       # DirectVulkan + improved-transparency-26.3
    ctest -V --no-tests=error -R '^MobileGLTraceReplay\.<case>\.<backend>$'
    
    (:700-723). These export MOBILEGL_* lines are the second proof that job env reaches the retrace test process — they are not in any ENVIRONMENT property (see A4) and they demonstrably take effect.
  • Artifacts: retrace-core-dumps-* on failure, retrace-result-<backend>-<case> always (:725-741).
  • retrace-summary (:743-782) renders render_retrace_summary.mjs over all retrace-result-*.
  • remove-artifact-clutter (:784-833) deletes trace-fixture-* and retrace-result-* artifacts, retaining fixtures only for failed retrace jobs.

pipe-gates (test.yml:835-887) — the MGPipe source gate, where P1's new gates belong

Deliberately independent of build-linux (:837-838). Four steps:

  1. python3 scripts/gen_pipe.py + git diff --exit-code -- MobileGL/MG_Pipe/generated (:850-853).
  2. stdio-instrumentation grep over MobileGL/MG_Backend and MobileGL/MG_State (:862-869) — bans fprintf(stderr|stdout, printf(, puts(, std::cout|cerr, with no exceptions list.
  3. python3 scripts/gen_pipe_dirty_surface.py --summary — informational, and the comment says explicitly "It becomes a gate in P1, when the mapping file exists to diff against" (:871-874).
  4. python3 scripts/check_doc_citations.py docs/Disaggregated/*.md || true — warning-only until the docs settle (:876-887).

A2. ctest labels

Only three labels exist in the tree:

  • unit — every MG_Test target (MobileGL/MG_Test/CMakeLists.txt:65, Pipe/CMakeLists.txt:28, Util/CMakeLists.txt:56-58, Purity/CMakeLists.txt:13).
  • integration-gpu — every registration in MobileGL/MG_IntegrationTest/CMakeLists.txt (13 of them).
  • benchmark — MG_Benchmark.
  • The trace-replay tests carry NO label at all (tools/trace_replay/CMakeLists.txt:350-379 sets only ENVIRONMENT). They are selected purely by the -R '^MobileGLTraceReplay\.…$' regex.

A3. MobileGL/MG_IntegrationTest/ — registration, env plumbing, harness

Target and sources

add_executable(MobileGLIntegrationTest …) with Main.cpp, two Harness TUs and 79 scenario TUs (CMakeLists.txt:51-130). Links GTest::gtest (not gtest_main) plus MobileGL_s on desktop / MobileGL (shared) on Android (:29-36, :137-141). Main.cpp installs a HarnessBanner testing::Environment and calls RUN_ALL_TESTS() (Main.cpp:62-66); the banner prints backend, renderer, surface and EGL_PLATFORM, and states whether it is skipping or failing every scenario.

371 TEST_F cases across Scenarios/*.cpp (counted by grep -cE '^\s*TEST_F\(').

The 13 ctest registrations (CMakeLists.txt:400-633)

All share LABELS integration-gpu, DISCOVERY_TIMEOUT 30, TIMEOUT ${MGL_ITEST_TIMEOUT} where MGL_ITEST_TIMEOUT = 120 (:393, with the comment "a GPU test that wedges must fail the run, not hang it").

Prefix TEST_FILTER cases ENVIRONMENT var
DirectGLES. — (all) 371 MGL_ITEST_GLES_ENVIRONMENT (:400-407)
DirectVulkan. — (all) 371 MGL_ITEST_VULKAN_ENVIRONMENT (:409-416)
DirectVulkan.AsyncCompile. XfbAfterClipDistanceScenario.* 5 …VULKAN_ASYNC_ENVIRONMENT (:429-437)
DirectGLES.ForcedDepthStencilEmulation. DepthStencilReadback*Scenario.* 20 …GLES_FORCED_DS_ENVIRONMENT (:447-455)
DirectGLES.UnlocatedIoBlocks. UnlocatedIoBlockScenario.* 3 …GLES_UNLOCATED_IO_BLOCKS_ENVIRONMENT (:464-472)
DirectGLES.AsyncOn. / DirectVulkan.AsyncOn. AsyncCompileScenario.* 6 + 6 :484-501
DirectGLES.AsyncOff. / DirectVulkan.AsyncOff. one case 1 + 1 :511-528
DirectGLES.OptimisticShaderStatus. / DirectVulkan.… one case 1 + 1 :537-554
DirectGLES.NoViewportArrayEmulation. one case 1 :561-569
DirectVulkan.PrimGenReroute. PrimitivesGeneratedNoXfbScenario.* 8 :580-588
DirectGLES.WidenedPacked16. CopyImagePacked16Scenario.* 3 :598-606
DirectGLES.PointSizeDemotion. / DirectVulkan.… PointSizeDemotionScenario.* 5 + 5 :615-633

Total ctest entries under integration-gpu: 808. Each entry is one process launch of MobileGLIntegrationTest --gtest_filter=<one case>, and each process does a full EGL bring-up preceded by a fork() pre-flight (Harness/HeadlessGL.cpp:323-343). Process count, not GL work, is the dominant cost of the lane.

The env-join idiom — copy it exactly

mgl_itest_join_environment(outVar …) (:307-317) escapes each ; as \\; because gtest_discover_tests forwards PROPERTIES as a flat list; without it "only MOBILEGL_BACKEND_TYPE reaches the test and the vendor/ICD pinning is lost" (:301-306).

MGL_ITEST_COMMON_ENV (:274-281) = __EGL_VENDOR_LIBRARY_FILENAMES=<detected> + MOBILEGL_ITEST_REQUIRE_GPU=1 only when the CMake option is ON (:279-281; the option itself at :258-259 defaults OFF, which is why CI passes it as job env instead). MGL_ITEST_VULKAN_ENV (:283-299) adds VK_ICD_FILENAMES=<pin> and, when the pin is lavapipe, the three iterationRP repairs.

CMakeLists.txt:339-344 states the rule in bold: every mode list appends to MGL_ITEST_COMMON_ENV / MGL_ITEST_VULKAN_ENV, because "A ctest ENVIRONMENT property REPLACES the job environment rather than adding to it, so an entry that lists only its mode variable would silently lose the EGL vendor and Vulkan ICD pinning".

How a scenario reads configuration (the MG_Config question)

It does not read MG_Config. Harness/ScenarioFixture.h:34-65 defines AmbientQuirk / AmbientQuirkFromEnvironment(name) which re-implements ConfigLoader's truthiness rule character for character, and the header states the two reasons (:38-48): the feature table is an internal symbol and the Android build links the shipping -fvisibility=hidden library; and an in-process poke is already too late for anything latched at initialization. The one exception is Harness/BackendCapsPeek.h:20-28, which peeks at the active backend's caps block and returns false on Android, touching nothing — the sanctioned shape for a desktop-only internal peek from this module.

ScenarioTest::SetUp (ScenarioFixture.h:69-104): FAIL() when MOBILEGL_ITEST_REQUIRE_GPU is set and the harness is unusable, else GTEST_SKIP(); derived fixtures must gate on Ready(), never IsSkipped() (:106-109, with the SIGSEGV story that motivated it).

The "arming assertion" idiom — the model for proving verify is actually on

Three lanes pin a quirk and a per-lane log file so the test can prove the library armed it: MGL_ITEST_GLES_UNLOCATED_IO_BLOCKS_ENVIRONMENT (:366-369), MGL_ITEST_VULKAN_PRIMGEN_REROUTE_ENVIRONMENT (:374-377), MGL_ITEST_{GLES,VULKAN}_POINT_SIZE_DEMOTION_ENVIRONMENT (:383-390). The comment at :361-365 explains why: the arming signal is a latched MGLOG_I and there is no other way for a test process to learn it fired. Consumption side: Scenarios/UnlocatedIoBlockScenario.cpp:354-360 and :489, Scenarios/PrimitivesGeneratedNoXfbScenario.cpp:253 and :526, Scenarios/PointSizeDemotionScenario.cpp:304-308 and :489 — each reads MOBILEGL_LOG_FILE_PATH, trusts only bytes written after the case started, and GTEST_SKIP()s with an explicit message when the path is unset.

Local runner

MobileGL/MG_IntegrationTest/run_integration_test.sh — ./run_integration_test.sh espryt|magma [gtest args]; pins __EGL_VENDOR_LIBRARY_FILENAMES / VK_ICD_FILENAMES from MGL_EGL_VENDOR / MGL_VK_ICD (defaults are NVIDIA paths), sets EGL_PLATFORM=${EGL_PLATFORM:-x11}, and honours MOBILEGL_ITEST_REQUIRE_GPU. Note it is not the ctest path: it does not carry the iterationRP pins or any mode variable.

A4. tools/trace_replay/ — the retrace gate

Case manifest

trace_cases.json: defaults (854x480, ssim 0.99, timeout_seconds: 900) + 40 cases.

  • One case is CI-excluded: minecraft-1.21.4-rd12-odinlite-in-world has "ci": false (trace_cases.json:63-70, the flag itself at :69).
  • One case is backend-restricted: minecraft-1.21.4-fabric-iris-iterationrp-in-world has "ci_backends": ["DirectVulkan"] (:286-295, the key at :288).
  • timeout_seconds distribution over the 39 CI cases: 17×900, 17×1800, 5×180.

Counts — reconcile these three numbers before writing any job

  • trace_cases.py --ci --format github-test-matrix → 77 entries (39 CI cases × 2 backends − 1 restricted). This is what the retrace matrix runs today (trace_cases.py:71-98).
  • The CMake generator is called without --ci (tools/trace_replay/CMakeLists.txt:389-394) and add_trace_replay_test_for_backends always registers both backends (:382-385), so a local ctest -R '^MobileGLTraceReplay\.' in build-retrace registers 80 tests.
  • The 79 in the task statement = 80 minus the iterationrp × DirectGLES pair that ci_backends marks unsupported. That is the desktop gate the WSL harness (retrace_gate.py) drives. Implementers should confirm which of 77 / 79 / 80 the P1 acceptance means; the docs consistently say "40 trace" (ROADMAP.md:17, ARCHITECTURE.md:502, :507).

Registration

add_trace_replay_test(CASE_NAME BACKEND …) (tools/trace_replay/CMakeLists.txt:311-380):

add_test(NAME MobileGLTraceReplay.${CASE_NAME}.${BACKEND}
         COMMAND cmake -DTRACE_REPLAY_EXE=$<TARGET_FILE:mobilegl_trace_replay>
                       -DMOBILEGL_LIBRARY=${mobilegl_trace_replay_mobilegl_library}
                       -DTRACE_CASE_NAME/-DTRACE_ARCHIVE/-DTRACE_FILE/-DTRACE_GOLDEN/
                       -DTRACE_ALTERNATE_GOLDEN/-DTRACE_BACKEND/-DTRACE_TARGET_CALL/
                       -DTRACE_WIDTH/-DTRACE_HEIGHT/-DTRACE_SSIM_THRESHOLD/-DTRACE_CROP_*/
                       -DTRACE_COHERENT_AS_FLUSH/-DTRACE_OUTPUT_DIR/-DTRACE_ARTIFACT_DIR
                       -P run_trace_case.cmake)                                      (:350-372)

Per-test ENVIRONMENT (:373-379):

  • DirectGLES: EGL_PLATFORM=surfaceless;LIBGL_ALWAYS_SOFTWARE=1;MESA_GL_VERSION_OVERRIDE=3.3;MESA_GLSL_VERSION_OVERRIDE=330
  • DirectVulkan: the same minus EGL_PLATFORM.

No MOBILEGL_* variable is named in either property, and no TIMEOUT property is set. Two consequences:

  1. MOBILEGL_PIPE_VERIFY / MOBILEGL_PIPE_PUSH exported by a job reach the replay process unimpeded — the same mechanism MOBILEGL_MAGMA_R11G11B10F_FALLBACK already uses (test.yml:705-707).
  2. Every trace test runs under ctest's default 1500 s timeout. The timeout_seconds field in trace_cases.json is consumed only by the Android lane (apk.yml:449 --timeout-seconds) — emit_cmake's key list (trace_cases.py:122-136) does not include it. So the 1800 s cases already sit above the desktop ctest default; anything that slows retrace down by 5–10× must raise it explicitly.

mobilegl_trace_replay_mobilegl_library is MOBILEGL_TRACE_REPLAY_MOBILEGL_LIBRARY when set, else $<TARGET_FILE:MobileGL> (:303-308).

run_trace_case.cmake — how a case actually runs

  • Required -D variables checked up front (:1-5); defaults filled for SSIM/crop (:7-21).
  • Wipes and recreates ${TRACE_OUTPUT_DIR}/{input,output} (:31-35), extracts the .tgz, asserts the trace file exists (:37-52).
  • execute_process() the replay exe with --trace/--golden/--diff/--output/--backend/--mobilegl-library/ --target-call/--width/--height/--ssim-threshold/--crop-* (+--coherent-as-flush) (:54-74). execute_process inherits the environment — this is the hop MOBILEGL_PIPE_VERIFY travels through.
  • Echoes MOBILEGL_TRACE_GL_* identity lines from output/retrace.log (:79-86).
  • Copies actual/golden/result.json/retrace.log/mobilegl.log into TRACE_ARTIFACT_DIR (:88-110).
  • Fails if output/result.json is missing (:112-125) or if the replay exit status is non-zero (:127-129). Both are message(FATAL_ERROR), i.e. a red ctest test.

output/mobilegl.log exists because trace_replay_core.cpp:1011 does setenv("MOBILEGL_LOG_FILE_PATH", mobileGlLogPath.c_str(), 1) — so a retrace run already has a library log the harness can grep, which is exactly what a verify-arming assertion needs.

Desktop CLI has no --env

trace_replay_cli.cpp:116-179 lists every option; there is no --env. The K=V;K=V passthrough (ApplyEnvOverrides, android-plugin/app/src/trace/cpp/trace_replay_core.cpp:143-155, called at :237) is reached only from the Android side (android-plugin/trace-replay-ci.sh:174 and :415 --es mobilegl_env, tools/trace_replay/run_android_retrace_local.py:192-196, :302). Its parser is pinned by a build-time self-running check: mobilegl_trace_env_overrides_test is built and executed as a POST_BUILD command, and mobilegl_trace_replay depends on it (tools/trace_replay/CMakeLists.txt:291-301), so cmake --build … --target mobilegl_trace_replay runs it even though the job never invokes ctest. trace_env_overrides_test.cpp:73-92 already uses MOBILEGL_PIPE_PUSH / MOBILEGL_PIPE_VERIFY as its example strings.

The Android retrace lane (apk.yml)

apk.yml:417-470: per-(backend, case) matrix, timeout-minutes: 75 (:331), env-per-knob at :420-425, and android-plugin/trace-replay-ci.sh invoked with the case's fields including --timeout-seconds. The generic --env "K=V;K=V" hook (trace-replay-ci.sh:35, :64, :174, :415) is where a device-side MOBILEGL_PIPE_VERIFY=1 would go. Trap already documented at MEASUREMENTS.md §5.3: an --env value containing /data/... needs MSYS_NO_PATHCONV=1 on a Windows host.

A5. The Fatal / assert idiom

  • MGLOG_F is MOBILEGL_LOG_INTERNAL("FATAL", ANDROID_LOG_FATAL, …) and is live at every build level (MobileGL/MG_Util/Debug/Log.h:87-92). It logs; it does not terminate.
  • MOBILEGL_ASSERT(cond, …) = MGLOG_F ×2 + TRAP (MobileGL/Defines.h:104-112), and is compiled to nothing unless MOBILEGL_LOG_ACTIVE_LEVEL <= MOBILEGL_LOG_LEVEL_DEBUG (:113-115). CI builds at MOBILEGL_LOG_LEVEL_INFO (test.yml:85), so every MOBILEGL_ASSERT in the tree is inert in CI. TRAP is __builtin_debugtrap() under clang, raise(SIGTRAP) elsewhere, assert(false) on MSVC (Defines.h:88-95).
  • The poison Fatal already exists in generated form:
    [[noreturn]] inline void MGPipeInputPoisonFatal(MGPipeInputField field, const char* verb) {
        MGLOG_F("MGPipe: Fatal{UnmigratedPipeInput, \"%s@%s\"}",
                kMGPipeInputFieldNames[static_cast<SizeT>(field)], verb);
        std::abort();
    }
    
    MobileGL/MG_Pipe/generated/PipeFilled.inc:298-302 (emitted by scripts/gen_pipe.py:497-503). It is MGLOG_F + std::abort(), not MOBILEGL_ASSERT — deliberately, since the assert would vanish in an INFO build. Companion state and predicate: MGPipeFilledState{CurrentVerbSerial, FilledGen[61]} (PipeFilled.inc:293-296) and MGPipeInputFieldIsFresh (:304-307), with the sticky table at :163 and the "which call fills it" table at :229.
  • The wire-protocol Fatal has the same shape: MGPipeWireProtocolFatal ends in std::abort() (scripts/gen_pipe.py:340, emitted into PipeWire.inc).
  • Termination therefore = SIGABRT. A test that wants to observe it must run it in a child process.

Death tests: there are none

grep -rn "EXPECT_DEATH|ASSERT_DEATH|ExitedWithCode|KilledBySignal|EXPECT_EXIT|DeathTest" over MobileGL/ and tools/ returns zero hits. The project's established way to observe a child's exit is raw fork()/waitpid():

  • MobileGL/MG_Test/Wire/FdPassingTest.cpp:62 (fork), :85 (_exit(status)), :130-132 (waitpid + WIFEXITED + WEXITSTATUS == kChildOk).
  • MobileGL/MG_IntegrationTest/Harness/HeadlessGL.cpp:323-423 — the pre-flight: fork, child _exit(step) (:365-368), parent waitpid with WNOHANG polling (:380-388), WTERMSIG (:410), WIFEXITED / WEXITSTATUS (:420-423). Its comment at :323-325 is the exact rationale P1 needs: "SIGTRAP is a datum instead of a crash. fork() gives exactly that."

A6. Existing negative-control patterns (the shapes to imitate)

  1. check_include_closure.py --self-test (scripts/check_include_closure.py:435-548): three canned TUs that must trip (negative_controls(), :469-497), a parser check over canned -H output (:447-467), a "synthesize a third glslang:: token and confirm the budget trips" control (:499-515), and — the piece that makes it a gate — if trips == 0: error("negative control did not trip: the closure gate is not checking anything") (:541-543). It runs in two places: the include-graph-check job (test.yml:349) and as a ctest under unit (MobileGL/MG_Test/Purity/CMakeLists.txt:9-13), where the comment reads "NO ENVIRONMENT property: it replaces the job env (ARCHITECTURE.md:567)".
  2. gen_pipe.py + git diff --exit-code (test.yml:850-853) — regeneration as the anti-drift gate, and gen_pipe.py refuses a catalogue whose payload has no field list, "a payload the G4 comparator cannot see is a payload MOBILEGL_PIPE_VERIFY is blind to" (scripts/gen_pipe.py:29-30, enforced at :194).
  3. Comparator unit tests: MobileGL/MG_Test/Pipe/PipeCatalogueTest.cpp:184-207 — MGPipeVerify(a, b, &field) true for equal payloads, false after b.InstanceCount = 7 with EXPECT_STREQ(field, "InstanceCount"), plus a padding control (equal despite padding) and an array/nested control (right.Color[3].Level = 2 → field "Color").
  4. Poison unit test: PipeCatalogueTest.cpp:224-235 — PipeInputFieldsStartUnfilled, including the line that matters ("The next verb makes the same value stale, which a written-once bitmap could not see").
  5. Env-pinned negative-control ctest lanes: DirectGLES.NoViewportArrayEmulation. registers the one case that describes behaviour with the emulation off (MG_IntegrationTest/CMakeLists.txt:556-569); DirectGLES.AsyncOff. registers the one case that has anything to say with async off (:503-528).
  6. Log-file arming assertions — A3 above.
  7. LogLevelTest (MobileGL/MG_Test/Util/LogLevelTest.cpp): links GTest::gtest (not gtest_main) so its own main() can point MOBILEGL_LOG_FILE_PATH at a temp file before anything latches the sink (MG_Test/Util/CMakeLists.txt:19-22); EmitAndRead() (:87-107) emits five markers and greps the file; SinkIsReachableAtAll (:110-118) exists purely so the suppression assertions cannot pass vacuously. This is the closest existing thing to a Fatal-capture hook and the cheapest template for one.
  8. A build-time self-running check: mobilegl_trace_env_overrides_test POST_BUILD (tools/trace_replay/CMakeLists.txt:291-301) — a gate that runs even in a job that never calls ctest.

A7. scripts/symbol_report.py (P0.5)

  • Purpose (docstring :10-16): per-symbol nm --defined-only -S and .text attribution between two libMobileGL.so builds; --strip-scope folds de-nesting renames into a "renamed-only" bucket.
  • Options (:266-296): --before, --after, --nm, --size, --cxxfilt, --markdown, --json, --strip-scope (repeatable), --rename-map OLD=NEW, --only-names, --threshold, --fail-on-added-bytes, --self-test.
  • It always exits 0: --fail-on-added-bytes is reserved; this run stays informational (:342, docstring :33-35). Making it a gate means either implementing the flag or asserting on --json in the job.
  • Guard rails it documents (:23-31): identical CMAKE_BUILD_TYPE (visibility presets differ, CMakeLists.txt:578-598), MOBILEGL_ENABLE_LTO OFF on both sides (CMakeLists.txt:108,137-146), same compiler and stdlib. It prints both paths and byte sizes in the report header.
  • Parsers: nm --defined-only -S sized/unsized rows (:46-52), size --format=sysv rows for .text/.data/.bss/.rodata/Total (:54-58).
  • P0.5's measured result, for calibration: "测试名零删除、MG_Backend 零 diff、.text 字节不变、符号 0 增 / 0 删 / 42 重命名" (ROADMAP.md:16).
  • The job name is already reserved: monolith-symbol-report (ARCHITECTURE.md:568); it does not exist in test.yml yet.

A8. What P1 is contractually required to show

ROADMAP.md:17 (the P1 row), acceptance column verbatim:

pull 构建 nm --defined-only 不变、.text 差异逐行归因(空守卫/三元重写推迟到 P2);40 trace + 全部集成测试在 MOBILEGL_PIPE_VERIFY=1 下零分歧;故意损坏一个快照字段能让 verify 变红;故意在 glGenerateMipmap 的填充表漏 一个字段能在那条 verb 上触发 poison Fatal

Supporting design: ARCHITECTURE.md:502 (verify is "第三种 CI 模式,40 个 trace + 全部集成测试,~5–10× 慢, 永不出货"), :335 (#if MOBILEGL_PIPE_PUSH / MGB_CTX), the phase table at :337-345, the poison paragraph at :367 ("debug 与 disaggregated 构建里读一个当前 verb 未填的字段是 Fatal{UnmigratedPipeInput, "GetStencilState@DrawVbo"}"), the switch tables at :579 (MOBILEGL_PIPE_VERIFY is a build-time CMake option, planned, not yet existing) and :588-589 (the runtime env vars, already landed).

Runtime plumbing that already exists: MobileGL/Config.h:319-328 (Uint64 PipePush, Bool PipeVerify), MobileGL/ConfigLoader.cpp:245-246, and the note at :243-244 that no allow-list entry is needed because every MOBILEGL_-prefixed variable is accepted by construction. QueryEnvUint64 rejects octal and negatives (ConfigLoader.cpp:167-190).


Part B — proposals

B1. Running the whole integration suite a third time under verify

B1.0 The prerequisite nobody can skip: the build

MOBILEGL_PIPE_VERIFY is a build-time option (ARCHITECTURE.md:579) and the poison is gated on MOBILEGL_DEBUG || MOBILEGL_BUILD_DISAGGREGATED (ARCHITECTURE.md:335 region). Today CI builds Release/INFO with MOBILEGL_BUILD_DISAGGREGATED OFF (test.yml:71-93, CMakeLists.txt:23), so both the verify comparator and the poison would be compiled out of every existing job. A job-env MOBILEGL_PIPE_VERIFY=1 against today's binary is a no-op that looks green — the exact failure mode this project keeps writing comments about.

So P1 needs a fourth build job:

  build-linux-verify:
    runs-on: ubuntu-latest
    permissions: { actions: write, contents: read }
    env:
      BUILD_DIR: build-verify
      CCACHE_DIR: ${{ github.workspace }}/.ccache
      CCACHE_BASEDIR: ${{ github.workspace }}
      CCACHE_COMPRESS: "true"
      CCACHE_MAXSIZE: 4G
      CCACHE_NOHASHDIR: "true"
    # steps 1-8 copied verbatim from build-linux (swap, checkout --recursive, get-cmake,
    # ccache restore with key ${{ runner.os }}-test-${{ github.job }}-ccache-v1,
    # Vulkan SDK, update_glslang_sources.py, apt deps, toolchain banner)
    # Configure: build-linux's flags PLUS
    #   -DMOBILEGL_PIPE_VERIFY=ON            # new option: compiles in SnapshotFromGLContext + G4
    #   -DMOBILEGL_BUILD_DISAGGREGATED=ON    # arms the per-verb poison; needs 3rdparty/flatbuffers
    # Package as ci-artifacts/mobilegl-linux-runtime-verify.tgz, same file list as :124-147
    # Upload as artifact `mobilegl-linux-runtime-verify`

Notes:

  • MOBILEGL_BUILD_DISAGGREGATED=ON also compiles MG_Remote/** and registers the MG_Test/Wire suite (CMakeLists.txt:454-469, MG_Test/CMakeLists.txt:93-95); it is force-reset to OFF if 3rdparty/flatbuffers/include is missing (CMakeLists.txt:440-451), which with submodules: recursive it is not. If you would rather not couple the two, give MOBILEGL_PIPE_VERIFY=ON its own -DMOBILEGL_PIPE_POISON=1 and make the generated MGPipeInputPoisonFatal call sites depend on MOBILEGL_DEBUG || MOBILEGL_BUILD_DISAGGREGATED || MOBILEGL_PIPE_VERIFY.
  • Keep -DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO. Do not build the verify lane at ..._DEBUG: that would turn on every MOBILEGL_ASSERT in the tree (Defines.h:104) and change what the lane is measuring. Verify's own reporting must therefore use MGLOG_E/MGLOG_F, never MGLOG_D.
  • Its own ccache key. Do not share build-linux's: different -D flags, different objects.

B1.1 Registering the verify entries — do NOT reuse the existing ones with job env

Job env would reach the process (proved by test.yml:274-288), but two things break:

  • the per-test TIMEOUT 120 (MG_IntegrationTest/CMakeLists.txt:393) is a property, so ctest --timeout cannot raise it, and 5–10× slower scenarios will start timing out;
  • there is no ctest-visible name that says "this was the verify run", so ctest -N parity checks and the retrace-summary style reporting cannot tell the two apart.

Instead add a fourteenth and fifteenth registration in MobileGL/MG_IntegrationTest/CMakeLists.txt, following the file's own idiom exactly:

# The third CI mode (ARCHITECTURE.md:502): the same scenarios, with the per-draw per-field
# shadow comparison armed. Registered rather than driven by job env for two reasons: a ctest
# ENVIRONMENT property is the only spelling that survives `mgl_itest_join_environment`'s
# escaping intact (CMakeLists.txt:301-306), and these entries need a TIMEOUT of their own -
# verify is 5-10x slower and the ambient 120 s would start killing scenarios.
option(MOBILEGL_PIPE_VERIFY "Compile in SnapshotFromGLContext() and the G4 comparators" OFF)
set(MGL_ITEST_VERIFY_TIMEOUT 900)

if (MOBILEGL_PIPE_VERIFY)
    mgl_itest_join_environment(MGL_ITEST_GLES_VERIFY_ENVIRONMENT
        "MOBILEGL_BACKEND_TYPE=DirectGLES" "MOBILEGL_PIPE_VERIFY=1"
        "MOBILEGL_LOG_FILE_PATH=${CMAKE_CURRENT_BINARY_DIR}/pipe-verify-gles.log"
        ${MGL_ITEST_COMMON_ENV})
    mgl_itest_join_environment(MGL_ITEST_VULKAN_VERIFY_ENVIRONMENT
        "MOBILEGL_BACKEND_TYPE=DirectVulkan" "MOBILEGL_PIPE_VERIFY=1"
        "MOBILEGL_LOG_FILE_PATH=${CMAKE_CURRENT_BINARY_DIR}/pipe-verify-vulkan.log"
        ${MGL_ITEST_VULKAN_ENV})

    gtest_discover_tests(MobileGLIntegrationTest
        TEST_PREFIX "DirectGLES.Verify."
        DISCOVERY_TIMEOUT 30
        PROPERTIES LABELS "integration-gpu;integration-verify"
                   TIMEOUT ${MGL_ITEST_VERIFY_TIMEOUT}
                   ENVIRONMENT "${MGL_ITEST_GLES_VERIFY_ENVIRONMENT}")
    gtest_discover_tests(MobileGLIntegrationTest
        TEST_PREFIX "DirectVulkan.Verify."
        DISCOVERY_TIMEOUT 30
        PROPERTIES LABELS "integration-gpu;integration-verify"
                   TIMEOUT ${MGL_ITEST_VERIFY_TIMEOUT}
                   ENVIRONMENT "${MGL_ITEST_VULKAN_VERIFY_ENVIRONMENT}")
endif()

Points of care, each one a rake this repo has already stepped on:

  • The ${MGL_ITEST_COMMON_ENV} / ${MGL_ITEST_VULKAN_ENV} tail is mandatory (CMakeLists.txt:339-344). Omitting it loses the EGL vendor pin, the lavapipe ICD pin, MOBILEGL_ITEST_REQUIRE_GPU, and the three iterationRP repairs — and the lane would still look green.
  • Two labels: keep integration-gpu so nothing that enumerates the lane loses these, and add integration-verify so a job can select only them. A ;-separated LABELS value inside a gtest_discover_tests PROPERTIES list needs the same escaping treatment as ENVIRONMENT if it ever grows a variable — safest is to write it as LABELS integration-gpu integration-verify (multi-value properties accept repeated tokens in gtest_discover_tests); verify with ctest -N -L integration-verify in the first build.
  • Name parity: DirectGLES.Verify.<Suite>.<Case> keeps the suffix identical to DirectGLES.<Suite>.<Case>, so ctest -N -L integration-gpu | sed 's/\.Verify\././' parity is mechanical — the same test ARCHITECTURE.md:507-3 demands between DirectGLES. and DirectGLES.Pipe./DirectGLES.Split. later.
  • Arming assertion (otherwise the lane is unfalsifiable, the recurring failure of this suite): add one scenario, e.g. Scenarios/PipeVerifyArmingScenario.cpp, that (a) skips unless AmbientQuirkFromEnvironment("MOBILEGL_PIPE_VERIFY") == AmbientQuirk::On (ScenarioFixture.h:56-65), (b) skips with an explicit message when MOBILEGL_LOG_FILE_PATH is unset (copy UnlocatedIoBlockScenario.cpp:354-360 and its message at :489), (c) draws once and asserts the library's latched MGLOG_I("MGPipe: verify armed, N fields, K payloads") appears in the bytes written after the case started, and that no Fatal{PipeVerifyDiffer line does. This is what turns "the option was compiled out" and "the build ran with MOBILEGL_PIPE_VERIFY spelled wrong" from silent green into red.

B1.2 The CI job

  integration-verify:
    runs-on: ubuntu-latest
    needs: build-linux-verify
    steps:
      # identical to `integration` (test.yml:205-296) except:
      #   - download/unpack artifact `mobilegl-linux-runtime-verify` into build-verify/
      #   - repeat the "Normalize CTest command paths" step over build-verify (test.yml:234-244)
      #   - working-directory: build-verify
      env:
        MOBILEGL_ITEST_REQUIRE_GPU: "1"
        MOBILEGL_MAGMA_FIX_ITERATIONRP_SUBGROUP_SCRATCH: "1"
        MOBILEGL_MAGMA_DERIVE_NUM_SUBGROUPS: "1"
        MOBILEGL_MAGMA_ITERATIONRP_FIX_BARRIER: "1"
      run: |
        ulimit -c unlimited
        sudo sysctl -w kernel.core_pattern='/tmp/core.%e.%p'
        ctest --output-on-failure -L integration-verify --no-tests=error

--no-tests=error is doing real work here: if the option was OFF at configure the label matches nothing and the job fails instead of passing empty. Keep the integration-core-dumps upload — a verify divergence that aborts leaves a core, and that core names the diverging draw.

B1.3 Cost

  • Entries: 371 × 2 = 742 new ctest entries (the verify lane covers the two ambient registrations; the 11 filtered mode lanes stay pull-only — they pin other quirks and doubling them would multiply the matrix for no new information).
  • Each entry is one process: full EGL/Vulkan bring-up plus one fork() pre-flight (HeadlessGL.cpp:323-343). Process startup dominates today; verify multiplies only the in-scenario GL work by the documented 5–10× (ARCHITECTURE.md:502). Expect the job to land somewhere between 2× and 5× the existing integration job's wall time — measure it on the first run and set timeout-minutes from that (the whole file has none today, so a wedged verify job would burn 6 h).
  • If it proves too slow for every push: split by backend into two jobs (-L integration-verify -R '^DirectGLES\.' and ^DirectVulkan\.), which halves wall time at no extra runner-minute cost.

B1.4 Locally

cmake -S . -B build-verify -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO \
  -DMOBILEGL_BUILD_TEST=ON -DMOBILEGL_BUILD_INTEGRATION_TEST=ON \
  -DMOBILEGL_PIPE_VERIFY=ON -DMOBILEGL_BUILD_DISAGGREGATED=ON \
  -DMOBILEGL_ITEST_REQUIRE_GPU=ON \
  -DMOBILEGL_ITEST_EGL_VENDOR=/usr/share/glvnd/egl_vendor.d/<vendor>.json \
  -DMOBILEGL_ITEST_VK_ICD=/usr/share/vulkan/icd.d/<vendor>.json
cmake --build build-verify --parallel "$(nproc)"
ctest --test-dir build-verify -L integration-verify --output-on-failure

-DMOBILEGL_ITEST_REQUIRE_GPU=ON here (not env) because locally the option bakes it into the property (MG_IntegrationTest/CMakeLists.txt:279-281) and you want the same thing CI gets. Do not use run_integration_test.sh for this: it carries neither the mode variables nor the iterationRP pins.

B2. Running the retrace gate a third time under verify

B2.1 The good news: no CMake change is needed for the env

No trace test's ENVIRONMENT property names any MOBILEGL_* variable (tools/trace_replay/CMakeLists.txt:373-379), and the existing job already exports three of them (test.yml:705-722). So export MOBILEGL_PIPE_VERIFY=1 in the job step reaches the replay process, through run_trace_case.cmake's execute_process (:54-74), which inherits.

B2.2 The bad news: the timeout, the library, and the silence

  1. Timeout. Trace tests have no TIMEOUT property, so they run under ctest's 1500 s default. Because the property is absent, ctest --timeout does apply — this is the one place it is unambiguous: ctest -V --no-tests=error --timeout 10800 -R '^MobileGLTraceReplay\.…$'. Also raise the job's own timeout-minutes (there is none today).
  2. Which library. The trace CTestTestfile.cmake hard-codes -DMOBILEGL_LIBRARY=<workspace>/build-linux/ libMobileGL.so (baked at configure from test.yml:424 / :501). The cheapest correct trick: in the verify retrace job, unpack the verify runtime tarball so that the verify libMobileGL.so lands at build-linux/libMobileGL.so, and unpack mobilegl-trace-replay unchanged. The replay binary itself does not need rebuilding — it dlopens the library named on the command line. Assert the swap actually happened (test -f build-linux/libMobileGL.so && nm -D build-linux/libMobileGL.so | grep -q MGPipeVerify or the arming line, see 3).
  3. Silence. Nothing today makes a verify divergence red. Decide and implement one contract:
    • Recommended: divergence is fatal by default — MGLOG_F("MGPipe: Fatal{PipeVerifyDiffer, \"%s@%s\", draw=%llu}", field, verb, serial) then std::abort(), exactly like MGPipeInputPoisonFatal (PipeFilled.inc:298-302). The replay process dies with SIGABRT → replay_result != 0 → run_trace_case.cmake:127-129 FATAL_ERROR → red ctest test, and /tmp/core.* is uploaded by the existing failure step. One env escape hatch, MOBILEGL_PIPE_VERIFY_FATAL=0 (log-and-count, for triage and for the integration arming lane which must survive to read its own log), added next to features.PipeVerify in ConfigLoader.cpp:246.
    • Plus an arming check in the harness, so a build with the option off cannot pass this lane silently. Add to run_trace_case.cmake, guarded on a new -DTRACE_PIPE_VERIFY=ON passed only by the verify job:
      if(TRACE_PIPE_VERIFY)
          if(NOT EXISTS "${mobilegl_log}")
              message(FATAL_ERROR "verify run produced no mobilegl.log")
          endif()
          file(STRINGS "${mobilegl_log}" armed REGEX "MGPipe: verify armed")
          if(NOT armed)
              message(FATAL_ERROR "MOBILEGL_PIPE_VERIFY=1 but the library never reported arming")
          endif()
          file(STRINGS "${mobilegl_log}" differ REGEX "Fatal\\{PipeVerifyDiffer")
          if(differ)
              message(FATAL_ERROR "verify divergence: ${differ}")
          endif()
      endif()
      
      mobilegl_log is already defined at run_trace_case.cmake:80 and already copied to the artifact dir at :107-109. -DTRACE_PIPE_VERIFY needs one line in add_trace_replay_test (tools/trace_replay/CMakeLists.txt:350-372) forwarding a cache variable, or simply -DTRACE_PIPE_VERIFY=$ENV{MOBILEGL_PIPE_VERIFY}-style detection inside the script (if(DEFINED ENV{MOBILEGL_PIPE_VERIFY} AND NOT "$ENV{MOBILEGL_PIPE_VERIFY}" STREQUAL "0")) — the latter needs no registration change at all and is the smaller diff.

B2.3 Job shape

  retrace-verify:
    name: retrace verify (${{ matrix.backend }}, ${{ matrix.case }})
    runs-on: ubuntu-latest
    needs: [build-linux-verify, build-retrace, trace-cases, trace-fixtures]
    if: ${{ always() && needs.build-linux-verify.result == 'success' && needs.build-retrace.result == 'success' }}
    timeout-minutes: 240
    strategy:
      fail-fast: false
      max-parallel: 4
      matrix: ${{ fromJSON(needs.trace-cases.outputs.matrix) }}   # or a reduced matrix, see below
    steps:
      # identical to `retrace` (test.yml:651-741) except:
      #   - download `mobilegl-linux-runtime-verify` and unpack it so the verify .so lands at
      #     build-linux/libMobileGL.so
      #   - export MOBILEGL_PIPE_VERIFY=1 alongside the existing per-case exports
      #   - ctest ... --timeout 10800
      #   - artifact names suffixed -verify so retrace-summary and remove-artifact-clutter
      #     (test.yml:784-833, which greps job names beginning "retrace (") are unaffected

It reuses the same trace-fixture-<case> artifacts the retrace matrix downloads, so the ~800 MB of fixtures (94 files in tools/trace_replay/fixtures/, individual .tgz 5–25 MB) are fetched once per run, not twice — provided retrace-verify runs in the same workflow run and before remove-artifact-clutter. Add retrace-verify to that job's needs (currently needs: retrace-summary, test.yml:787) or it will delete the fixtures out from under the verify matrix.

B2.4 Cost and cadence

  • Full third pass = +77 matrix jobs (or 79/80 locally, see A4), at max-parallel: 4 ≈ 20 sequential waves, each wave running a case that already takes minutes, now 5–10× slower. This is not a per-push cost anyone should pay.
  • Recommended cadence, which still satisfies ROADMAP.md:17 ("40 trace … 零分歧" is an exit criterion):
    • per push on feat/disaggregated: a reduced verify matrix of ~8 cases × 2 backends, chosen for path coverage rather than pixels — OpenRA, minecraft-1.21.4-startup, minecraft-1.21.4-main-menu, minecraft-1.21.4-in-world, minecraft-1.21.4-fabric-sodium-in-world, one Iris case, minecraft-1.21.1-neoforge-create-indirect-in-world, improved-transparency-minecraft-26.3. Express it as a new trace_cases.py --format github-verify-matrix (or a "verify": true key in trace_cases.json) so the selection is data, not YAML, and pipe-gates' regeneration discipline covers it.
    • nightly / workflow_dispatch: the full matrix.
    • P1 exit and every stage boundary: the full matrix, recorded in MEASUREMENTS.md.
  • Known non-P1 red to exclude or expect: minecraft-1.21.1-neoforge-create-indirect-in-world fails on both devices on the dev@81b17c0b baseline (MEASUREMENTS.md §5.5) — it is a device-lane fact, not a Linux-lane one, but keep it in mind before attributing a verify red to P1.

B2.5 Locally

cmake -S . -B build-retrace -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DMOBILEGL_BUILD_TEST=OFF -DMOBILEGL_BUILD_BENCHMARK=OFF -DMOBILEGL_BUILD_TRACE_REPLAY=ON \
  -DMOBILEGL_TRACE_REPLAY_MOBILEGL_LIBRARY=$PWD/build-verify/libMobileGL.so
cmake --build build-retrace --target mobilegl_trace_replay --parallel "$(nproc)"
MOBILEGL_PIPE_VERIFY=1 ctest --test-dir build-retrace/tools/trace_replay \
  -R '^MobileGLTraceReplay\.' --timeout 10800 --output-on-failure -j2

(-j2, not -j$(nproc): each case unpacks a multi-GB trace into TRACE_OUTPUT_DIR, which run_trace_case.cmake:31-33 wipes per run.) While the trace tests carry no label, consider adding LABELS retrace in tools/trace_replay/CMakeLists.txt:374-379 as a one-line P1 quality-of-life change — every other lane in the tree is label-selectable and this one is not.

B3. Negative control 1 — "corrupting one snapshot field turns verify red"

Build it in three layers, cheapest first. Layers 1 and 2 run on every CI; layer 3 runs in the verify lanes.

The injection knob (shared by all three layers)

Add, next to features.PipeVerify (ConfigLoader.cpp:246) and compiled only under #if MOBILEGL_PIPE_VERIFY so it can never exist in a shipped build:

MOBILEGL_PIPE_VERIFY_CORRUPT=<FieldName>[.<Member>]   # perturb exactly this field in the SNAPSHOT arm
MOBILEGL_PIPE_VERIFY_FATAL=0|1                         # default 1: first divergence aborts

<FieldName> is one of the 61 names already generated into kMGPipeInputFieldNames (PipeFilled.inc, list emitted by gen_pipe.py:473-477), so the knob's vocabulary is generated, not hand-typed, and a renamed accessor breaks the control loudly. Perturb the snapshot side, never the pushed side: corrupting the pushed side would also change what the backend renders and the control would be testing two things at once.

Layer 1 — unit, no GPU, MG_Test/Pipe (runs in the test job today)

Extend MobileGL/MG_Test/Pipe/PipeCatalogueTest.cpp (the comparator precedent is at :184-207) with a test that drives the harness entry point rather than a payload comparator:

TEST(PipeVerify, CorruptedSnapshotFieldIsNamedWithItsDrawSerial) {
    PipeInputs pushed{};  PipeInputs snapshot{};
    FillBothWithAKnownState(pushed, snapshot);
    const char* field = nullptr; Uint64 serial = 0;
    EXPECT_TRUE(MGPipeVerifyInputs(pushed, snapshot, &field, &serial));   // clean arm first
    ApplyVerifyCorruption(snapshot, "GetRenderStateParameters");          // the injector, not env
    EXPECT_FALSE(MGPipeVerifyInputs(pushed, snapshot, &field, &serial));
    EXPECT_STREQ(field, "GetRenderStateParameters");
}

The clean arm is not decoration: without it the test passes when the comparator is stuck returning false. Design requirement this imposes on P1: expose the comparison as a pure function over two PipeInputs (Bool MGPipeVerifyInputs(const PipeInputs&, const PipeInputs&, const char** outField, Uint64* outSerial)) and the corruption as a callable (ApplyVerifyCorruption(PipeInputs&, const char* fieldName)) that the env knob merely calls. If the only way to reach verify is a live GL context, this whole layer is impossible. Registration: PipeCatalogueTest is already LABELS unit (MG_Test/Pipe/CMakeLists.txt:28) — but the new test only compiles when MOBILEGL_PIPE_VERIFY is ON, so either guard the test body with #if MOBILEGL_PIPE_VERIFY … #else GTEST_SKIP() … (visible skip, honest) or add a PipeVerifyTest target registered inside if (MOBILEGL_PIPE_VERIFY). Prefer the guard: a test that disappears is a test nobody notices is gone.

Layer 2 — end-to-end "the gate goes red", cheapest possible

One extra step in integration-verify, run after the green pass:

      - name: Negative control - a corrupted snapshot field must turn the lane red
        working-directory: build-verify
        env:
          MOBILEGL_ITEST_REQUIRE_GPU: "1"
          MOBILEGL_PIPE_VERIFY_CORRUPT: GetRenderStateParameters
        run: |
          if ctest -L integration-verify -R 'ClearThenReadPixelsScenario' --no-tests=error; then
            echo "::error::verify did not go red with a corrupted snapshot field"; exit 1
          fi
          echo "negative control tripped as expected"

MOBILEGL_PIPE_VERIFY_CORRUPT is named by no ENVIRONMENT property, so the job env reaches the process — the MOBILEGL_ESPRYT_DISABLE_INVALIDATE_FLUSH precedent (test.yml:274-288) verbatim. Pick a short, always-drawing scenario (ClearThenReadPixelsScenario) so this costs seconds. The identical step belongs in the retrace verify job on one cheap case (OpenRA, 180 s timeout, trace_cases.json:14-27), inverted the same way.

Layer 3 — the in-suite version, so it survives outside CI too

A PipeVerifyArmingScenario case that runs only when MOBILEGL_PIPE_VERIFY_CORRUPT is set, with MOBILEGL_PIPE_VERIFY_FATAL=0, draws once, and asserts the per-lane log contains Fatal{PipeVerifyDiffer, "GetRenderStateParameters@…" — the MOBILEGL_LOG_FILE_PATH idiom from UnlocatedIoBlockScenario.cpp:354-360. Register it as its own tiny lane (DirectGLES.VerifyCorrupted. / DirectVulkan.VerifyCorrupted., TEST_FILTER "PipeVerifyArmingScenario.CorruptedFieldIsReported") with MOBILEGL_PIPE_VERIFY_CORRUPT and MOBILEGL_PIPE_VERIFY_FATAL=0 in its ENVIRONMENT — again appending ${MGL_ITEST_COMMON_ENV}. Two entries, seconds of runtime, and it makes the control reproducible with one ctest -R on a dev box.

B4. Negative control 2 — "omitting a field in glGenerateMipmap's fill table triggers the poison Fatal on that verb"

The injection knob

MOBILEGL_PIPE_POISON_OMIT=<CallName>:<FieldName> (e.g. GenerateMipmap:GetPixelStoreParameters), read next to features.PipeVerify and compiled only where the poison is compiled. The generated per-verb filler (G5, PipeFilled.inc + the ~93 boundary fill sites, ROADMAP.md:17) consults it and skips stamping that one field's FilledGen for that one call — it must skip the stamp, not the value, so the omission is indistinguishable from a filler that genuinely forgot. <CallName> is validated against the catalogue (PipeCalls.def, opcode enum in PipeWire.inc) and <FieldName> against kMGPipeInputFieldNames; an unknown name is itself a Fatal, so a typo cannot silently disarm the control.

Layer 1 — unit, no GPU, no death test (runs in the test job today)

Extend PipeCatalogueTest.cpp:224-235 (PipeInputFieldsStartUnfilled already establishes the pattern):

TEST(PipeCatalogue, OmittingOneFieldForOneVerbLeavesExactlyThatFieldStale) {
    MGPipeFilledState state{};
    state.CurrentVerbSerial = 7;
    FillForVerb(state, MGPWireOp::GenerateMipmap);            // the generated filler
    EXPECT_TRUE(MGPipeInputFieldIsFresh(state, kOmittedField));   // clean arm
    EXPECT_TRUE(MGPipeInputFieldIsFresh(state, kSiblingField));

    MGPipeFilledState omitted{};
    omitted.CurrentVerbSerial = 7;
    SetPoisonOmission("GenerateMipmap", "GetPixelStoreParameters");
    FillForVerb(omitted, MGPWireOp::GenerateMipmap);
    EXPECT_FALSE(MGPipeInputFieldIsFresh(omitted, kOmittedField));  // the control
    EXPECT_TRUE (MGPipeInputFieldIsFresh(omitted, kSiblingField));  // and only that one
}

This is the layer that proves "on that verb": the sibling assertion is what distinguishes a targeted omission from a filler that stopped working entirely.

Layer 2 — the Fatal actually fires and kills the process

The tree has zero gtest death tests (A6), and its own idiom for observing a child's death is fork()/waitpid() (MG_Test/Wire/FdPassingTest.cpp:62-132, HeadlessGL.cpp:323-423). Use that, not EXPECT_DEATH:

TEST(PipeCatalogue, ReadingAnOmittedFieldAbortsNamingTheVerb) {
#if !defined(MOBILEGL_PIPE_POISON)
    GTEST_SKIP() << "poison compiled out (needs MOBILEGL_DEBUG or MOBILEGL_BUILD_DISAGGREGATED)";
#else
    const pid_t pid = ::fork();
    ASSERT_NE(pid, -1);
    if (pid == 0) {
        MGPipeFilledState state{};  state.CurrentVerbSerial = 1;   // nothing filled
        if (!MGPipeInputFieldIsFresh(state, MGPipeInputField::GetPixelStoreParameters)) {
            MGPipeInputPoisonFatal(MGPipeInputField::GetPixelStoreParameters, "GenerateMipmap");
        }
        ::_exit(0);                       // reaching here is the failure
    }
    int status = 0;
    ASSERT_EQ(::waitpid(pid, &status, 0), pid);
    EXPECT_TRUE(WIFSIGNALED(status)) << "the poison did not terminate the process";
    EXPECT_EQ(WTERMSIG(status), SIGABRT);
#endif
}

Why not EXPECT_DEATH: it would work on Linux, but (a) nothing in this tree uses it, so nobody maintains the death-test-style/threading constraints, (b) the "fast" style forks a process that has already initialized gtest and any static GL state, and (c) _exit(0) on the non-fatal path gives a positive failure signal that EXPECT_DEATH's "died as expected" phrasing hides. Pair it with a clean-arm sibling test (a filled field must let the child reach _exit(0)), or the test passes whenever MGPipeInputFieldIsFresh is broken in the false direction. To also assert the message (the @GenerateMipmap half): point MOBILEGL_LOG_FILE_PATH at a temp file in the suite's own main() — the LogLevelTest shape (MG_Test/Util/CMakeLists.txt:19-22, LogLevelTest.cpp:76-107) — and after reaping, grep the file for Fatal{UnmigratedPipeInput, "GetPixelStoreParameters@GenerateMipmap"}, the exact string PipeFilled.inc:299-300 formats. Note MOBILEGL_LOG_ENABLE_CONSOLE is 0 (Defines.h:71), so the file is the only channel; and the child must let the sink flush before aborting (the MGLOG_F write precedes std::abort(), but confirm the file sink is unbuffered or flushes per line in MG_Util/Debug/Log.cpp).

Layer 3 — on the real verb, through the real backend, per backend

An integration scenario PoisonOmissionScenario in the verify build: parent forks, child sets up a texture and calls glGenerateMipmap, parent asserts WIFSIGNALED && WTERMSIG == SIGABRT and that the per-lane log carries …@GenerateMipmap. Registered as its own lane with MOBILEGL_PIPE_POISON_OMIT=GenerateMipmap:<field> + MOBILEGL_LOG_FILE_PATH=<per-lane> in ENVIRONMENT (appending ${MGL_ITEST_COMMON_ENV} / ${MGL_ITEST_VULKAN_ENV}), two entries, one per backend. The forking is already precedented inside this module (HeadlessGL.cpp:343), and the GL context is per-process so the child inherits a usable one — but fork the child before any scenario-local GPU work and keep the child's work minimal, and use _exit, never exit (the reason is written at HeadlessGL.cpp:365-368). Its complement — the same scenario with the omission unset, expecting _exit(0) and an empty log — is what keeps this from passing on a build where glGenerateMipmap aborts for some unrelated reason.

Where these gates live

Both controls' layer-1 tests belong in MG_Test/Pipe (label unit, already in the test job). Both controls' layer-2 CI steps belong in the verify jobs. Register the whole thing the way check_include_closure.py --self-test is registered — always on, not behind a manual flag (test.yml:330-331, Purity/CMakeLists.txt:9-13). If the omission/corruption knobs ever become optional, add the trips == 0 → error check (check_include_closure.py:541-543) so a control that stopped tripping fails instead of silently reporting nothing.

B5. nm --defined-only / .text attribution for P1's "pull build unchanged"

What the acceptance actually says

ROADMAP.md:17: pull build nm --defined-only unchanged, .text differences attributable line by line, with the empty-guard / ternary rewrites explicitly deferred to P2. ARCHITECTURE.md:507 adds that byte identity is dead as a general gate but two byte-level equalities survive, and ARCHITECTURE.md:568 reserves the job name monolith-symbol-report.

Why this is even achievable: phase A is a mechanical sed of MG_State::pGLContext-> → MGB_CTX-> at 293 sites (ARCHITECTURE.md:337-341), and in the pull build MGB_CTX expands to ::MG_State::pGLContext (ARCHITECTURE.md:335), so the preprocessed token stream is identical. The only expected .text movement comes from the 58 hand-converted non-arrow uses — ~34 deleted MOBILEGL_ASSERTs (already no-ops in an INFO build, so zero .text effect there), 7 empty guards, 3 ternaries, .get() captures, 14 != nullptr, 1 comment.

Concrete procedure

# both sides: same compiler, same build type, LTO off (CMakeLists.txt:108 default OFF), not stripped
for ref in <P0.5-baseline-sha> <P1-head-sha>; do
  git -C <worktree> checkout "$ref"
  cmake -S . -B build-sym-$ref -G Ninja -DCMAKE_BUILD_TYPE=Release \
        -DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO \
        -DMOBILEGL_BUILD_TEST=OFF -DMOBILEGL_BUILD_BENCHMARK=OFF \
        -DMOBILEGL_BUILD_INTEGRATION_TEST=OFF -DMOBILEGL_BUILD_TRACE_REPLAY=OFF \
        -DMOBILEGL_BUILD_DISAGGREGATED=OFF -DMOBILEGL_PIPE_VERIFY=OFF -DMOBILEGL_ENABLE_LTO=OFF \
        -DCMAKE_C_COMPILER=clang-20 -DCMAKE_CXX_COMPILER=clang++-20
  cmake --build build-sym-$ref --target MobileGL --parallel "$(nproc)"
done
python3 scripts/symbol_report.py \
  --before build-sym-<baseline>/libMobileGL.so \
  --after  build-sym-<p1>/libMobileGL.so \
  --threshold 0 \
  --markdown docs/Disaggregated/P1-symbol-attribution.md \
  --json     /tmp/p1-symbols.json
  • -DMOBILEGL_BUILD_DISAGGREGATED=OFF and -DMOBILEGL_PIPE_VERIFY=OFF are the whole point: the gate is about the pull build. Comparing a verify build against the P0.5 baseline is meaningless — verify compiles in SnapshotFromGLContext() and the G4 comparators by construction (ARCHITECTURE.md:579).
  • No --strip-scope / --rename-map here: unlike P0.5 (which de-nested five types and therefore renamed 42 mangled names, ROADMAP.md:16), P1 moves no type out of a scope. If the report shows a renamed-only bucket, something unintended de-nested and that is itself a finding.
  • --only-names is useful for a focused pass over the accessor names in PipeInputs, not for the gate.

Making it a gate rather than a report

symbol_report.py always exits 0 (:342). Two options:

  • Minimal (recommended): keep the script informational and assert in the job on its --json:
    python3 - <<'PY'
    import json,sys
    r=json.load(open('/tmp/p1-symbols.json'))
    added=r['added']; removed=r['removed']       # confirm the exact key names against the writer
    if added or removed:
        print("::error::pull build symbol set changed: +%d / -%d" % (len(added),len(removed))); sys.exit(1)
    PY
    
  • Fuller: implement --fail-on-added-bytes (the flag is already declared, :287-288) and pass 0. Note this changes a tool P0.5 shipped as informational; if you do it, keep the default behaviour unchanged.

The job

  monolith-symbol-report:
    name: monolith symbol / .text attribution
    runs-on: ubuntu-latest
    steps:
      - checkout (submodules: recursive), get-cmake, apt clang-20 + binutils, Vulkan SDK,
        update_glslang_sources.py                                  # same prologue as build-linux
      - build the baseline .so (git worktree at the pinned baseline sha) and the head .so with
        the identical flags above; two ccache keys
      - python3 scripts/symbol_report.py --before … --after … --markdown … --json …
      - the json assertion above (P1 only; informational before and after)
      - upload the markdown as an artifact
      - the two surviving byte-level equalities (ARCHITECTURE.md:507):
          nm --defined-only build-sym-head/libMobileGL.so | grep -q MG_Remote && exit 1   # must be empty

Because this needs two full builds it is the most expensive gate in the file. Run it on workflow_dispatch + at stage exit, not per push. Pinning the baseline as a stored artifact (upload the P0.5 libMobileGL.so once, download it here) halves the cost and removes the "was the baseline built the same way" doubt — the guard rails at symbol_report.py:23-31 exist precisely because that doubt is the usual way this comparison goes wrong.

The deliverable P1 owes

The .text "line-by-line attribution" is a document, not a script output: take the per-symbol size table from the markdown and add a reason column drawn from the 58-site conversion list. Put it in docs/Disaggregated/ so scripts/check_doc_citations.py (test.yml:876-887) lints its citations, and cite ROADMAP.md:17 from it.


Part C — trap list

  1. A ctest ENVIRONMENT property wins over the job env for the variables it names, and only those. test.yml:252-259 (the ICD pin that must not be exported) vs test.yml:274-288 (the flush flag that is exported inline and does reach the process). MOBILEGL_PIPE_VERIFY is named by no property today, so job-env works — but the moment you add it to a property, a job-level export of the same name becomes a lie.
  2. ARCHITECTURE.md:567 and MG_IntegrationTest/CMakeLists.txt:339-344 say the property "REPLACES" the job env. Whatever the precise CMake semantics, the project's rule is: every mode list appends ${MGL_ITEST_COMMON_ENV} / ${MGL_ITEST_VULKAN_ENV}. Follow the rule; do not relitigate it in a P1 CMake diff.
  3. ; inside a gtest_discover_tests PROPERTIES value must be escaped — use mgl_itest_join_environment (MG_IntegrationTest/CMakeLists.txt:307-317). Symptom of getting it wrong: only the first variable survives and the pins vanish silently.
  4. ctest --timeout cannot raise a TIMEOUT property. Integration entries have TIMEOUT 120 (:393) → new registrations need their own. Trace entries have none → --timeout works there, and they currently run under ctest's 1500 s default even though trace_cases.json declares 1800 s for 17 cases.
  5. MOBILEGL_ASSERT is a no-op in every CI build (Defines.h:104-115, test.yml:85). Neither negative control may be expressed with it. The generated poison already gets this right (MGLOG_F + std::abort(), PipeFilled.inc:298-302); the ~93 hand-written fill sites must too.
  6. MGLOG_D is compiled out at INFO (Log.h:59-64). Verify's per-draw diagnostics must be MGLOG_E/MGLOG_F, or you will build the verify lane at DEBUG and accidentally also turn on every assert in the tree.
  7. MOBILEGL_LOG_ENABLE_CONSOLE is 0 (Defines.h:71) and the desktop default log path is empty (Defines.h:85). Any log-reading assertion needs MOBILEGL_LOG_FILE_PATH set in the lane's ENVIRONMENT — per-lane, so nothing else appends (MG_IntegrationTest/CMakeLists.txt:361-365).
  8. The poison is compiled out of today's CI builds (MOBILEGL_DEBUG || MOBILEGL_BUILD_DISAGGREGATED, ARCHITECTURE.md:335; CI is Release/INFO with DISAGGREGATED OFF). Without a build change, the glGenerateMipmap control is vacuous.
  9. Any job that unpacks the runtime tgz and runs ctest must repeat the "Normalize CTest command paths" step (test.yml:174-184 and three copies).
  10. remove-artifact-clutter deletes trace-fixture-* (test.yml:784-833) and currently only waits on retrace-summary. Add any new retrace matrix to its needs.
  11. feat/disaggregated is not a push trigger (test.yml:3-9). Either add it or drive P1's gates with workflow_dispatch.
  12. No job in test.yml sets timeout-minutes. A wedged verify job burns 6 h of runner time.
  13. ENABLE_INTEGRATION_TESTS is referenced at MG_Test/CMakeLists.txt:100 and defined nowhere in the tree (grep over *.txt/*.cmake/*.yml finds only that use), so MG_Test/Backend/DirectVulkan is never built by any lane. Not a P1 blocker, but if P1 wants a DirectVulkan-side unit test that variable is a dead end.
  14. gen_pipe.py regeneration + git diff --exit-code is a gate (test.yml:850-853). Any new generated table (poison-omission vocabulary, verify field ids) must be emitted by gen_pipe.py, or the gate will either miss it or go red on an untracked file.
  15. pipe-gates' stdio ban covers MobileGL/MG_Backend and MobileGL/MG_State with no exceptions (test.yml:862-869). The verify reporter and the fill sites must not print.
  16. gen_pipe_dirty_surface.py --summary becomes a gate in P1 (test.yml:871-874). Its only flag today is --summary (scripts/gen_pipe_dirty_surface.py:140); making it a gate means adding a mapping file plus a --check mode with a self-test, in the check_include_closure.py shape.
  17. The trace-replay tests carry no ctest label (tools/trace_replay/CMakeLists.txt:373-379) — selection is by name regex only.
  18. Reconcile 77 / 79 / 80 before writing the verify matrix (A4).

Part D — files an implementer will touch

File Why
.github/workflows/test.yml new build-linux-verify, integration-verify, retrace-verify, monolith-symbol-report jobs; remove-artifact-clutter needs; trigger branch; timeout-minutes
CMakeLists.txt (:5-24, :108, :440-469, :735-762) option(MOBILEGL_PIPE_VERIFY …), compile definition, interaction with MOBILEGL_BUILD_DISAGGREGATED
MobileGL/MG_IntegrationTest/CMakeLists.txt (:274-393, :400-633) two verify registrations + the two negative-control lanes; MGL_ITEST_VERIFY_TIMEOUT
MobileGL/MG_IntegrationTest/Scenarios/PipeVerifyArmingScenario.cpp (new) arming assertion + corrupted-field control
MobileGL/MG_IntegrationTest/Scenarios/PoisonOmissionScenario.cpp (new) fork/waitpid glGenerateMipmap poison control
MobileGL/MG_Test/Pipe/PipeCatalogueTest.cpp (:184-235) the two layer-1 unit controls
MobileGL/MG_Test/Pipe/CMakeLists.txt (:28) if a separate PipeVerifyTest target is preferred
tools/trace_replay/run_trace_case.cmake (:79-129) verify arming + divergence check
tools/trace_replay/CMakeLists.txt (:311-380) optional LABELS retrace; optional -DTRACE_PIPE_VERIFY forward
tools/trace_replay/trace_cases.json / trace_cases.py (:71-136, :151-196) the reduced verify matrix as data
MobileGL/Config.h (:319-328) / MobileGL/ConfigLoader.cpp (:243-256) PipeVerifyFatal, PipeVerifyCorrupt, PipePoisonOmit
scripts/gen_pipe.py (:340, :375-440, :462-510) G4 harness entry point, G5 omission vocabulary
scripts/symbol_report.py (:266-296, :342) --fail-on-added-bytes, if the gate is made hard
docs/Disaggregated/{ROADMAP,ARCHITECTURE,MEASUREMENTS}.md record the lane, the costs and the attribution table