mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-17 23:13:22 +09:00
Two things land together: the plan and contract draft P6 needs before it starts, and the
working notes that until now existed on exactly one machine.
1. P6's plan and contract draft (neither started; not one line of code)
docs/Disaggregated/P6-SPAWN-PLAN.md the package plan
docs/Disaggregated/P6-CONTRACT-DRAFT.md the contract draft (moves to
MG_Remote/CONTRACT-P6.md when c6 lands)
The three findings that mattered, all written into the documents:
a. P6 is SMALLER than its roadmap cell suggests. The transport primitives are already in
the tree and already tested: SocketDoorbell including hangup detection and the two
lost-wakeup fences (Doorbell.cpp:211-360, four cases in FdPassingTest), SCM_RIGHTS fd
passing, ShmSegment::Adopt, and the SurfaceOp / WindowKind schema already defined in
protocol.fbs. inproc uses a real second shm mapping rather than an alias, and
SessionRings.h:22-28 says that was done so today is not the first time it runs. P6 is
assembly plus a control plane plus a process, not "write a socket transport".
b. The evidence for "P6 is only a transport swap" has EXPIRED. It was bought by P5c, whose
audit found 59 direct memory accesses in a tree that already looked clean. P5e changed
the tree again. So the first package, a6, is a read-only audit that writes no code, and
every contract row that depends on it is marked PENDING a6 rather than decided in
advance.
c. P6 does not get a game running on the phone. ANativeWindow* is meaningless in the child,
and protocol.fbs:186-196's own comment says Android transfers the window out of band -
that band is P12. P6 lands pbuffer / surfaceless / offscreen, which is what HeadlessGL,
the split-equivalent lane and trace replay need, and which is exactly what P6's own exit
gate measures. A real window reaching the server is a named refusal. This gets its own
section because "P6 is done" would otherwise read as "the game runs now".
The contract adds rule G: a role may not name a process-local handle. It extends P5c's
rule E (a role may not name the other role's memory) from memory to identity - EGLDisplay /
EGLSurface / EGLContext / ANativeWindow* / fd number / pid / mapped address either cross as
a token this contract defines or they do not cross, and a descriptor crosses only through
SCM_RIGHTS, where the kernel renumbers it.
What P5e left for P6 gets its own section. The latch semantics are already designed
(ARCHITECTURE.md:504) and are copied unchanged. What is new is that after run-ahead the
client no longer waits while the peer process can DIE: the client holds records that were
published and will never be applied, and Android's freezer suspends the child without
closing anything. So slow and dead must be distinguishable WITHOUT a timeout threshold.
That is P6's only new design; everything else is carriage. It shares a predicate with E1's
outstanding re-specification (ID-122), and the two should be designed together.
Two gates carried over from P5e's lessons: G2 requires integration-spawn's test-name set to
match integration-split's name for name, because a differing set means the lane is quietly
running less; and the arm-proof gate requires every entry to record the child pid and
transport=spawn in its OWN private log - ID-124 established that this kind of false green
really happens, and a spawn lane that actually ran monolith would pass every other gate.
S2's shape must be settled before it is written, and the document says why: E1 is the
cautionary case.
2. The working notes move into the repository (docs/Disaggregated/notes/, 415 .md files)
ROADMAP / CURRENT_STAGE_PROGRESS / MEASUREMENTS cite ruling numbers, audit line numbers and
the provenance of gate figures from ~/w7/notes/, a directory that exists on one machine and
is under no version control. A record cited hundreds of times that lives only locally is not
a record, so it moves in whole, with its directory structure intact.
Only .md moves. Scripts, run logs, census and gate result JSON, patches, APKs and nm dumps
stay where they are: they are evidence DATA rather than documents - large, regenerated per
run, and mostly carrying absolute paths and device serials that mean something only on the
machine that produced them. The mapping and this split are written in
docs/Disaggregated/notes/README.md.
Two exceptions are recorded there too: P6's BRIEF has been promoted to a first-class
document and keeps no copy under notes/, so the two cannot drift apart; and recovered/ holds
material salvaged from failed workflow runs, which may duplicate the like-named phase
directories and is kept because a few of those were the only complete version at the time.
CI's citation lint takes docs/Disaggregated/*.md only (non-recursive), and notes/ is
deliberately outside it: those notes' file:line citations were written against their own
code heads, and resolving them against today's HEAD would fail wholesale - which is not an
error, it is their correct state as a historical record. To check one, use the revision it
declares (--rev).
3. Inconsistencies fixed in passing
- README.md's status line still said P5e was in progress while the stage table already said
closed.
- CURRENT_STAGE_PROGRESS's next-step item 1 still listed the exit gate as pending, though
(a) and (b) were run in the previous commit; rewritten to closed plus the one item that
actually remains (E1).
- ROADMAP's P6 row still had the pre-phase plan in three of its cells; it now carries the
package order, the scope boundary and the two new gates.
Checked with check_doc_citations.py: 125 citations across docs/Disaggregated/*.md, 0 problems.
1024 lines
68 KiB
Markdown
1024 lines
68 KiB
Markdown
# P1 verification infrastructure — scout report
|
||
|
||
Worktree: `C:/Users/geekerwan/AndroidStudioProjects/FoldCraftLauncher/MobileGL-disagg`, branch `feat/disaggregated`.
|
||
Every claim below carries a `path:line` I opened. Read-only pass; nothing in the tree was modified.
|
||
|
||
Reading order: **Part A** is the inventory (what exists, exactly where). **Part B** is the proposal for the three
|
||
things the task asks for. **Part C** is the trap list — read it before writing any CMake, because three of the
|
||
traps have already cost this project a shipped bug or a silently-green lane.
|
||
|
||
---
|
||
|
||
# Part A — inventory
|
||
|
||
## A1. `.github/workflows/test.yml` — the whole job graph
|
||
|
||
Triggers: `push` to `dev`, `Feat/Backend-Direct-GLES`, `Feat/Backend-Direct-Vulkan`, plus `workflow_dispatch`
|
||
(`.github/workflows/test.yml:3-9`). **`feat/disaggregated` is not a trigger branch** — P1 work does not run this
|
||
workflow on push unless the branch list is extended or `workflow_dispatch` is used. There is no `pull_request`
|
||
trigger and no `timeout-minutes` on any job in this file (`grep timeout-minutes` finds only `apk.yml:331`), so
|
||
every job inherits GitHub's 6 h default.
|
||
|
||
Nine jobs:
|
||
|
||
### `build-linux` (`test.yml:12-147`)
|
||
- `runs-on: ubuntu-latest`, 32 GB swap (`:26-29`), `submodules: recursive` (`:31-34`).
|
||
- ccache via job `env` (`:17-23`): `BUILD_DIR=build-linux`, `CCACHE_DIR=${{ github.workspace }}/.ccache`, key
|
||
`${{ runner.os }}-test-${{ github.job }}-ccache-v1` (`:39-45`), saved only on the default branch (`:109-122`).
|
||
- Toolchain: clang-20 / libc++-20 / lld-20 / ninja / mesa + lavapipe + Vulkan loader (`:58-61`), Vulkan SDK
|
||
1.4.304.1 headers+loader (`:47-52`), `python update_glslang_sources.py` (`:54-56`).
|
||
- Configure (`:71-93`) — the flags P1 has to mirror:
|
||
```
|
||
-DCMAKE_BUILD_TYPE=Release (Debug only when ACTIONS_STEP_DEBUG=true, :73-77)
|
||
-DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO
|
||
-DMOBILEGL_BUILD_TEST=ON
|
||
-DMOBILEGL_BUILD_BENCHMARK=ON
|
||
-DMOBILEGL_BUILD_INTEGRATION_TEST=ON
|
||
-DMOBILEGL_ITEST_VK_ICD=/usr/share/vulkan/icd.d/lvp_icd.json
|
||
-DMOBILEGL_BUILD_TRACE_REPLAY=OFF
|
||
-DBENCHMARK_DOWNLOAD_DEPENDENCIES=ON -DBENCHMARK_ENABLE_TESTING=OFF
|
||
-DCMAKE_POLICY_VERSION_MINIMUM=3.5
|
||
```
|
||
Note what is **absent**: `MOBILEGL_BUILD_DISAGGREGATED` is never passed, so it stays OFF
|
||
(`CMakeLists.txt:23`), and `MOBILEGL_ITEST_REQUIRE_GPU` is never passed at configure time (it arrives as job
|
||
env instead — see `integration` below). `MOBILEGL_ENABLE_LTO` defaults OFF (`CMakeLists.txt:108`).
|
||
- Packaging (`:124-147`): `ci-artifacts/mobilegl-linux-runtime.tgz` containing the root
|
||
`build-linux/CTestTestfile.cmake`, the three test/benchmark/integration subtrees and **every** `*.so` found
|
||
under the build dir. Uploaded as artifact `mobilegl-linux-runtime`.
|
||
|
||
### `test` (`test.yml:149-203`) — the unit lane
|
||
- `needs: build-linux`; installs runtime deps only (`:160-163`); downloads + unpacks the runtime tgz.
|
||
- **"Normalize CTest command paths"** (`:174-184`): a python inline script rewriting
|
||
`"…/cmake-…/bin/cmake"` → `"cmake"` in every `CTestTestfile.cmake` under `build-linux`. Repeated verbatim in
|
||
`integration` (`:234-244`), `benchmark` (`:380-390`) and `build-retrace` (`:526-536`). Any new job that
|
||
unpacks the runtime artifact and calls ctest must repeat this step.
|
||
- Run (`:186-195`): cores armed (`ulimit -c unlimited`, `kernel.core_pattern=/tmp/core.%e.%p`), then
|
||
`ctest --output-on-failure -L unit --no-tests=error` (`-V` under ACTIONS_STEP_DEBUG).
|
||
- Failure-only core-dump artifact `unit-core-dumps` (`:197-203`).
|
||
|
||
### `integration` (`test.yml:205-296`) — the GPU-scenario lane, the one P1 doubles
|
||
- `needs: build-linux`. Extra runtime dep `libegl-mesa0`, with the reason spelled out (`:216-223`).
|
||
- Job `env` (`:266-270`):
|
||
```
|
||
MOBILEGL_ITEST_REQUIRE_GPU: "1"
|
||
MOBILEGL_MAGMA_FIX_ITERATIONRP_SUBGROUP_SCRATCH: "1"
|
||
MOBILEGL_MAGMA_DERIVE_NUM_SUBGROUPS: "1"
|
||
MOBILEGL_MAGMA_ITERATIONRP_FIX_BARRIER: "1"
|
||
```
|
||
- The **ENVIRONMENT-property warning** lives here in full (`:252-259`): the lavapipe ICD pin is passed at
|
||
configure (`-DMOBILEGL_ITEST_VK_ICD`) because the configure bakes it into each test's ctest `ENVIRONMENT`
|
||
property, "and a property entry OVERRIDES the job environment — a `VK_ICD_FILENAMES` exported here would be
|
||
silently ignored while looking like it works."
|
||
- The **counter-example is in the same job** (`:274-288`): the second, filtered pass exports
|
||
`MOBILEGL_ESPRYT_DISABLE_INVALIDATE_FLUSH=1` *inline* with the comment "The flag is NOT baked into the ctest
|
||
ENVIRONMENT properties, so an inline env reaches the test processes (unlike the ICD pin above)."
|
||
**This is the decisive evidence for P1**: a job-env variable that no `ENVIRONMENT` property names does reach
|
||
the test process. `MOBILEGL_PIPE_VERIFY` / `MOBILEGL_PIPE_PUSH` are named by no property today.
|
||
- Two ctest invocations:
|
||
```
|
||
ctest --output-on-failure -L integration-gpu --no-tests=error
|
||
MOBILEGL_ESPRYT_DISABLE_INVALIDATE_FLUSH=1 ctest --output-on-failure -L integration-gpu \
|
||
-R 'Buffer|Readback|Atomic|Ssbo|Arena' --no-tests=error
|
||
```
|
||
- `integration-core-dumps` on failure (`:290-296`).
|
||
|
||
### `flatc-check` (`test.yml:298-323`)
|
||
Independent of `build-linux`; regenerates `protocol_generated.h` with `scripts/gen_protocol.py` and
|
||
`git diff --exit-code`. The model for a source-level gate that must not be hidden by a broken build.
|
||
|
||
### `include-graph-check` (`test.yml:332-349`) — the P0.5 purity gate, and the template for P1's controls
|
||
- No `needs`, three header submodules only (`:340-343`), clang-20 (`:345-346`), then:
|
||
`python3 scripts/check_include_closure.py --mode both --compiler clang++-20 --self-test --require-all`.
|
||
- The job comment states the doctrine P1 must satisfy: "The script's own `--self-test` is always on: a negative
|
||
control that stopped tripping fails the job, because **a gate that cannot go red is not a gate**"
|
||
(`test.yml:330-331`, quoting `ROADMAP.md:7`).
|
||
|
||
### `benchmark` (`test.yml:351-405`)
|
||
`needs: build-linux`; `ctest -V -C Release -L benchmark --no-tests=error`.
|
||
|
||
### `build-retrace` (`test.yml:407-550`)
|
||
- `needs: build-linux, test, benchmark, integration` — i.e. the retrace lane only starts after the three
|
||
ctest lanes are green.
|
||
- Job env `MOBILEGL_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so` (`:424`); the runtime artifact
|
||
is unpacked and that path asserted to exist (`:478-481`).
|
||
- Second configure (`:483-502`), separate build dir `build-retrace`, separate ccache key:
|
||
```
|
||
-DMOBILEGL_BUILD_TEST=OFF -DMOBILEGL_BUILD_BENCHMARK=OFF
|
||
-DMOBILEGL_BUILD_TRACE_REPLAY=ON
|
||
-DMOBILEGL_TRACE_REPLAY_MOBILEGL_LIBRARY="${MOBILEGL_LIBRARY}"
|
||
```
|
||
- Builds only `--target mobilegl_trace_replay` (`:504-505`), normalizes CTest paths (`:526-536`), and packages
|
||
**exactly two files** (`:538-543`): `build-retrace/tools/trace_replay/mobilegl_trace_replay` and
|
||
`build-retrace/tools/trace_replay/CTestTestfile.cmake`. Artifact `mobilegl-trace-replay`.
|
||
- Consequence for P1: the *absolute paths* of the trace case commands (replay exe, `MOBILEGL_LIBRARY`, fixture
|
||
paths) are frozen into that `CTestTestfile.cmake` at configure time. A later job re-creates the same paths by
|
||
unpacking into the same workspace layout, it does not reconfigure.
|
||
|
||
### `trace-cases` / `trace-fixtures` / `retrace` / `retrace-summary` / `remove-artifact-clutter`
|
||
- `trace-cases` (`:552-570`) emits two outputs from `tools/trace_replay/trace_cases.py --ci`:
|
||
`matrix` (`--format github-test-matrix`) and `names` (`--format names`).
|
||
- `trace-fixtures` (`:572-635`) is a per-**case** matrix (names only): derive cache key
|
||
(`.github/scripts/trace-fixture-cache.sh key`), restore, verify, else `fetch-trace-fixture-lfs.sh`, stage the
|
||
files listed by `trace_cases.py --format fixture-files --case <name>`, upload as `trace-fixture-<case>`.
|
||
- `retrace` (`:637-741`) is the per-`(backend, case)` matrix, `fail-fast: false`, **`max-parallel: 4`**.
|
||
It downloads `trace-fixture-<case>` into `tools/trace_replay/fixtures/` (`:660-669`), unpacks both runtime
|
||
tarballs (`:693-698`), then in `build-retrace/tools/trace_replay`:
|
||
```
|
||
ulimit -c unlimited; sysctl kernel.core_pattern=/tmp/core.%e.%p
|
||
export MOBILEGL_MAGMA_R11G11B10F_FALLBACK=1 # DirectVulkan
|
||
export MOBILEGL_MAGMA_{FIX_ITERATIONRP_SUBGROUP_SCRATCH,DERIVE_NUM_SUBGROUPS,ITERATIONRP_FIX_BARRIER}=1
|
||
# DirectVulkan + iterationrp case
|
||
export MOBILEGL_MAGMA_DISABLE_BLENDED_DEPTH_WRITE=1 # DirectVulkan + improved-transparency-26.3
|
||
ctest -V --no-tests=error -R '^MobileGLTraceReplay\.<case>\.<backend>$'
|
||
```
|
||
(`:700-723`). **These `export MOBILEGL_*` lines are the second proof that job env reaches the retrace test
|
||
process** — they are not in any `ENVIRONMENT` property (see A4) and they demonstrably take effect.
|
||
- Artifacts: `retrace-core-dumps-*` on failure, `retrace-result-<backend>-<case>` always (`:725-741`).
|
||
- `retrace-summary` (`:743-782`) renders `render_retrace_summary.mjs` over all `retrace-result-*`.
|
||
- `remove-artifact-clutter` (`:784-833`) deletes `trace-fixture-*` and `retrace-result-*` artifacts, retaining
|
||
fixtures only for failed retrace jobs.
|
||
|
||
### `pipe-gates` (`test.yml:835-887`) — the MGPipe source gate, where P1's new gates belong
|
||
Deliberately independent of `build-linux` (`:837-838`). Four steps:
|
||
1. `python3 scripts/gen_pipe.py` + `git diff --exit-code -- MobileGL/MG_Pipe/generated` (`:850-853`).
|
||
2. stdio-instrumentation grep over `MobileGL/MG_Backend` and `MobileGL/MG_State` (`:862-869`) — bans
|
||
`fprintf(stderr|stdout`, `printf(`, `puts(`, `std::cout|cerr`, with **no exceptions list**.
|
||
3. `python3 scripts/gen_pipe_dirty_surface.py --summary` — informational, and the comment says explicitly
|
||
"It becomes a gate in P1, when the mapping file exists to diff against" (`:871-874`).
|
||
4. `python3 scripts/check_doc_citations.py docs/Disaggregated/*.md || true` — warning-only until the docs
|
||
settle (`:876-887`).
|
||
|
||
## A2. ctest labels
|
||
|
||
Only three labels exist in the tree:
|
||
- `unit` — every `MG_Test` target (`MobileGL/MG_Test/CMakeLists.txt:65`, `Pipe/CMakeLists.txt:28`,
|
||
`Util/CMakeLists.txt:56-58`, `Purity/CMakeLists.txt:13`).
|
||
- `integration-gpu` — every registration in `MobileGL/MG_IntegrationTest/CMakeLists.txt` (13 of them).
|
||
- `benchmark` — MG_Benchmark.
|
||
- **The trace-replay tests carry NO label at all** (`tools/trace_replay/CMakeLists.txt:350-379` sets only
|
||
`ENVIRONMENT`). They are selected purely by the `-R '^MobileGLTraceReplay\.…$'` regex.
|
||
|
||
## A3. `MobileGL/MG_IntegrationTest/` — registration, env plumbing, harness
|
||
|
||
### Target and sources
|
||
`add_executable(MobileGLIntegrationTest …)` with `Main.cpp`, two Harness TUs and **79 scenario TUs**
|
||
(`CMakeLists.txt:51-130`). Links `GTest::gtest` (not `gtest_main`) plus `MobileGL_s` on desktop /
|
||
`MobileGL` (shared) on Android (`:29-36`, `:137-141`). `Main.cpp` installs a `HarnessBanner`
|
||
`testing::Environment` and calls `RUN_ALL_TESTS()` (`Main.cpp:62-66`); the banner prints backend, renderer,
|
||
surface and `EGL_PLATFORM`, and states whether it is skipping or failing every scenario.
|
||
|
||
**371 `TEST_F` cases** across `Scenarios/*.cpp` (counted by `grep -cE '^\s*TEST_F\('`).
|
||
|
||
### The 13 ctest registrations (`CMakeLists.txt:400-633`)
|
||
All share `LABELS integration-gpu`, `DISCOVERY_TIMEOUT 30`, `TIMEOUT ${MGL_ITEST_TIMEOUT}` where
|
||
`MGL_ITEST_TIMEOUT = 120` (`:393`, with the comment "a GPU test that wedges must fail the run, not hang it").
|
||
|
||
| Prefix | `TEST_FILTER` | cases | ENVIRONMENT var |
|
||
|---|---|---|---|
|
||
| `DirectGLES.` | — (all) | 371 | `MGL_ITEST_GLES_ENVIRONMENT` (`:400-407`) |
|
||
| `DirectVulkan.` | — (all) | 371 | `MGL_ITEST_VULKAN_ENVIRONMENT` (`:409-416`) |
|
||
| `DirectVulkan.AsyncCompile.` | `XfbAfterClipDistanceScenario.*` | 5 | `…VULKAN_ASYNC_ENVIRONMENT` (`:429-437`) |
|
||
| `DirectGLES.ForcedDepthStencilEmulation.` | `DepthStencilReadback*Scenario.*` | 20 | `…GLES_FORCED_DS_ENVIRONMENT` (`:447-455`) |
|
||
| `DirectGLES.UnlocatedIoBlocks.` | `UnlocatedIoBlockScenario.*` | 3 | `…GLES_UNLOCATED_IO_BLOCKS_ENVIRONMENT` (`:464-472`) |
|
||
| `DirectGLES.AsyncOn.` / `DirectVulkan.AsyncOn.` | `AsyncCompileScenario.*` | 6 + 6 | `:484-501` |
|
||
| `DirectGLES.AsyncOff.` / `DirectVulkan.AsyncOff.` | one case | 1 + 1 | `:511-528` |
|
||
| `DirectGLES.OptimisticShaderStatus.` / `DirectVulkan.…` | one case | 1 + 1 | `:537-554` |
|
||
| `DirectGLES.NoViewportArrayEmulation.` | one case | 1 | `:561-569` |
|
||
| `DirectVulkan.PrimGenReroute.` | `PrimitivesGeneratedNoXfbScenario.*` | 8 | `:580-588` |
|
||
| `DirectGLES.WidenedPacked16.` | `CopyImagePacked16Scenario.*` | 3 | `:598-606` |
|
||
| `DirectGLES.PointSizeDemotion.` / `DirectVulkan.…` | `PointSizeDemotionScenario.*` | 5 + 5 | `:615-633` |
|
||
|
||
**Total ctest entries under `integration-gpu`: 808.** Each entry is one process launch of
|
||
`MobileGLIntegrationTest --gtest_filter=<one case>`, and each process does a full EGL bring-up preceded by a
|
||
`fork()` pre-flight (`Harness/HeadlessGL.cpp:323-343`). Process count, not GL work, is the dominant cost of the
|
||
lane.
|
||
|
||
### The env-join idiom — copy it exactly
|
||
`mgl_itest_join_environment(outVar …)` (`:307-317`) escapes each `;` as `\\;` because
|
||
`gtest_discover_tests` forwards `PROPERTIES` as a flat list; without it "only `MOBILEGL_BACKEND_TYPE` reaches
|
||
the test and the vendor/ICD pinning is lost" (`:301-306`).
|
||
|
||
`MGL_ITEST_COMMON_ENV` (`:274-281`) = `__EGL_VENDOR_LIBRARY_FILENAMES=<detected>` +
|
||
`MOBILEGL_ITEST_REQUIRE_GPU=1` *only when the CMake option is ON* (`:279-281`; the option itself at `:258-259`
|
||
defaults OFF, which is why CI passes it as job env instead). `MGL_ITEST_VULKAN_ENV` (`:283-299`) adds
|
||
`VK_ICD_FILENAMES=<pin>` and, when the pin is lavapipe, the three iterationRP repairs.
|
||
|
||
`CMakeLists.txt:339-344` states the rule in bold: every mode list **appends** to `MGL_ITEST_COMMON_ENV` /
|
||
`MGL_ITEST_VULKAN_ENV`, because "A ctest ENVIRONMENT property REPLACES the job environment rather than adding
|
||
to it, so an entry that lists only its mode variable would silently lose the EGL vendor and Vulkan ICD pinning".
|
||
|
||
### How a scenario reads configuration (the `MG_Config` question)
|
||
It does **not** read `MG_Config`. `Harness/ScenarioFixture.h:34-65` defines `AmbientQuirk` /
|
||
`AmbientQuirkFromEnvironment(name)` which re-implements `ConfigLoader`'s truthiness rule character for
|
||
character, and the header states the two reasons (`:38-48`): the feature table is an internal symbol and the
|
||
Android build links the shipping `-fvisibility=hidden` library; and an in-process poke is already too late for
|
||
anything latched at initialization. The one exception is `Harness/BackendCapsPeek.h:20-28`, which peeks at the
|
||
active backend's caps block and **returns false on Android**, touching nothing — the sanctioned shape for a
|
||
desktop-only internal peek from this module.
|
||
|
||
`ScenarioTest::SetUp` (`ScenarioFixture.h:69-104`): `FAIL()` when `MOBILEGL_ITEST_REQUIRE_GPU` is set and the
|
||
harness is unusable, else `GTEST_SKIP()`; derived fixtures must gate on `Ready()`, never `IsSkipped()`
|
||
(`:106-109`, with the SIGSEGV story that motivated it).
|
||
|
||
### The "arming assertion" idiom — the model for proving verify is actually on
|
||
Three lanes pin a quirk *and* a per-lane log file so the test can prove the library armed it:
|
||
`MGL_ITEST_GLES_UNLOCATED_IO_BLOCKS_ENVIRONMENT` (`:366-369`),
|
||
`MGL_ITEST_VULKAN_PRIMGEN_REROUTE_ENVIRONMENT` (`:374-377`),
|
||
`MGL_ITEST_{GLES,VULKAN}_POINT_SIZE_DEMOTION_ENVIRONMENT` (`:383-390`).
|
||
The comment at `:361-365` explains why: the arming signal is a latched `MGLOG_I` and there is no other way for
|
||
a test process to learn it fired. Consumption side:
|
||
`Scenarios/UnlocatedIoBlockScenario.cpp:354-360` and `:489`,
|
||
`Scenarios/PrimitivesGeneratedNoXfbScenario.cpp:253` and `:526`,
|
||
`Scenarios/PointSizeDemotionScenario.cpp:304-308` and `:489` — each reads `MOBILEGL_LOG_FILE_PATH`, trusts only
|
||
bytes written after the case started, and `GTEST_SKIP()`s with an explicit message when the path is unset.
|
||
|
||
### Local runner
|
||
`MobileGL/MG_IntegrationTest/run_integration_test.sh` — `./run_integration_test.sh espryt|magma [gtest args]`;
|
||
pins `__EGL_VENDOR_LIBRARY_FILENAMES` / `VK_ICD_FILENAMES` from `MGL_EGL_VENDOR` / `MGL_VK_ICD` (defaults are
|
||
NVIDIA paths), sets `EGL_PLATFORM=${EGL_PLATFORM:-x11}`, and honours `MOBILEGL_ITEST_REQUIRE_GPU`. Note it is
|
||
*not* the ctest path: it does not carry the iterationRP pins or any mode variable.
|
||
|
||
## A4. `tools/trace_replay/` — the retrace gate
|
||
|
||
### Case manifest
|
||
`trace_cases.json`: `defaults` (854x480, ssim 0.99, `timeout_seconds: 900`) + **40 cases**.
|
||
- One case is CI-excluded: `minecraft-1.21.4-rd12-odinlite-in-world` has `"ci": false`
|
||
(`trace_cases.json:63-70`, the flag itself at `:69`).
|
||
- One case is backend-restricted: `minecraft-1.21.4-fabric-iris-iterationrp-in-world` has
|
||
`"ci_backends": ["DirectVulkan"]` (`:286-295`, the key at `:288`).
|
||
- `timeout_seconds` distribution over the 39 CI cases: 17×900, 17×1800, 5×180.
|
||
|
||
### Counts — reconcile these three numbers before writing any job
|
||
- `trace_cases.py --ci --format github-test-matrix` → **77** entries (39 CI cases × 2 backends − 1 restricted).
|
||
This is what the `retrace` matrix runs today (`trace_cases.py:71-98`).
|
||
- The CMake generator is called **without** `--ci` (`tools/trace_replay/CMakeLists.txt:389-394`) and
|
||
`add_trace_replay_test_for_backends` always registers both backends (`:382-385`), so a local
|
||
`ctest -R '^MobileGLTraceReplay\.'` in `build-retrace` registers **80** tests.
|
||
- The **79** in the task statement = 80 minus the `iterationrp` × `DirectGLES` pair that `ci_backends` marks
|
||
unsupported. That is the desktop gate the WSL harness (`retrace_gate.py`) drives. Implementers should confirm
|
||
which of 77 / 79 / 80 the P1 acceptance means; the docs consistently say "40 trace"
|
||
(`ROADMAP.md:17`, `ARCHITECTURE.md:502`, `:507`).
|
||
|
||
### Registration
|
||
`add_trace_replay_test(CASE_NAME BACKEND …)` (`tools/trace_replay/CMakeLists.txt:311-380`):
|
||
```
|
||
add_test(NAME MobileGLTraceReplay.${CASE_NAME}.${BACKEND}
|
||
COMMAND cmake -DTRACE_REPLAY_EXE=$<TARGET_FILE:mobilegl_trace_replay>
|
||
-DMOBILEGL_LIBRARY=${mobilegl_trace_replay_mobilegl_library}
|
||
-DTRACE_CASE_NAME/-DTRACE_ARCHIVE/-DTRACE_FILE/-DTRACE_GOLDEN/
|
||
-DTRACE_ALTERNATE_GOLDEN/-DTRACE_BACKEND/-DTRACE_TARGET_CALL/
|
||
-DTRACE_WIDTH/-DTRACE_HEIGHT/-DTRACE_SSIM_THRESHOLD/-DTRACE_CROP_*/
|
||
-DTRACE_COHERENT_AS_FLUSH/-DTRACE_OUTPUT_DIR/-DTRACE_ARTIFACT_DIR
|
||
-P run_trace_case.cmake) (:350-372)
|
||
```
|
||
Per-test `ENVIRONMENT` (`:373-379`):
|
||
- DirectGLES: `EGL_PLATFORM=surfaceless;LIBGL_ALWAYS_SOFTWARE=1;MESA_GL_VERSION_OVERRIDE=3.3;MESA_GLSL_VERSION_OVERRIDE=330`
|
||
- DirectVulkan: the same minus `EGL_PLATFORM`.
|
||
|
||
**No `MOBILEGL_*` variable is named in either property, and no `TIMEOUT` property is set.** Two consequences:
|
||
1. `MOBILEGL_PIPE_VERIFY` / `MOBILEGL_PIPE_PUSH` exported by a job reach the replay process unimpeded — the
|
||
same mechanism `MOBILEGL_MAGMA_R11G11B10F_FALLBACK` already uses (`test.yml:705-707`).
|
||
2. Every trace test runs under **ctest's default 1500 s timeout**. The `timeout_seconds` field in
|
||
`trace_cases.json` is consumed only by the Android lane (`apk.yml:449` `--timeout-seconds`) — `emit_cmake`'s
|
||
key list (`trace_cases.py:122-136`) does not include it. So the 1800 s cases already sit above the desktop
|
||
ctest default; anything that slows retrace down by 5–10× must raise it explicitly.
|
||
|
||
`mobilegl_trace_replay_mobilegl_library` is `MOBILEGL_TRACE_REPLAY_MOBILEGL_LIBRARY` when set, else
|
||
`$<TARGET_FILE:MobileGL>` (`:303-308`).
|
||
|
||
### `run_trace_case.cmake` — how a case actually runs
|
||
- Required `-D` variables checked up front (`:1-5`); defaults filled for SSIM/crop (`:7-21`).
|
||
- Wipes and recreates `${TRACE_OUTPUT_DIR}/{input,output}` (`:31-35`), extracts the `.tgz`, asserts the trace
|
||
file exists (`:37-52`).
|
||
- `execute_process()` the replay exe with `--trace/--golden/--diff/--output/--backend/--mobilegl-library/
|
||
--target-call/--width/--height/--ssim-threshold/--crop-*` (+`--coherent-as-flush`) (`:54-74`).
|
||
`execute_process` inherits the environment — this is the hop `MOBILEGL_PIPE_VERIFY` travels through.
|
||
- Echoes `MOBILEGL_TRACE_GL_*` identity lines from `output/retrace.log` (`:79-86`).
|
||
- Copies actual/golden/result.json/`retrace.log`/`mobilegl.log` into `TRACE_ARTIFACT_DIR` (`:88-110`).
|
||
- **Fails** if `output/result.json` is missing (`:112-125`) or if the replay exit status is non-zero
|
||
(`:127-129`). Both are `message(FATAL_ERROR)`, i.e. a red ctest test.
|
||
|
||
`output/mobilegl.log` exists because `trace_replay_core.cpp:1011` does
|
||
`setenv("MOBILEGL_LOG_FILE_PATH", mobileGlLogPath.c_str(), 1)` — so a retrace run already has a library log the
|
||
harness can grep, which is exactly what a verify-arming assertion needs.
|
||
|
||
### Desktop CLI has no `--env`
|
||
`trace_replay_cli.cpp:116-179` lists every option; there is no `--env`. The `K=V;K=V` passthrough
|
||
(`ApplyEnvOverrides`, `android-plugin/app/src/trace/cpp/trace_replay_core.cpp:143-155`, called at `:237`) is
|
||
reached only from the Android side (`android-plugin/trace-replay-ci.sh:174` and `:415` `--es mobilegl_env`,
|
||
`tools/trace_replay/run_android_retrace_local.py:192-196`, `:302`). Its parser is pinned by a build-time
|
||
self-running check: `mobilegl_trace_env_overrides_test` is built and **executed as a POST_BUILD command**, and
|
||
`mobilegl_trace_replay` depends on it (`tools/trace_replay/CMakeLists.txt:291-301`), so `cmake --build … --target
|
||
mobilegl_trace_replay` runs it even though the job never invokes ctest. `trace_env_overrides_test.cpp:73-92`
|
||
already uses `MOBILEGL_PIPE_PUSH` / `MOBILEGL_PIPE_VERIFY` as its example strings.
|
||
|
||
### The Android retrace lane (`apk.yml`)
|
||
`apk.yml:417-470`: per-`(backend, case)` matrix, `timeout-minutes: 75` (`:331`), env-per-knob at `:420-425`,
|
||
and `android-plugin/trace-replay-ci.sh` invoked with the case's fields including `--timeout-seconds`. The
|
||
generic `--env "K=V;K=V"` hook (`trace-replay-ci.sh:35`, `:64`, `:174`, `:415`) is where a device-side
|
||
`MOBILEGL_PIPE_VERIFY=1` would go. Trap already documented at `MEASUREMENTS.md` §5.3: an `--env` value
|
||
containing `/data/...` needs `MSYS_NO_PATHCONV=1` on a Windows host.
|
||
|
||
## A5. The Fatal / assert idiom
|
||
|
||
- `MGLOG_F` is `MOBILEGL_LOG_INTERNAL("FATAL", ANDROID_LOG_FATAL, …)` and **is live at every build level**
|
||
(`MobileGL/MG_Util/Debug/Log.h:87-92`). It logs; it does not terminate.
|
||
- `MOBILEGL_ASSERT(cond, …)` = `MGLOG_F` ×2 + `TRAP` (`MobileGL/Defines.h:104-112`), and is **compiled to
|
||
nothing** unless `MOBILEGL_LOG_ACTIVE_LEVEL <= MOBILEGL_LOG_LEVEL_DEBUG` (`:113-115`). CI builds at
|
||
`MOBILEGL_LOG_LEVEL_INFO` (`test.yml:85`), so **every `MOBILEGL_ASSERT` in the tree is inert in CI.**
|
||
`TRAP` is `__builtin_debugtrap()` under clang, `raise(SIGTRAP)` elsewhere, `assert(false)` on MSVC
|
||
(`Defines.h:88-95`).
|
||
- The **poison Fatal already exists in generated form**:
|
||
```cpp
|
||
[[noreturn]] inline void MGPipeInputPoisonFatal(MGPipeInputField field, const char* verb) {
|
||
MGLOG_F("MGPipe: Fatal{UnmigratedPipeInput, \"%s@%s\"}",
|
||
kMGPipeInputFieldNames[static_cast<SizeT>(field)], verb);
|
||
std::abort();
|
||
}
|
||
```
|
||
`MobileGL/MG_Pipe/generated/PipeFilled.inc:298-302` (emitted by `scripts/gen_pipe.py:497-503`).
|
||
It is `MGLOG_F` + `std::abort()`, **not** `MOBILEGL_ASSERT` — deliberately, since the assert would vanish in
|
||
an INFO build. Companion state and predicate: `MGPipeFilledState{CurrentVerbSerial, FilledGen[61]}`
|
||
(`PipeFilled.inc:293-296`) and `MGPipeInputFieldIsFresh` (`:304-307`), with the sticky table at `:163` and
|
||
the "which call fills it" table at `:229`.
|
||
- The wire-protocol Fatal has the same shape: `MGPipeWireProtocolFatal` ends in `std::abort()`
|
||
(`scripts/gen_pipe.py:340`, emitted into `PipeWire.inc`).
|
||
- Termination therefore = `SIGABRT`. A test that wants to observe it must run it in a child process.
|
||
|
||
### Death tests: there are none
|
||
`grep -rn "EXPECT_DEATH|ASSERT_DEATH|ExitedWithCode|KilledBySignal|EXPECT_EXIT|DeathTest"` over
|
||
`MobileGL/` and `tools/` returns **zero hits**. The project's established way to observe a child's exit is
|
||
raw `fork()`/`waitpid()`:
|
||
- `MobileGL/MG_Test/Wire/FdPassingTest.cpp:62` (`fork`), `:85` (`_exit(status)`), `:130-132`
|
||
(`waitpid` + `WIFEXITED` + `WEXITSTATUS == kChildOk`).
|
||
- `MobileGL/MG_IntegrationTest/Harness/HeadlessGL.cpp:323-423` — the pre-flight: fork, child `_exit(step)`
|
||
(`:365-368`), parent `waitpid` with `WNOHANG` polling (`:380-388`), `WTERMSIG` (`:410`), `WIFEXITED` /
|
||
`WEXITSTATUS` (`:420-423`). Its comment at `:323-325` is the exact rationale P1 needs: "SIGTRAP is a datum
|
||
instead of a crash. fork() gives exactly that."
|
||
|
||
## A6. Existing negative-control patterns (the shapes to imitate)
|
||
|
||
1. **`check_include_closure.py --self-test`** (`scripts/check_include_closure.py:435-548`): three canned TUs
|
||
that must trip (`negative_controls()`, `:469-497`), a parser check over canned `-H` output (`:447-467`), a
|
||
"synthesize a third `glslang::` token and confirm the budget trips" control (`:499-515`), and — the piece
|
||
that makes it a gate — `if trips == 0: error("negative control did not trip: the closure gate is not
|
||
checking anything")` (`:541-543`). It runs in two places: the `include-graph-check` job (`test.yml:349`) and
|
||
as a ctest under `unit` (`MobileGL/MG_Test/Purity/CMakeLists.txt:9-13`), where the comment reads
|
||
"**NO ENVIRONMENT property: it replaces the job env (ARCHITECTURE.md:567)**".
|
||
2. **`gen_pipe.py` + `git diff --exit-code`** (`test.yml:850-853`) — regeneration as the anti-drift gate, and
|
||
`gen_pipe.py` refuses a catalogue whose payload has no field list, "a payload the G4 comparator cannot see
|
||
is a payload MOBILEGL_PIPE_VERIFY is blind to" (`scripts/gen_pipe.py:29-30`, enforced at `:194`).
|
||
3. **Comparator unit tests**: `MobileGL/MG_Test/Pipe/PipeCatalogueTest.cpp:184-207` —
|
||
`MGPipeVerify(a, b, &field)` true for equal payloads, false after `b.InstanceCount = 7` with
|
||
`EXPECT_STREQ(field, "InstanceCount")`, plus a padding control (equal despite padding) and an array/nested
|
||
control (`right.Color[3].Level = 2` → field `"Color"`).
|
||
4. **Poison unit test**: `PipeCatalogueTest.cpp:224-235` — `PipeInputFieldsStartUnfilled`, including the
|
||
line that matters ("The next verb makes the same value stale, which a written-once bitmap could not see").
|
||
5. **Env-pinned negative-control ctest lanes**: `DirectGLES.NoViewportArrayEmulation.` registers the *one* case
|
||
that describes behaviour with the emulation off (`MG_IntegrationTest/CMakeLists.txt:556-569`);
|
||
`DirectGLES.AsyncOff.` registers the one case that has anything to say with async off (`:503-528`).
|
||
6. **Log-file arming assertions** — A3 above.
|
||
7. **`LogLevelTest`** (`MobileGL/MG_Test/Util/LogLevelTest.cpp`): links `GTest::gtest` (not `gtest_main`) so its
|
||
own `main()` can point `MOBILEGL_LOG_FILE_PATH` at a temp file before anything latches the sink
|
||
(`MG_Test/Util/CMakeLists.txt:19-22`); `EmitAndRead()` (`:87-107`) emits five markers and greps the file;
|
||
`SinkIsReachableAtAll` (`:110-118`) exists purely so the suppression assertions cannot pass vacuously.
|
||
**This is the closest existing thing to a Fatal-capture hook and the cheapest template for one.**
|
||
8. **A build-time self-running check**: `mobilegl_trace_env_overrides_test` POST_BUILD
|
||
(`tools/trace_replay/CMakeLists.txt:291-301`) — a gate that runs even in a job that never calls ctest.
|
||
|
||
## A7. `scripts/symbol_report.py` (P0.5)
|
||
|
||
- Purpose (docstring `:10-16`): per-symbol `nm --defined-only -S` and `.text` attribution between two
|
||
`libMobileGL.so` builds; `--strip-scope` folds de-nesting renames into a "renamed-only" bucket.
|
||
- Options (`:266-296`): `--before`, `--after`, `--nm`, `--size`, `--cxxfilt`, `--markdown`, `--json`,
|
||
`--strip-scope` (repeatable), `--rename-map OLD=NEW`, `--only-names`, `--threshold`,
|
||
`--fail-on-added-bytes`, `--self-test`.
|
||
- **It always exits 0**: `--fail-on-added-bytes is reserved; this run stays informational` (`:342`, docstring
|
||
`:33-35`). Making it a gate means either implementing the flag or asserting on `--json` in the job.
|
||
- Guard rails it documents (`:23-31`): identical `CMAKE_BUILD_TYPE` (visibility presets differ,
|
||
`CMakeLists.txt:578-598`), `MOBILEGL_ENABLE_LTO` OFF on both sides (`CMakeLists.txt:108,137-146`), same
|
||
compiler and stdlib. It prints both paths and byte sizes in the report header.
|
||
- Parsers: `nm --defined-only -S` sized/unsized rows (`:46-52`), `size --format=sysv` rows for
|
||
`.text/.data/.bss/.rodata/Total` (`:54-58`).
|
||
- P0.5's measured result, for calibration: "测试名零删除、`MG_Backend` 零 diff、`.text` 字节不变、符号 0 增 /
|
||
0 删 / 42 重命名" (`ROADMAP.md:16`).
|
||
- The job name is already reserved: `monolith-symbol-report` (`ARCHITECTURE.md:568`); it does not exist in
|
||
`test.yml` yet.
|
||
|
||
## A8. What P1 is contractually required to show
|
||
|
||
`ROADMAP.md:17` (the P1 row), acceptance column verbatim:
|
||
> pull 构建 `nm --defined-only` 不变、`.text` 差异逐行归因(空守卫/三元重写推迟到 P2);40 trace + 全部集成测试在
|
||
> `MOBILEGL_PIPE_VERIFY=1` 下零分歧;故意损坏一个快照字段能让 verify 变红;故意在 `glGenerateMipmap` 的填充表漏
|
||
> 一个字段能在**那条 verb** 上触发 poison Fatal
|
||
|
||
Supporting design: `ARCHITECTURE.md:502` (verify is "第三种 CI 模式,40 个 trace + 全部集成测试,~5–10× 慢,
|
||
永不出货"), `:335` (`#if MOBILEGL_PIPE_PUSH` / `MGB_CTX`), the phase table at `:337-345`, the poison paragraph
|
||
at `:367` ("debug 与 disaggregated 构建里读一个当前 verb 未填的字段是 `Fatal{UnmigratedPipeInput,
|
||
"GetStencilState@DrawVbo"}`"), the switch tables at `:579` (`MOBILEGL_PIPE_VERIFY` is a **build-time** CMake
|
||
option, planned, not yet existing) and `:588-589` (the runtime env vars, already landed).
|
||
|
||
Runtime plumbing that already exists: `MobileGL/Config.h:319-328` (`Uint64 PipePush`, `Bool PipeVerify`),
|
||
`MobileGL/ConfigLoader.cpp:245-246`, and the note at `:243-244` that no allow-list entry is needed because
|
||
every `MOBILEGL_`-prefixed variable is accepted by construction. `QueryEnvUint64` rejects octal and negatives
|
||
(`ConfigLoader.cpp:167-190`).
|
||
|
||
---
|
||
|
||
# Part B — proposals
|
||
|
||
## B1. Running the whole integration suite a third time under verify
|
||
|
||
### B1.0 The prerequisite nobody can skip: the build
|
||
`MOBILEGL_PIPE_VERIFY` is a **build-time** option (`ARCHITECTURE.md:579`) and the poison is gated on
|
||
`MOBILEGL_DEBUG || MOBILEGL_BUILD_DISAGGREGATED` (`ARCHITECTURE.md:335` region). Today CI builds Release/INFO
|
||
with `MOBILEGL_BUILD_DISAGGREGATED` OFF (`test.yml:71-93`, `CMakeLists.txt:23`), so **both the verify
|
||
comparator and the poison would be compiled out of every existing job**. A job-env `MOBILEGL_PIPE_VERIFY=1`
|
||
against today's binary is a no-op that looks green — the exact failure mode this project keeps writing
|
||
comments about.
|
||
|
||
So P1 needs a fourth build job:
|
||
|
||
```yaml
|
||
build-linux-verify:
|
||
runs-on: ubuntu-latest
|
||
permissions: { actions: write, contents: read }
|
||
env:
|
||
BUILD_DIR: build-verify
|
||
CCACHE_DIR: ${{ github.workspace }}/.ccache
|
||
CCACHE_BASEDIR: ${{ github.workspace }}
|
||
CCACHE_COMPRESS: "true"
|
||
CCACHE_MAXSIZE: 4G
|
||
CCACHE_NOHASHDIR: "true"
|
||
# steps 1-8 copied verbatim from build-linux (swap, checkout --recursive, get-cmake,
|
||
# ccache restore with key ${{ runner.os }}-test-${{ github.job }}-ccache-v1,
|
||
# Vulkan SDK, update_glslang_sources.py, apt deps, toolchain banner)
|
||
# Configure: build-linux's flags PLUS
|
||
# -DMOBILEGL_PIPE_VERIFY=ON # new option: compiles in SnapshotFromGLContext + G4
|
||
# -DMOBILEGL_BUILD_DISAGGREGATED=ON # arms the per-verb poison; needs 3rdparty/flatbuffers
|
||
# Package as ci-artifacts/mobilegl-linux-runtime-verify.tgz, same file list as :124-147
|
||
# Upload as artifact `mobilegl-linux-runtime-verify`
|
||
```
|
||
Notes:
|
||
- `MOBILEGL_BUILD_DISAGGREGATED=ON` also compiles `MG_Remote/**` and registers the `MG_Test/Wire` suite
|
||
(`CMakeLists.txt:454-469`, `MG_Test/CMakeLists.txt:93-95`); it is force-reset to OFF if
|
||
`3rdparty/flatbuffers/include` is missing (`CMakeLists.txt:440-451`), which with `submodules: recursive` it
|
||
is not. If you would rather not couple the two, give `MOBILEGL_PIPE_VERIFY=ON` its own
|
||
`-DMOBILEGL_PIPE_POISON=1` and make the generated `MGPipeInputPoisonFatal` call sites depend on
|
||
`MOBILEGL_DEBUG || MOBILEGL_BUILD_DISAGGREGATED || MOBILEGL_PIPE_VERIFY`.
|
||
- Keep `-DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO`. Do **not** build the verify lane at
|
||
`..._DEBUG`: that would turn on every `MOBILEGL_ASSERT` in the tree (`Defines.h:104`) and change what the
|
||
lane is measuring. Verify's own reporting must therefore use `MGLOG_E`/`MGLOG_F`, never `MGLOG_D`.
|
||
- Its own ccache key. Do not share `build-linux`'s: different `-D` flags, different objects.
|
||
|
||
### B1.1 Registering the verify entries — do NOT reuse the existing ones with job env
|
||
Job env *would* reach the process (proved by `test.yml:274-288`), but two things break:
|
||
- the per-test `TIMEOUT 120` (`MG_IntegrationTest/CMakeLists.txt:393`) is a property, so `ctest --timeout`
|
||
cannot raise it, and 5–10× slower scenarios will start timing out;
|
||
- there is no ctest-visible name that says "this was the verify run", so `ctest -N` parity checks and the
|
||
retrace-summary style reporting cannot tell the two apart.
|
||
|
||
Instead add a fourteenth and fifteenth registration in
|
||
`MobileGL/MG_IntegrationTest/CMakeLists.txt`, following the file's own idiom exactly:
|
||
|
||
```cmake
|
||
# The third CI mode (ARCHITECTURE.md:502): the same scenarios, with the per-draw per-field
|
||
# shadow comparison armed. Registered rather than driven by job env for two reasons: a ctest
|
||
# ENVIRONMENT property is the only spelling that survives `mgl_itest_join_environment`'s
|
||
# escaping intact (CMakeLists.txt:301-306), and these entries need a TIMEOUT of their own -
|
||
# verify is 5-10x slower and the ambient 120 s would start killing scenarios.
|
||
option(MOBILEGL_PIPE_VERIFY "Compile in SnapshotFromGLContext() and the G4 comparators" OFF)
|
||
set(MGL_ITEST_VERIFY_TIMEOUT 900)
|
||
|
||
if (MOBILEGL_PIPE_VERIFY)
|
||
mgl_itest_join_environment(MGL_ITEST_GLES_VERIFY_ENVIRONMENT
|
||
"MOBILEGL_BACKEND_TYPE=DirectGLES" "MOBILEGL_PIPE_VERIFY=1"
|
||
"MOBILEGL_LOG_FILE_PATH=${CMAKE_CURRENT_BINARY_DIR}/pipe-verify-gles.log"
|
||
${MGL_ITEST_COMMON_ENV})
|
||
mgl_itest_join_environment(MGL_ITEST_VULKAN_VERIFY_ENVIRONMENT
|
||
"MOBILEGL_BACKEND_TYPE=DirectVulkan" "MOBILEGL_PIPE_VERIFY=1"
|
||
"MOBILEGL_LOG_FILE_PATH=${CMAKE_CURRENT_BINARY_DIR}/pipe-verify-vulkan.log"
|
||
${MGL_ITEST_VULKAN_ENV})
|
||
|
||
gtest_discover_tests(MobileGLIntegrationTest
|
||
TEST_PREFIX "DirectGLES.Verify."
|
||
DISCOVERY_TIMEOUT 30
|
||
PROPERTIES LABELS "integration-gpu;integration-verify"
|
||
TIMEOUT ${MGL_ITEST_VERIFY_TIMEOUT}
|
||
ENVIRONMENT "${MGL_ITEST_GLES_VERIFY_ENVIRONMENT}")
|
||
gtest_discover_tests(MobileGLIntegrationTest
|
||
TEST_PREFIX "DirectVulkan.Verify."
|
||
DISCOVERY_TIMEOUT 30
|
||
PROPERTIES LABELS "integration-gpu;integration-verify"
|
||
TIMEOUT ${MGL_ITEST_VERIFY_TIMEOUT}
|
||
ENVIRONMENT "${MGL_ITEST_VULKAN_VERIFY_ENVIRONMENT}")
|
||
endif()
|
||
```
|
||
Points of care, each one a rake this repo has already stepped on:
|
||
- **The `${MGL_ITEST_COMMON_ENV}` / `${MGL_ITEST_VULKAN_ENV}` tail is mandatory** (`CMakeLists.txt:339-344`).
|
||
Omitting it loses the EGL vendor pin, the lavapipe ICD pin, `MOBILEGL_ITEST_REQUIRE_GPU`, and the three
|
||
iterationRP repairs — and the lane would still look green.
|
||
- Two labels: keep `integration-gpu` so nothing that enumerates the lane loses these, and add
|
||
`integration-verify` so a job can select only them. A `;`-separated LABELS value inside a
|
||
`gtest_discover_tests PROPERTIES` list needs the same escaping treatment as ENVIRONMENT if it ever grows a
|
||
variable — safest is to write it as `LABELS integration-gpu integration-verify` (multi-value properties
|
||
accept repeated tokens in `gtest_discover_tests`); verify with `ctest -N -L integration-verify` in the first
|
||
build.
|
||
- **Name parity**: `DirectGLES.Verify.<Suite>.<Case>` keeps the suffix identical to `DirectGLES.<Suite>.<Case>`,
|
||
so `ctest -N -L integration-gpu | sed 's/\.Verify\././'` parity is mechanical — the same test
|
||
`ARCHITECTURE.md:507-3` demands between `DirectGLES.` and `DirectGLES.Pipe.`/`DirectGLES.Split.` later.
|
||
- **Arming assertion** (otherwise the lane is unfalsifiable, the recurring failure of this suite):
|
||
add one scenario, e.g. `Scenarios/PipeVerifyArmingScenario.cpp`, that
|
||
(a) skips unless `AmbientQuirkFromEnvironment("MOBILEGL_PIPE_VERIFY") == AmbientQuirk::On`
|
||
(`ScenarioFixture.h:56-65`),
|
||
(b) skips with an explicit message when `MOBILEGL_LOG_FILE_PATH` is unset (copy
|
||
`UnlocatedIoBlockScenario.cpp:354-360` and its message at `:489`),
|
||
(c) draws once and asserts the library's latched `MGLOG_I("MGPipe: verify armed, N fields, K payloads")`
|
||
appears in the bytes written after the case started, and that no `Fatal{PipeVerifyDiffer` line does.
|
||
This is what turns "the option was compiled out" and "the build ran with `MOBILEGL_PIPE_VERIFY` spelled
|
||
wrong" from silent green into red.
|
||
|
||
### B1.2 The CI job
|
||
```yaml
|
||
integration-verify:
|
||
runs-on: ubuntu-latest
|
||
needs: build-linux-verify
|
||
steps:
|
||
# identical to `integration` (test.yml:205-296) except:
|
||
# - download/unpack artifact `mobilegl-linux-runtime-verify` into build-verify/
|
||
# - repeat the "Normalize CTest command paths" step over build-verify (test.yml:234-244)
|
||
# - working-directory: build-verify
|
||
env:
|
||
MOBILEGL_ITEST_REQUIRE_GPU: "1"
|
||
MOBILEGL_MAGMA_FIX_ITERATIONRP_SUBGROUP_SCRATCH: "1"
|
||
MOBILEGL_MAGMA_DERIVE_NUM_SUBGROUPS: "1"
|
||
MOBILEGL_MAGMA_ITERATIONRP_FIX_BARRIER: "1"
|
||
run: |
|
||
ulimit -c unlimited
|
||
sudo sysctl -w kernel.core_pattern='/tmp/core.%e.%p'
|
||
ctest --output-on-failure -L integration-verify --no-tests=error
|
||
```
|
||
`--no-tests=error` is doing real work here: if the option was OFF at configure the label matches nothing and
|
||
the job fails instead of passing empty. Keep the `integration-core-dumps` upload — a verify divergence that
|
||
aborts leaves a core, and that core names the diverging draw.
|
||
|
||
### B1.3 Cost
|
||
- Entries: 371 × 2 = **742 new ctest entries** (the verify lane covers the two ambient registrations; the 11
|
||
filtered mode lanes stay pull-only — they pin *other* quirks and doubling them would multiply the matrix for
|
||
no new information).
|
||
- Each entry is one process: full EGL/Vulkan bring-up plus one `fork()` pre-flight (`HeadlessGL.cpp:323-343`).
|
||
Process startup dominates today; verify multiplies only the in-scenario GL work by the documented 5–10×
|
||
(`ARCHITECTURE.md:502`). Expect the job to land somewhere between 2× and 5× the existing `integration`
|
||
job's wall time — measure it on the first run and set `timeout-minutes` from that (the whole file has none
|
||
today, so a wedged verify job would burn 6 h).
|
||
- If it proves too slow for every push: split by backend into two jobs (`-L integration-verify -R '^DirectGLES\.'`
|
||
and `^DirectVulkan\.`), which halves wall time at no extra runner-minute cost.
|
||
|
||
### B1.4 Locally
|
||
```
|
||
cmake -S . -B build-verify -G Ninja \
|
||
-DCMAKE_BUILD_TYPE=Release -DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO \
|
||
-DMOBILEGL_BUILD_TEST=ON -DMOBILEGL_BUILD_INTEGRATION_TEST=ON \
|
||
-DMOBILEGL_PIPE_VERIFY=ON -DMOBILEGL_BUILD_DISAGGREGATED=ON \
|
||
-DMOBILEGL_ITEST_REQUIRE_GPU=ON \
|
||
-DMOBILEGL_ITEST_EGL_VENDOR=/usr/share/glvnd/egl_vendor.d/<vendor>.json \
|
||
-DMOBILEGL_ITEST_VK_ICD=/usr/share/vulkan/icd.d/<vendor>.json
|
||
cmake --build build-verify --parallel "$(nproc)"
|
||
ctest --test-dir build-verify -L integration-verify --output-on-failure
|
||
```
|
||
`-DMOBILEGL_ITEST_REQUIRE_GPU=ON` here (not env) because locally the option bakes it into the property
|
||
(`MG_IntegrationTest/CMakeLists.txt:279-281`) and you want the same thing CI gets. Do **not** use
|
||
`run_integration_test.sh` for this: it carries neither the mode variables nor the iterationRP pins.
|
||
|
||
## B2. Running the retrace gate a third time under verify
|
||
|
||
### B2.1 The good news: no CMake change is needed for the env
|
||
No trace test's `ENVIRONMENT` property names any `MOBILEGL_*` variable
|
||
(`tools/trace_replay/CMakeLists.txt:373-379`), and the existing job already exports three of them
|
||
(`test.yml:705-722`). So `export MOBILEGL_PIPE_VERIFY=1` in the job step reaches the replay process, through
|
||
`run_trace_case.cmake`'s `execute_process` (`:54-74`), which inherits.
|
||
|
||
### B2.2 The bad news: the timeout, the library, and the silence
|
||
1. **Timeout.** Trace tests have no `TIMEOUT` property, so they run under ctest's 1500 s default. Because the
|
||
property is absent, `ctest --timeout` *does* apply — this is the one place it is unambiguous:
|
||
`ctest -V --no-tests=error --timeout 10800 -R '^MobileGLTraceReplay\.…$'`.
|
||
Also raise the job's own `timeout-minutes` (there is none today).
|
||
2. **Which library.** The trace `CTestTestfile.cmake` hard-codes `-DMOBILEGL_LIBRARY=<workspace>/build-linux/
|
||
libMobileGL.so` (baked at configure from `test.yml:424` / `:501`). The cheapest correct trick: in the
|
||
verify retrace job, unpack the **verify** runtime tarball so that the verify `libMobileGL.so` lands at
|
||
`build-linux/libMobileGL.so`, and unpack `mobilegl-trace-replay` unchanged. The replay binary itself does
|
||
not need rebuilding — it `dlopen`s the library named on the command line. Assert the swap actually happened
|
||
(`test -f build-linux/libMobileGL.so && nm -D build-linux/libMobileGL.so | grep -q MGPipeVerify` or the
|
||
arming line, see 3).
|
||
3. **Silence.** Nothing today makes a verify divergence red. Decide and implement one contract:
|
||
- **Recommended**: divergence is fatal by default — `MGLOG_F("MGPipe: Fatal{PipeVerifyDiffer, \"%s@%s\",
|
||
draw=%llu}", field, verb, serial)` then `std::abort()`, exactly like `MGPipeInputPoisonFatal`
|
||
(`PipeFilled.inc:298-302`). The replay process dies with SIGABRT → `replay_result != 0` →
|
||
`run_trace_case.cmake:127-129` `FATAL_ERROR` → red ctest test, and `/tmp/core.*` is uploaded by the
|
||
existing failure step. One env escape hatch, `MOBILEGL_PIPE_VERIFY_FATAL=0` (log-and-count, for triage
|
||
and for the integration arming lane which must survive to read its own log), added next to
|
||
`features.PipeVerify` in `ConfigLoader.cpp:246`.
|
||
- **Plus an arming check in the harness**, so a build with the option off cannot pass this lane silently.
|
||
Add to `run_trace_case.cmake`, guarded on a new `-DTRACE_PIPE_VERIFY=ON` passed only by the verify job:
|
||
```cmake
|
||
if(TRACE_PIPE_VERIFY)
|
||
if(NOT EXISTS "${mobilegl_log}")
|
||
message(FATAL_ERROR "verify run produced no mobilegl.log")
|
||
endif()
|
||
file(STRINGS "${mobilegl_log}" armed REGEX "MGPipe: verify armed")
|
||
if(NOT armed)
|
||
message(FATAL_ERROR "MOBILEGL_PIPE_VERIFY=1 but the library never reported arming")
|
||
endif()
|
||
file(STRINGS "${mobilegl_log}" differ REGEX "Fatal\\{PipeVerifyDiffer")
|
||
if(differ)
|
||
message(FATAL_ERROR "verify divergence: ${differ}")
|
||
endif()
|
||
endif()
|
||
```
|
||
`mobilegl_log` is already defined at `run_trace_case.cmake:80` and already copied to the artifact dir at
|
||
`:107-109`. `-DTRACE_PIPE_VERIFY` needs one line in `add_trace_replay_test`
|
||
(`tools/trace_replay/CMakeLists.txt:350-372`) forwarding a cache variable, or simply
|
||
`-DTRACE_PIPE_VERIFY=$ENV{MOBILEGL_PIPE_VERIFY}`-style detection inside the script
|
||
(`if(DEFINED ENV{MOBILEGL_PIPE_VERIFY} AND NOT "$ENV{MOBILEGL_PIPE_VERIFY}" STREQUAL "0")`) — the latter
|
||
needs no registration change at all and is the smaller diff.
|
||
|
||
### B2.3 Job shape
|
||
```yaml
|
||
retrace-verify:
|
||
name: retrace verify (${{ matrix.backend }}, ${{ matrix.case }})
|
||
runs-on: ubuntu-latest
|
||
needs: [build-linux-verify, build-retrace, trace-cases, trace-fixtures]
|
||
if: ${{ always() && needs.build-linux-verify.result == 'success' && needs.build-retrace.result == 'success' }}
|
||
timeout-minutes: 240
|
||
strategy:
|
||
fail-fast: false
|
||
max-parallel: 4
|
||
matrix: ${{ fromJSON(needs.trace-cases.outputs.matrix) }} # or a reduced matrix, see below
|
||
steps:
|
||
# identical to `retrace` (test.yml:651-741) except:
|
||
# - download `mobilegl-linux-runtime-verify` and unpack it so the verify .so lands at
|
||
# build-linux/libMobileGL.so
|
||
# - export MOBILEGL_PIPE_VERIFY=1 alongside the existing per-case exports
|
||
# - ctest ... --timeout 10800
|
||
# - artifact names suffixed -verify so retrace-summary and remove-artifact-clutter
|
||
# (test.yml:784-833, which greps job names beginning "retrace (") are unaffected
|
||
```
|
||
It reuses the **same** `trace-fixture-<case>` artifacts the `retrace` matrix downloads, so the ~800 MB of
|
||
fixtures (94 files in `tools/trace_replay/fixtures/`, individual `.tgz` 5–25 MB) are fetched once per run, not
|
||
twice — provided `retrace-verify` runs in the same workflow run and before `remove-artifact-clutter`. Add
|
||
`retrace-verify` to that job's `needs` (currently `needs: retrace-summary`, `test.yml:787`) or it will delete
|
||
the fixtures out from under the verify matrix.
|
||
|
||
### B2.4 Cost and cadence
|
||
- Full third pass = **+77 matrix jobs** (or 79/80 locally, see A4), at `max-parallel: 4` ≈ 20 sequential waves,
|
||
each wave running a case that already takes minutes, now 5–10× slower. This is not a per-push cost anyone
|
||
should pay.
|
||
- Recommended cadence, which still satisfies `ROADMAP.md:17` ("40 trace … 零分歧" is an *exit* criterion):
|
||
- **per push on `feat/disaggregated`**: a reduced verify matrix of ~8 cases × 2 backends, chosen for path
|
||
coverage rather than pixels — `OpenRA`, `minecraft-1.21.4-startup`, `minecraft-1.21.4-main-menu`,
|
||
`minecraft-1.21.4-in-world`, `minecraft-1.21.4-fabric-sodium-in-world`, one Iris case,
|
||
`minecraft-1.21.1-neoforge-create-indirect-in-world`, `improved-transparency-minecraft-26.3`. Express it as
|
||
a new `trace_cases.py --format github-verify-matrix` (or a `"verify": true` key in
|
||
`trace_cases.json`) so the selection is data, not YAML, and `pipe-gates`' regeneration discipline covers it.
|
||
- **nightly / `workflow_dispatch`**: the full matrix.
|
||
- **P1 exit and every stage boundary**: the full matrix, recorded in `MEASUREMENTS.md`.
|
||
- Known non-P1 red to exclude or expect: `minecraft-1.21.1-neoforge-create-indirect-in-world` fails on both
|
||
devices on the `dev@81b17c0b` baseline (`MEASUREMENTS.md` §5.5) — it is a device-lane fact, not a Linux-lane
|
||
one, but keep it in mind before attributing a verify red to P1.
|
||
|
||
### B2.5 Locally
|
||
```
|
||
cmake -S . -B build-retrace -G Ninja -DCMAKE_BUILD_TYPE=Release \
|
||
-DMOBILEGL_BUILD_TEST=OFF -DMOBILEGL_BUILD_BENCHMARK=OFF -DMOBILEGL_BUILD_TRACE_REPLAY=ON \
|
||
-DMOBILEGL_TRACE_REPLAY_MOBILEGL_LIBRARY=$PWD/build-verify/libMobileGL.so
|
||
cmake --build build-retrace --target mobilegl_trace_replay --parallel "$(nproc)"
|
||
MOBILEGL_PIPE_VERIFY=1 ctest --test-dir build-retrace/tools/trace_replay \
|
||
-R '^MobileGLTraceReplay\.' --timeout 10800 --output-on-failure -j2
|
||
```
|
||
(`-j2`, not `-j$(nproc)`: each case unpacks a multi-GB trace into `TRACE_OUTPUT_DIR`, which
|
||
`run_trace_case.cmake:31-33` wipes per run.)
|
||
While the trace tests carry no label, consider adding `LABELS retrace` in
|
||
`tools/trace_replay/CMakeLists.txt:374-379` as a one-line P1 quality-of-life change — every other lane in the
|
||
tree is label-selectable and this one is not.
|
||
|
||
## B3. Negative control 1 — "corrupting one snapshot field turns verify red"
|
||
|
||
Build it in three layers, cheapest first. Layers 1 and 2 run on **every** CI; layer 3 runs in the verify lanes.
|
||
|
||
### The injection knob (shared by all three layers)
|
||
Add, next to `features.PipeVerify` (`ConfigLoader.cpp:246`) and compiled only under
|
||
`#if MOBILEGL_PIPE_VERIFY` so it can never exist in a shipped build:
|
||
```
|
||
MOBILEGL_PIPE_VERIFY_CORRUPT=<FieldName>[.<Member>] # perturb exactly this field in the SNAPSHOT arm
|
||
MOBILEGL_PIPE_VERIFY_FATAL=0|1 # default 1: first divergence aborts
|
||
```
|
||
`<FieldName>` is one of the 61 names already generated into `kMGPipeInputFieldNames`
|
||
(`PipeFilled.inc`, list emitted by `gen_pipe.py:473-477`), so the knob's vocabulary is generated, not
|
||
hand-typed, and a renamed accessor breaks the control loudly. Perturb the **snapshot** side, never the pushed
|
||
side: corrupting the pushed side would also change what the backend renders and the control would be testing
|
||
two things at once.
|
||
|
||
### Layer 1 — unit, no GPU, `MG_Test/Pipe` (runs in the `test` job today)
|
||
Extend `MobileGL/MG_Test/Pipe/PipeCatalogueTest.cpp` (the comparator precedent is at `:184-207`) with a test
|
||
that drives the harness entry point rather than a payload comparator:
|
||
```cpp
|
||
TEST(PipeVerify, CorruptedSnapshotFieldIsNamedWithItsDrawSerial) {
|
||
PipeInputs pushed{}; PipeInputs snapshot{};
|
||
FillBothWithAKnownState(pushed, snapshot);
|
||
const char* field = nullptr; Uint64 serial = 0;
|
||
EXPECT_TRUE(MGPipeVerifyInputs(pushed, snapshot, &field, &serial)); // clean arm first
|
||
ApplyVerifyCorruption(snapshot, "GetRenderStateParameters"); // the injector, not env
|
||
EXPECT_FALSE(MGPipeVerifyInputs(pushed, snapshot, &field, &serial));
|
||
EXPECT_STREQ(field, "GetRenderStateParameters");
|
||
}
|
||
```
|
||
The clean arm is not decoration: without it the test passes when the comparator is stuck returning false.
|
||
**Design requirement this imposes on P1**: expose the comparison as a pure function over two `PipeInputs`
|
||
(`Bool MGPipeVerifyInputs(const PipeInputs&, const PipeInputs&, const char** outField, Uint64* outSerial)`)
|
||
and the corruption as a callable (`ApplyVerifyCorruption(PipeInputs&, const char* fieldName)`) that the env
|
||
knob merely calls. If the only way to reach verify is a live GL context, this whole layer is impossible.
|
||
Registration: `PipeCatalogueTest` is already `LABELS unit` (`MG_Test/Pipe/CMakeLists.txt:28`) — but the new
|
||
test only compiles when `MOBILEGL_PIPE_VERIFY` is ON, so either guard the test body with
|
||
`#if MOBILEGL_PIPE_VERIFY … #else GTEST_SKIP() …` (visible skip, honest) or add a
|
||
`PipeVerifyTest` target registered inside `if (MOBILEGL_PIPE_VERIFY)`. Prefer the guard: a test that
|
||
disappears is a test nobody notices is gone.
|
||
|
||
### Layer 2 — end-to-end "the gate goes red", cheapest possible
|
||
One extra step in `integration-verify`, run after the green pass:
|
||
```yaml
|
||
- name: Negative control - a corrupted snapshot field must turn the lane red
|
||
working-directory: build-verify
|
||
env:
|
||
MOBILEGL_ITEST_REQUIRE_GPU: "1"
|
||
MOBILEGL_PIPE_VERIFY_CORRUPT: GetRenderStateParameters
|
||
run: |
|
||
if ctest -L integration-verify -R 'ClearThenReadPixelsScenario' --no-tests=error; then
|
||
echo "::error::verify did not go red with a corrupted snapshot field"; exit 1
|
||
fi
|
||
echo "negative control tripped as expected"
|
||
```
|
||
`MOBILEGL_PIPE_VERIFY_CORRUPT` is named by no `ENVIRONMENT` property, so the job env reaches the process —
|
||
the `MOBILEGL_ESPRYT_DISABLE_INVALIDATE_FLUSH` precedent (`test.yml:274-288`) verbatim. Pick a short,
|
||
always-drawing scenario (`ClearThenReadPixelsScenario`) so this costs seconds.
|
||
The identical step belongs in the retrace verify job on one cheap case (`OpenRA`, 180 s timeout,
|
||
`trace_cases.json:14-27`), inverted the same way.
|
||
|
||
### Layer 3 — the in-suite version, so it survives outside CI too
|
||
A `PipeVerifyArmingScenario` case that runs only when `MOBILEGL_PIPE_VERIFY_CORRUPT` is set, with
|
||
`MOBILEGL_PIPE_VERIFY_FATAL=0`, draws once, and asserts the per-lane log contains
|
||
`Fatal{PipeVerifyDiffer, "GetRenderStateParameters@…"` — the `MOBILEGL_LOG_FILE_PATH` idiom from
|
||
`UnlocatedIoBlockScenario.cpp:354-360`. Register it as its own tiny lane
|
||
(`DirectGLES.VerifyCorrupted.` / `DirectVulkan.VerifyCorrupted.`, `TEST_FILTER
|
||
"PipeVerifyArmingScenario.CorruptedFieldIsReported"`) with `MOBILEGL_PIPE_VERIFY_CORRUPT` and
|
||
`MOBILEGL_PIPE_VERIFY_FATAL=0` in its ENVIRONMENT — again appending `${MGL_ITEST_COMMON_ENV}`.
|
||
Two entries, seconds of runtime, and it makes the control reproducible with one `ctest -R` on a dev box.
|
||
|
||
## B4. Negative control 2 — "omitting a field in `glGenerateMipmap`'s fill table triggers the poison Fatal on that verb"
|
||
|
||
### The injection knob
|
||
`MOBILEGL_PIPE_POISON_OMIT=<CallName>:<FieldName>` (e.g. `GenerateMipmap:GetPixelStoreParameters`), read next
|
||
to `features.PipeVerify` and compiled only where the poison is compiled. The generated per-verb filler
|
||
(G5, `PipeFilled.inc` + the ~93 boundary fill sites, `ROADMAP.md:17`) consults it and skips *stamping* that one
|
||
field's `FilledGen` for that one call — it must skip the **stamp**, not the value, so the omission is
|
||
indistinguishable from a filler that genuinely forgot. `<CallName>` is validated against the catalogue
|
||
(`PipeCalls.def`, opcode enum in `PipeWire.inc`) and `<FieldName>` against `kMGPipeInputFieldNames`; an
|
||
unknown name is itself a Fatal, so a typo cannot silently disarm the control.
|
||
|
||
### Layer 1 — unit, no GPU, no death test (runs in the `test` job today)
|
||
Extend `PipeCatalogueTest.cpp:224-235` (`PipeInputFieldsStartUnfilled` already establishes the pattern):
|
||
```cpp
|
||
TEST(PipeCatalogue, OmittingOneFieldForOneVerbLeavesExactlyThatFieldStale) {
|
||
MGPipeFilledState state{};
|
||
state.CurrentVerbSerial = 7;
|
||
FillForVerb(state, MGPWireOp::GenerateMipmap); // the generated filler
|
||
EXPECT_TRUE(MGPipeInputFieldIsFresh(state, kOmittedField)); // clean arm
|
||
EXPECT_TRUE(MGPipeInputFieldIsFresh(state, kSiblingField));
|
||
|
||
MGPipeFilledState omitted{};
|
||
omitted.CurrentVerbSerial = 7;
|
||
SetPoisonOmission("GenerateMipmap", "GetPixelStoreParameters");
|
||
FillForVerb(omitted, MGPWireOp::GenerateMipmap);
|
||
EXPECT_FALSE(MGPipeInputFieldIsFresh(omitted, kOmittedField)); // the control
|
||
EXPECT_TRUE (MGPipeInputFieldIsFresh(omitted, kSiblingField)); // and only that one
|
||
}
|
||
```
|
||
This is the layer that proves "on **that** verb": the sibling assertion is what distinguishes a targeted
|
||
omission from a filler that stopped working entirely.
|
||
|
||
### Layer 2 — the Fatal actually fires and kills the process
|
||
The tree has **zero** gtest death tests (A6), and its own idiom for observing a child's death is
|
||
`fork()`/`waitpid()` (`MG_Test/Wire/FdPassingTest.cpp:62-132`, `HeadlessGL.cpp:323-423`). Use that, not
|
||
`EXPECT_DEATH`:
|
||
```cpp
|
||
TEST(PipeCatalogue, ReadingAnOmittedFieldAbortsNamingTheVerb) {
|
||
#if !defined(MOBILEGL_PIPE_POISON)
|
||
GTEST_SKIP() << "poison compiled out (needs MOBILEGL_DEBUG or MOBILEGL_BUILD_DISAGGREGATED)";
|
||
#else
|
||
const pid_t pid = ::fork();
|
||
ASSERT_NE(pid, -1);
|
||
if (pid == 0) {
|
||
MGPipeFilledState state{}; state.CurrentVerbSerial = 1; // nothing filled
|
||
if (!MGPipeInputFieldIsFresh(state, MGPipeInputField::GetPixelStoreParameters)) {
|
||
MGPipeInputPoisonFatal(MGPipeInputField::GetPixelStoreParameters, "GenerateMipmap");
|
||
}
|
||
::_exit(0); // reaching here is the failure
|
||
}
|
||
int status = 0;
|
||
ASSERT_EQ(::waitpid(pid, &status, 0), pid);
|
||
EXPECT_TRUE(WIFSIGNALED(status)) << "the poison did not terminate the process";
|
||
EXPECT_EQ(WTERMSIG(status), SIGABRT);
|
||
#endif
|
||
}
|
||
```
|
||
Why not `EXPECT_DEATH`: it would work on Linux, but (a) nothing in this tree uses it, so nobody maintains the
|
||
death-test-style/threading constraints, (b) the "fast" style forks a process that has already initialized
|
||
gtest and any static GL state, and (c) `_exit(0)` on the non-fatal path gives a *positive* failure signal that
|
||
`EXPECT_DEATH`'s "died as expected" phrasing hides. Pair it with a clean-arm sibling test (a *filled* field
|
||
must let the child reach `_exit(0)`), or the test passes whenever `MGPipeInputFieldIsFresh` is broken in the
|
||
false direction.
|
||
To also assert the **message** (the `@GenerateMipmap` half): point `MOBILEGL_LOG_FILE_PATH` at a temp file in
|
||
the suite's own `main()` — the `LogLevelTest` shape (`MG_Test/Util/CMakeLists.txt:19-22`,
|
||
`LogLevelTest.cpp:76-107`) — and after reaping, grep the file for
|
||
`Fatal{UnmigratedPipeInput, "GetPixelStoreParameters@GenerateMipmap"}`, the exact string
|
||
`PipeFilled.inc:299-300` formats. Note `MOBILEGL_LOG_ENABLE_CONSOLE` is 0 (`Defines.h:71`), so the file is
|
||
the only channel; and the child must let the sink flush before aborting (the `MGLOG_F` write precedes
|
||
`std::abort()`, but confirm the file sink is unbuffered or flushes per line in `MG_Util/Debug/Log.cpp`).
|
||
|
||
### Layer 3 — on the real verb, through the real backend, per backend
|
||
An integration scenario `PoisonOmissionScenario` in the verify build: parent forks, child sets up a texture and
|
||
calls `glGenerateMipmap`, parent asserts `WIFSIGNALED && WTERMSIG == SIGABRT` and that the per-lane log carries
|
||
`…@GenerateMipmap`. Registered as its own lane with
|
||
`MOBILEGL_PIPE_POISON_OMIT=GenerateMipmap:<field>` + `MOBILEGL_LOG_FILE_PATH=<per-lane>` in ENVIRONMENT
|
||
(appending `${MGL_ITEST_COMMON_ENV}` / `${MGL_ITEST_VULKAN_ENV}`), two entries, one per backend. The forking
|
||
is already precedented inside this module (`HeadlessGL.cpp:343`), and the GL context is per-process so the
|
||
child inherits a usable one — but fork the child **before** any scenario-local GPU work and keep the child's
|
||
work minimal, and use `_exit`, never `exit` (the reason is written at `HeadlessGL.cpp:365-368`).
|
||
Its complement — the same scenario with the omission unset, expecting `_exit(0)` and an empty log — is what
|
||
keeps this from passing on a build where `glGenerateMipmap` aborts for some unrelated reason.
|
||
|
||
### Where these gates live
|
||
Both controls' layer-1 tests belong in `MG_Test/Pipe` (label `unit`, already in the `test` job). Both
|
||
controls' layer-2 CI steps belong in the verify jobs. Register the whole thing the way
|
||
`check_include_closure.py --self-test` is registered — **always on**, not behind a manual flag
|
||
(`test.yml:330-331`, `Purity/CMakeLists.txt:9-13`). If the omission/corruption knobs ever become optional,
|
||
add the `trips == 0 → error` check (`check_include_closure.py:541-543`) so a control that stopped tripping
|
||
fails instead of silently reporting nothing.
|
||
|
||
## B5. `nm --defined-only` / `.text` attribution for P1's "pull build unchanged"
|
||
|
||
### What the acceptance actually says
|
||
`ROADMAP.md:17`: **pull** build `nm --defined-only` unchanged, `.text` differences attributable line by line,
|
||
with the empty-guard / ternary rewrites explicitly deferred to P2. `ARCHITECTURE.md:507` adds that byte
|
||
identity is dead as a general gate but two byte-level equalities survive, and `ARCHITECTURE.md:568` reserves
|
||
the job name `monolith-symbol-report`.
|
||
|
||
Why this is even achievable: phase A is a mechanical `sed` of `MG_State::pGLContext->` → `MGB_CTX->` at 293
|
||
sites (`ARCHITECTURE.md:337-341`), and in the **pull** build `MGB_CTX` expands to `::MG_State::pGLContext`
|
||
(`ARCHITECTURE.md:335`), so the preprocessed token stream is identical. The only expected `.text` movement
|
||
comes from the 58 hand-converted non-arrow uses — ~34 deleted `MOBILEGL_ASSERT`s (already no-ops in an INFO
|
||
build, so **zero** `.text` effect there), 7 empty guards, 3 ternaries, `.get()` captures, 14 `!= nullptr`,
|
||
1 comment.
|
||
|
||
### Concrete procedure
|
||
```bash
|
||
# both sides: same compiler, same build type, LTO off (CMakeLists.txt:108 default OFF), not stripped
|
||
for ref in <P0.5-baseline-sha> <P1-head-sha>; do
|
||
git -C <worktree> checkout "$ref"
|
||
cmake -S . -B build-sym-$ref -G Ninja -DCMAKE_BUILD_TYPE=Release \
|
||
-DMOBILEGL_LOG_ACTIVE_LEVEL=MOBILEGL_LOG_LEVEL_INFO \
|
||
-DMOBILEGL_BUILD_TEST=OFF -DMOBILEGL_BUILD_BENCHMARK=OFF \
|
||
-DMOBILEGL_BUILD_INTEGRATION_TEST=OFF -DMOBILEGL_BUILD_TRACE_REPLAY=OFF \
|
||
-DMOBILEGL_BUILD_DISAGGREGATED=OFF -DMOBILEGL_PIPE_VERIFY=OFF -DMOBILEGL_ENABLE_LTO=OFF \
|
||
-DCMAKE_C_COMPILER=clang-20 -DCMAKE_CXX_COMPILER=clang++-20
|
||
cmake --build build-sym-$ref --target MobileGL --parallel "$(nproc)"
|
||
done
|
||
python3 scripts/symbol_report.py \
|
||
--before build-sym-<baseline>/libMobileGL.so \
|
||
--after build-sym-<p1>/libMobileGL.so \
|
||
--threshold 0 \
|
||
--markdown docs/Disaggregated/P1-symbol-attribution.md \
|
||
--json /tmp/p1-symbols.json
|
||
```
|
||
- **`-DMOBILEGL_BUILD_DISAGGREGATED=OFF` and `-DMOBILEGL_PIPE_VERIFY=OFF` are the whole point**: the gate is
|
||
about the *pull* build. Comparing a verify build against the P0.5 baseline is meaningless — verify compiles
|
||
in `SnapshotFromGLContext()` and the G4 comparators by construction (`ARCHITECTURE.md:579`).
|
||
- No `--strip-scope` / `--rename-map` here: unlike P0.5 (which de-nested five types and therefore renamed 42
|
||
mangled names, `ROADMAP.md:16`), P1 moves no type out of a scope. If the report shows a renamed-only bucket,
|
||
something unintended de-nested and that is itself a finding.
|
||
- `--only-names` is useful for a focused pass over the accessor names in `PipeInputs`, not for the gate.
|
||
|
||
### Making it a gate rather than a report
|
||
`symbol_report.py` always exits 0 (`:342`). Two options:
|
||
- **Minimal (recommended)**: keep the script informational and assert in the job on its `--json`:
|
||
```bash
|
||
python3 - <<'PY'
|
||
import json,sys
|
||
r=json.load(open('/tmp/p1-symbols.json'))
|
||
added=r['added']; removed=r['removed'] # confirm the exact key names against the writer
|
||
if added or removed:
|
||
print("::error::pull build symbol set changed: +%d / -%d" % (len(added),len(removed))); sys.exit(1)
|
||
PY
|
||
```
|
||
- **Fuller**: implement `--fail-on-added-bytes` (the flag is already declared, `:287-288`) and pass `0`.
|
||
Note this changes a tool P0.5 shipped as informational; if you do it, keep the default behaviour unchanged.
|
||
|
||
### The job
|
||
```yaml
|
||
monolith-symbol-report:
|
||
name: monolith symbol / .text attribution
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- checkout (submodules: recursive), get-cmake, apt clang-20 + binutils, Vulkan SDK,
|
||
update_glslang_sources.py # same prologue as build-linux
|
||
- build the baseline .so (git worktree at the pinned baseline sha) and the head .so with
|
||
the identical flags above; two ccache keys
|
||
- python3 scripts/symbol_report.py --before … --after … --markdown … --json …
|
||
- the json assertion above (P1 only; informational before and after)
|
||
- upload the markdown as an artifact
|
||
- the two surviving byte-level equalities (ARCHITECTURE.md:507):
|
||
nm --defined-only build-sym-head/libMobileGL.so | grep -q MG_Remote && exit 1 # must be empty
|
||
```
|
||
Because this needs two full builds it is the most expensive gate in the file. Run it on
|
||
`workflow_dispatch` + at stage exit, not per push. Pinning the baseline as a **stored artifact** (upload the
|
||
P0.5 `libMobileGL.so` once, download it here) halves the cost and removes the "was the baseline built the same
|
||
way" doubt — the guard rails at `symbol_report.py:23-31` exist precisely because that doubt is the usual way
|
||
this comparison goes wrong.
|
||
|
||
### The deliverable P1 owes
|
||
The `.text` "line-by-line attribution" is a document, not a script output: take the per-symbol size table from
|
||
the markdown and add a reason column drawn from the 58-site conversion list. Put it in
|
||
`docs/Disaggregated/` so `scripts/check_doc_citations.py` (`test.yml:876-887`) lints its citations, and cite
|
||
`ROADMAP.md:17` from it.
|
||
|
||
---
|
||
|
||
# Part C — trap list
|
||
|
||
1. **A ctest `ENVIRONMENT` property wins over the job env for the variables it names, and only those.**
|
||
`test.yml:252-259` (the ICD pin that must not be exported) vs `test.yml:274-288` (the flush flag that is
|
||
exported inline and does reach the process). `MOBILEGL_PIPE_VERIFY` is named by no property today, so
|
||
job-env works — but the moment you add it to a property, a job-level export of the same name becomes a lie.
|
||
2. **`ARCHITECTURE.md:567` and `MG_IntegrationTest/CMakeLists.txt:339-344` say the property "REPLACES" the job
|
||
env.** Whatever the precise CMake semantics, the project's rule is: every mode list appends
|
||
`${MGL_ITEST_COMMON_ENV}` / `${MGL_ITEST_VULKAN_ENV}`. Follow the rule; do not relitigate it in a P1 CMake diff.
|
||
3. **`;` inside a `gtest_discover_tests PROPERTIES` value must be escaped** — use
|
||
`mgl_itest_join_environment` (`MG_IntegrationTest/CMakeLists.txt:307-317`). Symptom of getting it wrong:
|
||
only the first variable survives and the pins vanish silently.
|
||
4. **`ctest --timeout` cannot raise a `TIMEOUT` property.** Integration entries have `TIMEOUT 120`
|
||
(`:393`) → new registrations need their own. Trace entries have none → `--timeout` works there, and they
|
||
currently run under ctest's 1500 s default even though `trace_cases.json` declares 1800 s for 17 cases.
|
||
5. **`MOBILEGL_ASSERT` is a no-op in every CI build** (`Defines.h:104-115`, `test.yml:85`). Neither negative
|
||
control may be expressed with it. The generated poison already gets this right (`MGLOG_F` + `std::abort()`,
|
||
`PipeFilled.inc:298-302`); the ~93 hand-written fill sites must too.
|
||
6. **`MGLOG_D` is compiled out at INFO** (`Log.h:59-64`). Verify's per-draw diagnostics must be `MGLOG_E`/`MGLOG_F`,
|
||
or you will build the verify lane at DEBUG and accidentally also turn on every assert in the tree.
|
||
7. **`MOBILEGL_LOG_ENABLE_CONSOLE` is 0** (`Defines.h:71`) and the desktop default log path is empty
|
||
(`Defines.h:85`). Any log-reading assertion needs `MOBILEGL_LOG_FILE_PATH` set in the lane's ENVIRONMENT —
|
||
per-lane, so nothing else appends (`MG_IntegrationTest/CMakeLists.txt:361-365`).
|
||
8. **The poison is compiled out of today's CI builds** (`MOBILEGL_DEBUG || MOBILEGL_BUILD_DISAGGREGATED`,
|
||
`ARCHITECTURE.md:335`; CI is Release/INFO with DISAGGREGATED OFF). Without a build change, the
|
||
`glGenerateMipmap` control is vacuous.
|
||
9. **Any job that unpacks the runtime tgz and runs ctest must repeat the "Normalize CTest command paths" step**
|
||
(`test.yml:174-184` and three copies).
|
||
10. **`remove-artifact-clutter` deletes `trace-fixture-*`** (`test.yml:784-833`) and currently only waits on
|
||
`retrace-summary`. Add any new retrace matrix to its `needs`.
|
||
11. **`feat/disaggregated` is not a push trigger** (`test.yml:3-9`). Either add it or drive P1's gates with
|
||
`workflow_dispatch`.
|
||
12. **No job in `test.yml` sets `timeout-minutes`.** A wedged verify job burns 6 h of runner time.
|
||
13. **`ENABLE_INTEGRATION_TESTS` is referenced at `MG_Test/CMakeLists.txt:100` and defined nowhere in the
|
||
tree** (`grep` over `*.txt`/`*.cmake`/`*.yml` finds only that use), so `MG_Test/Backend/DirectVulkan` is
|
||
never built by any lane. Not a P1 blocker, but if P1 wants a DirectVulkan-side unit test that variable is
|
||
a dead end.
|
||
14. **`gen_pipe.py` regeneration + `git diff --exit-code` is a gate** (`test.yml:850-853`). Any new generated
|
||
table (poison-omission vocabulary, verify field ids) must be emitted by `gen_pipe.py`, or the gate will
|
||
either miss it or go red on an untracked file.
|
||
15. **`pipe-gates`' stdio ban covers `MobileGL/MG_Backend` and `MobileGL/MG_State`** with no exceptions
|
||
(`test.yml:862-869`). The verify reporter and the fill sites must not print.
|
||
16. **`gen_pipe_dirty_surface.py --summary` becomes a gate in P1** (`test.yml:871-874`). Its only flag today is
|
||
`--summary` (`scripts/gen_pipe_dirty_surface.py:140`); making it a gate means adding a mapping file plus a
|
||
`--check` mode with a self-test, in the `check_include_closure.py` shape.
|
||
17. **The trace-replay tests carry no ctest label** (`tools/trace_replay/CMakeLists.txt:373-379`) — selection is
|
||
by name regex only.
|
||
18. **Reconcile 77 / 79 / 80** before writing the verify matrix (A4).
|
||
|
||
---
|
||
|
||
# Part D — files an implementer will touch
|
||
|
||
| File | Why |
|
||
|---|---|
|
||
| `.github/workflows/test.yml` | new `build-linux-verify`, `integration-verify`, `retrace-verify`, `monolith-symbol-report` jobs; `remove-artifact-clutter` `needs`; trigger branch; `timeout-minutes` |
|
||
| `CMakeLists.txt` (`:5-24`, `:108`, `:440-469`, `:735-762`) | `option(MOBILEGL_PIPE_VERIFY …)`, compile definition, interaction with `MOBILEGL_BUILD_DISAGGREGATED` |
|
||
| `MobileGL/MG_IntegrationTest/CMakeLists.txt` (`:274-393`, `:400-633`) | two verify registrations + the two negative-control lanes; `MGL_ITEST_VERIFY_TIMEOUT` |
|
||
| `MobileGL/MG_IntegrationTest/Scenarios/PipeVerifyArmingScenario.cpp` (new) | arming assertion + corrupted-field control |
|
||
| `MobileGL/MG_IntegrationTest/Scenarios/PoisonOmissionScenario.cpp` (new) | fork/waitpid `glGenerateMipmap` poison control |
|
||
| `MobileGL/MG_Test/Pipe/PipeCatalogueTest.cpp` (`:184-235`) | the two layer-1 unit controls |
|
||
| `MobileGL/MG_Test/Pipe/CMakeLists.txt` (`:28`) | if a separate `PipeVerifyTest` target is preferred |
|
||
| `tools/trace_replay/run_trace_case.cmake` (`:79-129`) | verify arming + divergence check |
|
||
| `tools/trace_replay/CMakeLists.txt` (`:311-380`) | optional `LABELS retrace`; optional `-DTRACE_PIPE_VERIFY` forward |
|
||
| `tools/trace_replay/trace_cases.json` / `trace_cases.py` (`:71-136`, `:151-196`) | the reduced verify matrix as data |
|
||
| `MobileGL/Config.h` (`:319-328`) / `MobileGL/ConfigLoader.cpp` (`:243-256`) | `PipeVerifyFatal`, `PipeVerifyCorrupt`, `PipePoisonOmit` |
|
||
| `scripts/gen_pipe.py` (`:340`, `:375-440`, `:462-510`) | G4 harness entry point, G5 omission vocabulary |
|
||
| `scripts/symbol_report.py` (`:266-296`, `:342`) | `--fail-on-added-bytes`, if the gate is made hard |
|
||
| `docs/Disaggregated/{ROADMAP,ARCHITECTURE,MEASUREMENTS}.md` | record the lane, the costs and the attribution table |
|