MobileGL/docs/Disaggregated/notes/perf-p15/HANDOFF.md

# P15 handoff (living document)

The plan and its measurements are in [PLAN-P15.md](PLAN-P15.md); the audit is in
[PARITY-AUDIT.md](PARITY-AUDIT.md).

**Target** (PLAN §0.1, revised by the user 2026-10-08):
- CPU scenes: FCL presenting-thread CPU at or below dev monolith (Magma <= 1.79, Espryt <= 2.14
  ms/frame), dev measured in the same session, ~5 % noise.
- GPU scene (Iris + BSL): fps within ~5 % of MobileGlues on both backends.

## RESUME HERE (2026-10-09 evening, end of the long P15 run)

**Rules from the user (2026-10-09):**
- A fresh agent per clear task boundary.
- Mechanical jobs (bench A/Bs that follow the existing scripts, builds, log/profile parsing, CI watching)
  go to sonnet/haiku subagents. Design and implementation stay on opus.
- No minSdk raise for the shipped build: API 29 is a SEPARATE artifact (its own library, plugin APK and
  FCL-embedded copy).
- No record bypass: monolith = record arm.
- TLS (superseded 2026-10-09): API 29 did not remove the TLS cost (native TLSDESC in a dlopen'ed library
  costs about the same as emulated TLS), so portable hot-path TLS elimination is approved for all API
  levels. No TLS-model attributes or API-level gates as the mechanism.
- No new Android API usage. Keep MobileGL platform-agnostic unless a platform path is measured and has
  no portable alternative.
- ASurfaceControl, frame-rate hints and ADPF are deferred.
- Optimization is driven by Perfetto traces (2026-10-09). Results are reported as a per-frame
  timeline: phases in order with offsets within a typical frame, the waits, and the critical path.
  Do not report them as ms tables. simpleperf is for function-level zoom-in inside a phase.

**Where the code is (worktree `.claude/worktrees/p15`, branch `p15impl`):**
- `origin/feat/disaggregated` = db346f28 plus the pushes listed in SCOREBOARD.
- Local on `p15impl`, not pushed:
  - 14d63881, docs.
  - 8d548d5b, window-scoped freqcheck.
  - ca0a3b74, a cherry-pick of the coordinator's LTO commit 190580f2 on `build-opt`. The coordinator
    pushes 190580f2, so drop ca0a3b74 when rebasing.
  - a0015bad, Magma 2a + 2b. Its message carries the m2ab result, or says it is pending.
- `wip/espryt-depthcopy-measured` (the two Espryt BSL GPU items rebased onto origin 9f610167, with the
  device numbers in its message; the old `wip/espryt-depthcopy` eb7572ff is the unmeasured original).
  **Measured 2026-10-09 on device, NOT landed: no gain on BSL.** x = with both items, b = without, both
  LTO builds of origin 9f610167; gpuhunt, 3 VALID reps each, one arm per cooldown:
  - BSL: x 51.4 fps (0.921 of MobileGlues 55.8), b 51.9 (0.930); GPU 100 % busy in all runs, GPU
    19.45 vs 19.29 ms/frame (MobileGlues 17.88). The target of 0.95 is not reached; the items are neutral.
  - Vanilla cpuhunt check (Espryt monolith): x 248.5 fps / 2.87 ms GL CPU, b 254.0 / 2.83 (-2.2 % fps,
    within noise; no CPU regression worth the name, but no gain).
  - Engaged: the simpleperf profile of the BSL render thread shows `BlitDepthTexture2D` 0.100 ms/frame in
    b and gone in x; `FramebufferImpl::SyncToBackendByHandle` 0.178 -> 0.162. The saved CPU is hidden by the
    GPU-bound frame, and the GPU time per frame did not move, so the Espryt BSL gap is not in these
    items. See SCOREBOARD (ebbsl).
  Original description:
  - (1) Depth glCopyImageSubData goes to the driver (ES 3.2 / EXT/OES_copy_image), with the
    scratch-FBO depth blit as fallback. Knob MOBILEGL_ESPRYT_NATIVE_DEPTH_COPY.
  - (2) A per-point attachment shadow on the Espryt FBO twin skips an attach call for a point that already
    holds the same surface. It is keyed on the surface handle {slot, gen} plus driver id, level, layer,
    call and re-mint generation; texture-side syncs still run. Knob MOBILEGL_ESPRYT_ATTACHMENT_SHADOW.
  - Tests: DepthCopyImageScenario (2 cases) and AttachmentReattachScenario (4 cases), both registered on
    the split arms. Red-once: a shifted native copy fails all DepthCopyImage cases on DirectGLES; dropping
    Level from the shadow key fails the level case; keying on the driver id alone fails the
    recycled-name case.
  - Host suites: 1672 DirectGLES ctest entries (monolith + Split) pass serially in WSL (362 skipped by
    design), including DepthCopyImage, AttachmentReattach, PassOrder and every blit case.
- `wip/api29-switch` (42922d21, on p15impl): `-Pmobilegl.androidApi=29` in build.gradle and the plugin's
  build.gradle.kts.
  - It sets minSdk 29 and therefore ANDROID_PLATFORM android-29; the plugin gets the `.api29` id suffix
    and `-api29` version suffix. The CMake API-guard comment is updated. Default stays 26.
  - Verified: the API 29 lib has 95 R_AARCH64_TLSDESC relocs and a TLS segment. __emutls remains only for
    3 libc++abi variables.
  - No A/B yet. The next agent runs the H2 A/B: API 26 vs 29, monolith + inproc, both backends,
    MobileGlues control.
  - Nothing in the tree needs a compile-time API gate yet, so none was added.
  - The FCL-embedded 29 copy needs a matching FCL-side switch (FCL's own minSdk).

**The m2ab device job was still running at hand-off. Do not interrupt it.**
- What it is:
  - Started as a background chain: `bash p15/m2ab_job.sh > p15/runs/m2ab_job.out; bash p15/espg_job.sh`.
  - `espg_job.sh` is parked: it only echoes "espg parked". The real Espryt BSL job is `espg_job.sh.next`.
  - m2ab_job waits for the device lock, cools to battery ≤ 33.0 °C, then runs
    `PROFILE=cpuhunt devjob2.sh p15/runs/m2ab.log bash p15/m2ab_session.sh`.
- Arms: cpuhunt H2, 3 interleaved reps, PF=1. Both libs are LTO builds of the same tree:
  - `n` = `libs/p15en`, without 2a/2b; monolith with sp=1, plus inproc;
  - `e` = `libs/p15e`, with 2a/2b (it also carries the Espryt items, which do nothing on Magma); monolith
    with sp=1, plus inproc;
  - MobileGlues.
- Results:
  - per run in `p15/runs/m2ab/<arm>-r<N>/` (fps.txt, valid.txt, perf.data on sp=1 runs);
  - the summary is appended to `p15/runs/m2ab.log` (look for the `n=` lines).
- Device restore: the session script runs `fcl_scene.sh restore` and `fcl_p14.sh restore`, and devjob2's
  exit trap stops the bench session and releases the lock.
- Interim at hand-off (GL CPU ms/frame):

  | arm | n (without 2a/2b) | e (with 2a/2b) |
  |---|---|---|
  | monolith r1 / r2 | 2.39 / 2.48 | 2.29 / 2.45 |
  | inproc GL thread | 1.20 / 1.20 | 1.17 |
  | inproc fps | 343.5 / 371.6 | 363.8 |

- When it ends:
  1. Summarize with `fclsum.py runs/m2ab`. Judge 2a/2b on monolith GL CPU and on the inproc apply thread
     (threads.txt `mgl-srv-apply`).
  2. Build the profile caches offline: `prof_prep_offline.sh <run> p15e|p15en runs/c1c2/x-DirectVulkan-r1`.
     Compare RenameBusyWireStore, VkBufferObject::Create, SweepDeferredWireReleases and
     IsSubmitIndexComplete.
  3. If neutral-or-better: amend a0015bad's message with the numbers. Push the p15impl commits WITHOUT
     ca0a3b74, i.e. cherry-pick onto origin after the coordinator pushes 190580f2. Then CI.
- Host suites for 2a/2b: done and at baseline (92 environment unit failures, 2 ColdStart).
  - The one extra unit failure seen once, CompositorRecovery.ACompositorOnTheServerWindowRecovers..., is a
    pre-existing load flake: under full CPU load it fails 6/20 without the change and 2/20 with it.

**Hot-path TLS cut (2026-10-09 night, worktree `.claude/worktrees/tlshoist`, branch `tlshoist` on origin
e70925d3, local, not pushed):**
- c0a9b2cd pGLContext and gPipeInputs become constant-initialized pointer slots (no TLS init wrapper or
  guard; the owning SharedPtr lives in a second thread_local, released before the slot is cleared at
  thread exit). 36f55af8 the applier lookup cache is one block. 6191cbf2 Magma session read and built-slot
  lookup inline. abb22eef Espryt bound serial and per-context cache share one block (epoch bump closes ABA).
  d4a00835 owner release order + SanityTest fix.
- Host: unit 92 failures, the same set as the baseline; integration 2 (Spawn.ColdStart.ClearThenReadPixels
  on both backends), as the baseline. Split-OFF NDK arm64 and the plugin LTO build compile.
- Device (SCOREBOARD tls): Magma +6.3 % fps, GL CPU -0.14 ms; Espryt +0.9 %, -0.05 ms. TLS resolver
  0.058 -> 0.034 (Magma), 0.049 -> 0.035 (Espryt) ms/frame.
- Remaining emutls per frame (~3-4k): pGLContext ~2.3k, gPipeInputs 0.6-1.1k, Magma session 0.6k
  (GetActiveRenderPass), ResolveAttributeFromBinding note bumps, SetupWireDraw. Each further cut is
  ~0.002 ms; not done.
- Perfetto traces of a second A/B (tlspf) are in `runs/tlspf/*/p15.pftrace` for the present-wait work.

**LTO and the server check (coordinator):**
- libMobileGLServer.so is built and packaged by the LTO-ON assemblePluginRelease. Its only libMobileGL
  import, `mobilegl_server_main`, is exported by the LTO lib.
- The spawn-transport smoke was **NOT run**. It needs the LTO-ON plugin APK (`libs/p15lto/plugin.apk`)
  installed, because the FCL mgdebug install has no libMobileGLServer.so.

**Open items, in order:**
1. CI fix: the reds of Test run 37925410578 on db346f28 (integration-split-strict, the two DirectVulkan
   retrace reds, skip census). Then push the LTO commit (190580f2 on `build-opt`) and the local p15impl
   commits (without ca0a3b74), and confirm CI is green.
2. `userguide/` at the repo root, an end-user guide for every setup: which setup to use, how to build,
   integrate and run it. Covers:
   - Android FCL-embedded;
   - the plugin APK (inproc, spawn with `MOBILEGL_IPC_SURFACE=server`, TCP);
   - the API 26 and API 29 artifacts;
   - Linux/anland;
   - the Windows WGL drop-in;
   - build options, user-facing knobs and troubleshooting.
3. DONE 2026-10-09: Espryt BSL GPU items (`wip/espryt-depthcopy-measured`): device A/B measured, no gain,
   not landed (see above and SCOREBOARD ebbsl). The Espryt BSL gap to MobileGlues (0.92) is still open;
   the next lead is the GPU side (render-stage trace of Espryt vs MobileGlues passes), not driver calls.
4. Small CPU batch: ValidateForVerb residue, Tracker::Update (0.018, mostly inherent),
   EmitGlobalConstants (0.026). Expected ≤0.03 ms in total.
   - DONE on host, branch `cpubatch` (2026-10-09). Measured on device 2026-10-09 (SCOREBOARD egc): EmitGlobalConstants 0.022 -> 0.014 ms/frame on both backends, consistent over 3 reps; whole-frame fps inside noise. Line-level
     profile of fbm2 h-r1..r3 (both backends, p15h syms) and m2ab e-DirectVulkan r1..r3 (LTO).
   - EmitGlobalConstants, cut: the (ShaderCso, Version) key is per handle (the applier keeps one
     block per record), so switching back to a program whose uniforms did not move sends nothing,
     and a moved one sends a span against its own held image instead of the whole block. The span
     is the range the frontend wrote (ProgramObject::MarkUBOContentDirtyRange / TakeUBODirtyRange),
     so the client shadow and its two-ended diff scan are gone. The scan plus shadow copies were
     ~0.0065 ms/frame; with the skipped switch-back sends, expect ~0.007-0.010 ms/frame.
   - Tracker::Update: inherent, skipped. ~30 ns per verb over ~25 frontend counters; the only real
     cut is the epoch skip (PLAN stage 1), which is not a small item.
   - ValidateForVerb residue: skipped. The largest named piece (~0.008 ms/frame) is the
     MOBILEGL_PIPE_POISON stamp walk, armed by MOBILEGL_BUILD_DISAGGREGATED. The bench libraries
     are the plugin build (split ON); the FCL-embedded build (split OFF, INFO) compiles it out.
     The rest (prologue, singleton guards, gates) is spread below 0.002 each.
5. DONE 2026-10-09: API 29 build plus its A/B (`wip/api29-switch`, 42922d21). Result in SCOREBOARD (api29):
   - cpuhunt vanilla, 3 interleaved reps, 27 of 27 VALID. fps api29/api26: Magma monolith 1.041, Magma inproc 1.044,
     Espryt monolith 1.008, Espryt inproc 1.015; GL-thread CPU -3 % / -7 % / -0.6 % / -3.4 %.
   - The gain is not TLS: simpleperf puts `tlsdesc_resolver_dynamic` (api29) at the same 0.05 ms/frame, about 2 % of the
     render thread, as `__emutls_get_address` (api26), because a dlopen'ed library takes the dynamic resolver.
   - Open: whether a static TLS model (initial-exec) on the hot variables can remove the 0.05 ms in a dlopen'ed lib on
     bionic; and what actually makes the api29 Magma build faster (not isolated).
   - The shipped build stays API 26. The FCL-embedded 29 copy still needs FCL's own minSdk switch.
5b. DONE 2026-10-09 (offline), see [PRESENT-WAIT.md](PRESENT-WAIT.md). Cause: the display buffer release (~every other frame renders into the buffer just handed back) gates the whole single-batch frame, then ~0.8 ms GPU start latency + 0.5 ms execution; the next queueBuffer throttles on it. MobileGL CPU is not on that path. Options: more Magma swapchain images, gate only the window pass, present off the GL thread. Original scope: present-wait timeline from the existing P15 traces (fbm, c4, c1c2: sched, kgsl submit/retire, gfx atrace, SF frametimeline). Confirm against the TLS agent's base-vs-change Perfetto traces on e70925d3.
   - Finding (egc profiles, e70925d3, cpuhunt, monolith): render-thread on-CPU + off-CPU = frame time.
     | | Espryt | Magma |
     |---|---|---|
     | on-CPU ms/frame | 3.08 | 2.63 |
     | off-CPU ms/frame | 0.89 | 1.09 |
     | of which in Present | 0.82 | 1.03 |
   - The blocked stack is the same on both backends: `queueBuffer`'s producer throttle
     (`Fence::waitForever`) on the previous frame's GPU-completion fence.
   - So frame time = CPU + present wait. CPU cuts reach fps only if the wait does not absorb them.
   - The GPU is only ~17 % busy at cpuhunt, so the fence signals late rather than the GPU running
     long. Candidates:
     - work submitted only at swap (~1 batch/frame);
     - the GPU waiting on the compositor's buffer release.
   - Trace (both backends, monolith + inproc, MobileGlues control):
     - MobileGL submit/flush points;
     - kgsl GPU busy spans and frequency;
     - fence create/signal times;
     - SurfaceFlinger latch/release.
   - Decide whether earlier submission (or another portable change) shrinks the wait.
5c. QUEUED 2026-10-10, after the quick before/after fps run:
   - Capture: one Perfetto run per arm (Espryt monolith base vs pwgate+pmflush; Magma monolith base vs
     pwgate+pmflush+extra 2), traces in scratchpad p15/runs/pwlat.
   - Derive offline: a rough per-frame timeline, and the input-to-present latency from the SF
     frametimeline.
   - Write it up in docs/Disaggregated/pwait/: what the levers do, the effect on fps and latency,
     the knob, and the FIFO caveat (extra images add latency only with vsync on).
6. Smaller follow-ups:
   - DONE 2026-10-09: `bench_session.sh stop` now runs `pin_clocks.sh restore` first (verified over a
     17-session run: max clocks back to stock after every session, daemons running, no session file).
     The scratchpad `devjob2.sh` / `fcl_bench.sh` hard-code the p15 worktree's copy of the harness;
     point them at a tree that has this commit.
   - The `Fatal{UnmigratedSurface, "AndroidNativeWindow@P12"}` text and the tests asserting on it still
     point at P12. The rule is permanent: a client window cannot cross processes, so windowed spawn uses
     `MOBILEGL_IPC_SURFACE=server`. Reword the message to say that.
   - DONE 2026-10-09: the windowed out-of-process smoke of the LTO plugin APK passes on both backends.
     The plugin is debug-signed and replaced the installed one, with the user's OK. Setup:
     - server: the plugin's `MobileGLDisplayActivity`, i.e. the `:mglwin` in-process server on
       `tcp://127.0.0.1:40613`;
     - client: FCL with `MOBILEGL_TRANSPORT=spawn`, `MOBILEGL_IPC_CONTROL=tcp://127.0.0.1:40613` and
       `MOBILEGL_IPC_SURFACE=server`. ServerOwned 1280x800 window, Espryt and Magma both in-world.

     A plain spawned child has no display (`NoServerDisplay`). Evidence is in the coordinator scratchpad,
     `p15/runs/spawn_smoke2/`.
   - Open from that run: every run logs `MG_Remote server: shared image 1 could not be sent on the aux
     socket (rc=3)`. Rendering continues, apparently on the non-zero-copy fallback. Find out why the
     loopback-TCP aux-socket send fails.

**Harness rules (all in scratchpad `p15/`; the tools live in `tools/device_bench/disagg/`):**
- H2 = `fcl_bench.sh`, driven by `fclab2.sh` for interleaved arms; summaries come from `fclsum.py`.
- Every session goes through `devjob2.sh` (device lock, bench session, restore on exit). Profiles:
  - `PROFILE=cpuhunt` for CPU work (CPU 1.5/1.25 GHz, GPU 903);
  - `PROFILE=gpuhunt` for BSL (CPU 2.4/1.8, GPU 578).
- BSL: **one arm per cooldown** (battery ≤ 33.0 °C between runs), arms interleaved across cooldowns. At
  gpuhunt the scene hits thermal limits by the second back-to-back rep.
- Validity: `freqcheck.py` judges clocks over the measurement windows only (measure_raw.txt spans) and
  thermal over the whole run.
- Stop a session on thr>0, a thermal_pwrlevel below the pin, or "max capped". Discard INVALID runs and go
  on. Never stop on tpl alone: it is 6 by construction at gpuhunt.
- Build simpleperf binary caches only after a session, or offline with `prof_prep_offline.sh`
  (donor cache plus the build-id entry in build_id_list). Never adb-pull during a session.
- Never edit a bash script that a running job will still read (bash reads scripts incrementally).
- TaskStop does not reliably kill child bash processes on this host: kill the PIDs, and only the inner
  session processes, so devjob's cleanup runs.
- Libraries live in `scratchpad/p15/libs/<tag>/lib/arm64-v8a/libMobileGL.so`, with symbols in
  `syms/<tag>/` and runs in `runs/<session>/`. Tags:

  | tag | contents |
  |---|---|
  | p15e | LTO + 2a/2b + Espryt items |
  | p15en | p15e without 2a/2b |
  | p15lto / p15off | LTO ON / OFF, both with 2a/2b |
  | p15a29 | API 29 |
  | p15h | the pipe batch |
  | p15y | 1c |
  | p15t | before 1c |

- Profile tools: `topinc.py`, `callers.py`, `callees.py`, `layerfns.py`, `layerdiff.py`, `diffprof.py`,
  `appside.py`, `pftstate.py` (Perfetto SQL), `threadstate.py`.

**Device state at hand-off:**
- At hand-off the m2ab session was still running; it restores FCL, the scene and the bench session itself.
- The device lock is released by devjob2 when m2ab ends.
- The plugin installs `top.mobilegl.plugin.*.trace` from earlier phases are still on the device.

## Open: Espryt small-store write rename (branch `pmflush`, 2026-10-09, host-verified, needs device)

**Not a p15 regression.** MONOLITH-DIFF's top Espryt row (`FlushPendingRangesFrom` driver 0.091 + libc
0.025, "dev 0") is the rename the doc's own caveat lists: dev's `FlushPendingRangesNow` costs 0.086 + 0.019
in the same profiles. Both are the same three-tier ladder, unchanged since dev f973008c. Inclusive:
dev 0.066 / p15 0.078 (feat-esp-1), 0.106 on the newer api29ab e26 run. So the drain is real cost that
dev pays too.

**What the drain is.** Under it, 0.080 of 0.106 ms/frame is one Adreno driver routine reached from
`glCopyBufferSubData` (dispatch slot 0x670 -> 0x3250c0 -> 0x2ee9d0, with heap allocations inside).
So it is the upload-ring tier: one GPU copy job per queued partial range. The 1.21.5 in-world trace
(5998912 calls, last 252 frames) shows the shape: 162 glBufferSubData a frame. 122 of them are partial
writes from offset 0, mostly 96 or 192 bytes into 288-byte vertex stores, and ~3 KB into 6.7 KB ones.
That is Minecraft's per-format immediate vertex buffer. Magma already renames these stores
(`RenameBusyWireStore`, S0 M4); `WriteWireBuffer` costs 0.031 ms/frame there, so Magma needs nothing.

**Change.** `RenameSmallStoreForDrain` (Managers.cpp) runs before the draw-time drain. For a store
<= 64 KiB whose unwritten bytes are <= 8 KiB, it replaces the queue with [0, store). The pinned ladder
then takes its tier 1, the whole-buffer orphan-map: no copy job, and recorded draws keep the old
storage. It refuses when the store holds bytes the host copy does not have: `serverGpuWritten`,
queued resident writes, persistent/immutable/imported stores, or split staged coverage short of the
whole store. On the trace this renames 108 of 122 partial writes a frame, at +101 KiB of host memcpy a
frame; a plain 64 KiB bound would cost +811 KiB, because of 128-byte writes into a 53 KiB index store.
The ladder bodies and the G5 pins are untouched.

**Tests.** EsprytFlushLadder has 3 new unit cases. There are 2 new scenarios, one per backend on every
arm: AtomicCounterScenario.APartialSubDataBetweenDispatchesKeepsTheShaderWrittenNeighbour and
CrossFrameBufferScenario.APartialSubDataBetweenTwoDrawsOfOneFrameKeepsTheFirstDrawsBytes. The skip
census mirrors their siblings' skipped lanes. Red-once: dropping the `serverGpuWritten` refusal reds
the atomic case on DirectGLES.Spawn, which reads back 16 instead of 32; monolith and inproc pass
because the frontend reads GPU writes back before a partial CPU write.

**Suite results.** WSL archlinux, clang Release, lavapipe, env MOBILEGL_ITEST_REQUIRE_GPU=1, suites run
one at a time:
- unit 2907/2907;
- integration 5300/5300 on the labels gpu, split, magma-all-split, magma-full-split,
  monolith-control, cold-start, cpu-mip-split, clientarrays-split and remint-fallback-split;
- 354/354 spawn+TCP buffer, SSBO, atomic, persistent, XFB and index scenarios;
- 18/18 Espryt retraces (OpenRA, 1.21.4-in-world, 1.21.11-main-menu, sodium, create-instancing, Iris
  BSL; each monolith, SPLIT and SPAWN);
- both Android arm64 shapes compile (disaggregated ON and OFF, android-26).

A first run with the dozen ICD (REQUIRE_GPU=ON in the cache) failed 1488 DirectVulkan cases. That is
the ICD, not this change: there were 0 DirectGLES failures.

**Host blocker for retrace evidence (pre-existing).** Espryt on llvmpipe segfaults in the 1.21.5 fixture
at call 75682 (a glDrawElements right after vertex buffer 8 is deleted and recreated). This happens on
the base tree too. Magma completes (SSIM 0.985 on dozen). So Espryt cannot be checked for SSIM on this
trace on the host.

**Device ask (Perfetto first, per the 2026-10-09 rule).** FCL MC 1.21.5 vanilla bench scene, monolith,
cpuhunt pinned, base = origin e70925d3 vs pmflush, same LTO settings. Run Espryt (the arm under test)
and Magma (control, expected identical). Capture a Perfetto trace per arm with:
- sched_switch/wakeup, cpu_frequency;
- atrace gfx/view/hal;
- kgsl gpu_frequency and GPU busy/slices;
- dma-fence signals;
- SurfaceFlinger frame timeline.

Report one typical frame as a timeline: render thread runnable/running/blocked, queueBuffer/dequeueBuffer
waits, the GPU slice and its end against the next frame's throttle wait. The question is whether
removing ~100 copy jobs a frame shortens the GPU slice and with it the ~0.8-1.0 ms queueBuffer wait (a
shorter frame), or only cuts CPU (a longer wait, the same frame). simpleperf zoom: `FlushPendingRangesFrom`
inclusive and its callees, expected to drop from ~0.106 towards ~0.03 ms/frame (the orphan-map plus the
extra memcpy). Also check the driver's map path (slot 0x550 -> 0x1db8a0) does not grow into new
allocations per rename. Correctness: SSIM/golden on the device retrace of minecraft-1.21.5-vanilla-fcl
plus the Iris/BSL cases, Espryt.

## Present-wait levers A + B (branch `pwgate`, worktree `.claude/worktrees/pwgate`, 2026-10-09)

Built on origin/feat/disaggregated a759b4b7. Host-only so far; the device plan is below. Mechanism:
PRESENT-WAIT.md (p15/pwait): every other frame renders into the buffer the display just released,
and the whole frame batch was gated behind that buffer's release fence.

**Commits (local, not pushed):**
- 168451bc Magma: submit a frame's off-screen work ahead of the swapchain acquire wait.
- 66adc7aa Espryt: flush once per frame ahead of the window buffer's first use.
- dd3e608a Tests: WindowPassGateScenario covers frames split around the window pass.
- cdd2160f Magma: MOBILEGL_MAGMA_EXTRA_SWAPCHAIN_IMAGES asks for swapchain images beyond the
  surface minimum.

**Lever A, Magma (no knob).**
- `FrameData::acquiredImageReferenced` means the open recording uses the acquired swapchain image.
  It is reset at every acquire (`WaitAndAcquireNextImage`, `AcquireNextImageInCurrentSlot`).
- `SubmitPendingCommandBuffer` waits on `imageAvailableSemaphore` only when the recording references
  the image, or when the caller forces the wait (`ParkActiveTarget`'s wait-out). It marks the
  semaphore consumed only when it waited. Present's own submission still waits whenever the
  semaphore is unconsumed.
- `SubmitAheadOfAcquiredImage()` runs before the frame's first use of the acquired image. It flushes
  what is recorded so far without the wait, then marks the image referenced. It is called from:
  - `SetupDraw` when the draw framebuffer is the default one, next to `SplitOversizedRecording`,
    before anything of the draw is recorded. Splitting later in setup lands after the draw's
    descriptor bind, because the attachment pre-resolve reads the default framebuffer's read index.
    That was measured as VUID-vkCmdDraw-None-08600 plus an llvmpipe segfault.
  - `ClearWireFramebuffer` for the default framebuffer.
  - `ResolveWireImage` for the acquired colour image. This is the net under blits, copies, readbacks
    and shared-image presents.
  - `Present`. A frame that never touched the window submits its work unwaited, and the present's
    own submission carries the wait, possibly with no command buffer.
  - Inside a draw's preparation (`m_wirePreparationDepth != 0`) it only marks the image; it never
    submits.
- Wait stage: every acquire wait now uses `FrameContext::kAcquireWaitStages`
  (COLOR_ATTACHMENT_OUTPUT | TRANSFER). Before, mid-frame flushes waited at ALL_COMMANDS and the
  present submit at TOP_OF_PIPE, which waits for nothing.
- `TransitionWireImage` adds those stages to the source scope when a default colour image leaves
  PRESENT_SRC_KHR or UNDEFINED.
- Engagement proof: one latched log line,
  `DirectVulkan: a frame's off-screen work goes to the queue ahead of its window pass ...`.
- Acquire timing is unchanged: the next image is still acquired at the end of Present.
  - Acquiring later cannot move the gate. The buffer queue's free list is FIFO, so a later dequeue
    gets the same buffer, or a more recently released one, with the same release time R.
  - It would also break the buffer-age contract: `CurrentDrawBufferAge` answers for the image
    acquired at the previous present.
  - So no device A/B was spent on it.

**Sync-correctness argument (Magma).**
- One wait per signal. Each acquire signals imageAvailableSemaphore once. Exactly one submission
  waits on it: the first whose recording references the image, or the present's, or the park
  wait-out. All three check and set `imageAvailableSemaphoreConsumed`. The next acquire on the slot
  comes after the slot's fence wait, so by then the semaphore is unsignaled with no pending wait.
- Recordings that never reference the acquired image carry no wait. That is correct because none of
  their commands access it.
  - Reads of `m_defaultFramebufferImageIndex` that name the previously presented image touch a
    different image.
  - The old code did not order those reads against an acquire either.
- Every first access of an acquired image is a layout transition out of PRESENT_SRC_KHR or UNDEFINED:
  - `TransitionWireImage`, whose source scope includes kAcquireWaitStages;
  - `TransitionToPresent`, whose source scope is ALL_COMMANDS.
- The semaphore wait's second scope covers kAcquireWaitStages in its own batch and in every later
  batch. So the transition is ordered after the presentation engine's release, even when the use is
  in a later submission than the wait. Through the transition's ALL_COMMANDS destination, everything
  after it is ordered too.
- renderFinished is signaled by the present submission. That is last in submission order, so its
  first scope covers the off-screen submissions too.
- Unchanged by this work:
  - Frames in flight: split submissions take pooled fences and advance `lastSubmitIndex`, and the
    present's slot wait covers them.
  - Wire render-pass guard: `FlushPendingCommands` ends the open pass.
  - Pass-barrier coalescing: barriers order across submissions.
  - Deferred clears: the default framebuffer never defers.
- Not machine-checked. lavapipe syncval does not flag a broken acquire chain.
  - Positive control: removing the source-stage fix while keeping the narrowed wait still gave
    0 hazards.
  - It never flagged the old present submit's TOP_OF_PIPE either.
  - The argument above rests on the spec.

**Lever A, Espryt (no knob).**
- `FlushAheadOfWindowBufferUse()` issues one `glFlush` per frame, just before `BindCurrentFBO` or
  `SyncAndBindFramebufferByHandle` moves the driver's binding to framebuffer 0. A thread-local flag
  limits it to once per frame and is reset at the present's swap.
- Window surfaces only: a pbuffer has no lazy dequeue.
- Not issued on MobileGL's own scoped binding restores, or when the binding is already 0. A frame
  that renders straight into framebuffer 0 has nothing to flush ahead.
- No wait and no glFinish. Adreno dequeues lazily at the first default-framebuffer use.
- Host coverage is pixels only, because the harness has no window surface.

**Lever B (knob, default 0).**
- `MOBILEGL_MAGMA_EXTRA_SWAPCHAIN_IMAGES` (0-16) requests
  `max(frames-in-flight hint, surface minImageCount + extra)` images, clamped to maxImageCount.
- Logged as `Set minImageCount = N (surface min M, max X, extra K)`.
- Documented in userguide/configuration.md and the README knob table.
- Cost per extra image at 2560x1600: one RGBA colour buffer (~16 MB) plus the default framebuffer's
  same-size depth/stencil image.
- Vulkan only. Espryt's EGL buffer count needs a platform API, so it is deferred.

**Host verification.** Setup: WSL archlinux, clang, CMake 3.31.10, `~/mgl-pwgate-ci31`, lavapipe,
suites run serially.

| Check | Result | Baseline |
|---|---|---|
| Unit | 2904 tests, 92 failed (ServerSpawn / PeerLatch / FuzzArm2) | same 92, environmental |
| Integration: `integration-gpu`, `-magma-all-split`, `-magma-full-split`, minus spawn and tcp, `MOBILEGL_ITEST_REQUIRE_GPU=1` | 3883 tests, 2 failed (both Spawn.ColdStart ClearThenReadPixels) | same 2 |
| Syncval (submit-time, shader heuristic): WindowPassGate + CrossFrameBuffer + DefaultFramebufferAcrossSwap.ADraw*, monolith and inproc | 0 hazards, 0 VUIDs; also clean with EXTRA_SWAPCHAIN_IMAGES=2 and FRAMES_IN_FLIGHT=2 | — |
| CI gates `magma_runahead_checks.py negative`, `validation`, `magma_cache_checks.py` | rc 0 | — |
| EXTRA_SWAPCHAIN_IMAGES=2 | 85 present-shaped Magma entries pass; minImageCount 4+2 = 6 on the headless surface | — |
| Android arm64 plugin release (`assemblePluginRelease`), API 26 and API 29 | both rc 0 | — |

- The integration run includes WindowPassGateScenario on DirectGLES and DirectVulkan (monolith,
  Split, Split.Full), plus the `DirectVulkan.WindowPassGate.` arming entry.
- Verb tables are unchanged, so split_coverage was not needed.
- The spawn and tcp lanes fail on this host for every scenario: TcpServer.Start fails, and spawn
  bring-up aborts with CapsBeforeFirstSnapshot.
- Pre-existing, not from this work: DefaultFramebufferAcrossSwap.AReadbackAfterASwap reads an image
  already handed to presentation, which syncval reports as WRITE_AFTER_PRESENT. That is why the new
  scenario reads back before its swaps.

**Device plan (not run; another agent has the phone).**
- Libs: both are API 26 default plugin builds.
  - base: origin/feat/disaggregated a759b4b7.
  - A: pwgate cdd2160f.
- Setup: same FCL, MC 1.21.5 vanilla frozen world, cpuhunt clocks, FCL pinned to cpu2-6.
- Validity rules: `fcl_bench.sh` VALID, freq and thermal sampled every 1-2 s, throttled runs dropped.
- Perfetto (PF=1): `p15_perfetto.cfg` with every `kgsl/*` ftrace event removed (not on this kernel).
  Keep:
  - sched (switch, waking, wakeup, blocked_reason), cpu_frequency and cpu_idle;
  - atrace `gfx` + `view` for the FCL package and surfaceflinger (gfx is what creates the
    `GPU completion` / `HWC release` FenceMonitor threads);
  - process_stats and `android.surfaceflinger.frametimeline`;
  - 5 s duration.
- Arms, 3 interleaved reps each:
  1. Espryt monolith: base, A.
  2. Espryt inproc: base, A.
  3. Magma monolith: base, A, A+`MOBILEGL_MAGMA_EXTRA_SWAPCHAIN_IMAGES=1`, A+`=2`.
  4. Magma inproc: base, A, A+1, A+2.
- Engagement proof:
  - Magma A: logcat has the latched "goes to the queue ahead of its window pass" line, and the trace
    shows QueueSubmit per frame rising from ~1.33 to ~2.3 (one submit just before the blit's draw,
    one at present).
  - Extra images: logcat shows `Set minImageCount = 6/7 (surface min 5 ...)`.
  - Espryt A has no marker. Take one extra `gpu.renderstages` capture per Espryt arm (it works on
    this device); it should show A's off-screen stages running before R.
- Analysis, with the scratchpad `p15/pwait/` scripts:
  - `cycle.py <label> <runs...>`
  - `pairs.py`
  - `gexec.py`
  - `sim.py <label> <start> <exec> <runs...>`
- Report per-frame timelines, not ms tables.
- Expected for lever A (PRESENT-WAIT, "What a fresh trace should confirm"):
  - Gated frames stay at ~50 %.
  - For gated frames, E - R falls from ~1.35 to ~0.9 ms: start latency plus the window pass only.
  - Back-to-back execution is unchanged at ~0.5 ms.
  - sim's throttle, ms/frame: Espryt mono 0.62 -> 0.49, Magma mono 0.72 -> 0.54,
    Magma inproc 1.31 -> 1.09, Espryt inproc 1.09 -> 0.91.
  - If E - R does not fall, either the driver gates the off-screen submission too, or the split is
    not engaged. Check the submit count, and when the GPU work starts relative to R.
- Expected for lever B:
  - `HWC release fence N has signaled` at >= 85 % of AcquireNextImageKHR.
  - Gated frames < 20 %.
  - Throttle < 0.2 ms/frame on monolith.
  - inproc may need +2 (sim: 23 % still gated at k=2).
  - The user picks lever B's default from these curves.

## Earlier (2026-10-09): cuts 1b + 1c, cut 4, pipe-record pieces

- **Landed locally (pushed after the BSL gpuhunt check):**
  - 66e3b2c1 (rebased as 65d4b808): Espryt surface-size memo, GL CPU 3.18 -> 3.05.
  - 533a2564: Magma deferred clears (1c) plus coalesced pass-begin barriers (1b).
    - Per frame: passes 11.5 -> 4.3 (dev 4.1), barriers 15.5 -> 9.0 (dev 3.4).
    - Inproc +5.0 % fps, apply thread -0.17 ms; monolith GL CPU -0.09 ms.
- **Landed 2026-10-09 afternoon:**
  - 485dff14: pipe client batch, i.e. the per-framebuffer FB-state memo (MOBILEGL_PIPE_FRAMEBUFFER_STATE_MEMO),
    handle hints, per-buffer handle reuse and the per-verb gate cache.
  - c85c9439: Magma hash TLS hoist.
  - Effect: H2 GL CPU Magma 2.68 -> 2.62, Espryt 3.03 -> 3.00. The profile is in SCOREBOARD (fbm2).
  - 9eed5a27 / 9ff97c2e: CI branch fixes. The E1 control follows ApplierReset, and ScenarioIsolation is
    listed as unarmed in verify-split.
- **Small-cut policy:** pieces below about 0.03 ms are verified by their own before/after function profile
  plus a red-once, and get one H2 A/B per batch.
- **Order (coordinator, 2026-10-09):** FB-state memo, then handle cache on objects, vertex input,
  program, samplers, accessor/TLS hoist, the ValidateForVerb dirty walk, Tracker.
  - No record-bypass design change until the user decides.
- **Parked: the Espryt attrib-format split** (MONOLITH-DIFF.md).
- **Remaining Magma barriers:** CPU ceiling ≤0.02 ms. Revisit only if BSL shows a Magma GPU gap.
- **Harness gotchas found today:**
  - `tpl` in the freq sampler is kgsl `thermal_pwrlevel`. Under gpuhunt it is 6 by construction
    (the 578 MHz pin); freqcheck already maps it. Stop a session on INVALID or `thr > 0`, never on
    `tpl > 0`.
  - Never run `binary_cache_builder` (adb pulls) while a session is measuring. Build binary caches
    after the session ends.
  - FCL sometimes dies at bootstrap with signal 34 (seen on every build including MobileGlues);
    fclab2 retries, and a run that never reaches the world is lost.
- **Monolith floor estimate (MONOLITH-DIFF.md, structural):**
  - Pipe record: 0.35 -> about 0.10-0.13 if every local piece lands.
  - Whole Magma GL thread: about 2.35-2.40 ms (native TLS: about 2.30), against dev 2.00.

## Earlier (2026-10-08 late): harness H2, dev reference, scoreboard

- **Every FCL vanilla number now goes in [SCOREBOARD.md](SCOREBOARD.md).** It has one row per
  session, as fps ratio to MobileGlues plus CPU ms/frame. It also holds the fixed dev reference, so
  "vs dev" no longer needs dev in every session. The per-function dev baseline is in
  [MONOLITH-DIFF.md](MONOLITH-DIFF.md).
- **The old probe's variance was core placement.** The scheduler gave MobileGL's monolith render
  thread the X4 core 72-90 % of the time and MobileGlues' 47-60 %. MobileGlues' fps tracked its X4
  share (fifi: 60 % -> 378, 47 % -> 344).
- **Harness H2 (scratchpad `p15/fcl_bench.sh`, `fclab2.sh`, `benchwin.py`):** the same for every
  arm.
  - After world entry, `taskset -a -p 7c` pins FCL to cpu2-6 (A720).
  - fps comes from the SurfaceView BLAST layer `frame=` in `dumpsys SurfaceFlinger`. It agrees
    with the MobileGL fps log over the same span (252.1 vs ~253).
  - Warmup: 20 s, then 5 s windows until two consecutive ones agree within 3 %.
  - Measurement: two 5 s windows with no tracer; CPU from schedstat.
  - atrace, then the optional simpleperf (`sp=1`), run only after the windows.
- **Frozen bench saves** (1.21.5 and 1.21.4-Fabric, scratchpad `p15/freeze_world.py`):
  - Rules: no daylight, weather or mob-spawn cycle, randomTickSpeed 0, no fire tick.
  - DayTime 6000, clear weather.
  - Before this, the vanilla save's day cycle was running.
  - The originals are on the device as `level.dat.p15-orig`.
- **Spread over 3 interleaved reps, (max - min) / mean:**
  - Old probe: Espryt 9.3 %, Magma 5.6 %, MobileGlues 1.3 %; MobileGlues spanned 322-376 across
    sessions.
  - H2: Espryt 1.2 %, Magma 2.9 %, MobileGlues 2.0 %, dev 6-7 %.
- **Dev reference (H2, safter, all VALID):** MobileGlues 321.5 fps / 1.70 ms. dev Magma is 0.950
  of that (2.01 ms) and dev Espryt 0.887 (2.38 ms).
- **On A720 the monolith gap is larger than the X4 hid:** p15s Magma is 0.768 of MobileGlues (0.81
  of dev, 2.91 ms) and Espryt 0.746 (0.84 of dev, 3.18 ms). That is +0.90 / +0.80 ms/frame of CPU
  over dev.
- **Inproc p15s Magma on H2:** 327.7 fps against MobileGlues ~321.
- **Emulated TLS:** `__emutls_get_address` costs ~0.06 ms/frame of self time on the p15 monolith
  render thread, spread over many callers. dev shows almost none. The structural fix is native TLS
  (minSdk), which is the user's call.

## Earlier (2026-10-08, after S0)

Every number below carries its profile and passed the validity check (`state.txt`). `cpuhunt` =
CPU about 1.5/1.25 GHz, GPU 903 MHz (cleanly CPU-bound); interval 0, anland off, daemons stopped.

- **The S0 milestone's absolute numbers are superseded.** The perf stack's `msm_performance` caps
  (787 / 1286 / 1075 / 1248 MHz) were clamping the CPU under the pin for some of that session. Only
  same-session deltas from it stand. Measurement now uses `bench_session.sh` and the sampler (PLAN
  §1), and the probe reports only clean fps windows taken before any tracing (`P15PROBE
  trace-start` marker, 25 s settle).
- **FCL vanilla 1.21.5, cpuhunt, presenting-thread CPU ms/frame:**

  | build | Espryt | Magma |
  |---|---|---|
  | dev | 1.76 | 1.60 |
  | p15k (S0 + VAO shadow) | 2.10 | 2.03 |
  | p15l (+ vertex-elements dedup) | 2.10 | 2.08 |

  MobileGlues: 1.58 in an earlier cpuhunt session (needs a same-session re-run with clean windows).
  Floor gap: Espryt +0.34, Magma +0.43 ms.
- **Driver-call redundancy:**
  - Measured with a diagnostic build that counts every host GLES/EGL call and redundant re-sets
    (`p15diag` worktree, never committed; HostCount.inl and VkCount.cpp).
  - Espryt sent 3717 host calls/frame against MC's ~874: 2118 `glDisableVertexAttribArray`, 378
    enable, 378 divisor, all re-sets.
  - Two fixes, each independently removes the walk churn:
    - b4e209e1: a shadow of each driver VAO (850 calls/frame);
    - c8a5abe5: the client no longer re-creates an identical vertex-elements record on every
      `glBindVertexBuffer`. With the shadow off it takes 3724 down to 1228.
  - CPU gain is small (about 0.1 ms): the calls were cheap in the driver.
  - Magma issues about 380 Vulkan calls/frame with essentially none redundant.
- **Where Espryt's gap to dev is now:** the driver time equals dev's (0.50 vs 0.48). The excess is
  libMobileGL itself, 0.75 vs 0.32: client half 0.51, backend 0.24, applier 0.02. The client half
  is spread thin (validate, tracker, emitters, residual copy): S1's draw-delta.
- **Inproc (cpuhunt):**
  - Espryt reaches 306 fps vs MobileGlues about 319; GL thread 1.68 ms, apply thread 1.83 ms of a
    3.27 ms frame.
  - Neither thread is saturated. The apply thread waits in `queueBuffer` on a GPU fence
    (present pacing), and the GL thread waits on the apply thread.
  - `MOBILEGL_IPC_PRESENT_CREDIT=2` changes nothing (300 fps).
  - Beating MobileGlues on inproc needs a present-path change (stage P territory, dropped by the
    user), not CPU work.
- **S0b:** the fixture `minecraft-1.21.5-vanilla-fcl-in-world` is landed (4e27342e) and is in the
  SSIM gate as `fcl` (32/32 on p15j).
- **Knobs on FCL:** `adb shell setprop debug.mobilegl.env "MOBILEGL_A=1;MOBILEGL_B=0"` (49c6394d);
  `fclab.sh` takes a 6th field `K=V,K=V`.
- **MC's AFK frame limit:** `inactivityFpsLimit:"afk"` (MC 1.21.2+) made later probe windows and
  profiles spin in `System.nanoTime`. `fcl_scene.sh` now sets `"minimized"` for bench scenes and
  restores the original options files (`/data/local/tmp/p15-opts-121{5,4}.txt`). Profiles taken
  before this are partly AFK-polluted.
- **BSL at `gpuhunt`** (CPU 2380/1689 MHz, GPU 578 MHz, all 100 % GPU busy):
  - MobileGlues 53.4 fps;
  - dev Magma 53.2;
  - p15l Magma 51.8 (3 reps, -3.0 % vs MobileGlues: target met, -2.6 % vs dev to bisect);
  - Espryt dev and p15l 48.3 (-9.6 %): the GPU gap, inherited from dev.
- **Same-session cpuhunt, apply thread:**

  | build | fps | apply ms/frame |
  |---|---|---|
  | MobileGlues | 329 | CPU 1.55 |
  | Magma inproc p15m | 300 | 1.6 |
  | Espryt inproc | 296 | 1.65-1.8 |

  - Inproc beats MobileGlues only when the apply thread's CPU is below MobileGlues' (both pay the
    same queueBuffer wait). About 0.3 ms per frame beyond the apply CPU is still unexplained.
  - 500b63ea: per-store reuse of renamed-away wire buffers (Magma apply -0.13 ms).
  - ONE_TIME_SUBMIT: no effect (A/B'd, knob dropped).
  - Magma's driver work is about 2x dev's: vkResetCommandBuffer 0.09 + vkEndCommandBuffer 0.11
    ms/frame.
- **Inproc serialization found (2026-10-08, p15l cpuhunt profiles):** every blit reached the
  client as `BlitNamedFramebuffer`, a verb with no `MGP_VERB_OP_LIST` row, so
  `ClientVerbIsBarriered` answered barriered and the GL thread quiesced the applier once per frame
  (1.65 ms/frame parked in `WaitForApplyToCatchUp`, plus 0.11 from the following `glBindTexture`).
  The apply thread then idled 0.24 ms parked plus 0.2 ms spinning in `Doorbell::Wait`. e00db8e6
  replaces the inverse and its barriered fallback with a verb -> record-op table
  (`MG_Pipe/VerbRecordOps.h`, one row per verb by static_assert; no knob, user decision).
  - Same-session blab (cpuhunt): Magma inproc 319.6 -> 334.9 fps (+4.8 %), Espryt inproc
    324.4 -> 338.1 (+4.2 %).
  - Wait census (`MOBILEGL_IPC_WAIT_STATS=1`, 7acf3ea8): FCL vanilla inproc waits only on the
    present credit (1.00/frame). BSL waits 7 times per frame on three kWaitApplied rows plus 22
    follow-on quiesces; see [WAIT-AUDIT.md](WAIT-AUDIT.md). User decision applied in 2fbcc224:
    fire-and-forget rows async, inproc creates and PBO readbacks no longer wait, and the dual block
    (`MOBILEGL_IPC_ROLE_SPLIT_STATE`, now default on) lets a barriered verb cost one wait. BSL
    census after it still owed (gpuhunt only; FCL's 1.21.4-Fabric profile exited with signal 34 at
    JVM start on both backends in the last attempt).
  - db915f76: a client parked on the present credit is rung only by the credit (apply-thread
    futex wakes 0.056 ms/frame before).
  - Frames in flight (0b96a3ef): `MOBILEGL_FRAMES_IN_FLIGHT` (default 3) drives Magma's frame
    contexts, Espryt's post-present fence wait and the split present credit max(1, N-2); the Magma
    name is an alias. Default latency unchanged; the FIF A/B is the user's input for the default.
  - Present credit 2 (cens, cpuhunt, all VALID): Magma 309.0 -> 319.9, Espryt 299.1 -> 319.1 fps
    vs MobileGlues 321.9 in the same session; apply CPU 1.40-1.43 vs MobileGlues' GL thread 1.57.
    Credit 2 costs a frame of latency: user decision. The remaining inproc gap is the queueBuffer
    fence wait (GPU-side), not CPU.
  - BSL at cpuhunt (GPU 903 MHz) reaches GPU 95 C and kgsl thermal_pwrlevel 1 (834 MHz cap): the
    census run was INVALID twice and the job was stopped. Run BSL only at gpuhunt.
  - Apply-thread frame (Magma inproc): on-CPU 1.75 (0.2 of it spin), queueBuffer fence wait 1.45,
    parked 0.24 ms.
  - Magma apply on-CPU by verb: draws 0.55 (SetupWireDraw 0.40, of it descriptor binding 0.13),
    present 0.37 (End CB 0.10, queueBuffer CPU 0.08, submit 0.07, wire-store sweep 0.035),
    clear 0.19 (the frame's recording begin 0.09 + pass begin 0.05), buffer writes 0.09, blit
    0.06 (two image views and a VkFramebuffer created and retired per blit).
- **Device contention, root cause (2026-10-08):** the anland app was relaunched by `fclab.sh`'s own
  end-of-run `monkey` (launchedFromUid=0); its activity takes the foreground back whenever paused,
  so FCL launched over it never reaches the world (RenderDoc job: 0 of 4 worlds). c41dfdda:
  `bench_session.sh start` force-stops it and `stop` relaunches it only if it ran before.
- **Open user decision:** Espryt's apply thread spends 0.06 ms/frame in `LatchIfGuiltyBeforeApply`
  (glGetGraphicsResetStatus is a kgsl ioctl, queried on every wake). Rate-limiting it to 2 ms
  breaks `ServerLoopEglLatchTest.AGuiltyResetEndsTheSessionBeforeItsNextRecord...`, the designed
  promptness guarantee, so it was reverted. Trading that guarantee for 0.06 ms is the user's call.
- **CI:** 4a8ecf9a green. 49c6394d: one job failed on a hosted runner's `apt-get update` timeout
  (45 min, infrastructure); the failed job is re-run.
- **Next:**
  - RenderDoc (rdc GLES layer, remote server stopped): Espryt vs MobileGlues BSL pass structure;
  - S1 (client validate 0.25 ms) and S3 (backend by-handle syncs, descriptor resolve) for the CPU
    floor and the inproc apply thread.

## Status (2026-10-08)

**Branch:** `p15impl` (worktree `.claude/worktrees/p15`, short path `W:/p15`), pushed to
`feat/disaggregated`.

**S0.1 parity audit:** written.

**S0.2, Magma wire non-draw ports:**
- **M1a, clears become part of the draw pass** (`ClearWireFramebufferInPass`; `PrepareWireDrawPass`
  split out of `SetupWireDraw`):
  - A whole-area clear that begins the pass begins it with a LOAD_OP_CLEAR variant of the pass's
    render pass (`RenderPassEntry::clearVariants`, compatible with the framebuffer).
  - Any other clear (into the open pass, or scissored) is `vkCmdClearAttachments` in the pass. A
    forced pass restart for depth clears (an LRZ hypothesis) was tried and reverted: it cost CPU
    and fixed nothing.
  - Kill switches: `MOBILEGL_MAGMA_INPASS_CLEAR=0`; `MOBILEGL_MAGMA_INPASS_CLEAR_DEFAULT=1` puts
    default-framebuffer clears in the pass too (off by default, see the open item).
- **Open item: default-framebuffer clears stay standalone.** With them in the pass, the openra
  retrace on Adreno 750 loses its palette-sampled sprites (SSIM 0.956378, deterministic). What was
  measured:
  - Both `vkCmdClearAttachments` and LOAD_OP_CLEAR fail; so does splitting the clear calls.
  - Colour-only or depth-only in the pass, with the other part standalone, passes.
  - Event traces: the first draw flushes the recording (the palette `glTexImage2D`'s upload flush,
    `FlushPendingCommands`) right after the clear. Command order otherwise matches the standalone
    path.
  - Not explained yet. Lavapipe and the InPassClear scenarios on FBOs are clean.
  - Ruled out by reading: a missing ordering edge. Default-framebuffer images get the same
    pass-begin barrier as FBO images (a full ALL_COMMANDS memory barrier, or their transition),
    with their layout tracked per swapchain index through the swapchain object.
  - Still open: the in-pass path commits the freshly acquired swapchain image
    (`DefaultFramebufferWriteIndex(true)` in `PrepareWireDrawPass`) at the clear; check whether
    the standalone path's clear and the in-pass clear address the same index on the trace route.
  - Cost of leaving it: small for FCL (MC renders into its own FBO and blits to the default one).
- **M2, draw-pass cache across frames:** one cache, entries pruned when their image epochs move,
  destroyed after their last submit. Kill switch: `MOBILEGL_MAGMA_PASS_CACHE=0`.
- **M3, blit descriptor sets:** one pool per frame slot.
- **M9, converted vertex streams** (wrongly marked "have" in the first audit). The wire arm
  re-converted the whole remaining stream on every draw: Sodium's multi-draws over one arena cost
  49 ms/frame of CPU on BSL (19.5 fps against dev's 64.7).
  - Now one conversion per (store content serial, layout), kept across frames in
    `VkBufferManager` (`LookupConvertedStream` / `StoreConvertedStream`).
  - Each entry owns a vertex buffer. Bounded at 192 MB (least recently used first); entries idle
    for 256 frames go. A replaced buffer is released when its frame slot comes round.
  - The content serial is process-wide monotonic and moves on every host write, respecify,
    GPU-write mark, shader-writable bind and donation.
  - Kill switch: `MOBILEGL_MAGMA_VERTEX_CONVERSION_CACHE=0`.
- **M4, write renaming.** A busy `glBufferSubData` used to take a staged copy, which ended the
  render pass with two full barriers (FCL: ~5.6 per frame, all on one 288-byte store). Now:
  - A store of 64 KB or less keeps a host shadow (64 MB budget), from respecify on.
  - A busy write into a shadowed store gives it a fresh VkBuffer filled from the shadow and
    releases the old one like a glBufferData orphan. No copy is recorded and the pass stays open.
  - GPU-written marks, shader-writable binds, donation and import drop the shadow.
  - `ReadWireBuffer` serves shadowed stores from the shadow, so no host wait.
  - Kill switch: `MOBILEGL_MAGMA_WRITE_RENAME=0`.
- **Tests:**
  - `InPassClearScenario`: 4 cases, monolith and split, both backends.
  - `ConvertedVertexStreamScenario`: 3 cases (reuse, same-frame sub-data, cross-frame sub-data).
    Red-once: a lookup that ignores the content serial turns both sub-data cases red.
  - `WriteRenameScenario`: 4 cases (order and carry, one-byte writes, uniform and index buffers,
    multi-frame). Red-once: a rename that skips the shadow fill turns all 4 red, which also
    proves the rename engages.
  - Red-once checks: the colour index and the scissor (2 cases red); the depth clear-load bit
    (depth/stencil case red).
  - Syncval gates green (cache 7/7, run-ahead 14/14), plus the InPassClear cases under syncval.
- **Gates:**
  - SSIM 24/24 (openra, rd12, bsl × 4 arms × 2 backends) on `p15e`.
  - Host unit and integration-gpu green, apart from the known environmental `ServerSpawn` /
    `FuzzArm2` / `PeerLatch` / spawn ColdStart failures.
  - `CompositorRecovery` flakes under -j16 and passes alone 3/3.

**S0 milestone (one device session, FCL, interval 0, 2 reps; `runs/ms-s0i`):**

| scene | build | Magma CPU ms/frame | Magma fps | Espryt CPU ms/frame | Espryt fps | GPU busy |
|---|---|---|---|---|---|---|
| vanilla 1.21.5 (CPU) | fin | 3.27 | 170 | 2.81 | 190 | 12-15 % |
| | p15h (S0 without M4) | 2.59 | 191 | - | - | 15 % |
| | p15i (S0) | 2.56 | 190 | 2.77 | 193 | 13-15 % |
| | dev | 1.79 (n=1) | 234 | 2.14 | 218 | 14 % |
| | MobileGlues | 1.59 | 256 | | | 15 % |
| BSL (GPU) | p15i | 7.26 | 77.2 | 7.61 | 70.6 | 99 % |
| | dev | 13.85 | 66.1 | 7.17 | 70.5 | 85-100 % |
| | MobileGlues | 6.25 | 78.2 | | | 98-100 % |

(BSL before the converted-stream cache: Magma 19.5 fps at 26 % GPU busy, 49 ms CPU/frame.)

- Espryt's `base` lib (p15plan, unchanged) read 2.84 in the same session: the earlier "p15e
  Espryt regression" was session variance. S0 does not touch Espryt.
- **Target status.** Magma BSL is met (within 1.3 % of MobileGlues, GPU-bound). Espryt BSL is
  GPU-bound at 99 % and 10 % below MobileGlues, same as dev: a GPU-side gap still to explain (the
  load/store and clear audit of PLAN §0.1). CPU scene: Magma 2.56 vs MobileGlues 1.59, Espryt 2.77.
- **Where Magma's CPU goes now** (p15i profile, `runs/prof-hi`, ms/frame inclusive):
  `VerbChannel::Port` 0.85, `OnDrawVbo` 0.62, `SetupWireDraw` 0.46, `Present` 0.32,
  `MGPipeValidateForVerb` 0.27, in-pass clears 0.18. The draw path is S1-S3's; S0's non-draw ports
  are done. Proof that M4 engages: `StagedWireRangeCopy` 0.067 is gone and `RenameBusyWireStore`
  costs 0.048 (a VMA allocation per rename; a recycle pool could take most of it).

**Stage P:** dropped for the target.

## Harness

| purpose | where |
|---|---|
| FCL probes and reducers | `tools/device_bench/disagg/` (README "P15 FCL probes") |
| FCL vanilla A/B, H2 (pinned, SF frame source, stability warmup, `sp=1` profiles after the windows) | scratchpad `p15/fclab2.sh "<name>:<lib>:<backend>[:sp[:transport[:K=V,...]]] ..." <reps>` -> `fcl_bench.sh`; `fclsum.py`, `residency.py`, `diffprof.py`, `prof_prep.sh` |
| FCL milestone (CPU + BSL scenes, fin / new / dev / MobileGlues) | scratchpad `p15/fcl_milestone.sh`, `fcl_scene.sh`, `fclab.sh`, `fclsum.py` |
| SSIM gate (openra, rd12, bsl × 4 arms × 2 backends) | scratchpad `p15/ssim_gate.sh`; `matrix_p14.sh` now takes `PREP_WLS` and a `bsl` workload |
| host CI-shaped build (WSL archlinux) | `~/mgl-p15-ci31`, scratchpad `p15/wsl_ci.sh`, `w_suite.sh`, `w_syncval.sh`, `w_sv_one.sh`, `w_one.sh` |
| trace APK builds | scratchpad `build_trace.sh W:/p15 .<tag>` (rerun once on the `IncrementalSplitterRunnable` failure) |

**Host-build trap:** editing a header while ninja is compiling leaves objects newer than the header
but built from its old text. This happened once and gave a `bad_alloc` in every scenario. Never
edit sources during a build; `touch` the edited headers if in doubt.

**Device facts:**
- Clocks: CPU 2035.2 MHz on policies 2/5/7 and 1574.4 MHz on policy0, GPU 680 MHz (new default since
  2026-10-08, pending the soak). Pinned per session by `bench_session.sh`, which every `devjob.sh`
  job runs.
- Frequency daemons are stopped only inside a bench session: `thermal-engine`, `perf2-hal-1-0`,
  `vendor.perfservice`, `performance` and `hyperschedule_hal_service`. Their prior state is in
  `/data/local/tmp/p15-bench-session` while a session is open, and they are restarted and verified
  on exit. If that file exists with no job running, a session was left open: run
  `bench_session.sh stop`.
- The perf stack's game-mode caps in `/sys/kernel/msm_performance/parameters/cpu_max_freq` survive
  its services and silently override the pin. The session releases them.
- FCL's original lib md5 is `00c09f0a…`.
- MC 1.21.5 options md5 is `93b3f708…`; it is backed up at `/data/local/tmp/p15-options.txt`.
- Iris properties are backed up at `/data/local/tmp/p15-iris.properties`.
- `fcl_scene.sh restore` restores Iris; `fcl_p14.sh restore` restores the lib and `config.json`.
- The anland desktop is stopped during runs and restarted afterwards.


## pwab: device A/B of the present-wait levers (2026-10-10)

Result table, engagement evidence and the failed lever-B expectations are in [SCOREBOARD.md](SCOREBOARD.md) (section pwab). Summary: before a759b4b7, after pwgate cdd2160f + pmflush eec5e6e8 (+ `MOBILEGL_MAGMA_EXTRA_SWAPCHAIN_IMAGES=2` on Magma). fps after/before: Magma inproc 1.32 (1.495 of MobileGlues), Magma mono 1.34 (1.176), Espryt inproc 1.09 (1.102), Espryt mono 1.12 on reps 1+3 (0.896; before rep 2 NO WORLD, not re-run).
- Libs: `p15/libs/{pwbase,pwA,pwF,pwAF}`, symbols in `p15/syms/`. Worktrees `.claude/worktrees/{pwbase,pwF,pwAF}` hold the builds (pwF/pwAF are detached HEADs with eec5e6e8 cherry-picked, not pushed). The 3rdparty copy also needs `include/ska` (a gitlink) copied into a fresh worktree.
- Harness: `p15/pw/dev/` (chain.sh, session.sh, fcl_bench.sh with the post-Perfetto atrace stage made opt-in, devjob.sh), run data in `p15/runs/pwab`, Perfetto in `p15/runs/pwlat`.
- Not done: the per-arm timeline analysis (cycle.py/pairs.py/gexec.py) and the separate lever A / F / extra=1 arms, both cut by the user. Next: derive the timelines from `runs/pwlat` and find why the Magma gain exceeds the lever-B model.
- Harness note: a launch can fail with NO WORLD after four 240 s attempts (b-DirectGLES-r2); fclab2.sh only retries INVALID runs, not NO WORLD.