[Docs] (P15): S0 handoff, parity audit corrections and the bsl workload in matrix_p14.

docs/Disaggregated/notes/perf-p15/HANDOFF.md

@@ -0,0 +1,128 @@
+# P15 handoff (living document)
+
+The plan and its measurements are in [PLAN-P15.md](PLAN-P15.md); the audit is in
+[PARITY-AUDIT.md](PARITY-AUDIT.md).
+
+**Target** (PLAN §0.1):
+- CPU scenes: FCL presenting-thread CPU on both backends within ~5 % of the MobileGlues plugin.
+- GPU scene (Iris + BSL): fps within ~5 % of the better of MobileGlues and dev.
+
+## Status (2026-10-08)
+
+**Branch:** `p15impl` (worktree `.claude/worktrees/p15`, short path `W:/p15`), pushed to
+`feat/disaggregated`.
+
+**S0.1 parity audit:** written.
+
+**S0.2, Magma wire non-draw ports:**
+- **M1a, clears become part of the draw pass** (`ClearWireFramebufferInPass`; `PrepareWireDrawPass`
+  split out of `SetupWireDraw`):
+  - A whole-area clear that begins the pass begins it with a LOAD_OP_CLEAR variant of the pass's
+    render pass (`RenderPassEntry::clearVariants`, compatible with the framebuffer).
+  - Any other clear (into the open pass, or scissored) is `vkCmdClearAttachments` in the pass. A
+    forced pass restart for depth clears (an LRZ hypothesis) was tried and reverted: it cost CPU
+    and fixed nothing.
+  - Kill switches: `MOBILEGL_MAGMA_INPASS_CLEAR=0`; `MOBILEGL_MAGMA_INPASS_CLEAR_DEFAULT=1` puts
+    default-framebuffer clears in the pass too (off by default, see the open item).
+- **Open item: default-framebuffer clears stay standalone.** With them in the pass, the openra
+  retrace on Adreno 750 loses its palette-sampled sprites (SSIM 0.956378, deterministic). What was
+  measured:
+  - Both `vkCmdClearAttachments` and LOAD_OP_CLEAR fail; so does splitting the clear calls.
+  - Colour-only or depth-only in the pass, with the other part standalone, passes.
+  - Event traces: the first draw flushes the recording (the palette `glTexImage2D`'s upload flush,
+    `FlushPendingCommands`) right after the clear. Command order otherwise matches the standalone
+    path.
+  - Not explained yet. Lavapipe and the InPassClear scenarios on FBOs are clean.
+  - Ruled out by reading: a missing ordering edge. Default-framebuffer images get the same
+    pass-begin barrier as FBO images (a full ALL_COMMANDS memory barrier, or their transition),
+    with their layout tracked per swapchain index through the swapchain object.
+  - Still open: the in-pass path commits the freshly acquired swapchain image
+    (`DefaultFramebufferWriteIndex(true)` in `PrepareWireDrawPass`) at the clear; check whether
+    the standalone path's clear and the in-pass clear address the same index on the trace route.
+  - Cost of leaving it: small for FCL (MC renders into its own FBO and blits to the default one).
+- **M2, draw-pass cache across frames:** one cache, entries pruned when their image epochs move,
+  destroyed after their last submit. Kill switch: `MOBILEGL_MAGMA_PASS_CACHE=0`.
+- **M3, blit descriptor sets:** one pool per frame slot.
+- **M9, converted vertex streams** (wrongly marked "have" in the first audit). The wire arm
+  re-converted the whole remaining stream on every draw: Sodium's multi-draws over one arena cost
+  49 ms/frame of CPU on BSL (19.5 fps against dev's 64.7).
+  - Now one conversion per (store content serial, layout), kept across frames in
+    `VkBufferManager` (`LookupConvertedStream` / `StoreConvertedStream`).
+  - Each entry owns a vertex buffer. Bounded at 192 MB (least recently used first); entries idle
+    for 256 frames go. A replaced buffer is released when its frame slot comes round.
+  - The content serial is process-wide monotonic and moves on every host write, respecify,
+    GPU-write mark, shader-writable bind and donation.
+  - Kill switch: `MOBILEGL_MAGMA_VERTEX_CONVERSION_CACHE=0`.
+- **M4, write renaming.** A busy `glBufferSubData` used to take a staged copy, which ended the
+  render pass with two full barriers (FCL: ~5.6 per frame, all on one 288-byte store). Now:
+  - A store of 64 KB or less keeps a host shadow (64 MB budget), from respecify on.
+  - A busy write into a shadowed store gives it a fresh VkBuffer filled from the shadow and
+    releases the old one like a glBufferData orphan. No copy is recorded and the pass stays open.
+  - GPU-written marks, shader-writable binds, donation and import drop the shadow.
+  - `ReadWireBuffer` serves shadowed stores from the shadow, so no host wait.
+  - Kill switch: `MOBILEGL_MAGMA_WRITE_RENAME=0`.
+- **Tests:**
+  - `InPassClearScenario`: 4 cases, monolith and split, both backends.
+  - `ConvertedVertexStreamScenario`: 3 cases (reuse, same-frame sub-data, cross-frame sub-data).
+    Red-once: a lookup that ignores the content serial turns both sub-data cases red.
+  - `WriteRenameScenario`: 4 cases (order and carry, one-byte writes, uniform and index buffers,
+    multi-frame). Red-once: a rename that skips the shadow fill turns all 4 red, which also
+    proves the rename engages.
+  - Red-once checks: the colour index and the scissor (2 cases red); the depth clear-load bit
+    (depth/stencil case red).
+  - Syncval gates green (cache 7/7, run-ahead 14/14), plus the InPassClear cases under syncval.
+- **Gates:**
+  - SSIM 24/24 (openra, rd12, bsl × 4 arms × 2 backends) on `p15e`.
+  - Host unit and integration-gpu green, apart from the known environmental `ServerSpawn` /
+    `FuzzArm2` / `PeerLatch` / spawn ColdStart failures.
+  - `CompositorRecovery` flakes under -j16 and passes alone 3/3.
+
+**S0 milestone (one device session, FCL, interval 0, 2 reps; `runs/ms-s0i`):**
+
+| scene | build | Magma CPU ms/frame | Magma fps | Espryt CPU ms/frame | Espryt fps | GPU busy |
+|---|---|---|---|---|---|---|
+| vanilla 1.21.5 (CPU) | fin | 3.27 | 170 | 2.81 | 190 | 12-15 % |
+| | p15h (S0 without M4) | 2.59 | 191 | - | - | 15 % |
+| | p15i (S0) | 2.56 | 190 | 2.77 | 193 | 13-15 % |
+| | dev | 1.79 (n=1) | 234 | 2.14 | 218 | 14 % |
+| | MobileGlues | 1.59 | 256 | | | 15 % |
+| BSL (GPU) | p15i | 7.26 | 77.2 | 7.61 | 70.6 | 99 % |
+| | dev | 13.85 | 66.1 | 7.17 | 70.5 | 85-100 % |
+| | MobileGlues | 6.25 | 78.2 | | | 98-100 % |
+
+(BSL before the converted-stream cache: Magma 19.5 fps at 26 % GPU busy, 49 ms CPU/frame.)
+
+- Espryt's `base` lib (p15plan, unchanged) read 2.84 in the same session: the earlier "p15e
+  Espryt regression" was session variance. S0 does not touch Espryt.
+- **Target status.** Magma BSL is met (within 1.3 % of MobileGlues, GPU-bound). Espryt BSL is
+  GPU-bound at 99 % and 10 % below MobileGlues, same as dev: a GPU-side gap still to explain (the
+  load/store and clear audit of PLAN §0.1). CPU scene: Magma 2.56 vs MobileGlues 1.59, Espryt 2.77.
+- **Where Magma's CPU goes now** (p15i profile, `runs/prof-hi`, ms/frame inclusive):
+  `VerbChannel::Port` 0.85, `OnDrawVbo` 0.62, `SetupWireDraw` 0.46, `Present` 0.32,
+  `MGPipeValidateForVerb` 0.27, in-pass clears 0.18. The draw path is S1-S3's; S0's non-draw ports
+  are done. Proof that M4 engages: `StagedWireRangeCopy` 0.067 is gone and `RenameBusyWireStore`
+  costs 0.048 (a VMA allocation per rename; a recycle pool could take most of it).
+
+**Stage P:** dropped for the target.
+
+## Harness
+
+| purpose | where |
+|---|---|
+| FCL probes and reducers | `tools/device_bench/disagg/` (README "P15 FCL probes") |
+| FCL milestone (CPU + BSL scenes, fin / new / dev / MobileGlues) | scratchpad `p15/fcl_milestone.sh`, `fcl_scene.sh`, `fclab.sh`, `fclsum.py` |
+| SSIM gate (openra, rd12, bsl × 4 arms × 2 backends) | scratchpad `p15/ssim_gate.sh`; `matrix_p14.sh` now takes `PREP_WLS` and a `bsl` workload |
+| host CI-shaped build (WSL archlinux) | `~/mgl-p15-ci31`, scratchpad `p15/wsl_ci.sh`, `w_suite.sh`, `w_syncval.sh`, `w_sv_one.sh`, `w_one.sh` |
+| trace APK builds | scratchpad `build_trace.sh W:/p15 .<tag>` (rerun once on the `IncrementalSplitterRunnable` failure) |
+
+**Host-build trap:** editing a header while ninja is compiling leaves objects newer than the header
+but built from its old text. This happened once and gave a `bad_alloc` in every scenario. Never
+edit sources during a build; `touch` the edited headers if in doubt.
+
+**Device facts:**
+- Clocks are pinned.
+- FCL's original lib md5 is `00c09f0a…`.
+- MC 1.21.5 options md5 is `93b3f708…`; it is backed up at `/data/local/tmp/p15-options.txt`.
+- Iris properties are backed up at `/data/local/tmp/p15-iris.properties`.
+- `fcl_scene.sh restore` restores Iris; `fcl_p14.sh restore` restores the lib and `config.json`.
+- The anland desktop is stopped during runs and restarted afterwards.

docs/Disaggregated/notes/perf-p15/PARITY-AUDIT.md

@@ -0,0 +1,56 @@
+# P15 S0.1: parity audit of the deleted frontend-arm mechanisms
+
+**Question.** P13 W4 moved monolith onto the record/wire arm, and W6e / `de6ab167` then deleted the
+frontend arm as dead code. Which performance mechanisms did that arm have, and does the wire arm
+have an equivalent?
+
+**Sources:**
+- `origin/dev` and the pre-W6 tree `88a11b81` (the frontend arm);
+- the HEAD wire arm;
+- [PARITY-SEED.md](PARITY-SEED.md), the regex-extracted list of deleted definitions;
+- the profiles in PLAN-P15.md §2.2: FCL MC 1.21.5 and rd12, render thread, ms/frame.
+
+**Verdicts:**
+- **port**: measured as missing, and the port is planned;
+- **have**: the wire arm has an equivalent;
+- **skip**: not shown in any profile; reconsider only if one shows it.
+
+## Magma (DirectVulkan)
+
+| # | mechanism (dev / 88a11b81) | wire arm at HEAD | profile evidence | verdict, cost |
+|---|---|---|---|---|
+| M1 | **Clears.** `VkClearManager`: a glClear inside the active pass on the same attachments became `ClearAttachmentsOnActiveRenderPass`; otherwise a pending clear keyed (texture identity, level, layer) turned the next pass's load op into CLEAR, materialised before reads, blits and copies | `ClearWireFramebuffer`: ends the draw pass; per surface, a new view + render pass + framebuffer, its own pass, then retire | FCL 0.42-0.46 ms (dev ~0.05); rd12 ~0.27 ms. Dev's per-draw `HasPendingClear` probe cost ~0.6 ms/frame on rd12 | **port, in two steps.** (a) The in-pass half: the clear takes the draw pass a draw to the same framebuffer would take (`PrepareWireDrawPass`) and records `vkCmdClearAttachments` in it. No new state, nothing deferred, so no read path has to materialise anything. (b) LOAD_OP_CLEAR only if a GPU-side reason appears; CPU-wise (a) removes the object churn and the pass break. Small cost, low risk |
+| M2 | **Render pass + framebuffer cache.** `VkRenderPassManager::GetOrCreateRenderPass`, hash-keyed `RenderPassEntry` (render pass, framebuffer), garbage-collected at present | `m_wireDrawPassCaches`: exact-key reuse within one frame slot, emptied when the slot's frame begins (`ClearWireDrawPassCache`). Every frame rebuilds each framebuffer's pass, views and VkFramebuffer | FCL: vkCreateFramebuffer in SetupWireDraw 0.04, `~RenderPassEntry` 0.04-0.05, plus views | **port**: keep entries across frames while the key's image epochs hold, and retire through the submit-indexed list when an image goes away. Medium cost, medium risk (view lifetime against image release) |
+| M3 | **Blit resources.** `InitializeBlitResources`, `TryBlitToDefaultFramebufferWithShader` (persistent pipeline and pool) | `BlitWireColorImage`: persistent pipelines, but per blit a descriptor **pool** + set, 2 views and a framebuffer, all retired | FCL 0.13 ms (vkCreateDescriptorPool 0.084 + destroy) vs dev 0.02 | **port**: one pool per frame slot, reset when the slot's frame begins; a full pool falls back to the old path. Views and framebuffer are next if they show up. Small cost, low risk |
+| M4 | **Buffer sub-data.** `VkBufferManager::OnSubData`: a busy resident buffer took a staged copy (`StagedRangeCopy`, as now). A *streamed* buffer (`AcquireStreamedSlice`) updated only the CPU shadow, and the next draw streamed the whole buffer into a fresh arena slice, so no copy and no pass break | `WriteWireBuffer`: busy → `StagedWireRangeCopy`, which ends the pass and records two full barriers; idle → host memcpy. No streamed/renamed buffers on the wire | FCL 0.28 ms (11 sub-data calls/frame, ~25 µs each), and each one splits the pass on the GPU. WBUF probe: 4.5 host + 6.7 staged writes/frame; ~5.6 of the staged ones hit one 288-byte store | **ported**: write renaming for stores of 64 KB or less with a host shadow; the rename goes through the same orphan release as glBufferData, which every bind memo already handles (VkBuffer compare + destroy epoch) |
+| M5 | **Draw fast path.** `TrySetupDrawFastPath` / `SetupDrawSnapshot` (4 snapshots on program lifetime id) | P14 wire memos: pass continuation (`WirePassMemoHolds`), unit sampler memo, pipeline / vertex / index bind dedupe, vertex input on its record | FCL SetupWireDraw 0.55 ≈ dev SetupDraw 0.54 | **have** |
+| M6 | **Uniform rebinds.** `UniformManager` `FastRebindMemo`, `SampledBindingsUnchanged`, `CollectSampledTextures` walk-skip | P14 descriptor-set rebind with fresh offsets, walk-skip | small | **have** |
+| M7 | **Vertex input.** `VertexInputStateFactory` per-VAO memos (`VaoBackendMemos`, memoised hash) | vertex input kept on its record (`ResolveWireVertexInput`) | small | **have** |
+| M8 | **Texture sync.** `VkTextureManager` `SyncedTextureMemoEntry`, `DrawSyncScope` | P14 direct-mapped texture fronts, record-serial checks | small | **have** |
+| M9 | **Vertex streams.** `UploadAndBindVertexBuffers`, `ConvertedVertexStream` cache (per frame, pointer + changeSerial), `LookupVaoDrawMemo` | wire vertex slices + bind shadow, but NO converted-stream cache: every draw re-converted the whole remaining stream | FCL: none (no conversion). BSL: `ConvertWireVertexStream` 49 ms/frame, Sodium multi-draws over one arena (Magma 19.5 fps vs dev 64.7) | **ported** (corrected from "have"): per-frame cache keyed on a monotonic store content serial, then kept across frames in `VkBufferManager` |
+| M10 | **Depth mipmap / MSAA resolve / YUV**: per-op descriptor pools existed in dev too | same per-op pools | not in FCL/rd12 profiles | **skip** |
+| M11 | **Command-buffer reset/end** | larger, more fragmented recordings | FCL +0.16 ms | re-measure after M1-M4 (follows from the pass count) |
+
+## Espryt (DirectGLES)
+
+| # | mechanism | wire/record arm at HEAD | profile evidence | verdict |
+|---|---|---|---|---|
+| E1 | `StateBackendObjectRegistry` (pointer-keyed twins, GC) | twins keyed on applier serials / lifetime ids (P14) | backend +0.09 ms on FCL | have |
+| E2 | `SyncCurrentProgram`, `BindCurrentProgramWithResources` | `SyncCurrentProgramByHandle` + P14 memos | rd12 +0.05/+0.07 | covered by S3 (backend draw-delta) |
+| E3 | `SyncVaoAttributeBuffersByHandle` | `SyncVaoAttributeBuffersByRecord`, `AdoptUnchangedBufferSets` | rd12 +0.09 | covered by S2/S3 |
+| E4 | `BackendFramebufferObject::SyncToBackend` (dirty-bit sync) | record-serial sync, clean-attachment skip (P14) | small | have |
+| E5 | clear / blit (host GL) | same host calls | FCL clear 0.16 vs 0.13, blit 0.125 vs 0.11 | have (no Espryt port; user confirmed) |
+| E6 | buffer sub-data (`Ops_SubDataTracked`) | `Ops_H_SubDataTracked` via record | 0.024 vs 0.007 | skip |
+
+Espryt's FCL gap is the client half of the draw path (+0.39 ms), which stages S1-S3 address.
+
+## Order of ports (S0.2)
+
+1. M1a: in-pass clear.
+2. M3: blit frame pools.
+3. M2: cross-frame pass cache.
+4. M4: sub-data renaming.
+5. Re-measure M11.
+
+Every step: FCL presenting-thread CPU before and after, in the same session, with the MobileGlues
+and dev reference rows.

docs/Disaggregated/notes/perf-p15/PLAN-P15.md

@@ -49,11 +49,12 @@ This is the plan, plus the measurements it rests on (section 2). Nothing here is
 Stage order:
 - 0: parity audit + wire non-draw fast path;
 - 0b: an FCL 1.21.5 trace fixture;
-- P: swap-interval-correct presentation. It is independent of 1-4 and can run in parallel after 0;
 - 1: client draw-delta prototype;
 - 2: per-VAO / per-program slices;
 - 3: applier + backend draw-delta;
-- 4: split arms, verify, default-on.
+- 4: split arms, verify, default-on;
+- P (swap-interval-correct presentation): **not needed for the target; dropped unless the user asks**
+  (2026-10-08). MobileGlues pays the same ~2 ms present wait, so matching it does not need P.
 
 Decisions taken with the user (2026-10-08):
 - Presenting-thread CPU ms/frame is the primary FCL metric, with fps beside it.
@@ -62,6 +63,36 @@ Decisions taken with the user (2026-10-08):
 - The probe tools are committed under `tools/device_bench/disagg/`.
 - No Espryt deferred-clear port: its clear and blit costs match dev's.
 
+## 0.1 Target and stopping rule (user, 2026-10-08)
+
+**Target:** "We only need to match MobileGlues' performance."
+
+**Success criterion:** both backends reach the MobileGlues plugin (`com.fcl.plugin.mobileglues`)
+in FCL MC 1.21.5, on the same pinned device, same world, same swap interval. Its presenting-thread
+CPU there is ~1.7 ms/frame (~248 fps) at interval 0 and ~1.9 ms/frame at interval 1.
+
+| build | interval 0 CPU ms/frame | needed |
+|---|---|---|
+| feat Magma | 3.12 | −1.4 |
+| feat Espryt | 2.73 | −1.0 |
+| dev Magma | 1.76 | (reference) |
+| dev Espryt | 2.14 | (reference) |
+
+Espryt has to go below dev here.
+
+**Success criterion has two parts (user, 2026-10-08):**
+- CPU-bound scenes: within ~5 % of MobileGlues presenting-thread CPU ms/frame.
+- GPU-bound shaderpack scenes (Iris + BSL): within ~5 % of the better of MobileGlues and dev fps.
+
+**Stopping rule:** P15 ends when FCL presenting-thread CPU on both backends is within noise (~5 %)
+of MobileGlues at the same interval, and the BSL scene meets its fps bar. It also ends if a remaining gap is shown to be irreducible,
+with the reason written down.
+
+**Reference rows in every FCL milestone,** measured in the same session:
+- MobileGlues (primary);
+- dev (second).
+MobileGlues also runs on the trace fixtures where the harness can load it.
+
 ## 1. Measurement rules for P15
 
 - **FCL:**
@@ -77,6 +108,23 @@ Decisions taken with the user (2026-10-08):
     thread idles between frames. Compare CPU only within one interval setting.
   - Take readings only from in-world steady windows. The probe waits for MC's `Time elapsed:`
     line and then four fps-log windows of more than 150 frames each.
+- **GPU-bound scene (user, 2026-10-08):**
+  - FCL `1.21.4-Fabric` with Iris and `BSL_v10.1.3`, at the same world and interval, measured live
+    with `fcl_scene.sh bsl` and restored afterwards.
+  - The BSL fixture `minecraft-1.21.4-fabric-iris-bsl-in-world` joins the SSIM gate. It is a
+    single-frame trim, so it is a correctness check, not a benchmark.
+  - Show the scene is GPU-bound, with evidence:
+    - kgsl busy near 100 % (`gpu_clock_stats`, kgsl cmdbatch union from ftrace);
+    - the presenting thread waiting in fences or present;
+    - the `gsl_syncobj_wait` share.
+  - On GPU-bound runs, fps is the metric.
+  - A GPU regression on BSL blocks a change even if CPU improved.
+- **GPU metrics in every milestone:**
+  - kgsl GPU busy % for every FCL run (`fclsum.py`);
+  - for GPU-bound runs, fps against dev and MobileGlues.
+- **Load and store ops:** check every pass change in GPU terms. Prefer LOAD_OP_CLEAR or DONT_CARE
+  over loading and clearing on the tiler, and use DONT_CARE stores for depth/stencil that nothing
+  reads after the pass.
 - **Trace (rd12, openra, and the FCL 1.21.5 fixture from stage 0b):**
   - fps, best of 3.
   - Per-layer CPU in ms/frame from `profile_p14.sh` / `layers.py`. Remember that rd12 carries
@@ -352,6 +400,9 @@ Espryt's deleted mechanisms. Port only those that show in profiles.
 
 ### Stage P: swap-interval-correct presentation (both backends)
 
+**Status: not needed for the target; dropped unless the user asks (2026-10-08).** MobileGlues
+pays the same ~2 ms present wait, so the target (section 0.1) does not need this stage.
+
 **Goal:**
 - With interval 0, presentation never paces the app: FCL fps becomes 1 / presenting-thread CPU.
 - With interval ≥1 the app is still paced to the display: 165 fps ±1 % on this device, with no

tools/device_bench/disagg/matrix_p14.sh

@@ -49,6 +49,8 @@ wl_args() { # trace golden w h target crop
   case "$1" in
     openra) echo "openra openra.0000031249.png 640 480 31249 1 1 638 478" ;;
     rd12) echo "rd12 minecraft-1.21.4-rd12-odinlite-in-world.0004660351.png 854 480 4660351 0 0 0 0" ;;
+    # P15: GPU-bound shaderpack scene (a single-frame trim: ssim only, not a benchmark).
+    bsl) echo "bsl minecraft-1.21.4-fabric-iris-bsl-in-world.0000110725.png 854 480 110725 0 0 0 0" ;;
   esac
 }
 
@@ -74,8 +76,13 @@ cool_down() {
 prep_pkg() {
   local pkg="$1" app; app=$(app_of "$pkg")
   local tmp="$ROOT/fixtures"; mkdir -p "$tmp"
-  for wl in openra rd12; do
-    local tgz="$FIXTURES/$( [ $wl = openra ] && echo openra.tgz || echo minecraft-1.21.4-rd12-odinlite-in-world.tgz)"
+  for wl in ${PREP_WLS:-openra rd12}; do
+    local tgz
+    case $wl in
+      openra) tgz="$FIXTURES/openra.tgz" ;;
+      rd12) tgz="$FIXTURES/minecraft-1.21.4-rd12-odinlite-in-world.tgz" ;;
+      bsl) tgz="$FIXTURES/minecraft-1.21.4-fabric-iris-bsl-in-world.tgz" ;;
+    esac
     set -- $(wl_args $wl)
     local golden="$FIXTURES/$2"
     if [ ! -f "$tmp/$wl.trace" ]; then