MobileGL/docs/Disaggregated/notes/perf-p15/HANDOFF.md

# P15 handoff (living document)

The plan and its measurements are in [PLAN-P15.md](PLAN-P15.md); the audit is in
[PARITY-AUDIT.md](PARITY-AUDIT.md).

**Target** (PLAN §0.1, revised by the user 2026-10-08):
- CPU scenes: FCL presenting-thread CPU at or below dev monolith (Magma <= 1.79, Espryt <= 2.14
  ms/frame), dev measured in the same session, ~5 % noise.
- GPU scene (Iris + BSL): fps within ~5 % of MobileGlues on both backends.

## Latest (2026-10-08, after S0)

Every number below carries its profile and passed the validity check (`state.txt`). `cpuhunt` =
CPU about 1.5/1.25 GHz, GPU 903 MHz (cleanly CPU-bound); interval 0, anland off, daemons stopped.

- **The S0 milestone's absolute numbers are superseded.** The perf stack's `msm_performance` caps
  (787 / 1286 / 1075 / 1248 MHz) were clamping the CPU under the pin for some of that session. Only
  same-session deltas from it stand. Measurement now uses `bench_session.sh` and the sampler (PLAN
  §1), and the probe reports only clean fps windows taken before any tracing (`P15PROBE
  trace-start` marker, 25 s settle).
- **FCL vanilla 1.21.5, cpuhunt, presenting-thread CPU ms/frame:**

  | build | Espryt | Magma |
  |---|---|---|
  | dev | 1.76 | 1.60 |
  | p15k (S0 + VAO shadow) | 2.10 | 2.03 |
  | p15l (+ vertex-elements dedup) | 2.10 | 2.08 |

  MobileGlues: 1.58 in an earlier cpuhunt session (needs a same-session re-run with clean windows).
  Floor gap: Espryt +0.34, Magma +0.43 ms.
- **Driver-call redundancy:**
  - Measured with a diagnostic build that counts every host GLES/EGL call and redundant re-sets
    (`p15diag` worktree, never committed; HostCount.inl and VkCount.cpp).
  - Espryt sent 3717 host calls/frame against MC's ~874: 2118 `glDisableVertexAttribArray`, 378
    enable, 378 divisor, all re-sets.
  - Two fixes, each independently removes the walk churn:
    - b4e209e1: a shadow of each driver VAO (850 calls/frame);
    - c8a5abe5: the client no longer re-creates an identical vertex-elements record on every
      `glBindVertexBuffer`. With the shadow off it takes 3724 down to 1228.
  - CPU gain is small (about 0.1 ms): the calls were cheap in the driver.
  - Magma issues about 380 Vulkan calls/frame with essentially none redundant.
- **Where Espryt's gap to dev is now:** the driver time equals dev's (0.50 vs 0.48). The excess is
  libMobileGL itself, 0.75 vs 0.32: client half 0.51, backend 0.24, applier 0.02. The client half
  is spread thin (validate, tracker, emitters, residual copy): S1's draw-delta.
- **Inproc (cpuhunt):**
  - Espryt reaches 306 fps vs MobileGlues about 319; GL thread 1.68 ms, apply thread 1.83 ms of a
    3.27 ms frame.
  - Neither thread is saturated. The apply thread waits in `queueBuffer` on a GPU fence
    (present pacing), and the GL thread waits on the apply thread.
  - `MOBILEGL_IPC_PRESENT_CREDIT=2` changes nothing (300 fps).
  - Beating MobileGlues on inproc needs a present-path change (stage P territory, dropped by the
    user), not CPU work.
- **S0b:** the fixture `minecraft-1.21.5-vanilla-fcl-in-world` is landed (4e27342e) and is in the
  SSIM gate as `fcl` (32/32 on p15j).
- **Knobs on FCL:** `adb shell setprop debug.mobilegl.env "MOBILEGL_A=1;MOBILEGL_B=0"` (49c6394d);
  `fclab.sh` takes a 6th field `K=V,K=V`.
- **MC's AFK frame limit:** `inactivityFpsLimit:"afk"` (MC 1.21.2+) made later probe windows and
  profiles spin in `System.nanoTime`. `fcl_scene.sh` now sets `"minimized"` for bench scenes and
  restores the original options files (`/data/local/tmp/p15-opts-121{5,4}.txt`). Profiles taken
  before this are partly AFK-polluted.
- **BSL at `gpuhunt`** (CPU 2380/1689 MHz, GPU 578 MHz, all 100 % GPU busy):
  - MobileGlues 53.4 fps;
  - dev Magma 53.2;
  - p15l Magma 51.8 (3 reps, -3.0 % vs MobileGlues: target met, -2.6 % vs dev to bisect);
  - Espryt dev and p15l 48.3 (-9.6 %): the GPU gap, inherited from dev.
- **Same-session cpuhunt, apply thread:**

  | build | fps | apply ms/frame |
  |---|---|---|
  | MobileGlues | 329 | CPU 1.55 |
  | Magma inproc p15m | 300 | 1.6 |
  | Espryt inproc | 296 | 1.65-1.8 |

  - Inproc beats MobileGlues only when the apply thread's CPU is below MobileGlues' (both pay the
    same queueBuffer wait). About 0.3 ms per frame beyond the apply CPU is still unexplained.
  - 500b63ea: per-store reuse of renamed-away wire buffers (Magma apply -0.13 ms).
  - ONE_TIME_SUBMIT: no effect (A/B'd, knob dropped).
  - Magma's driver work is about 2x dev's: vkResetCommandBuffer 0.09 + vkEndCommandBuffer 0.11
    ms/frame.
- **Inproc serialization found (2026-10-08, p15l cpuhunt profiles):** every blit reached the
  client as `BlitNamedFramebuffer`, a verb with no `MGP_VERB_OP_LIST` row, so
  `ClientVerbIsBarriered` answered barriered and the GL thread quiesced the applier once per frame
  (1.65 ms/frame parked in `WaitForApplyToCatchUp`, plus 0.11 from the following `glBindTexture`).
  The apply thread then idled 0.24 ms parked plus 0.2 ms spinning in `Doorbell::Wait`. e00db8e6
  replaces the inverse and its barriered fallback with a verb -> record-op table
  (`MG_Pipe/VerbRecordOps.h`, one row per verb by static_assert; no knob, user decision).
  - Same-session blab (cpuhunt): Magma inproc 319.6 -> 334.9 fps (+4.8 %), Espryt inproc
    324.4 -> 338.1 (+4.2 %).
  - Wait census (`MOBILEGL_IPC_WAIT_STATS=1`, 7acf3ea8): FCL vanilla inproc waits only on the
    present credit (1.00/frame). BSL waits 7 times per frame on three kWaitApplied rows plus 22
    follow-on quiesces; see [WAIT-AUDIT.md](WAIT-AUDIT.md). User decision applied in 2fbcc224:
    fire-and-forget rows async, inproc creates and PBO readbacks no longer wait, and the dual block
    (`MOBILEGL_IPC_ROLE_SPLIT_STATE`, now default on) lets a barriered verb cost one wait. BSL
    census after it still owed (gpuhunt only; FCL's 1.21.4-Fabric profile exited with signal 34 at
    JVM start on both backends in the last attempt).
  - db915f76: a client parked on the present credit is rung only by the credit (apply-thread
    futex wakes 0.056 ms/frame before).
  - Frames in flight (0b96a3ef): `MOBILEGL_FRAMES_IN_FLIGHT` (default 3) drives Magma's frame
    contexts, Espryt's post-present fence wait and the split present credit max(1, N-2); the Magma
    name is an alias. Default latency unchanged; the FIF A/B is the user's input for the default.
  - Present credit 2 (cens, cpuhunt, all VALID): Magma 309.0 -> 319.9, Espryt 299.1 -> 319.1 fps
    vs MobileGlues 321.9 in the same session; apply CPU 1.40-1.43 vs MobileGlues' GL thread 1.57.
    Credit 2 costs a frame of latency: user decision. The remaining inproc gap is the queueBuffer
    fence wait (GPU-side), not CPU.
  - BSL at cpuhunt (GPU 903 MHz) reaches GPU 95 C and kgsl thermal_pwrlevel 1 (834 MHz cap): the
    census run was INVALID twice and the job was stopped. Run BSL only at gpuhunt.
  - Apply-thread frame (Magma inproc): on-CPU 1.75 (0.2 of it spin), queueBuffer fence wait 1.45,
    parked 0.24 ms.
  - Magma apply on-CPU by verb: draws 0.55 (SetupWireDraw 0.40, of it descriptor binding 0.13),
    present 0.37 (End CB 0.10, queueBuffer CPU 0.08, submit 0.07, wire-store sweep 0.035),
    clear 0.19 (the frame's recording begin 0.09 + pass begin 0.05), buffer writes 0.09, blit
    0.06 (two image views and a VkFramebuffer created and retired per blit).
- **Device contention, root cause (2026-10-08):** the anland app was relaunched by `fclab.sh`'s own
  end-of-run `monkey` (launchedFromUid=0); its activity takes the foreground back whenever paused,
  so FCL launched over it never reaches the world (RenderDoc job: 0 of 4 worlds). c41dfdda:
  `bench_session.sh start` force-stops it and `stop` relaunches it only if it ran before.
- **Open user decision:** Espryt's apply thread spends 0.06 ms/frame in `LatchIfGuiltyBeforeApply`
  (glGetGraphicsResetStatus is a kgsl ioctl, queried on every wake). Rate-limiting it to 2 ms
  breaks `ServerLoopEglLatchTest.AGuiltyResetEndsTheSessionBeforeItsNextRecord...`, the designed
  promptness guarantee, so it was reverted. Trading that guarantee for 0.06 ms is the user's call.
- **CI:** 4a8ecf9a green. 49c6394d: one job failed on a hosted runner's `apt-get update` timeout
  (45 min, infrastructure); the failed job is re-run.
- **Next:**
  - RenderDoc (rdc GLES layer, remote server stopped): Espryt vs MobileGlues BSL pass structure;
  - S1 (client validate 0.25 ms) and S3 (backend by-handle syncs, descriptor resolve) for the CPU
    floor and the inproc apply thread.

## Status (2026-10-08)

**Branch:** `p15impl` (worktree `.claude/worktrees/p15`, short path `W:/p15`), pushed to
`feat/disaggregated`.

**S0.1 parity audit:** written.

**S0.2, Magma wire non-draw ports:**
- **M1a, clears become part of the draw pass** (`ClearWireFramebufferInPass`; `PrepareWireDrawPass`
  split out of `SetupWireDraw`):
  - A whole-area clear that begins the pass begins it with a LOAD_OP_CLEAR variant of the pass's
    render pass (`RenderPassEntry::clearVariants`, compatible with the framebuffer).
  - Any other clear (into the open pass, or scissored) is `vkCmdClearAttachments` in the pass. A
    forced pass restart for depth clears (an LRZ hypothesis) was tried and reverted: it cost CPU
    and fixed nothing.
  - Kill switches: `MOBILEGL_MAGMA_INPASS_CLEAR=0`; `MOBILEGL_MAGMA_INPASS_CLEAR_DEFAULT=1` puts
    default-framebuffer clears in the pass too (off by default, see the open item).
- **Open item: default-framebuffer clears stay standalone.** With them in the pass, the openra
  retrace on Adreno 750 loses its palette-sampled sprites (SSIM 0.956378, deterministic). What was
  measured:
  - Both `vkCmdClearAttachments` and LOAD_OP_CLEAR fail; so does splitting the clear calls.
  - Colour-only or depth-only in the pass, with the other part standalone, passes.
  - Event traces: the first draw flushes the recording (the palette `glTexImage2D`'s upload flush,
    `FlushPendingCommands`) right after the clear. Command order otherwise matches the standalone
    path.
  - Not explained yet. Lavapipe and the InPassClear scenarios on FBOs are clean.
  - Ruled out by reading: a missing ordering edge. Default-framebuffer images get the same
    pass-begin barrier as FBO images (a full ALL_COMMANDS memory barrier, or their transition),
    with their layout tracked per swapchain index through the swapchain object.
  - Still open: the in-pass path commits the freshly acquired swapchain image
    (`DefaultFramebufferWriteIndex(true)` in `PrepareWireDrawPass`) at the clear; check whether
    the standalone path's clear and the in-pass clear address the same index on the trace route.
  - Cost of leaving it: small for FCL (MC renders into its own FBO and blits to the default one).
- **M2, draw-pass cache across frames:** one cache, entries pruned when their image epochs move,
  destroyed after their last submit. Kill switch: `MOBILEGL_MAGMA_PASS_CACHE=0`.
- **M3, blit descriptor sets:** one pool per frame slot.
- **M9, converted vertex streams** (wrongly marked "have" in the first audit). The wire arm
  re-converted the whole remaining stream on every draw: Sodium's multi-draws over one arena cost
  49 ms/frame of CPU on BSL (19.5 fps against dev's 64.7).
  - Now one conversion per (store content serial, layout), kept across frames in
    `VkBufferManager` (`LookupConvertedStream` / `StoreConvertedStream`).
  - Each entry owns a vertex buffer. Bounded at 192 MB (least recently used first); entries idle
    for 256 frames go. A replaced buffer is released when its frame slot comes round.
  - The content serial is process-wide monotonic and moves on every host write, respecify,
    GPU-write mark, shader-writable bind and donation.
  - Kill switch: `MOBILEGL_MAGMA_VERTEX_CONVERSION_CACHE=0`.
- **M4, write renaming.** A busy `glBufferSubData` used to take a staged copy, which ended the
  render pass with two full barriers (FCL: ~5.6 per frame, all on one 288-byte store). Now:
  - A store of 64 KB or less keeps a host shadow (64 MB budget), from respecify on.
  - A busy write into a shadowed store gives it a fresh VkBuffer filled from the shadow and
    releases the old one like a glBufferData orphan. No copy is recorded and the pass stays open.
  - GPU-written marks, shader-writable binds, donation and import drop the shadow.
  - `ReadWireBuffer` serves shadowed stores from the shadow, so no host wait.
  - Kill switch: `MOBILEGL_MAGMA_WRITE_RENAME=0`.
- **Tests:**
  - `InPassClearScenario`: 4 cases, monolith and split, both backends.
  - `ConvertedVertexStreamScenario`: 3 cases (reuse, same-frame sub-data, cross-frame sub-data).
    Red-once: a lookup that ignores the content serial turns both sub-data cases red.
  - `WriteRenameScenario`: 4 cases (order and carry, one-byte writes, uniform and index buffers,
    multi-frame). Red-once: a rename that skips the shadow fill turns all 4 red, which also
    proves the rename engages.
  - Red-once checks: the colour index and the scissor (2 cases red); the depth clear-load bit
    (depth/stencil case red).
  - Syncval gates green (cache 7/7, run-ahead 14/14), plus the InPassClear cases under syncval.
- **Gates:**
  - SSIM 24/24 (openra, rd12, bsl × 4 arms × 2 backends) on `p15e`.
  - Host unit and integration-gpu green, apart from the known environmental `ServerSpawn` /
    `FuzzArm2` / `PeerLatch` / spawn ColdStart failures.
  - `CompositorRecovery` flakes under -j16 and passes alone 3/3.

**S0 milestone (one device session, FCL, interval 0, 2 reps; `runs/ms-s0i`):**

| scene | build | Magma CPU ms/frame | Magma fps | Espryt CPU ms/frame | Espryt fps | GPU busy |
|---|---|---|---|---|---|---|
| vanilla 1.21.5 (CPU) | fin | 3.27 | 170 | 2.81 | 190 | 12-15 % |
| | p15h (S0 without M4) | 2.59 | 191 | - | - | 15 % |
| | p15i (S0) | 2.56 | 190 | 2.77 | 193 | 13-15 % |
| | dev | 1.79 (n=1) | 234 | 2.14 | 218 | 14 % |
| | MobileGlues | 1.59 | 256 | | | 15 % |
| BSL (GPU) | p15i | 7.26 | 77.2 | 7.61 | 70.6 | 99 % |
| | dev | 13.85 | 66.1 | 7.17 | 70.5 | 85-100 % |
| | MobileGlues | 6.25 | 78.2 | | | 98-100 % |

(BSL before the converted-stream cache: Magma 19.5 fps at 26 % GPU busy, 49 ms CPU/frame.)

- Espryt's `base` lib (p15plan, unchanged) read 2.84 in the same session: the earlier "p15e
  Espryt regression" was session variance. S0 does not touch Espryt.
- **Target status.** Magma BSL is met (within 1.3 % of MobileGlues, GPU-bound). Espryt BSL is
  GPU-bound at 99 % and 10 % below MobileGlues, same as dev: a GPU-side gap still to explain (the
  load/store and clear audit of PLAN §0.1). CPU scene: Magma 2.56 vs MobileGlues 1.59, Espryt 2.77.
- **Where Magma's CPU goes now** (p15i profile, `runs/prof-hi`, ms/frame inclusive):
  `VerbChannel::Port` 0.85, `OnDrawVbo` 0.62, `SetupWireDraw` 0.46, `Present` 0.32,
  `MGPipeValidateForVerb` 0.27, in-pass clears 0.18. The draw path is S1-S3's; S0's non-draw ports
  are done. Proof that M4 engages: `StagedWireRangeCopy` 0.067 is gone and `RenameBusyWireStore`
  costs 0.048 (a VMA allocation per rename; a recycle pool could take most of it).

**Stage P:** dropped for the target.

## Harness

| purpose | where |
|---|---|
| FCL probes and reducers | `tools/device_bench/disagg/` (README "P15 FCL probes") |
| FCL milestone (CPU + BSL scenes, fin / new / dev / MobileGlues) | scratchpad `p15/fcl_milestone.sh`, `fcl_scene.sh`, `fclab.sh`, `fclsum.py` |
| SSIM gate (openra, rd12, bsl × 4 arms × 2 backends) | scratchpad `p15/ssim_gate.sh`; `matrix_p14.sh` now takes `PREP_WLS` and a `bsl` workload |
| host CI-shaped build (WSL archlinux) | `~/mgl-p15-ci31`, scratchpad `p15/wsl_ci.sh`, `w_suite.sh`, `w_syncval.sh`, `w_sv_one.sh`, `w_one.sh` |
| trace APK builds | scratchpad `build_trace.sh W:/p15 .<tag>` (rerun once on the `IncrementalSplitterRunnable` failure) |

**Host-build trap:** editing a header while ninja is compiling leaves objects newer than the header
but built from its old text. This happened once and gave a `bad_alloc` in every scenario. Never
edit sources during a build; `touch` the edited headers if in doubt.

**Device facts:**
- Clocks: CPU 2035.2 MHz on policies 2/5/7 and 1574.4 MHz on policy0, GPU 680 MHz (new default since
  2026-10-08, pending the soak). Pinned per session by `bench_session.sh`, which every `devjob.sh`
  job runs.
- Frequency daemons are stopped only inside a bench session: `thermal-engine`, `perf2-hal-1-0`,
  `vendor.perfservice`, `performance` and `hyperschedule_hal_service`. Their prior state is in
  `/data/local/tmp/p15-bench-session` while a session is open, and they are restarted and verified
  on exit. If that file exists with no job running, a session was left open: run
  `bench_session.sh stop`.
- The perf stack's game-mode caps in `/sys/kernel/msm_performance/parameters/cpu_max_freq` survive
  its services and silently override the pin. The session releases them.
- FCL's original lib md5 is `00c09f0a…`.
- MC 1.21.5 options md5 is `93b3f708…`; it is backed up at `/data/local/tmp/p15-options.txt`.
- Iris properties are backed up at `/data/local/tmp/p15-iris.properties`.
- `fcl_scene.sh restore` restores Iris; `fcl_p14.sh restore` restores the lib and `config.json`.
- The anland desktop is stopped during runs and restarted afterwards.