diff --git a/docs/Disaggregated/notes/perf-p15/HANDOFF.md b/docs/Disaggregated/notes/perf-p15/HANDOFF.md new file mode 100644 index 00000000..2d165bab --- /dev/null +++ b/docs/Disaggregated/notes/perf-p15/HANDOFF.md @@ -0,0 +1,128 @@ +# P15 handoff (living document) + +The plan and its measurements are in [PLAN-P15.md](PLAN-P15.md); the audit is in +[PARITY-AUDIT.md](PARITY-AUDIT.md). + +**Target** (PLAN §0.1): +- CPU scenes: FCL presenting-thread CPU on both backends within ~5 % of the MobileGlues plugin. +- GPU scene (Iris + BSL): fps within ~5 % of the better of MobileGlues and dev. + +## Status (2026-10-08) + +**Branch:** `p15impl` (worktree `.claude/worktrees/p15`, short path `W:/p15`), pushed to +`feat/disaggregated`. + +**S0.1 parity audit:** written. + +**S0.2, Magma wire non-draw ports:** +- **M1a, clears become part of the draw pass** (`ClearWireFramebufferInPass`; `PrepareWireDrawPass` + split out of `SetupWireDraw`): + - A whole-area clear that begins the pass begins it with a LOAD_OP_CLEAR variant of the pass's + render pass (`RenderPassEntry::clearVariants`, compatible with the framebuffer). + - Any other clear (into the open pass, or scissored) is `vkCmdClearAttachments` in the pass. A + forced pass restart for depth clears (an LRZ hypothesis) was tried and reverted: it cost CPU + and fixed nothing. + - Kill switches: `MOBILEGL_MAGMA_INPASS_CLEAR=0`; `MOBILEGL_MAGMA_INPASS_CLEAR_DEFAULT=1` puts + default-framebuffer clears in the pass too (off by default, see the open item). +- **Open item: default-framebuffer clears stay standalone.** With them in the pass, the openra + retrace on Adreno 750 loses its palette-sampled sprites (SSIM 0.956378, deterministic). What was + measured: + - Both `vkCmdClearAttachments` and LOAD_OP_CLEAR fail; so does splitting the clear calls. + - Colour-only or depth-only in the pass, with the other part standalone, passes. + - Event traces: the first draw flushes the recording (the palette `glTexImage2D`'s upload flush, + `FlushPendingCommands`) right after the clear. Command order otherwise matches the standalone + path. + - Not explained yet. Lavapipe and the InPassClear scenarios on FBOs are clean. + - Ruled out by reading: a missing ordering edge. Default-framebuffer images get the same + pass-begin barrier as FBO images (a full ALL_COMMANDS memory barrier, or their transition), + with their layout tracked per swapchain index through the swapchain object. + - Still open: the in-pass path commits the freshly acquired swapchain image + (`DefaultFramebufferWriteIndex(true)` in `PrepareWireDrawPass`) at the clear; check whether + the standalone path's clear and the in-pass clear address the same index on the trace route. + - Cost of leaving it: small for FCL (MC renders into its own FBO and blits to the default one). +- **M2, draw-pass cache across frames:** one cache, entries pruned when their image epochs move, + destroyed after their last submit. Kill switch: `MOBILEGL_MAGMA_PASS_CACHE=0`. +- **M3, blit descriptor sets:** one pool per frame slot. +- **M9, converted vertex streams** (wrongly marked "have" in the first audit). The wire arm + re-converted the whole remaining stream on every draw: Sodium's multi-draws over one arena cost + 49 ms/frame of CPU on BSL (19.5 fps against dev's 64.7). + - Now one conversion per (store content serial, layout), kept across frames in + `VkBufferManager` (`LookupConvertedStream` / `StoreConvertedStream`). + - Each entry owns a vertex buffer. Bounded at 192 MB (least recently used first); entries idle + for 256 frames go. A replaced buffer is released when its frame slot comes round. + - The content serial is process-wide monotonic and moves on every host write, respecify, + GPU-write mark, shader-writable bind and donation. + - Kill switch: `MOBILEGL_MAGMA_VERTEX_CONVERSION_CACHE=0`. +- **M4, write renaming.** A busy `glBufferSubData` used to take a staged copy, which ended the + render pass with two full barriers (FCL: ~5.6 per frame, all on one 288-byte store). Now: + - A store of 64 KB or less keeps a host shadow (64 MB budget), from respecify on. + - A busy write into a shadowed store gives it a fresh VkBuffer filled from the shadow and + releases the old one like a glBufferData orphan. No copy is recorded and the pass stays open. + - GPU-written marks, shader-writable binds, donation and import drop the shadow. + - `ReadWireBuffer` serves shadowed stores from the shadow, so no host wait. + - Kill switch: `MOBILEGL_MAGMA_WRITE_RENAME=0`. +- **Tests:** + - `InPassClearScenario`: 4 cases, monolith and split, both backends. + - `ConvertedVertexStreamScenario`: 3 cases (reuse, same-frame sub-data, cross-frame sub-data). + Red-once: a lookup that ignores the content serial turns both sub-data cases red. + - `WriteRenameScenario`: 4 cases (order and carry, one-byte writes, uniform and index buffers, + multi-frame). Red-once: a rename that skips the shadow fill turns all 4 red, which also + proves the rename engages. + - Red-once checks: the colour index and the scissor (2 cases red); the depth clear-load bit + (depth/stencil case red). + - Syncval gates green (cache 7/7, run-ahead 14/14), plus the InPassClear cases under syncval. +- **Gates:** + - SSIM 24/24 (openra, rd12, bsl × 4 arms × 2 backends) on `p15e`. + - Host unit and integration-gpu green, apart from the known environmental `ServerSpawn` / + `FuzzArm2` / `PeerLatch` / spawn ColdStart failures. + - `CompositorRecovery` flakes under -j16 and passes alone 3/3. + +**S0 milestone (one device session, FCL, interval 0, 2 reps; `runs/ms-s0i`):** + +| scene | build | Magma CPU ms/frame | Magma fps | Espryt CPU ms/frame | Espryt fps | GPU busy | +|---|---|---|---|---|---|---| +| vanilla 1.21.5 (CPU) | fin | 3.27 | 170 | 2.81 | 190 | 12-15 % | +| | p15h (S0 without M4) | 2.59 | 191 | - | - | 15 % | +| | p15i (S0) | 2.56 | 190 | 2.77 | 193 | 13-15 % | +| | dev | 1.79 (n=1) | 234 | 2.14 | 218 | 14 % | +| | MobileGlues | 1.59 | 256 | | | 15 % | +| BSL (GPU) | p15i | 7.26 | 77.2 | 7.61 | 70.6 | 99 % | +| | dev | 13.85 | 66.1 | 7.17 | 70.5 | 85-100 % | +| | MobileGlues | 6.25 | 78.2 | | | 98-100 % | + +(BSL before the converted-stream cache: Magma 19.5 fps at 26 % GPU busy, 49 ms CPU/frame.) + +- Espryt's `base` lib (p15plan, unchanged) read 2.84 in the same session: the earlier "p15e + Espryt regression" was session variance. S0 does not touch Espryt. +- **Target status.** Magma BSL is met (within 1.3 % of MobileGlues, GPU-bound). Espryt BSL is + GPU-bound at 99 % and 10 % below MobileGlues, same as dev: a GPU-side gap still to explain (the + load/store and clear audit of PLAN §0.1). CPU scene: Magma 2.56 vs MobileGlues 1.59, Espryt 2.77. +- **Where Magma's CPU goes now** (p15i profile, `runs/prof-hi`, ms/frame inclusive): + `VerbChannel::Port` 0.85, `OnDrawVbo` 0.62, `SetupWireDraw` 0.46, `Present` 0.32, + `MGPipeValidateForVerb` 0.27, in-pass clears 0.18. The draw path is S1-S3's; S0's non-draw ports + are done. Proof that M4 engages: `StagedWireRangeCopy` 0.067 is gone and `RenameBusyWireStore` + costs 0.048 (a VMA allocation per rename; a recycle pool could take most of it). + +**Stage P:** dropped for the target. + +## Harness + +| purpose | where | +|---|---| +| FCL probes and reducers | `tools/device_bench/disagg/` (README "P15 FCL probes") | +| FCL milestone (CPU + BSL scenes, fin / new / dev / MobileGlues) | scratchpad `p15/fcl_milestone.sh`, `fcl_scene.sh`, `fclab.sh`, `fclsum.py` | +| SSIM gate (openra, rd12, bsl × 4 arms × 2 backends) | scratchpad `p15/ssim_gate.sh`; `matrix_p14.sh` now takes `PREP_WLS` and a `bsl` workload | +| host CI-shaped build (WSL archlinux) | `~/mgl-p15-ci31`, scratchpad `p15/wsl_ci.sh`, `w_suite.sh`, `w_syncval.sh`, `w_sv_one.sh`, `w_one.sh` | +| trace APK builds | scratchpad `build_trace.sh W:/p15 .` (rerun once on the `IncrementalSplitterRunnable` failure) | + +**Host-build trap:** editing a header while ninja is compiling leaves objects newer than the header +but built from its old text. This happened once and gave a `bad_alloc` in every scenario. Never +edit sources during a build; `touch` the edited headers if in doubt. + +**Device facts:** +- Clocks are pinned. +- FCL's original lib md5 is `00c09f0a…`. +- MC 1.21.5 options md5 is `93b3f708…`; it is backed up at `/data/local/tmp/p15-options.txt`. +- Iris properties are backed up at `/data/local/tmp/p15-iris.properties`. +- `fcl_scene.sh restore` restores Iris; `fcl_p14.sh restore` restores the lib and `config.json`. +- The anland desktop is stopped during runs and restarted afterwards. diff --git a/docs/Disaggregated/notes/perf-p15/PARITY-AUDIT.md b/docs/Disaggregated/notes/perf-p15/PARITY-AUDIT.md new file mode 100644 index 00000000..87af28d1 --- /dev/null +++ b/docs/Disaggregated/notes/perf-p15/PARITY-AUDIT.md @@ -0,0 +1,56 @@ +# P15 S0.1: parity audit of the deleted frontend-arm mechanisms + +**Question.** P13 W4 moved monolith onto the record/wire arm, and W6e / `de6ab167` then deleted the +frontend arm as dead code. Which performance mechanisms did that arm have, and does the wire arm +have an equivalent? + +**Sources:** +- `origin/dev` and the pre-W6 tree `88a11b81` (the frontend arm); +- the HEAD wire arm; +- [PARITY-SEED.md](PARITY-SEED.md), the regex-extracted list of deleted definitions; +- the profiles in PLAN-P15.md §2.2: FCL MC 1.21.5 and rd12, render thread, ms/frame. + +**Verdicts:** +- **port**: measured as missing, and the port is planned; +- **have**: the wire arm has an equivalent; +- **skip**: not shown in any profile; reconsider only if one shows it. + +## Magma (DirectVulkan) + +| # | mechanism (dev / 88a11b81) | wire arm at HEAD | profile evidence | verdict, cost | +|---|---|---|---|---| +| M1 | **Clears.** `VkClearManager`: a glClear inside the active pass on the same attachments became `ClearAttachmentsOnActiveRenderPass`; otherwise a pending clear keyed (texture identity, level, layer) turned the next pass's load op into CLEAR, materialised before reads, blits and copies | `ClearWireFramebuffer`: ends the draw pass; per surface, a new view + render pass + framebuffer, its own pass, then retire | FCL 0.42-0.46 ms (dev ~0.05); rd12 ~0.27 ms. Dev's per-draw `HasPendingClear` probe cost ~0.6 ms/frame on rd12 | **port, in two steps.** (a) The in-pass half: the clear takes the draw pass a draw to the same framebuffer would take (`PrepareWireDrawPass`) and records `vkCmdClearAttachments` in it. No new state, nothing deferred, so no read path has to materialise anything. (b) LOAD_OP_CLEAR only if a GPU-side reason appears; CPU-wise (a) removes the object churn and the pass break. Small cost, low risk | +| M2 | **Render pass + framebuffer cache.** `VkRenderPassManager::GetOrCreateRenderPass`, hash-keyed `RenderPassEntry` (render pass, framebuffer), garbage-collected at present | `m_wireDrawPassCaches`: exact-key reuse within one frame slot, emptied when the slot's frame begins (`ClearWireDrawPassCache`). Every frame rebuilds each framebuffer's pass, views and VkFramebuffer | FCL: vkCreateFramebuffer in SetupWireDraw 0.04, `~RenderPassEntry` 0.04-0.05, plus views | **port**: keep entries across frames while the key's image epochs hold, and retire through the submit-indexed list when an image goes away. Medium cost, medium risk (view lifetime against image release) | +| M3 | **Blit resources.** `InitializeBlitResources`, `TryBlitToDefaultFramebufferWithShader` (persistent pipeline and pool) | `BlitWireColorImage`: persistent pipelines, but per blit a descriptor **pool** + set, 2 views and a framebuffer, all retired | FCL 0.13 ms (vkCreateDescriptorPool 0.084 + destroy) vs dev 0.02 | **port**: one pool per frame slot, reset when the slot's frame begins; a full pool falls back to the old path. Views and framebuffer are next if they show up. Small cost, low risk | +| M4 | **Buffer sub-data.** `VkBufferManager::OnSubData`: a busy resident buffer took a staged copy (`StagedRangeCopy`, as now). A *streamed* buffer (`AcquireStreamedSlice`) updated only the CPU shadow, and the next draw streamed the whole buffer into a fresh arena slice, so no copy and no pass break | `WriteWireBuffer`: busy → `StagedWireRangeCopy`, which ends the pass and records two full barriers; idle → host memcpy. No streamed/renamed buffers on the wire | FCL 0.28 ms (11 sub-data calls/frame, ~25 µs each), and each one splits the pass on the GPU. WBUF probe: 4.5 host + 6.7 staged writes/frame; ~5.6 of the staged ones hit one 288-byte store | **ported**: write renaming for stores of 64 KB or less with a host shadow; the rename goes through the same orphan release as glBufferData, which every bind memo already handles (VkBuffer compare + destroy epoch) | +| M5 | **Draw fast path.** `TrySetupDrawFastPath` / `SetupDrawSnapshot` (4 snapshots on program lifetime id) | P14 wire memos: pass continuation (`WirePassMemoHolds`), unit sampler memo, pipeline / vertex / index bind dedupe, vertex input on its record | FCL SetupWireDraw 0.55 ≈ dev SetupDraw 0.54 | **have** | +| M6 | **Uniform rebinds.** `UniformManager` `FastRebindMemo`, `SampledBindingsUnchanged`, `CollectSampledTextures` walk-skip | P14 descriptor-set rebind with fresh offsets, walk-skip | small | **have** | +| M7 | **Vertex input.** `VertexInputStateFactory` per-VAO memos (`VaoBackendMemos`, memoised hash) | vertex input kept on its record (`ResolveWireVertexInput`) | small | **have** | +| M8 | **Texture sync.** `VkTextureManager` `SyncedTextureMemoEntry`, `DrawSyncScope` | P14 direct-mapped texture fronts, record-serial checks | small | **have** | +| M9 | **Vertex streams.** `UploadAndBindVertexBuffers`, `ConvertedVertexStream` cache (per frame, pointer + changeSerial), `LookupVaoDrawMemo` | wire vertex slices + bind shadow, but NO converted-stream cache: every draw re-converted the whole remaining stream | FCL: none (no conversion). BSL: `ConvertWireVertexStream` 49 ms/frame, Sodium multi-draws over one arena (Magma 19.5 fps vs dev 64.7) | **ported** (corrected from "have"): per-frame cache keyed on a monotonic store content serial, then kept across frames in `VkBufferManager` | +| M10 | **Depth mipmap / MSAA resolve / YUV**: per-op descriptor pools existed in dev too | same per-op pools | not in FCL/rd12 profiles | **skip** | +| M11 | **Command-buffer reset/end** | larger, more fragmented recordings | FCL +0.16 ms | re-measure after M1-M4 (follows from the pass count) | + +## Espryt (DirectGLES) + +| # | mechanism | wire/record arm at HEAD | profile evidence | verdict | +|---|---|---|---|---| +| E1 | `StateBackendObjectRegistry` (pointer-keyed twins, GC) | twins keyed on applier serials / lifetime ids (P14) | backend +0.09 ms on FCL | have | +| E2 | `SyncCurrentProgram`, `BindCurrentProgramWithResources` | `SyncCurrentProgramByHandle` + P14 memos | rd12 +0.05/+0.07 | covered by S3 (backend draw-delta) | +| E3 | `SyncVaoAttributeBuffersByHandle` | `SyncVaoAttributeBuffersByRecord`, `AdoptUnchangedBufferSets` | rd12 +0.09 | covered by S2/S3 | +| E4 | `BackendFramebufferObject::SyncToBackend` (dirty-bit sync) | record-serial sync, clean-attachment skip (P14) | small | have | +| E5 | clear / blit (host GL) | same host calls | FCL clear 0.16 vs 0.13, blit 0.125 vs 0.11 | have (no Espryt port; user confirmed) | +| E6 | buffer sub-data (`Ops_SubDataTracked`) | `Ops_H_SubDataTracked` via record | 0.024 vs 0.007 | skip | + +Espryt's FCL gap is the client half of the draw path (+0.39 ms), which stages S1-S3 address. + +## Order of ports (S0.2) + +1. M1a: in-pass clear. +2. M3: blit frame pools. +3. M2: cross-frame pass cache. +4. M4: sub-data renaming. +5. Re-measure M11. + +Every step: FCL presenting-thread CPU before and after, in the same session, with the MobileGlues +and dev reference rows. diff --git a/docs/Disaggregated/notes/perf-p15/PLAN-P15.md b/docs/Disaggregated/notes/perf-p15/PLAN-P15.md index 07d01b59..0838fd48 100644 --- a/docs/Disaggregated/notes/perf-p15/PLAN-P15.md +++ b/docs/Disaggregated/notes/perf-p15/PLAN-P15.md @@ -49,11 +49,12 @@ This is the plan, plus the measurements it rests on (section 2). Nothing here is Stage order: - 0: parity audit + wire non-draw fast path; - 0b: an FCL 1.21.5 trace fixture; -- P: swap-interval-correct presentation. It is independent of 1-4 and can run in parallel after 0; - 1: client draw-delta prototype; - 2: per-VAO / per-program slices; - 3: applier + backend draw-delta; -- 4: split arms, verify, default-on. +- 4: split arms, verify, default-on; +- P (swap-interval-correct presentation): **not needed for the target; dropped unless the user asks** + (2026-10-08). MobileGlues pays the same ~2 ms present wait, so matching it does not need P. Decisions taken with the user (2026-10-08): - Presenting-thread CPU ms/frame is the primary FCL metric, with fps beside it. @@ -62,6 +63,36 @@ Decisions taken with the user (2026-10-08): - The probe tools are committed under `tools/device_bench/disagg/`. - No Espryt deferred-clear port: its clear and blit costs match dev's. +## 0.1 Target and stopping rule (user, 2026-10-08) + +**Target:** "We only need to match MobileGlues' performance." + +**Success criterion:** both backends reach the MobileGlues plugin (`com.fcl.plugin.mobileglues`) +in FCL MC 1.21.5, on the same pinned device, same world, same swap interval. Its presenting-thread +CPU there is ~1.7 ms/frame (~248 fps) at interval 0 and ~1.9 ms/frame at interval 1. + +| build | interval 0 CPU ms/frame | needed | +|---|---|---| +| feat Magma | 3.12 | −1.4 | +| feat Espryt | 2.73 | −1.0 | +| dev Magma | 1.76 | (reference) | +| dev Espryt | 2.14 | (reference) | + +Espryt has to go below dev here. + +**Success criterion has two parts (user, 2026-10-08):** +- CPU-bound scenes: within ~5 % of MobileGlues presenting-thread CPU ms/frame. +- GPU-bound shaderpack scenes (Iris + BSL): within ~5 % of the better of MobileGlues and dev fps. + +**Stopping rule:** P15 ends when FCL presenting-thread CPU on both backends is within noise (~5 %) +of MobileGlues at the same interval, and the BSL scene meets its fps bar. It also ends if a remaining gap is shown to be irreducible, +with the reason written down. + +**Reference rows in every FCL milestone,** measured in the same session: +- MobileGlues (primary); +- dev (second). +MobileGlues also runs on the trace fixtures where the harness can load it. + ## 1. Measurement rules for P15 - **FCL:** @@ -77,6 +108,23 @@ Decisions taken with the user (2026-10-08): thread idles between frames. Compare CPU only within one interval setting. - Take readings only from in-world steady windows. The probe waits for MC's `Time elapsed:` line and then four fps-log windows of more than 150 frames each. +- **GPU-bound scene (user, 2026-10-08):** + - FCL `1.21.4-Fabric` with Iris and `BSL_v10.1.3`, at the same world and interval, measured live + with `fcl_scene.sh bsl` and restored afterwards. + - The BSL fixture `minecraft-1.21.4-fabric-iris-bsl-in-world` joins the SSIM gate. It is a + single-frame trim, so it is a correctness check, not a benchmark. + - Show the scene is GPU-bound, with evidence: + - kgsl busy near 100 % (`gpu_clock_stats`, kgsl cmdbatch union from ftrace); + - the presenting thread waiting in fences or present; + - the `gsl_syncobj_wait` share. + - On GPU-bound runs, fps is the metric. + - A GPU regression on BSL blocks a change even if CPU improved. +- **GPU metrics in every milestone:** + - kgsl GPU busy % for every FCL run (`fclsum.py`); + - for GPU-bound runs, fps against dev and MobileGlues. +- **Load and store ops:** check every pass change in GPU terms. Prefer LOAD_OP_CLEAR or DONT_CARE + over loading and clearing on the tiler, and use DONT_CARE stores for depth/stencil that nothing + reads after the pass. - **Trace (rd12, openra, and the FCL 1.21.5 fixture from stage 0b):** - fps, best of 3. - Per-layer CPU in ms/frame from `profile_p14.sh` / `layers.py`. Remember that rd12 carries @@ -352,6 +400,9 @@ Espryt's deleted mechanisms. Port only those that show in profiles. ### Stage P: swap-interval-correct presentation (both backends) +**Status: not needed for the target; dropped unless the user asks (2026-10-08).** MobileGlues +pays the same ~2 ms present wait, so the target (section 0.1) does not need this stage. + **Goal:** - With interval 0, presentation never paces the app: FCL fps becomes 1 / presenting-thread CPU. - With interval ≥1 the app is still paced to the display: 165 fps ±1 % on this device, with no diff --git a/tools/device_bench/disagg/matrix_p14.sh b/tools/device_bench/disagg/matrix_p14.sh index 0b4a20a2..bca420fb 100644 --- a/tools/device_bench/disagg/matrix_p14.sh +++ b/tools/device_bench/disagg/matrix_p14.sh @@ -49,6 +49,8 @@ wl_args() { # trace golden w h target crop case "$1" in openra) echo "openra openra.0000031249.png 640 480 31249 1 1 638 478" ;; rd12) echo "rd12 minecraft-1.21.4-rd12-odinlite-in-world.0004660351.png 854 480 4660351 0 0 0 0" ;; + # P15: GPU-bound shaderpack scene (a single-frame trim: ssim only, not a benchmark). + bsl) echo "bsl minecraft-1.21.4-fabric-iris-bsl-in-world.0000110725.png 854 480 110725 0 0 0 0" ;; esac } @@ -74,8 +76,13 @@ cool_down() { prep_pkg() { local pkg="$1" app; app=$(app_of "$pkg") local tmp="$ROOT/fixtures"; mkdir -p "$tmp" - for wl in openra rd12; do - local tgz="$FIXTURES/$( [ $wl = openra ] && echo openra.tgz || echo minecraft-1.21.4-rd12-odinlite-in-world.tgz)" + for wl in ${PREP_WLS:-openra rd12}; do + local tgz + case $wl in + openra) tgz="$FIXTURES/openra.tgz" ;; + rd12) tgz="$FIXTURES/minecraft-1.21.4-rd12-odinlite-in-world.tgz" ;; + bsl) tgz="$FIXTURES/minecraft-1.21.4-fabric-iris-bsl-in-world.tgz" ;; + esac set -- $(wl_args $wl) local golden="$FIXTURES/$2" if [ ! -f "$tmp/$wl.trace" ]; then