From a0f4aa830e4009ee3b6f5aa8a09c01f4eaca2bef Mon Sep 17 00:00:00 2001 From: Swung0x48 Date: Wed, 7 Oct 2026 21:20:11 -0400 Subject: [PATCH] [Docs] (P14): close-out bench, README, next-phase draw-packet sketch, DEBTS rows and handoff (rd12 monolith Espryt 69.8 -> 91.1, Magma 45.4 -> 83.8 fps). --- docs/Disaggregated/notes/DEBTS.md | 9 +- docs/Disaggregated/notes/perf-p14/BENCH.md | 151 ++++++++++++++++++ docs/Disaggregated/notes/perf-p14/HANDOFF.md | 137 ++++++++-------- .../notes/perf-p14/NEXT-draw-packet-cache.md | 150 +++++++++++++++++ docs/Disaggregated/notes/perf-p14/README.md | 111 +++++++++++++ 5 files changed, 487 insertions(+), 71 deletions(-) create mode 100644 docs/Disaggregated/notes/perf-p14/BENCH.md create mode 100644 docs/Disaggregated/notes/perf-p14/NEXT-draw-packet-cache.md create mode 100644 docs/Disaggregated/notes/perf-p14/README.md diff --git a/docs/Disaggregated/notes/DEBTS.md b/docs/Disaggregated/notes/DEBTS.md index 6420deab..195bf8f4 100644 --- a/docs/Disaggregated/notes/DEBTS.md +++ b/docs/Disaggregated/notes/DEBTS.md @@ -1,5 +1,7 @@ # 仍开放的债务（跨阶段） +> **2026-10-07（P14 收官）**：W9 关闭（读数，见下）；ID-P13-2（Magma monolith）更新为 trace 已基本追平、FCL 仍差的现状；新增三行：Espryt monolith 记录臂的逐项构成与结构原因、driver 项的 cache 压力推断、测过未做的前端变更 epoch 快路径。下一阶段是 draw-packet 缓存重设计。 +> > **2026-10-07（P13 收官）关闭三行**：ID-P8-13（9 条 monolith 缺陷——monolith 换到记录臂后对应 monolith 登记在集成车道上全过，旧的 monolith 前端臂在 W6 删除）、ID-P8-14（`SyncClientSideVertexArraysForIndirectFetch` 的整 buffer 回读与 Magma monolith 深度 mip 的 view 类型都属于已删的 monolith 前端臂）、`MOBILEGL_PIPE_TEXEL_RETAIN_MB`（W3b 随 pull 构建删除）。新增两行：W9 缓存容量重调与 Magma monolith 性能，都排在 P13 之后的性能轮，基线是 `dev` 构建。 > > 路线图只列"有债务、看这里"。每条写明去向；已关闭的历史债与当时的完整债务表见 [`p5b/README.md`](p5b/README.md) 末节。开放问题另见 [`OPEN-QUESTIONS.md`](OPEN-QUESTIONS.md)。更新：2026-09-30（P8 第二波；按债务审查重排。同日：create-indirect 换成派生 fixture `minecraft-1.21.1-neoforge-create-indirect-in-world-align1024`（不是原抓取，16 字节对齐的 SSBO 范围绑定挪到 1024 对齐），ID-P8-14、ID-P8-19 两条关闭——红米两后端 monolith / spawn 全过，Magma spawn 不再丢设备）。去向「需裁定」= 等用户定；「可进行」= 随时可开的小修，不依赖别的阶段。 @@ -24,8 +26,11 @@ | P8 转出（E）：driver-written 标按整张纹理打（`GenerateMipmap` 也打），驱动拒读时连基级也具名拒绝，偏保守；跨格式拷贝 store 不跟随；拒绝仍是会话 Fatal（要改回包状态才能答 `GL_INVALID_OPERATION`） | 小修（按 level 打标）；其余按需 | | P8 转出（B）：`PipeCalls.def:260` `GenerateMipmap` 仍是 `kWaitApplied`，裁定 14 的理由（CPU 滤波碰 client 影子）已不成立，可改 `kWaitNone` | 性能工作（协议行为变更，需单独裁定） | | P8 转出（C）：server SSBO 标记过近似（窗口内每个绑定的 SSBO 都标）；一个留在 SSBO 绑定上又当 restart EBO 用的 buffer 每次 draw 都回读 | 性能工作 | -| P13 转出（W5 翻转后，ID-P13-2 记录项，用户裁定记为债务）：Magma monolith 走记录臂比前端臂慢约 16%——Y700 上 FCL MC 1.21.5、RD2 小场景、时钟未钉：记录臂 168 / 176 fps，同库强制前端臂 206 / 204 fps，即每帧 CPU 约 5.8 ms 对 4.9 ms（渲染线程都约占满一核）；Espryt 持平（224 / 226 对 222）。疑因：`VulkanRenderer::SetupDraw` 在记录臂直接进 `SetupWireDraw`，绕过只在前端臂跑的 `TrySetupDrawFastPath`（PLAN-P13 §1.3 已提示「换臂前先量」）。W6 删前端臂后这部分成本固定下来 | P13 收官后（用户裁定：P13 内不做性能工作）：性能工作的「之前」基线是 `dev` 分支构建（pull monolith），不是 W6 前的 feat 构建；在设备上钉时钟对比 dev 与 feat，场景为 FCL MC 与 anland glmark2，两个后端都测；再用 simpleperf profile Magma monolith 每 draw 成本，决定是否给 wire draw 臂补快路径 | -| P13 转出（W9，用户裁定：P13 内不做性能工作）：缓存容量重调（CSO / 描述符 / 管线缓存容量按 `MOBILEGL_PIPE_STATS` 读数定初值、设备复核）未做，容量仍是 P8 时的值 | P13 之后的性能轮：以 `dev` 构建为基线（同 Magma 16% 那行），先在主机语料取 `MOBILEGL_PIPE_STATS` 读数定初值，再在设备上钉时钟复核 | +| P13 转出（ID-P13-2，P14 更新）：Magma monolith 记录臂对 dev（pull）——trace 上已基本追平：rd12 83.8 对 87.0 fps（−4%），openra 747 对 293（dev 在那里等 GPU）；P14 最大一项是 `553eb86d`（纹理层上传不再在 CPU 上等 GPU 排空，rd12 70→83）。但 **FCL MC 上仍差**：monolith 171 对 dev 232（−26%），inproc 195；呈现线程 CPU 只占约 2.8 / 5.8 ms 帧，其余是等待——与 trace 不同的另一处等待 | 下一阶段第一项：FCL Magma monolith 做 off-CPU profile（`--trace-offcpu`，同找到上传等待的方法；P14 收官时两次采集都没抓到游戏帧），找出那 3 ms 等待；数字见 `perf-p14/README.md` | +| ~~P13 转出（W9）：缓存容量重调~~ **P14 关闭（读数）**：`MOBILEGL_PIPE_STATS` 在 rd12 / openra 上 render-state CSO 稳态每帧新铸 0 个，Magma 管线 memo 命中 99.0% / 92.7%，没有缓存抖动，P8 容量保留 | 已关闭（`perf-p14/BENCH.md` W9 节） | +| P14 转出：Espryt monolith 记录臂对 dev（pull）rd12 91.1 对 115.2 fps（−21%）；FCL MC 198 对 229（−13%）。渲染线程每帧 CPU 11.56 对 9.07 ms：client half +1.39（validate 0.92：EmitGlobalConstants 0.17、EmitVertexElements 0.13、Tracker::Update 0.12、CopyField 0.10、validate 自身 0.10、EmitVertexBuffers 0.07；verb port 约 0.2）、backend +0.45（SyncNeccessaryTextures +0.14、SyncNeccessaryBuffers +0.08、AdoptUnchangedBufferSets +0.09、BindCurrentProgramWithResources +0.07、SyncCurrentProgramByHandle +0.05）、driver +0.33、applier +0.07。每项都在帧时间 2% 以下；合起来是记录臂的结构成本：pull 的后端在 draw 时按指针读前端对象，push 要检测变化（tracker）、序列化（emitter、uniform span）、应用（applier）、再由后端按句柄经 slot 表重新解析，并且因为后端看不到前端脏位而对每个绑定对象做 clean 检查——每步每 draw 几十 ns、各自是不相干的冷 cache 行，乘以 1471 draw/帧 | 下一阶段：per-(VAO, program) draw-packet 缓存（设计草稿 `perf-p14/NEXT-draw-packet-cache.md`，估计 rd12 Espryt monolith 回收 1.0–1.6 ms/帧，约 91→100–106 fps）；先只做 client half 原型量 | +| P14 转出：Espryt monolith 的 driver 项 +0.33 ms/帧（2.9%），全在 `glDrawElements` 内部，GL 调用组合与 dev 相同。`simpleperf stat`（整次运行、渲染线程）：feat 比 dev 多执行 16.6% 指令、L1D refill 多 56%、L2D 多 23%（IPC 3.73 对 3.86）。推断：记录臂每 draw 更大的工作集（record、applier 状态、slot 表、twin）把驱动的状态挤出 cache——这是计数器推断，没有在驱动内部测量 | 随 draw-packet 缓存一起验证：落地前后量 L1D refill 与 driver ms/帧；如果 driver 项不降，推断不成立，再单独查 | +| P14 转出（测过未做）：Tracker / residual 的前端变更 epoch 快路径——在每个 `DECLARE_GL_FUNCTION_END` 入口（除 draw、`glBindVertexArray`、浮点 `glUniform*` 外）以及手写入口和 EGL 入口里 bump 一个上下文 epoch；epoch 未动时 validate 只重算 VAO / program 派生的 shutter（上限约 0.07 ms），再经审计哪些 residual 字段依赖 VAO / program 绑定后跳过 residual copy（合计 0.15–0.25 ms，1.5–2%） | 各自低于 2% 门槛，且漏 bump 一个入口就渲染陈旧状态；并入 draw-packet 设计（其 §3 的上下文 epoch 就是它），不单独做 | | P13 转出（FCL 浸泡，用户裁定：FCL 暂不改、MobileGL 不做规避）：FCL game process can die silently (exit SIGNALED status=34, SIGRTMIN+2): the JVM signal reset after renderer selection drops libcore's AsynchronousCloseMonitor handler, and FCL's okhttp/Okio watchdog then kills the process; renderer-independent (reproduced on P12 and P13 libraries, both backends; untested with other renderers)。浸泡：P13 库 30 次（Magma 22 / Espryt 8）死 18 次、P12 库 10 次死 2 次；P12 / P13 逐次交替的对照（同场景同网络时段）P12 4/10、P13 2/10，时间线（`Renderer:` 行到 MobileGL 首行约 6.2 s、晚死固定在其后约 9.4 s）两库相同，无证据表明 MobileGL 拉高概率；全部 status=34，无 tombstone、无 kgsl；证据与机理见 [`p13/fcl-soak/README.md`](p13/fcl-soak/README.md) | fix belongs in FCL (save/restore libcore's SIGRTMIN+2 sigaction around JVM creation); left for the user | | P8 转出（MD）：dev 的 `RingAllocateSlow` 在出错的上下文上无上限地排水（设备上挂在 `glFinish`） | 小修（dev） | | P8 转出：`LogForwardChannel.APeerThatStopsReadingCostsWarnLinesNotTheCallersTime` 在负载下超时（两次），单跑必过 | 小修（放宽时限或隔离） | diff --git a/docs/Disaggregated/notes/perf-p14/BENCH.md b/docs/Disaggregated/notes/perf-p14/BENCH.md new file mode 100644 index 00000000..b1d22e3d --- /dev/null +++ b/docs/Disaggregated/notes/perf-p14/BENCH.md @@ -0,0 +1,151 @@ +# P14 perf pass: final bench + +**Headline:** FCL Minecraft gives Espryt monolith 198 vs dev 229 and inproc 242; Magma +monolith 171 vs dev 232 and inproc 195. On the rd12 trace, Espryt monolith is 91 vs 115 and +Magma monolith 84 vs 87; inproc is 112 (Espryt) / 110 (Magma). Details below. + +Head `781a7446` on `feat/disaggregated` (trace APK `p14fin`) against `dev` at `08124c99` (the pull +monolith, the baseline). Lenovo Y700 (Snapdragon 8 Gen 3, Adreno 750), rooted, CPU and GPU clocks +pinned (`pin_clocks.sh`), no per-frame glFinish (`FINISH=false`), best of 3, steady frames only. +Raw runs: `.trace-work/perf-p14/final-p14/` (`matrix_p14.sh run`, `p14_report.py`). + +## Final matrix (fps, best of 3) + +| workload | backend | dev (pull) | monolith | inproc | shm | tcp | +|---|---|---|---|---|---|---| +| rd12 | Espryt | 115.2 | 91.1 | 111.9 | 87.8 | 68.7 | +| rd12 | Magma | 87.0 | 83.8 | 110.0 | 95.6 | 69.9 | +| openra | Espryt | 889.2 | 832.5 | 1073.9 | 1063.1 | 602.0 | +| openra | Magma | 293.1 | 747.4 | 1051.6 | 976.0 | 544.5 | + +Start of the pass (same harness, `nofinish` tag, `f8a9466b`): + +| workload | backend | monolith | inproc | shm | tcp | +|---|---|---|---|---|---| +| rd12 | Espryt | 69.8 | 71.1 | 62.8 | 50.0 | +| rd12 | Magma | 45.4 | 63.2 | 58.6 | 45.6 | +| openra | Espryt | 812.2 | 1067.5 | 953.8 | 529.2 | +| openra | Magma | 356.3 | 446.0 | 394.4 | 412.1 | + +So over the pass, rd12 went from 69.8 to 91.1 fps on Espryt monolith (+31%) and from 45.4 to 83.8 on +Magma monolith (+85%). Split arms: inproc +57% / +74%, shm +40% / +63%, tcp +37% / +53%. On openra, +Magma monolith went 356 -> 747 and inproc 446 -> 1052. + +## Against the three goals + +| goal | rd12 | openra | +|---|---|---| +| push = pull (monolith vs dev) | Espryt 91.1 vs 115.2 (-21%); Magma 83.8 vs 87.0 (-4%) | Espryt 832 vs 889 (-6%); Magma 747 vs 293 (2.5x dev) | +| inproc beats pull | Espryt 111.9 vs 115.2 (-3%); Magma 110.0 vs 87.0 (+26%) | Espryt 1074 vs 889 (+21%); Magma 1052 vs 293 (3.6x) | +| backend parity (feat) | monolith 91.1 vs 83.8; inproc 111.9 vs 110.0 | monolith 832 vs 747; inproc 1074 vs 1052 | + +## CPU- or GPU-bound + +Every arm on both workloads is CPU-bound: GPU busy is 19-39 % everywhere (`GPU busy` column of the +report), and the arm's top thread runs at 94-99 % of a core on the monolith and inproc arms +(97-99 % for the render thread, 94-95 % for `mgl-srv-apply` where it is the top thread). +- monolith: one thread, `MobileGLTraceRe`, at 99 % on all four cells. +- inproc / shm: two threads at about the same load (client 8.5-8.7, apply 8.5-8.6 ms/frame on rd12), + so the split arms are bound on whichever of the two is ahead; on rd12 Espryt that is the client + (harness parse + client half), on rd12 Magma the apply thread. +- tcp: 70-80 % top-thread utilisation, the rest is the socket ping-pong (credit-1 waits), as in P10. +- `dev` Magma on openra is the one cell below 95 %: 63 % utilisation, wide spread (13.6 %) - dev's + Magma waits on the GPU there, which the feat build no longer does (see "Magma upload wait"). + +Backend parity therefore means "both CPU-bound at the same CPU cost". On the feat build the two +backends are within 2-11 % of each other on every arm, and the remaining gap is CPU debt (rd12 +monolith: Magma backend + driver 3.93 vs Espryt 2.76 ms/frame, client half 1.91 vs 1.78). + +## Layer split, rd12 monolith (render thread, ms/frame, simpleperf at 2 kHz, steady tail) + +| layer | Espryt feat | Espryt dev | delta | Magma feat | Magma dev | delta | +|---|---|---|---|---|---|---| +| driver | 1.69 | 1.36 | +0.33 | 2.12 | 1.88 | +0.24 | +| backend | 1.07 | 0.62 | +0.45 | 1.81 | 2.56 | -0.75 | +| applier | 0.07 | 0 | +0.07 | 0.13 | 0 | +0.13 | +| client half | 1.78 | 0.39 | +1.39 | 1.91 | 0.70 | +1.21 | +| harness (trace parse) | 6.94 | 6.70 | +0.24 (same code) | 6.79 | 6.86 | -0.07 | +| total CPU | 11.56 | 9.07 | +2.49 | 12.77 | 12.01 | +0.76 | + +About 6.7-6.9 ms of every rd12 frame is the replay harness, so the MobileGL part of an Espryt frame +is 4.6 ms on feat against 2.4 ms on dev, and of a Magma frame 6.0 against 5.1. + +### What the Espryt monolith gap is made of (as % of an 11.2 ms frame) + +- Validate point, 0.92 ms (none of it exists in pull): EmitGlobalConstants 0.17 (1.5 %), + EmitVertexElements 0.13 (1.1 %, includes the inline apply of the bind), Tracker::Update 0.12 + (1.1 %), validate self 0.10 (0.8 %), CopyField 0.10 (0.9 %), EmitVertexBuffers 0.07 (0.6 %), + thread-local and atomic overhead about 0.07. +- Verb port, about 0.2 ms (1.8 %): EmitDrawRecord, ReadDrawBindings 0.06, MarkGpuWritesForDraw 0.05. +- Backend +0.45: SyncNeccessaryTextures 0.23 vs 0.10, SyncNeccessaryBuffers 0.16 vs 0.08, + AdoptUnchangedBufferSets 0.09, BindCurrentProgramWithResources 0.18 vs 0.11, + SyncCurrentProgramByHandle 0.09 vs 0.04. +- Driver +0.33 (2.9 %): inside glDrawElements itself, with the same GL call mix as dev. A + `simpleperf stat` over the whole run (render thread) has feat executing 16.6 % more instructions + than dev with 56 % more L1D and 23 % more L2D refills (IPC 3.73 vs 3.86): the record arm's larger + per-draw working set (records, applier state, slot tables, twins) evicting the driver's state is + the best reading, inferred from those counters, not measured inside the driver. + +Each item is under 2 % of the frame except the driver term. Why the sum is structural on the record +arm: pull's backend reads frontend objects by pointer at draw time; push has to detect what moved +(tracker), serialise it (emitters, uniform spans), apply it (applier), then let the backend resolve +it again by handle through slot tables, with per-bound-object clean checks because the backend no +longer sees the frontend's dirty bits. Each step is tens of ns per draw on cache lines no other +step shares, times 1471 draws a frame. + +Options left (none below the 2 % bar on its own, or each with a correctness risk the bar does not +pay for): +- a frontend mutation epoch (bumped in every `DECLARE_GL_FUNCTION_END` entry except draws, + glBindVertexArray and float glUniform*) so a validate with only VAO and uniform movement skips + the context-global shutters (at most 0.07 ms) and, after an audit of which residual fields + depend on the VAO/program bindings, the residual copies (0.15-0.25 ms together, 1.5-2 %); +- a per-(VAO, program) draw packet that keeps tracker, emitter, applier and backend state for a + draw on a few cache lines, keyed on lifetime ids - the structural answer to the cache-pressure + term, a redesign rather than a pass item; +- native TLS (minSdk / ANDROID_PLATFORM 29 instead of 26, the user's call): measured +1.7 % + Espryt, within noise on Magma (rd12 monolith, best of 3; whole APK, harness included). + +## Pass findings worth keeping + +- **Magma upload wait (553eb86d).** Every texture level upload flushed the recording and then + waited on the CPU for every submission to complete (`FlushWirePendingCommandsForTextureUpdate`, + added in P7 B4). The upload batch is submitted on the same queue after the flush and its first + barrier takes its source scope from the image's tracked layout, so the wait bought nothing; on + rd12 it was 1.8 ms/frame of fence stall (Minecraft updates its lightmap every frame). Removing it: + rd12 Magma monolith 70.4 -> 83.4, openra 367 -> 745. The preserve path keeps its wait. Proof: a + new between-draws upload scenario under the synchronization validator (red on a broken barrier), + sync validation of the texture/remint/copy scenarios (no new hazards), SSIM 16/16. +- **Sync validation needs submit-time validation on.** The Khronos layer (1.4.350) does not report + a hazard between two queue submissions unless `VK_KHRONOS_VALIDATION_SYNCVAL_SUBMIT_TIME_VALIDATION` + is set; the run-ahead and cache gates now set it (781a7446), and the run-ahead gate goes red on + the broken-barrier build where it used to stay green. +- **GPU-fault latch regression (549a34bb -> fixed in c24b3428).** Polling the reset status every + 256 records let a record published right after the previous acknowledgement run unpolled on a + guilty context. The drain now polls before any record the last poll could not have seen + (e058a450 hands the apply loop's poll to the drain so a woken drain does not poll twice). +- **Record-publish batching, measured and dropped.** Publishing an unbarriered validate's state + records with the draw record behind them was within +-2 % on every arm and opens a deadlock class + (a staging-retirement wait on a record that is written but not published - it hung rd12 Espryt + split until the stage wait published owed records). Not worth the surface; the patch and its + stress scenario are not in the tree. + +## W9 (cache capacities), closed with readings + +`MOBILEGL_PIPE_STATS` on rd12 and openra: the render-state CSO cache mints 0 per frame in the steady +state, and Magma's pipeline memo hits 99.0 % on rd12 and 92.7 % on openra. No cache thrashes; the +P8 capacities stand. + +## FCL (Minecraft in world, the representative app check) + +FCL's own build of the library (plugin flavor, no POISON), swapped into FCL. Readings come from +MobileGL's fps log: mean of two launches × three steady windows. Clocks pinned. Runs: +`.trace-work/perf-p14/fcl-final/`. + +| backend | dev (pull) | feat monolith | feat inproc | presenting-thread CPU, monolith feat / dev | +|---|---|---|---|---| +| Espryt | 228.6 | 198.3 (-13 %) | 241.5 (+6 %) | 2.43 / 1.85 ms/frame | +| Magma | 231.8 | 171.4 (-26 %) | 194.8 (-16 %) | 2.80 / 1.64 ms/frame | + +FCL Magma does not match the trace: the presenting thread is busy only about 2.8 ms of a 5.8 ms +frame, so about 3 ms is a wait the trace does not have. That is the next phase's first item: an +off-CPU profile, the method that found the trace's upload stall. diff --git a/docs/Disaggregated/notes/perf-p14/HANDOFF.md b/docs/Disaggregated/notes/perf-p14/HANDOFF.md index 1da0e66c..cf7d520f 100644 --- a/docs/Disaggregated/notes/perf-p14/HANDOFF.md +++ b/docs/Disaggregated/notes/perf-p14/HANDOFF.md @@ -1,74 +1,50 @@ -# P14 perf pass: handoff (checkpoint 2026-10-07) +# P14 perf pass: handoff (closed 2026-10-07) -Worktree `.claude/worktrees/perf` (branch `perf`, pushed to `origin/feat/disaggregated`). Estimate: -[ESTIMATE.md](ESTIMATE.md). BENCH.md / README.md are not written yet (see "Next"). +The pass is closed at `781a7446`, plus this docs commit. Results are in [BENCH.md](BENCH.md) and +[README.md](README.md). The next phase is the per-(VAO, program) draw-packet cache +([NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md)). Its first item is an off-CPU profile of +FCL Magma monolith (README "Headline numbers"). -## Head and running totals +## Where things stand (rd12, pinned, FINISH=false, best of 3) -Head: `94705adc` (plus this handoff commit). rd12, Lenovo Y700 (Adreno 750), clocks pinned, no -per-frame glFinish (`FINISH=false`, the primary mode; finish=true serialises the split arms on one -FenceWait round trip per frame), best of 3: - -| arm | Espryt start -> now | Magma start -> now | dev (pull) Espryt / Magma | +| arm | Espryt | Magma | dev Espryt / Magma | |---|---|---|---| -| monolith | 69.8 -> 88.6 | 45.4 -> 69.5 | 115.1 / 86.1 | -| inproc | 71.1 -> ~107 | 63.2 -> ~99 | | -| shm | 62.8 -> ~82 | 58.6 -> ~85 | | -| tcp | 50.0 -> ~64 | 45.6 -> ~61 | | +| monolith | 91.1 | 83.8 | 115.2 / 87.0 | +| inproc | 111.9 | 110.0 | | +| shm | 87.8 | 95.6 | | +| tcp | 68.7 | 69.9 | | -(split-arm "now" values are from the p14p A/B, before the last two monolith commits; re-measure.) -GPU busy 22-33 % everywhere: every arm is CPU-bound on one thread. +FCL MC in world: +- dev: Espryt 228.6, Magma 231.8; +- feat monolith: Espryt 198.3, Magma 171.4; +- feat inproc: Espryt 241.5, Magma 194.8. -## Latest layer split (rd12 monolith, render thread ms/frame, simpleperf, head `94705adc` = p14r) +All arms are CPU-bound (GPU 19-39 %). -| layer | Espryt feat | Espryt dev | Magma feat | Magma dev | -|---|---|---|---|---| -| driver | 1.56 | 1.40 | 2.15 | 1.98 | -| backend | 1.21 | 0.59 | 1.95 | 2.12 | -| applier | 0.07 | 0 | 0.13 | 0 | -| client half | 1.96 | 0.36 | 2.01 | 0.69 | -| harness | 6.46 | 6.27 | 6.50 | 6.25 | -| total CPU | 11.27 | 8.62 | 12.77 | 11.05 | +## Open items (DEBTS.md has the rows) -Client half top: validate self 0.22-0.26, verb port (MG_Record) 0.15-0.18, Tracker::Update 0.14-0.17, -CopyField 0.07-0.09, VAO memo probe 0.07-0.09. Espryt backend top: PrepareForDraw 0.10, -AdoptUnchangedBufferSets 0.10, PerNativeContext<...>::Current 0.084, SyncTextureToBackendByHandle -0.066, IsBufferDrawCleanByHandle 0.058. Magma backend top: SetupWireDraw self 0.35, -BindDescriptorSetDeduped 0.23 (dynamic UBO offset moves every chunk draw), AcquireWireSlice 0.19. - -Per-draw counts (instrumented build, `p14count2.patch` in the scratchpad, not committed): rd12 1471 -draws/frame, 3.3 GL calls/draw; after the protocol change 66.8 B/validate (bind 16 B + uniform span -~47 B), VertexBuffers/IndexBuffer 100 % no-op. - -## Commits this pass (newest first) - -94705adc inline applier / fill plan / word-wise spans / memo-answered draw bindings (+6 % Espryt, +3.7 % Magma); -81bdfb1f Espryt twin lookups w/o SharedPtr copy + one TLS block (+3.8 %); -032dcd24 Magma vertex-input layout kept on its record + slot-indexed buffer front; -c1497bb8 VAO-kept bindings, uniform spans, context-values gate (split arms +5-10 %; stream span merge fixes a tcp regression); -0158a115 Magma pass-continuation memo; 8883e496 fill pins + VAO hash memo; 80dc9d45 VAO memo; -5788a602 Magma descriptor rebind; afcf8d96 Espryt AdoptUnchangedBufferSets; e6281b26, 4ef9705e Magma -lookups/sampler memo; b556ec31, a9108e1a Espryt; 549a34bb server fault latch; 8346d809 slot allocator -front; 267211e8 Magma hash memo + scratch. - -## In flight / next - -1. Espryt backend lookups: `PerNativeContext::Current` (TLS cache per T + non-inline serial/epoch - calls), `IsBufferDrawCleanByHandle`, `AdoptUnchangedBufferSets` (compares the window every draw - because the bind re-apply bumps VertexBuffersSerial), `SyncTextureToBackendByHandle`. -2. Tracker::Update: 18 shutters evaluated per draw, 4 fire. Needs a cheap "nothing but the VAO and - the uniform moved" signal; no central frontend mutation epoch exists yet. -3. Residual fill: now a precomputed walk, but still ~11 CopyField per draw. -4. Magma: SetupWireDraw self, per-draw dynamic-offset descriptor bind. -5. Final full matrix (dev + 4 arms x 2 backends x openra/rd12, REPS=3), then BENCH.md, README.md, - DEBTS.md rows (Magma 16 %, W9: closed with readings - render-state CSO cache mints 0/frame, Magma - pipeline memo 99.0 % hit on rd12, 92.7 % on openra), FCL re-run on the final lib. +1. FCL Magma monolith waits about 3 ms/frame. The trace does not show it. Profile it off-CPU + first. Both close-out attempts caught no game frames, because FCL start-up ran past the + 60 s warm-up. +2. The draw-packet cache (design sketch). Prototype the client half on Espryt monolith first. +3. The Espryt driver +0.33 ms is a cache-pressure inference. Check it with L1D refills + before and after (2). +4. Native TLS, minSdk 29 vs 26, is the user's call: +1.7 % Espryt, noise on Magma. ## Harness -Scratchpad scripts (session-local): `build_trace.sh W:/perf .p14X` (trace APK, suffix = package -`top.mobilegl.plugin.p14X.trace`), `sign_install.sh `. W: is `subst` of -`.claude/worktrees` (MAX_PATH). Then: +Session scratchpad scripts: +- `build_trace.sh W:/perf .p14X`: trace APK, package `top.mobilegl.plugin.p14X.trace`. +- `sign_install.sh `. +- `devjob.sh `: runs a device job under a mkdir lock. Always use it; two concurrent + device jobs contaminate each other, and killing a job leaves its children running. +- `incl.py` / `children.py` / `lines.py` / `layers2.py` / `drvcall.py` / `offcpu.py`: + simpleperf analyses. The tail window comes from the render thread's `retraceCall` samples. +- `profile_offcpu.sh`: `--trace-offcpu` variant of `profile_p14.sh`. +- `stat_p14.sh`: `simpleperf stat` counters. +- `fcl_offcpu.sh`: FCL profile plus restore. + +Then: ``` cd tools/device_bench/disagg @@ -77,21 +53,44 @@ ARMS="monolith@p14a monolith@p14b" BACKENDS="DirectGLES DirectVulkan" WLS=rd12 R python p14_report.py ../../../.trace-work/perf-p14/ab ARMS="monolith@p14X inproc@p14X shm@p14X tcp@p14X" REPS=1 TAG=ssim bash matrix_p14.sh ssim # SSIM gate ARM=monolith BACKEND=DirectGLES WL=rd12 TAG=prof PKG_FEAT=... SYMDIR= bash profile_p14.sh -python layers.py MobileGLTraceRe --tail-ms=N [--detail=backend] ``` -FIXTURES must point at a checkout with real LFS goldens (worktrees have pointer files). Host tests: -WSL archlinux `~/mgl-perf-bld` (clang, split+inproc, tests on); ServerSpawnTest / FuzzArm2 / -PeerLatchTest fail there with Refuse{BuildFingerprint} (no build stamp in that tree), environmental. +The FCL library comes from `:app:assemblePluginRelease`. Unzip `lib/arm64-v8a/libMobileGL.so` from +`app/build/outputs/apk/plugin/release/`; the unstripped copy is under +`intermediates/merged_native_libs/pluginRelease`. Run `fcl_p14.sh run`, then `restore` +(md5-checked). + +Gradle's `packageTraceRelease` sometimes fails on the first try with +`IncrementalSplitterRunnable`; a rerun succeeds. + +Host: +- Unit tests: WSL archlinux `~/mgl-perf-bld` (clang, split + inproc). + - ServerSpawnTest, FuzzArm2 and PeerLatchTest fail there with `Refuse{BuildFingerprint}`; + that is environmental. + - The google-benchmark targets don't build with clang 22; also environmental. +- CI-shaped integration tree: `~/mgl-perf-ci31`, configured with CMake 3.31.10 from the + `~/mgl-perf-tools` venv. + - The system CMake 4.x joins `gtest_discover_tests` ENVIRONMENT lists into one variable, and + every entry then skips. Use 3.31.10. + - lavapipe ICD; spawn/tcp lanes are environmental there. + - To run the syncval gates: `scripts/ci/magma_runahead_checks.py validation --build-dir ~/mgl-perf-ci31 --execute`, + with `GIT_DIR=.../.git/modules/MobileGL/worktrees/perf` set for the cache script. +- `~/mgl-perf-ci` is a CMake 4 tree. Use it only through the scratchpad's `runentry.sh`, which + splits the joined environment. ## Device state -HA27Q3LQ: CPU/GPU clocks PINNED (`bash pin_clocks.sh restore` to undo). FCL lib/config/props restored -(md5 checked; `debug.mobilegl.*` props unset). Trace APKs `top.mobilegl.plugin.p14*.trace` installed. +- HA27Q3LQ CPU/GPU clocks are still PINNED: the next phase wants them pinned. + `bash pin_clocks.sh restore` undoes it; ask before unpinning. +- FCL is restored: lib md5 matches `/data/local/tmp/p14-fcl-orig.so`, config.json matches its + saved copy, and the `debug.mobilegl.*` properties are deleted. +- Trace APKs `top.mobilegl.plugin.p14*.trace` are installed; `p14fin` is the close-out build. ## CI -No run has completed since f8a9466b (each push cancelled the previous). workflow_dispatch runs on -the checkpoint head: see the milestone message / below. +Dispatched runs (workflow_dispatch, not cancelled by later pushes): +- 86ab2835: Test 37702990844, APK 37702995519. +- 781a7446: Test 37707221661. +- The close-out docs head: see the final report. -Dispatched on ff53ba3b: Test run 37674322398, MobileGL APK run 37674328683 (workflow_dispatch, not cancelled by later pushes). +There is only one self-hosted GPU runner (`minipc-8845-arch-wsl-gpu`), so GPU jobs queue behind it. diff --git a/docs/Disaggregated/notes/perf-p14/NEXT-draw-packet-cache.md b/docs/Disaggregated/notes/perf-p14/NEXT-draw-packet-cache.md new file mode 100644 index 00000000..8be07960 --- /dev/null +++ b/docs/Disaggregated/notes/perf-p14/NEXT-draw-packet-cache.md @@ -0,0 +1,150 @@ +# Next phase: a per-(VAO, program) draw-packet cache (design sketch) + +This is a starting point for the next phase, not an implementation. The numbers are from the P14 +close-out (`BENCH.md`, rd12 monolith, Y700, pinned, render thread, ms/frame). + +## 1. The problem + +On the record arm, a draw crosses five layers, and each layer keeps its own per-object state: + +| layer | Espryt feat | Espryt dev | delta | what it touches per draw | +|---|---|---|---|---| +| client half | 1.78 | 0.39 | +1.39 | tracker shutters (VAO, program, context gens), VaoMemo, emitter latches, slot allocator, residual block, verb port | +| applier | 0.07 | 0 | +0.07 | vertex-elements record, applier unit windows, global-constants block | +| backend | 1.07 | 0.62 | +0.45 | VAO twin, texture/sampler records + twins (clean checks), program twin, UBO ring | +| driver | 1.69 | 1.36 | +0.33 | the same GL calls as dev; slower, consistent with cache pressure (+56 % L1D refills, +16.6 % instructions, whole run) | + +Minecraft rd12 issues 1471 draws a frame, about 3.3 GL calls per draw. Between two consecutive +draws, usually only the bound VAO and the program's default-uniform block change. Each layer still +re-derives "what moved" from its own structures, and each of those structures is a different cold +cache line per VAO: the frontend VAO, the VaoMemo entry, the emitter latch, the applier record, and +the backend twin. Pull touches one of them, the frontend VAO. + +None of the individual items is above 2 % of the frame. The sum is the gap: on Espryt monolith, +91 vs 115 fps. + +## 2. What one draw packet holds + +There is one packet per (VAO lifetime id, program lifetime id) pair that has drawn. It lives in a +client-side table, and in the server's record space on the split arms. Its fields: + +- **Identity:** VAO lifetime id + config version, program lifetime id + link version, and the + context serial. +- **Client side:** + - the vertex-elements handle and its published config version; + - the buffer-set and index handles and their hash, which the VaoMemo holds today; + - the shader CSO handle; + - the residual-field values this pair contributed (the bound-VAO and program pins); + - the context-global epoch it was validated at (see section 3). +- **Applier side:** a pointer, or slot index plus generation, to the vertex-elements record and the + stored buffer/index window, so that bind_vertex_elements applies by reference instead of copying. +- **Backend side (Espryt):** + - the VAO twin; + - the resolved draw buffers; + - the program twin; + - the sampled-texture list with each entry's synced serials; + - the synced attachment list; + - the UBO block size. +- **Backend side (Magma):** the wire vertex-input state, the pipeline handle, the descriptor-set + layout, and the pass-compatibility id. +- **A "packet valid at" stamp per layer:** the serials and epochs each layer checks to trust its + slice of the packet. + +The goal is that a draw whose packet is valid at every layer touches the packet and the per-draw +inputs, and nothing else. The per-draw inputs are the uniform span, the draw arguments, and the UBO +ring slot. + +## 3. Keys and invalidation sources + +**Keys.** Lifetime ids only, never GL names or pointers. This is the ABA rule, already broken and +fixed three times: sampler, texture, program. The packet table key is (VAO lifetime id, program +lifetime id, context serial). + +Invalidation sources, by layer: + +- **Frontend context-global state.** This is everything the tracker reads that is neither VAO- nor + program-derived: render/pipeline state versions, texture content/params/bind generations, + framebuffer attachment and bind versions, buffer bind-point generations, TFB, pixel pack, patch, + vertex attribute defaults, and the program pipeline. + - This needs a context mutation epoch, bumped in every GL/EGL entry except the + whitelist: draws, glBindVertexArray, and float glUniform*. + - The bump fits in the `DECLARE_GL_FUNCTION_END` macros, with the whitelist as a constexpr name + test, plus the manual entries (glFinish/glFlush and the hand-written extension wrappers) and + the EGL entries. + - **Audit item:** every MOBILEGL_GL_API definition outside the macros. +- **VAO.** The config version (attribute pointers and enables), the index slot version, and buffer + respecification through `AnyVaoAttributeGeneration`. These are already lifetime- and + version-keyed. +- **Program.** Link version, backend state version (sampler units through glUniform1i), block + binding version, and image unit version. Float uniforms move only the UBO content version, + which is a per-draw input and not a packet invalidation. +- **Applier.** The record `ContentSerial`, `ContextSerial`, `MGPipeSlots().ResetEpoch()`, and the + vertex-buffers and index-buffer serials. These are already monotonic. +- **Backend.** + - Texture: record `Serial`, `ParamsSerial`, and `PendingUploads`; twin force-resync flags; + sampler CSO serial; shared-image state. + - Context generation. + - Magma only: texture image epoch, resource erase epoch, the image layout leaving GENERAL, and the + store destroy epoch. The P14 Magma memos already check these. + - The backend needs one "texture state moved" epoch, so that the per-bound-texture clean check + can be skipped when nothing in the texture family moved. This is the item the content-version + lesson warns about: glGenerateMipmap grew mips without a version bump. Every backend-side mutation + of a twin's synced state must bump the epoch too. + +## 4. Split arms and the verify comparator + +- **The packet is per role.** The client half of the packet (handles, latches, residual pins) + lives in the client. The applier and backend halves live wherever the applier runs: the same + thread on monolith, the apply thread on inproc, the server process on spawn and tcp. +- **Nothing in the packet crosses the wire.** The wire stays the record stream. A server-side + packet is keyed on the vertex-elements handle and the shader-CSO handle (slot + generation) + plus the applier context serial, because the server never sees frontend lifetime ids. +- **Run-ahead.** Client-side validity must not depend on server state. The client's half is + checked against client epochs only, exactly like today's latches. +- **The verify comparator (`MOBILEGL_PIPE_VERIFY`) compares the residual block against a live + snapshot at the verb.** + - A packet hit that skips residual copies would make the comparator diverge, unless the packet's + pins are what it compares. + - Proposal: under verify, take the slow path always and assert that the packet would have hit + with the same values. That gives a free cross-check of the epoch's coverage on every verify + lane. +- **POISON stamps (trace build only).** The stamp loop stays, or the packet records the stamp set + and applies it with one bulk write. + +## 5. Risks + +- **A missed epoch bump renders stale state.** This is the dominant risk. Mitigations: + - the verify-lane cross-check above; + - a negative control that removes the bump from one entry and must go red; + - SSIM 16/16 on every change; + - the existing retrace CI. +- **ABA on packet keys.** Lifetime ids only, plus context serial and slot reset epoch. +- **Memory.** One packet per drawn (VAO, program) pair. Minecraft has thousands of chunk VAOs × + a few programs, so a bounded table with LRU is about 100-200 B × 10k ≈ 2 MB. The table must + not itself become the next cache-miss source: entries go in a flat array with an index stored + on the frontend VAO. +- **Backend-side invalidation completeness** (section 3, backend): the same class as the reverted + e8e1521. +- **Code size and review cost.** It touches every layer. It should land behind a runtime knob, + default off until the CI and SSIM gates and an FCL soak are green. + +## 6. Rough gain estimate (rd12, ms/frame, Espryt monolith) + +| layer | removable on a packet hit | why not all of the delta | +|---|---|---| +| client half (+1.39) | 0.6-0.9 | the uniform span, bind record and draw record still have to be produced | +| applier (+0.07) | 0.03 | bind by reference | +| backend (+0.45) | 0.25-0.35 | the UBO ring slot and the GL calls stay | +| driver (+0.33) | 0.1-0.3, if the cache-pressure reading is right | unproven; measure L1D refills before and after | + +Total: about 1.0-1.6 ms of the 2.5 ms gap. That is roughly 91 -> 100-106 fps on Espryt monolith. +Magma's share is similar on the client half (+1.21). Its backend is already below dev, and its +driver term is +0.24. + +Inproc gains the client-half part on the client thread: that is the bound thread on Espryt rd12, +so 112 -> above dev's 115. Magma inproc is bound on the apply thread, which gains only the backend +and applier part. + +Prototype first, before the full design: the client-half packet (tracker + emitter + residual pins, +with the context epoch) on monolith Espryt only. Measure it against the 0.6-0.9 ms estimate, then +decide on the backend half. diff --git a/docs/Disaggregated/notes/perf-p14/README.md b/docs/Disaggregated/notes/perf-p14/README.md new file mode 100644 index 00000000..be078cb2 --- /dev/null +++ b/docs/Disaggregated/notes/perf-p14/README.md @@ -0,0 +1,111 @@ +# P14: the post-P13 performance pass + +P14 measured the record (push) arm against the `dev` pull monolith and cut its CPU cost. It ran on +`feat/disaggregated` from `f8a9466b` (start) to `781a7446` (close-out), on a Lenovo Y700 (Adreno +750) with pinned clocks. + +- [ESTIMATE.md](ESTIMATE.md): the code-based estimate, written before the first benchmark. +- [BENCH.md](BENCH.md): the final trace matrix, the layer split, CPU vs GPU binding, and what the + remaining gap is made of. +- [HANDOFF.md](HANDOFF.md): harness commands, device state, open items. +- [NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md): the design sketch for the next phase. + +## Headline numbers + +### Minecraft in FCL (the representative app check) + +Setup: +- FCL Minecraft, in world. +- The library is swapped into FCL, which runs it as a plugin build without `MOBILEGL_PIPE_POISON`. +- Readings come from MobileGL's own fps log. +- Two launches × three steady windows of about 2 s each. Dev Espryt is three windows: its second + launch died at start-up, twice. +- Clocks are pinned. Runs: `.trace-work/perf-p14/fcl-final/`. + +| backend | dev (pull) | feat monolith | feat inproc | feat monolith, presenting-thread CPU | +|---|---|---|---|---| +| Espryt | 228.6 | 198.3 (−13 %) | 241.5 (+6 %) | 2.43 ms/frame (dev 1.85) | +| Magma | 231.8 | 171.4 (−26 %) | 194.8 (−16 %) | 2.80 ms/frame (dev 1.64) | + +Earlier in the pass, at `94705adc`, the same measurement read: +- Espryt: monolith about 192, inproc about 233; +- Magma: monolith about 169, inproc about 179; +- dev: 209-245. + +FCL Magma monolith did not follow the trace's improvement. Its presenting thread is busy only +about 2.8 of the roughly 5.8 ms frame, so the rest is a wait. That is the first item for the next +phase: an off-CPU profile of FCL Magma monolith, the same method that found the upload stall in +the trace. Two attempts at it during close-out caught no game frames (FCL start-up). + +### Trace replay, rd12 (Minecraft 1.21.4 RD12, 1471 draws/frame), best of 3, fps + +| backend | dev (pull) | monolith | inproc | shm | tcp | +|---|---|---|---|---|---| +| Espryt | 115.2 | 91.1 (start 69.8) | 111.9 (71.1) | 87.8 (62.8) | 68.7 (50.0) | +| Magma | 87.0 | 83.8 (start 45.4) | 110.0 (63.2) | 95.6 (58.6) | 69.9 (45.6) | + +openra is in [BENCH.md](BENCH.md). On openra, Magma monolith now runs 2.5x faster than dev +(747 vs 293), and inproc 3.6x (1052 vs 293). + +Every arm is CPU-bound: GPU busy is 19-39 %, and the top thread sits at 94-99 % of a core +(70-80 % on tcp). + +## What landed + +The commits run from `f8a9466b` to `781a7446`. Per-commit numbers are in the commit subjects. + +- **Magma wire draw:** + - no per-draw SPIR-V rehash or allocation; + - unit sampler memo; unchanged pipeline, vertex and index binds skipped; + - direct-mapped texture and buffer fronts; + - descriptor-set rebind with fresh offsets; + - pass-continuation memo; + - vertex-input layout kept on its record; + - **texture level uploads no longer wait for the GPU to drain** (`553eb86d`, the largest + single gain of the pass: rd12 70 -> 83, openra 367 -> 745); + - slices stamped from the owner's submit counter. +- **Espryt:** + - VAO twin re-points only the moved buffers; + - clean attachments skip resync; + - unchanged buffer sets are adopted; + - twin lookups without SharedPtr copies; + - one thread-local block; + - one UBO-ring lookup per draw. +- **MG_Pipe client half:** + - lifetime-id slot fronts; + - per-VAO memo; + - fill pins; + - kept VAO bindings on the server; + - uniform spans; + - context-values gate; + - inline process applier; + - precomputed residual walk, with split copy and stamp loops; + - memo-answered draw bindings. +- **MG_Remote:** fault-verdict polling restructured. + - It first went from every record to every 256 records (549a34bb). + - The drain now polls before any record the last poll could not have seen (`c24b3428`). This + fixes the regression where a guilty context ran one more record. + - The apply loop hands its poll to the drain (`e058a450`). +- **Magma hang watch:** a marker slot is reused only once its submission's fence completes + (`ec973fba`). +- **Tests and CI:** + - TextureUploadBetweenDrawsScenario on every arm, plus inproc sync validation (`67f4f311`, + `86ab2835`); + - submit-time synchronization validation in the run-ahead and cache gates (`781a7446`); + - ServerLoopEglLatchTest now holds the race window open. +- **Tools:** + - pinned-clock matrix with a per-thread CPU and GPU-busy sampler; + - layer-split and off-CPU profile helpers; + - FCL harness; + - opt-in fps log and transport override properties. + +## Measured and not landed + +- **Record-publish batching:** publishing a validate's state records with the draw record. It came + in within ±2 % on every arm, and it opens a deadlock class (a staging-retirement wait on a + written but unpublished record). +- **Native TLS (minSdk / ANDROID_PLATFORM 29):** +1.7 % on Espryt, noise on Magma (rd12 monolith). + This is the user's call. +- **Tracker / residual mutation-epoch fast path:** estimated at most 0.07 ms from the tracker alone, + 0.15-0.25 ms with the residual copies. Each step is below the 2 % bar, and a missed epoch bump + renders stale. It is folded into the draw-packet design.