[Docs] (P14): close-out bench, README, next-phase draw-packet sketch, DEBTS rows and handoff (rd12 monolith Espryt 69.8 -> 91.1, Magma 45.4 -> 83.8 fps).

This commit is contained in:
2026-10-07 21:20:11 -04:00
parent 781a7446c4
commit a0f4aa830e
5 changed files with 487 additions and 71 deletions
+7 -2
View File
@@ -1,5 +1,7 @@
# 仍开放的债务(跨阶段)
> **2026-10-07(P14 收官)**:W9 关闭(读数,见下);ID-P13-2(Magma monolith)更新为 trace 已基本追平、FCL 仍差的现状;新增三行:Espryt monolith 记录臂的逐项构成与结构原因、driver 项的 cache 压力推断、测过未做的前端变更 epoch 快路径。下一阶段是 draw-packet 缓存重设计。
>
> **2026-10-07(P13 收官)关闭三行**:ID-P8-13(9 条 monolith 缺陷——monolith 换到记录臂后对应 monolith 登记在集成车道上全过,旧的 monolith 前端臂在 W6 删除)、ID-P8-14(`SyncClientSideVertexArraysForIndirectFetch` 的整 buffer 回读与 Magma monolith 深度 mip 的 view 类型都属于已删的 monolith 前端臂)、`MOBILEGL_PIPE_TEXEL_RETAIN_MB`(W3b 随 pull 构建删除)。新增两行:W9 缓存容量重调与 Magma monolith 性能,都排在 P13 之后的性能轮,基线是 `dev` 构建。
>
> 路线图只列"有债务、看这里"。每条写明去向;已关闭的历史债与当时的完整债务表见 [`p5b/README.md`](p5b/README.md) 末节。开放问题另见 [`OPEN-QUESTIONS.md`](OPEN-QUESTIONS.md)。更新:2026-09-30(P8 第二波;按债务审查重排。同日:create-indirect 换成派生 fixture `minecraft-1.21.1-neoforge-create-indirect-in-world-align1024`(不是原抓取,16 字节对齐的 SSBO 范围绑定挪到 1024 对齐),ID-P8-14、ID-P8-19 两条关闭——红米两后端 monolith / spawn 全过,Magma spawn 不再丢设备)。去向「需裁定」= 等用户定;「可进行」= 随时可开的小修,不依赖别的阶段。
@@ -24,8 +26,11 @@
| P8 转出(E):driver-written 标按整张纹理打(`GenerateMipmap` 也打),驱动拒读时连基级也具名拒绝,偏保守;跨格式拷贝 store 不跟随;拒绝仍是会话 Fatal(要改回包状态才能答 `GL_INVALID_OPERATION`) | 小修(按 level 打标);其余按需 |
| P8 转出(B):`PipeCalls.def:260` `GenerateMipmap` 仍是 `kWaitApplied`,裁定 14 的理由(CPU 滤波碰 client 影子)已不成立,可改 `kWaitNone` | 性能工作(协议行为变更,需单独裁定) |
| P8 转出(C):server SSBO 标记过近似(窗口内每个绑定的 SSBO 都标);一个留在 SSBO 绑定上又当 restart EBO 用的 buffer 每次 draw 都回读 | 性能工作 |
| P13 转出(W5 翻转后,ID-P13-2 记录项,用户裁定记为债务):Magma monolith 走记录臂比前端臂慢约 16%——Y700 上 FCL MC 1.21.5、RD2 小场景、时钟未钉:记录臂 168 / 176 fps,同库强制前端臂 206 / 204 fps,即每帧 CPU 约 5.8 ms 对 4.9 ms(渲染线程都约占满一核);Espryt 持平(224 / 226 对 222)。疑因:`VulkanRenderer::SetupDraw` 在记录臂直接进 `SetupWireDraw`,绕过只在前端臂跑的 `TrySetupDrawFastPath`(PLAN-P13 §1.3 已提示「换臂前先量」)。W6 删前端臂后这部分成本固定下来 | P13 收官后(用户裁定:P13 内不做性能工作):性能工作的「之前」基线是 `dev` 分支构建(pull monolith),不是 W6 前的 feat 构建;在设备上钉时钟对比 dev 与 feat,场景为 FCL MC 与 anland glmark2,两个后端都测;再用 simpleperf profile Magma monolith 每 draw 成本,决定是否给 wire draw 臂补快路径 |
| P13 转出(W9,用户裁定:P13 内不做性能工作):缓存容量重调(CSO / 描述符 / 管线缓存容量按 `MOBILEGL_PIPE_STATS` 读数定初值、设备复核)未做,容量仍是 P8 时的值 | P13 之后的性能轮:以 `dev` 构建为基线(同 Magma 16% 那行),先在主机语料取 `MOBILEGL_PIPE_STATS` 读数定初值,再在设备上钉时钟复核 |
| P13 转出(ID-P13-2,P14 更新):Magma monolith 记录臂对 dev(pull)——trace 上已基本追平:rd12 83.8 对 87.0 fps(−4%),openra 747 对 293(dev 在那里等 GPU);P14 最大一项是 `553eb86d`(纹理层上传不再在 CPU 上等 GPU 排空,rd12 70→83)。但 **FCL MC 上仍差**:monolith 171 对 dev 232(−26%),inproc 195;呈现线程 CPU 只占约 2.8 / 5.8 ms 帧,其余是等待——与 trace 不同的另一处等待 | 下一阶段第一项:FCL Magma monolith 做 off-CPU profile(`--trace-offcpu`,同找到上传等待的方法;P14 收官时两次采集都没抓到游戏帧),找出那 3 ms 等待;数字见 `perf-p14/README.md` |
| ~~P13 转出(W9):缓存容量重调~~ **P14 关闭(读数)**:`MOBILEGL_PIPE_STATS` 在 rd12 / openra 上 render-state CSO 稳态每帧新铸 0 个,Magma 管线 memo 命中 99.0% / 92.7%,没有缓存抖动,P8 容量保留 | 已关闭(`perf-p14/BENCH.md` W9 节) |
| P14 转出:Espryt monolith 记录臂对 dev(pull)rd12 91.1 对 115.2 fps(−21%);FCL MC 198 对 229(−13%)。渲染线程每帧 CPU 11.56 对 9.07 ms:client half +1.39(validate 0.92:EmitGlobalConstants 0.17、EmitVertexElements 0.13、Tracker::Update 0.12、CopyField 0.10、validate 自身 0.10、EmitVertexBuffers 0.07;verb port 约 0.2)、backend +0.45(SyncNeccessaryTextures +0.14、SyncNeccessaryBuffers +0.08、AdoptUnchangedBufferSets +0.09、BindCurrentProgramWithResources +0.07、SyncCurrentProgramByHandle +0.05)、driver +0.33、applier +0.07。每项都在帧时间 2% 以下;合起来是记录臂的结构成本:pull 的后端在 draw 时按指针读前端对象,push 要检测变化(tracker)、序列化(emitter、uniform span)、应用(applier)、再由后端按句柄经 slot 表重新解析,并且因为后端看不到前端脏位而对每个绑定对象做 clean 检查——每步每 draw 几十 ns、各自是不相干的冷 cache 行,乘以 1471 draw/帧 | 下一阶段:per-(VAO, program) draw-packet 缓存(设计草稿 `perf-p14/NEXT-draw-packet-cache.md`,估计 rd12 Espryt monolith 回收 1.0–1.6 ms/帧,约 91→100–106 fps);先只做 client half 原型量 |
| P14 转出:Espryt monolith 的 driver 项 +0.33 ms/帧(2.9%),全在 `glDrawElements` 内部,GL 调用组合与 dev 相同。`simpleperf stat`(整次运行、渲染线程):feat 比 dev 多执行 16.6% 指令、L1D refill 多 56%、L2D 多 23%(IPC 3.73 对 3.86)。推断:记录臂每 draw 更大的工作集(record、applier 状态、slot 表、twin)把驱动的状态挤出 cache——这是计数器推断,没有在驱动内部测量 | 随 draw-packet 缓存一起验证:落地前后量 L1D refill 与 driver ms/帧;如果 driver 项不降,推断不成立,再单独查 |
| P14 转出(测过未做):Tracker / residual 的前端变更 epoch 快路径——在每个 `DECLARE_GL_FUNCTION_END` 入口(除 draw、`glBindVertexArray`、浮点 `glUniform*` 外)以及手写入口和 EGL 入口里 bump 一个上下文 epoch;epoch 未动时 validate 只重算 VAO / program 派生的 shutter(上限约 0.07 ms),再经审计哪些 residual 字段依赖 VAO / program 绑定后跳过 residual copy(合计 0.15–0.25 ms,1.5–2%) | 各自低于 2% 门槛,且漏 bump 一个入口就渲染陈旧状态;并入 draw-packet 设计(其 §3 的上下文 epoch 就是它),不单独做 |
| P13 转出(FCL 浸泡,用户裁定:FCL 暂不改、MobileGL 不做规避):FCL game process can die silently (exit SIGNALED status=34, SIGRTMIN+2): the JVM signal reset after renderer selection drops libcore's AsynchronousCloseMonitor handler, and FCL's okhttp/Okio watchdog then kills the process; renderer-independent (reproduced on P12 and P13 libraries, both backends; untested with other renderers)。浸泡:P13 库 30 次(Magma 22 / Espryt 8)死 18 次、P12 库 10 次死 2 次;P12 / P13 逐次交替的对照(同场景同网络时段)P12 4/10、P13 2/10,时间线(`Renderer:` 行到 MobileGL 首行约 6.2 s、晚死固定在其后约 9.4 s)两库相同,无证据表明 MobileGL 拉高概率;全部 status=34,无 tombstone、无 kgsl;证据与机理见 [`p13/fcl-soak/README.md`](p13/fcl-soak/README.md) | fix belongs in FCL (save/restore libcore's SIGRTMIN+2 sigaction around JVM creation); left for the user |
| P8 转出(MD):dev 的 `RingAllocateSlow` 在出错的上下文上无上限地排水(设备上挂在 `glFinish`) | 小修(dev) |
| P8 转出:`LogForwardChannel.APeerThatStopsReadingCostsWarnLinesNotTheCallersTime` 在负载下超时(两次),单跑必过 | 小修(放宽时限或隔离) |
+151
View File
@@ -0,0 +1,151 @@
# P14 perf pass: final bench
**Headline:** FCL Minecraft gives Espryt monolith 198 vs dev 229 and inproc 242; Magma
monolith 171 vs dev 232 and inproc 195. On the rd12 trace, Espryt monolith is 91 vs 115 and
Magma monolith 84 vs 87; inproc is 112 (Espryt) / 110 (Magma). Details below.
Head `781a7446` on `feat/disaggregated` (trace APK `p14fin`) against `dev` at `08124c99` (the pull
monolith, the baseline). Lenovo Y700 (Snapdragon 8 Gen 3, Adreno 750), rooted, CPU and GPU clocks
pinned (`pin_clocks.sh`), no per-frame glFinish (`FINISH=false`), best of 3, steady frames only.
Raw runs: `.trace-work/perf-p14/final-p14/` (`matrix_p14.sh run`, `p14_report.py`).
## Final matrix (fps, best of 3)
| workload | backend | dev (pull) | monolith | inproc | shm | tcp |
|---|---|---|---|---|---|---|
| rd12 | Espryt | 115.2 | 91.1 | 111.9 | 87.8 | 68.7 |
| rd12 | Magma | 87.0 | 83.8 | 110.0 | 95.6 | 69.9 |
| openra | Espryt | 889.2 | 832.5 | 1073.9 | 1063.1 | 602.0 |
| openra | Magma | 293.1 | 747.4 | 1051.6 | 976.0 | 544.5 |
Start of the pass (same harness, `nofinish` tag, `f8a9466b`):
| workload | backend | monolith | inproc | shm | tcp |
|---|---|---|---|---|---|
| rd12 | Espryt | 69.8 | 71.1 | 62.8 | 50.0 |
| rd12 | Magma | 45.4 | 63.2 | 58.6 | 45.6 |
| openra | Espryt | 812.2 | 1067.5 | 953.8 | 529.2 |
| openra | Magma | 356.3 | 446.0 | 394.4 | 412.1 |
So over the pass, rd12 went from 69.8 to 91.1 fps on Espryt monolith (+31%) and from 45.4 to 83.8 on
Magma monolith (+85%). Split arms: inproc +57% / +74%, shm +40% / +63%, tcp +37% / +53%. On openra,
Magma monolith went 356 -> 747 and inproc 446 -> 1052.
## Against the three goals
| goal | rd12 | openra |
|---|---|---|
| push = pull (monolith vs dev) | Espryt 91.1 vs 115.2 (-21%); Magma 83.8 vs 87.0 (-4%) | Espryt 832 vs 889 (-6%); Magma 747 vs 293 (2.5x dev) |
| inproc beats pull | Espryt 111.9 vs 115.2 (-3%); Magma 110.0 vs 87.0 (+26%) | Espryt 1074 vs 889 (+21%); Magma 1052 vs 293 (3.6x) |
| backend parity (feat) | monolith 91.1 vs 83.8; inproc 111.9 vs 110.0 | monolith 832 vs 747; inproc 1074 vs 1052 |
## CPU- or GPU-bound
Every arm on both workloads is CPU-bound: GPU busy is 19-39 % everywhere (`GPU busy` column of the
report), and the arm's top thread runs at 94-99 % of a core on the monolith and inproc arms
(97-99 % for the render thread, 94-95 % for `mgl-srv-apply` where it is the top thread).
- monolith: one thread, `MobileGLTraceRe`, at 99 % on all four cells.
- inproc / shm: two threads at about the same load (client 8.5-8.7, apply 8.5-8.6 ms/frame on rd12),
so the split arms are bound on whichever of the two is ahead; on rd12 Espryt that is the client
(harness parse + client half), on rd12 Magma the apply thread.
- tcp: 70-80 % top-thread utilisation, the rest is the socket ping-pong (credit-1 waits), as in P10.
- `dev` Magma on openra is the one cell below 95 %: 63 % utilisation, wide spread (13.6 %) - dev's
Magma waits on the GPU there, which the feat build no longer does (see "Magma upload wait").
Backend parity therefore means "both CPU-bound at the same CPU cost". On the feat build the two
backends are within 2-11 % of each other on every arm, and the remaining gap is CPU debt (rd12
monolith: Magma backend + driver 3.93 vs Espryt 2.76 ms/frame, client half 1.91 vs 1.78).
## Layer split, rd12 monolith (render thread, ms/frame, simpleperf at 2 kHz, steady tail)
| layer | Espryt feat | Espryt dev | delta | Magma feat | Magma dev | delta |
|---|---|---|---|---|---|---|
| driver | 1.69 | 1.36 | +0.33 | 2.12 | 1.88 | +0.24 |
| backend | 1.07 | 0.62 | +0.45 | 1.81 | 2.56 | -0.75 |
| applier | 0.07 | 0 | +0.07 | 0.13 | 0 | +0.13 |
| client half | 1.78 | 0.39 | +1.39 | 1.91 | 0.70 | +1.21 |
| harness (trace parse) | 6.94 | 6.70 | +0.24 (same code) | 6.79 | 6.86 | -0.07 |
| total CPU | 11.56 | 9.07 | +2.49 | 12.77 | 12.01 | +0.76 |
About 6.7-6.9 ms of every rd12 frame is the replay harness, so the MobileGL part of an Espryt frame
is 4.6 ms on feat against 2.4 ms on dev, and of a Magma frame 6.0 against 5.1.
### What the Espryt monolith gap is made of (as % of an 11.2 ms frame)
- Validate point, 0.92 ms (none of it exists in pull): EmitGlobalConstants 0.17 (1.5 %),
EmitVertexElements 0.13 (1.1 %, includes the inline apply of the bind), Tracker::Update 0.12
(1.1 %), validate self 0.10 (0.8 %), CopyField 0.10 (0.9 %), EmitVertexBuffers 0.07 (0.6 %),
thread-local and atomic overhead about 0.07.
- Verb port, about 0.2 ms (1.8 %): EmitDrawRecord, ReadDrawBindings 0.06, MarkGpuWritesForDraw 0.05.
- Backend +0.45: SyncNeccessaryTextures 0.23 vs 0.10, SyncNeccessaryBuffers 0.16 vs 0.08,
AdoptUnchangedBufferSets 0.09, BindCurrentProgramWithResources 0.18 vs 0.11,
SyncCurrentProgramByHandle 0.09 vs 0.04.
- Driver +0.33 (2.9 %): inside glDrawElements itself, with the same GL call mix as dev. A
`simpleperf stat` over the whole run (render thread) has feat executing 16.6 % more instructions
than dev with 56 % more L1D and 23 % more L2D refills (IPC 3.73 vs 3.86): the record arm's larger
per-draw working set (records, applier state, slot tables, twins) evicting the driver's state is
the best reading, inferred from those counters, not measured inside the driver.
Each item is under 2 % of the frame except the driver term. Why the sum is structural on the record
arm: pull's backend reads frontend objects by pointer at draw time; push has to detect what moved
(tracker), serialise it (emitters, uniform spans), apply it (applier), then let the backend resolve
it again by handle through slot tables, with per-bound-object clean checks because the backend no
longer sees the frontend's dirty bits. Each step is tens of ns per draw on cache lines no other
step shares, times 1471 draws a frame.
Options left (none below the 2 % bar on its own, or each with a correctness risk the bar does not
pay for):
- a frontend mutation epoch (bumped in every `DECLARE_GL_FUNCTION_END` entry except draws,
glBindVertexArray and float glUniform*) so a validate with only VAO and uniform movement skips
the context-global shutters (at most 0.07 ms) and, after an audit of which residual fields
depend on the VAO/program bindings, the residual copies (0.15-0.25 ms together, 1.5-2 %);
- a per-(VAO, program) draw packet that keeps tracker, emitter, applier and backend state for a
draw on a few cache lines, keyed on lifetime ids - the structural answer to the cache-pressure
term, a redesign rather than a pass item;
- native TLS (minSdk / ANDROID_PLATFORM 29 instead of 26, the user's call): measured +1.7 %
Espryt, within noise on Magma (rd12 monolith, best of 3; whole APK, harness included).
## Pass findings worth keeping
- **Magma upload wait (553eb86d).** Every texture level upload flushed the recording and then
waited on the CPU for every submission to complete (`FlushWirePendingCommandsForTextureUpdate`,
added in P7 B4). The upload batch is submitted on the same queue after the flush and its first
barrier takes its source scope from the image's tracked layout, so the wait bought nothing; on
rd12 it was 1.8 ms/frame of fence stall (Minecraft updates its lightmap every frame). Removing it:
rd12 Magma monolith 70.4 -> 83.4, openra 367 -> 745. The preserve path keeps its wait. Proof: a
new between-draws upload scenario under the synchronization validator (red on a broken barrier),
sync validation of the texture/remint/copy scenarios (no new hazards), SSIM 16/16.
- **Sync validation needs submit-time validation on.** The Khronos layer (1.4.350) does not report
a hazard between two queue submissions unless `VK_KHRONOS_VALIDATION_SYNCVAL_SUBMIT_TIME_VALIDATION`
is set; the run-ahead and cache gates now set it (781a7446), and the run-ahead gate goes red on
the broken-barrier build where it used to stay green.
- **GPU-fault latch regression (549a34bb -> fixed in c24b3428).** Polling the reset status every
256 records let a record published right after the previous acknowledgement run unpolled on a
guilty context. The drain now polls before any record the last poll could not have seen
(e058a450 hands the apply loop's poll to the drain so a woken drain does not poll twice).
- **Record-publish batching, measured and dropped.** Publishing an unbarriered validate's state
records with the draw record behind them was within +-2 % on every arm and opens a deadlock class
(a staging-retirement wait on a record that is written but not published - it hung rd12 Espryt
split until the stage wait published owed records). Not worth the surface; the patch and its
stress scenario are not in the tree.
## W9 (cache capacities), closed with readings
`MOBILEGL_PIPE_STATS` on rd12 and openra: the render-state CSO cache mints 0 per frame in the steady
state, and Magma's pipeline memo hits 99.0 % on rd12 and 92.7 % on openra. No cache thrashes; the
P8 capacities stand.
## FCL (Minecraft in world, the representative app check)
FCL's own build of the library (plugin flavor, no POISON), swapped into FCL. Readings come from
MobileGL's fps log: mean of two launches × three steady windows. Clocks pinned. Runs:
`.trace-work/perf-p14/fcl-final/`.
| backend | dev (pull) | feat monolith | feat inproc | presenting-thread CPU, monolith feat / dev |
|---|---|---|---|---|
| Espryt | 228.6 | 198.3 (-13 %) | 241.5 (+6 %) | 2.43 / 1.85 ms/frame |
| Magma | 231.8 | 171.4 (-26 %) | 194.8 (-16 %) | 2.80 / 1.64 ms/frame |
FCL Magma does not match the trace: the presenting thread is busy only about 2.8 ms of a 5.8 ms
frame, so about 3 ms is a wait the trace does not have. That is the next phase's first item: an
off-CPU profile, the method that found the trace's upload stall.
+68 -69
View File
@@ -1,74 +1,50 @@
# P14 perf pass: handoff (checkpoint 2026-10-07)
# P14 perf pass: handoff (closed 2026-10-07)
Worktree `.claude/worktrees/perf` (branch `perf`, pushed to `origin/feat/disaggregated`). Estimate:
[ESTIMATE.md](ESTIMATE.md). BENCH.md / README.md are not written yet (see "Next").
The pass is closed at `781a7446`, plus this docs commit. Results are in [BENCH.md](BENCH.md) and
[README.md](README.md). The next phase is the per-(VAO, program) draw-packet cache
([NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md)). Its first item is an off-CPU profile of
FCL Magma monolith (README "Headline numbers").
## Head and running totals
## Where things stand (rd12, pinned, FINISH=false, best of 3)
Head: `94705adc` (plus this handoff commit). rd12, Lenovo Y700 (Adreno 750), clocks pinned, no
per-frame glFinish (`FINISH=false`, the primary mode; finish=true serialises the split arms on one
FenceWait round trip per frame), best of 3:
| arm | Espryt start -> now | Magma start -> now | dev (pull) Espryt / Magma |
| arm | Espryt | Magma | dev Espryt / Magma |
|---|---|---|---|
| monolith | 69.8 -> 88.6 | 45.4 -> 69.5 | 115.1 / 86.1 |
| inproc | 71.1 -> ~107 | 63.2 -> ~99 | |
| shm | 62.8 -> ~82 | 58.6 -> ~85 | |
| tcp | 50.0 -> ~64 | 45.6 -> ~61 | |
| monolith | 91.1 | 83.8 | 115.2 / 87.0 |
| inproc | 111.9 | 110.0 | |
| shm | 87.8 | 95.6 | |
| tcp | 68.7 | 69.9 | |
(split-arm "now" values are from the p14p A/B, before the last two monolith commits; re-measure.)
GPU busy 22-33 % everywhere: every arm is CPU-bound on one thread.
FCL MC in world:
- dev: Espryt 228.6, Magma 231.8;
- feat monolith: Espryt 198.3, Magma 171.4;
- feat inproc: Espryt 241.5, Magma 194.8.
## Latest layer split (rd12 monolith, render thread ms/frame, simpleperf, head `94705adc` = p14r)
All arms are CPU-bound (GPU 19-39 %).
| layer | Espryt feat | Espryt dev | Magma feat | Magma dev |
|---|---|---|---|---|
| driver | 1.56 | 1.40 | 2.15 | 1.98 |
| backend | 1.21 | 0.59 | 1.95 | 2.12 |
| applier | 0.07 | 0 | 0.13 | 0 |
| client half | 1.96 | 0.36 | 2.01 | 0.69 |
| harness | 6.46 | 6.27 | 6.50 | 6.25 |
| total CPU | 11.27 | 8.62 | 12.77 | 11.05 |
## Open items (DEBTS.md has the rows)
Client half top: validate self 0.22-0.26, verb port (MG_Record) 0.15-0.18, Tracker::Update 0.14-0.17,
CopyField 0.07-0.09, VAO memo probe 0.07-0.09. Espryt backend top: PrepareForDraw 0.10,
AdoptUnchangedBufferSets 0.10, PerNativeContext<...>::Current 0.084, SyncTextureToBackendByHandle
0.066, IsBufferDrawCleanByHandle 0.058. Magma backend top: SetupWireDraw self 0.35,
BindDescriptorSetDeduped 0.23 (dynamic UBO offset moves every chunk draw), AcquireWireSlice 0.19.
Per-draw counts (instrumented build, `p14count2.patch` in the scratchpad, not committed): rd12 1471
draws/frame, 3.3 GL calls/draw; after the protocol change 66.8 B/validate (bind 16 B + uniform span
~47 B), VertexBuffers/IndexBuffer 100 % no-op.
## Commits this pass (newest first)
94705adc inline applier / fill plan / word-wise spans / memo-answered draw bindings (+6 % Espryt, +3.7 % Magma);
81bdfb1f Espryt twin lookups w/o SharedPtr copy + one TLS block (+3.8 %);
032dcd24 Magma vertex-input layout kept on its record + slot-indexed buffer front;
c1497bb8 VAO-kept bindings, uniform spans, context-values gate (split arms +5-10 %; stream span merge fixes a tcp regression);
0158a115 Magma pass-continuation memo; 8883e496 fill pins + VAO hash memo; 80dc9d45 VAO memo;
5788a602 Magma descriptor rebind; afcf8d96 Espryt AdoptUnchangedBufferSets; e6281b26, 4ef9705e Magma
lookups/sampler memo; b556ec31, a9108e1a Espryt; 549a34bb server fault latch; 8346d809 slot allocator
front; 267211e8 Magma hash memo + scratch.
## In flight / next
1. Espryt backend lookups: `PerNativeContext<T>::Current` (TLS cache per T + non-inline serial/epoch
calls), `IsBufferDrawCleanByHandle`, `AdoptUnchangedBufferSets` (compares the window every draw
because the bind re-apply bumps VertexBuffersSerial), `SyncTextureToBackendByHandle`.
2. Tracker::Update: 18 shutters evaluated per draw, 4 fire. Needs a cheap "nothing but the VAO and
the uniform moved" signal; no central frontend mutation epoch exists yet.
3. Residual fill: now a precomputed walk, but still ~11 CopyField per draw.
4. Magma: SetupWireDraw self, per-draw dynamic-offset descriptor bind.
5. Final full matrix (dev + 4 arms x 2 backends x openra/rd12, REPS=3), then BENCH.md, README.md,
DEBTS.md rows (Magma 16 %, W9: closed with readings - render-state CSO cache mints 0/frame, Magma
pipeline memo 99.0 % hit on rd12, 92.7 % on openra), FCL re-run on the final lib.
1. FCL Magma monolith waits about 3 ms/frame. The trace does not show it. Profile it off-CPU
first. Both close-out attempts caught no game frames, because FCL start-up ran past the
60 s warm-up.
2. The draw-packet cache (design sketch). Prototype the client half on Espryt monolith first.
3. The Espryt driver +0.33 ms is a cache-pressure inference. Check it with L1D refills
before and after (2).
4. Native TLS, minSdk 29 vs 26, is the user's call: +1.7 % Espryt, noise on Magma.
## Harness
Scratchpad scripts (session-local): `build_trace.sh W:/perf .p14X` (trace APK, suffix = package
`top.mobilegl.plugin.p14X.trace`), `sign_install.sh <apk> <name>`. W: is `subst` of
`.claude/worktrees` (MAX_PATH). Then:
Session scratchpad scripts:
- `build_trace.sh W:/perf .p14X`: trace APK, package `top.mobilegl.plugin.p14X.trace`.
- `sign_install.sh <apk> <name>`.
- `devjob.sh <log> <cmd>`: runs a device job under a mkdir lock. Always use it; two concurrent
device jobs contaminate each other, and killing a job leaves its children running.
- `incl.py` / `children.py` / `lines.py` / `layers2.py` / `drvcall.py` / `offcpu.py`:
simpleperf analyses. The tail window comes from the render thread's `retraceCall` samples.
- `profile_offcpu.sh`: `--trace-offcpu` variant of `profile_p14.sh`.
- `stat_p14.sh`: `simpleperf stat` counters.
- `fcl_offcpu.sh`: FCL profile plus restore.
Then:
```
cd tools/device_bench/disagg
@@ -77,21 +53,44 @@ ARMS="monolith@p14a monolith@p14b" BACKENDS="DirectGLES DirectVulkan" WLS=rd12 R
python p14_report.py ../../../.trace-work/perf-p14/ab
ARMS="monolith@p14X inproc@p14X shm@p14X tcp@p14X" REPS=1 TAG=ssim bash matrix_p14.sh ssim # SSIM gate
ARM=monolith BACKEND=DirectGLES WL=rd12 TAG=prof PKG_FEAT=... SYMDIR=<unstripped libs> bash profile_p14.sh
python layers.py <perf.data> <binary_cache> MobileGLTraceRe <cpu ms/frame> --tail-ms=N [--detail=backend]
```
FIXTURES must point at a checkout with real LFS goldens (worktrees have pointer files). Host tests:
WSL archlinux `~/mgl-perf-bld` (clang, split+inproc, tests on); ServerSpawnTest / FuzzArm2 /
PeerLatchTest fail there with Refuse{BuildFingerprint} (no build stamp in that tree), environmental.
The FCL library comes from `:app:assemblePluginRelease`. Unzip `lib/arm64-v8a/libMobileGL.so` from
`app/build/outputs/apk/plugin/release/`; the unstripped copy is under
`intermediates/merged_native_libs/pluginRelease`. Run `fcl_p14.sh run`, then `restore`
(md5-checked).
Gradle's `packageTraceRelease` sometimes fails on the first try with
`IncrementalSplitterRunnable`; a rerun succeeds.
Host:
- Unit tests: WSL archlinux `~/mgl-perf-bld` (clang, split + inproc).
- ServerSpawnTest, FuzzArm2 and PeerLatchTest fail there with `Refuse{BuildFingerprint}`;
that is environmental.
- The google-benchmark targets don't build with clang 22; also environmental.
- CI-shaped integration tree: `~/mgl-perf-ci31`, configured with CMake 3.31.10 from the
`~/mgl-perf-tools` venv.
- The system CMake 4.x joins `gtest_discover_tests` ENVIRONMENT lists into one variable, and
every entry then skips. Use 3.31.10.
- lavapipe ICD; spawn/tcp lanes are environmental there.
- To run the syncval gates: `scripts/ci/magma_runahead_checks.py validation --build-dir ~/mgl-perf-ci31 --execute`,
with `GIT_DIR=.../.git/modules/MobileGL/worktrees/perf` set for the cache script.
- `~/mgl-perf-ci` is a CMake 4 tree. Use it only through the scratchpad's `runentry.sh`, which
splits the joined environment.
## Device state
HA27Q3LQ: CPU/GPU clocks PINNED (`bash pin_clocks.sh restore` to undo). FCL lib/config/props restored
(md5 checked; `debug.mobilegl.*` props unset). Trace APKs `top.mobilegl.plugin.p14*.trace` installed.
- HA27Q3LQ CPU/GPU clocks are still PINNED: the next phase wants them pinned.
`bash pin_clocks.sh restore` undoes it; ask before unpinning.
- FCL is restored: lib md5 matches `/data/local/tmp/p14-fcl-orig.so`, config.json matches its
saved copy, and the `debug.mobilegl.*` properties are deleted.
- Trace APKs `top.mobilegl.plugin.p14*.trace` are installed; `p14fin` is the close-out build.
## CI
No run has completed since f8a9466b (each push cancelled the previous). workflow_dispatch runs on
the checkpoint head: see the milestone message / below.
Dispatched runs (workflow_dispatch, not cancelled by later pushes):
- 86ab2835: Test 37702990844, APK 37702995519.
- 781a7446: Test 37707221661.
- The close-out docs head: see the final report.
Dispatched on ff53ba3b: Test run 37674322398, MobileGL APK run 37674328683 (workflow_dispatch, not cancelled by later pushes).
There is only one self-hosted GPU runner (`minipc-8845-arch-wsl-gpu`), so GPU jobs queue behind it.
@@ -0,0 +1,150 @@
# Next phase: a per-(VAO, program) draw-packet cache (design sketch)
This is a starting point for the next phase, not an implementation. The numbers are from the P14
close-out (`BENCH.md`, rd12 monolith, Y700, pinned, render thread, ms/frame).
## 1. The problem
On the record arm, a draw crosses five layers, and each layer keeps its own per-object state:
| layer | Espryt feat | Espryt dev | delta | what it touches per draw |
|---|---|---|---|---|
| client half | 1.78 | 0.39 | +1.39 | tracker shutters (VAO, program, context gens), VaoMemo, emitter latches, slot allocator, residual block, verb port |
| applier | 0.07 | 0 | +0.07 | vertex-elements record, applier unit windows, global-constants block |
| backend | 1.07 | 0.62 | +0.45 | VAO twin, texture/sampler records + twins (clean checks), program twin, UBO ring |
| driver | 1.69 | 1.36 | +0.33 | the same GL calls as dev; slower, consistent with cache pressure (+56 % L1D refills, +16.6 % instructions, whole run) |
Minecraft rd12 issues 1471 draws a frame, about 3.3 GL calls per draw. Between two consecutive
draws, usually only the bound VAO and the program's default-uniform block change. Each layer still
re-derives "what moved" from its own structures, and each of those structures is a different cold
cache line per VAO: the frontend VAO, the VaoMemo entry, the emitter latch, the applier record, and
the backend twin. Pull touches one of them, the frontend VAO.
None of the individual items is above 2 % of the frame. The sum is the gap: on Espryt monolith,
91 vs 115 fps.
## 2. What one draw packet holds
There is one packet per (VAO lifetime id, program lifetime id) pair that has drawn. It lives in a
client-side table, and in the server's record space on the split arms. Its fields:
- **Identity:** VAO lifetime id + config version, program lifetime id + link version, and the
context serial.
- **Client side:**
- the vertex-elements handle and its published config version;
- the buffer-set and index handles and their hash, which the VaoMemo holds today;
- the shader CSO handle;
- the residual-field values this pair contributed (the bound-VAO and program pins);
- the context-global epoch it was validated at (see section 3).
- **Applier side:** a pointer, or slot index plus generation, to the vertex-elements record and the
stored buffer/index window, so that bind_vertex_elements applies by reference instead of copying.
- **Backend side (Espryt):**
- the VAO twin;
- the resolved draw buffers;
- the program twin;
- the sampled-texture list with each entry's synced serials;
- the synced attachment list;
- the UBO block size.
- **Backend side (Magma):** the wire vertex-input state, the pipeline handle, the descriptor-set
layout, and the pass-compatibility id.
- **A "packet valid at" stamp per layer:** the serials and epochs each layer checks to trust its
slice of the packet.
The goal is that a draw whose packet is valid at every layer touches the packet and the per-draw
inputs, and nothing else. The per-draw inputs are the uniform span, the draw arguments, and the UBO
ring slot.
## 3. Keys and invalidation sources
**Keys.** Lifetime ids only, never GL names or pointers. This is the ABA rule, already broken and
fixed three times: sampler, texture, program. The packet table key is (VAO lifetime id, program
lifetime id, context serial).
Invalidation sources, by layer:
- **Frontend context-global state.** This is everything the tracker reads that is neither VAO- nor
program-derived: render/pipeline state versions, texture content/params/bind generations,
framebuffer attachment and bind versions, buffer bind-point generations, TFB, pixel pack, patch,
vertex attribute defaults, and the program pipeline.
- This needs a context mutation epoch, bumped in every GL/EGL entry except the
whitelist: draws, glBindVertexArray, and float glUniform*.
- The bump fits in the `DECLARE_GL_FUNCTION_END` macros, with the whitelist as a constexpr name
test, plus the manual entries (glFinish/glFlush and the hand-written extension wrappers) and
the EGL entries.
- **Audit item:** every MOBILEGL_GL_API definition outside the macros.
- **VAO.** The config version (attribute pointers and enables), the index slot version, and buffer
respecification through `AnyVaoAttributeGeneration`. These are already lifetime- and
version-keyed.
- **Program.** Link version, backend state version (sampler units through glUniform1i), block
binding version, and image unit version. Float uniforms move only the UBO content version,
which is a per-draw input and not a packet invalidation.
- **Applier.** The record `ContentSerial`, `ContextSerial`, `MGPipeSlots().ResetEpoch()`, and the
vertex-buffers and index-buffer serials. These are already monotonic.
- **Backend.**
- Texture: record `Serial`, `ParamsSerial`, and `PendingUploads`; twin force-resync flags;
sampler CSO serial; shared-image state.
- Context generation.
- Magma only: texture image epoch, resource erase epoch, the image layout leaving GENERAL, and the
store destroy epoch. The P14 Magma memos already check these.
- The backend needs one "texture state moved" epoch, so that the per-bound-texture clean check
can be skipped when nothing in the texture family moved. This is the item the content-version
lesson warns about: glGenerateMipmap grew mips without a version bump. Every backend-side mutation
of a twin's synced state must bump the epoch too.
## 4. Split arms and the verify comparator
- **The packet is per role.** The client half of the packet (handles, latches, residual pins)
lives in the client. The applier and backend halves live wherever the applier runs: the same
thread on monolith, the apply thread on inproc, the server process on spawn and tcp.
- **Nothing in the packet crosses the wire.** The wire stays the record stream. A server-side
packet is keyed on the vertex-elements handle and the shader-CSO handle (slot + generation)
plus the applier context serial, because the server never sees frontend lifetime ids.
- **Run-ahead.** Client-side validity must not depend on server state. The client's half is
checked against client epochs only, exactly like today's latches.
- **The verify comparator (`MOBILEGL_PIPE_VERIFY`) compares the residual block against a live
snapshot at the verb.**
- A packet hit that skips residual copies would make the comparator diverge, unless the packet's
pins are what it compares.
- Proposal: under verify, take the slow path always and assert that the packet would have hit
with the same values. That gives a free cross-check of the epoch's coverage on every verify
lane.
- **POISON stamps (trace build only).** The stamp loop stays, or the packet records the stamp set
and applies it with one bulk write.
## 5. Risks
- **A missed epoch bump renders stale state.** This is the dominant risk. Mitigations:
- the verify-lane cross-check above;
- a negative control that removes the bump from one entry and must go red;
- SSIM 16/16 on every change;
- the existing retrace CI.
- **ABA on packet keys.** Lifetime ids only, plus context serial and slot reset epoch.
- **Memory.** One packet per drawn (VAO, program) pair. Minecraft has thousands of chunk VAOs ×
a few programs, so a bounded table with LRU is about 100-200 B × 10k ≈ 2 MB. The table must
not itself become the next cache-miss source: entries go in a flat array with an index stored
on the frontend VAO.
- **Backend-side invalidation completeness** (section 3, backend): the same class as the reverted
e8e1521.
- **Code size and review cost.** It touches every layer. It should land behind a runtime knob,
default off until the CI and SSIM gates and an FCL soak are green.
## 6. Rough gain estimate (rd12, ms/frame, Espryt monolith)
| layer | removable on a packet hit | why not all of the delta |
|---|---|---|
| client half (+1.39) | 0.6-0.9 | the uniform span, bind record and draw record still have to be produced |
| applier (+0.07) | 0.03 | bind by reference |
| backend (+0.45) | 0.25-0.35 | the UBO ring slot and the GL calls stay |
| driver (+0.33) | 0.1-0.3, if the cache-pressure reading is right | unproven; measure L1D refills before and after |
Total: about 1.0-1.6 ms of the 2.5 ms gap. That is roughly 91 -> 100-106 fps on Espryt monolith.
Magma's share is similar on the client half (+1.21). Its backend is already below dev, and its
driver term is +0.24.
Inproc gains the client-half part on the client thread: that is the bound thread on Espryt rd12,
so 112 -> above dev's 115. Magma inproc is bound on the apply thread, which gains only the backend
and applier part.
Prototype first, before the full design: the client-half packet (tracker + emitter + residual pins,
with the context epoch) on monolith Espryt only. Measure it against the 0.6-0.9 ms estimate, then
decide on the backend half.
+111
View File
@@ -0,0 +1,111 @@
# P14: the post-P13 performance pass
P14 measured the record (push) arm against the `dev` pull monolith and cut its CPU cost. It ran on
`feat/disaggregated` from `f8a9466b` (start) to `781a7446` (close-out), on a Lenovo Y700 (Adreno
750) with pinned clocks.
- [ESTIMATE.md](ESTIMATE.md): the code-based estimate, written before the first benchmark.
- [BENCH.md](BENCH.md): the final trace matrix, the layer split, CPU vs GPU binding, and what the
remaining gap is made of.
- [HANDOFF.md](HANDOFF.md): harness commands, device state, open items.
- [NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md): the design sketch for the next phase.
## Headline numbers
### Minecraft in FCL (the representative app check)
Setup:
- FCL Minecraft, in world.
- The library is swapped into FCL, which runs it as a plugin build without `MOBILEGL_PIPE_POISON`.
- Readings come from MobileGL's own fps log.
- Two launches × three steady windows of about 2 s each. Dev Espryt is three windows: its second
launch died at start-up, twice.
- Clocks are pinned. Runs: `.trace-work/perf-p14/fcl-final/`.
| backend | dev (pull) | feat monolith | feat inproc | feat monolith, presenting-thread CPU |
|---|---|---|---|---|
| Espryt | 228.6 | 198.3 (−13 %) | 241.5 (+6 %) | 2.43 ms/frame (dev 1.85) |
| Magma | 231.8 | 171.4 (−26 %) | 194.8 (−16 %) | 2.80 ms/frame (dev 1.64) |
Earlier in the pass, at `94705adc`, the same measurement read:
- Espryt: monolith about 192, inproc about 233;
- Magma: monolith about 169, inproc about 179;
- dev: 209-245.
FCL Magma monolith did not follow the trace's improvement. Its presenting thread is busy only
about 2.8 of the roughly 5.8 ms frame, so the rest is a wait. That is the first item for the next
phase: an off-CPU profile of FCL Magma monolith, the same method that found the upload stall in
the trace. Two attempts at it during close-out caught no game frames (FCL start-up).
### Trace replay, rd12 (Minecraft 1.21.4 RD12, 1471 draws/frame), best of 3, fps
| backend | dev (pull) | monolith | inproc | shm | tcp |
|---|---|---|---|---|---|
| Espryt | 115.2 | 91.1 (start 69.8) | 111.9 (71.1) | 87.8 (62.8) | 68.7 (50.0) |
| Magma | 87.0 | 83.8 (start 45.4) | 110.0 (63.2) | 95.6 (58.6) | 69.9 (45.6) |
openra is in [BENCH.md](BENCH.md). On openra, Magma monolith now runs 2.5x faster than dev
(747 vs 293), and inproc 3.6x (1052 vs 293).
Every arm is CPU-bound: GPU busy is 19-39 %, and the top thread sits at 94-99 % of a core
(70-80 % on tcp).
## What landed
The commits run from `f8a9466b` to `781a7446`. Per-commit numbers are in the commit subjects.
- **Magma wire draw:**
- no per-draw SPIR-V rehash or allocation;
- unit sampler memo; unchanged pipeline, vertex and index binds skipped;
- direct-mapped texture and buffer fronts;
- descriptor-set rebind with fresh offsets;
- pass-continuation memo;
- vertex-input layout kept on its record;
- **texture level uploads no longer wait for the GPU to drain** (`553eb86d`, the largest
single gain of the pass: rd12 70 -> 83, openra 367 -> 745);
- slices stamped from the owner's submit counter.
- **Espryt:**
- VAO twin re-points only the moved buffers;
- clean attachments skip resync;
- unchanged buffer sets are adopted;
- twin lookups without SharedPtr copies;
- one thread-local block;
- one UBO-ring lookup per draw.
- **MG_Pipe client half:**
- lifetime-id slot fronts;
- per-VAO memo;
- fill pins;
- kept VAO bindings on the server;
- uniform spans;
- context-values gate;
- inline process applier;
- precomputed residual walk, with split copy and stamp loops;
- memo-answered draw bindings.
- **MG_Remote:** fault-verdict polling restructured.
- It first went from every record to every 256 records (549a34bb).
- The drain now polls before any record the last poll could not have seen (`c24b3428`). This
fixes the regression where a guilty context ran one more record.
- The apply loop hands its poll to the drain (`e058a450`).
- **Magma hang watch:** a marker slot is reused only once its submission's fence completes
(`ec973fba`).
- **Tests and CI:**
- TextureUploadBetweenDrawsScenario on every arm, plus inproc sync validation (`67f4f311`,
`86ab2835`);
- submit-time synchronization validation in the run-ahead and cache gates (`781a7446`);
- ServerLoopEglLatchTest now holds the race window open.
- **Tools:**
- pinned-clock matrix with a per-thread CPU and GPU-busy sampler;
- layer-split and off-CPU profile helpers;
- FCL harness;
- opt-in fps log and transport override properties.
## Measured and not landed
- **Record-publish batching:** publishing a validate's state records with the draw record. It came
in within ±2 % on every arm, and it opens a deadlock class (a staging-retirement wait on a
written but unpublished record).
- **Native TLS (minSdk / ANDROID_PLATFORM 29):** +1.7 % on Espryt, noise on Magma (rd12 monolith).
This is the user's call.
- **Tracker / residual mutation-epoch fast path:** estimated at most 0.07 ms from the tracker alone,
0.15-0.25 ms with the residual copies. Each step is below the 2 % bar, and a missed epoch bump
renders stale. It is folded into the draw-packet design.