[Docs] (.claude/skills, .github/workflows, CMakeLists.txt, docs/archive, docs/Disaggregated, docs/MobileVA, docs/Performance, MG_Backend, MG_Backend/DirectGLES, MG_Backend/DirectVulkan, MG_Backend/Record, MG_Gbm, MG_Impl, MG_Impl/EGLImpl, MG_Impl/GLImpl, MG_Impl/GLXImpl, MG_Impl/Pipe, MG_IntegrationTest, MG_IntegrationTest/Harness, MG_IntegrationTest/Scenarios, MG_Pipe, MG_Pipe/generated, MG_Remote, MG_Remote/Client, MG_Remote/Server, MG_Remote/Transport, MG_Remote/Wire, MG_State/GLState, MG_Test/Pipe, MG_Test/Util, MG_Test/Wire, MobileGL, scripts, scripts/data, tools/device_bench, userguide): archive the planning notes and agent contracts under docs/archive

- DEBTS.md and OPEN-QUESTIONS.md stay maintained, now in docs/Disaggregated.
This commit is contained in:
2026-10-15 20:01:45 -04:00
parent 0e3e2b7761
commit aabca3c171
1035 changed files with 410 additions and 398 deletions
+151
View File
@@ -0,0 +1,151 @@
# P14 perf pass: final bench
**Headline:** FCL Minecraft gives Espryt monolith 198 vs dev 229 and inproc 242; Magma
monolith 171 vs dev 232 and inproc 195. On the rd12 trace, Espryt monolith is 91 vs 115 and
Magma monolith 84 vs 87; inproc is 112 (Espryt) / 110 (Magma). Details below.
Head `781a7446` on `feat/disaggregated` (trace APK `p14fin`) against `dev` at `08124c99` (the pull
monolith, the baseline). Lenovo Y700 (Snapdragon 8 Gen 3, Adreno 750), rooted, CPU and GPU clocks
pinned (`pin_clocks.sh`), no per-frame glFinish (`FINISH=false`), best of 3, steady frames only.
Raw runs: `.trace-work/perf-p14/final-p14/` (`matrix_p14.sh run`, `p14_report.py`).
## Final matrix (fps, best of 3)
| workload | backend | dev (pull) | monolith | inproc | shm | tcp |
|---|---|---|---|---|---|---|
| rd12 | Espryt | 115.2 | 91.1 | 111.9 | 87.8 | 68.7 |
| rd12 | Magma | 87.0 | 83.8 | 110.0 | 95.6 | 69.9 |
| openra | Espryt | 889.2 | 832.5 | 1073.9 | 1063.1 | 602.0 |
| openra | Magma | 293.1 | 747.4 | 1051.6 | 976.0 | 544.5 |
Start of the pass (same harness, `nofinish` tag, `f8a9466b`):
| workload | backend | monolith | inproc | shm | tcp |
|---|---|---|---|---|---|
| rd12 | Espryt | 69.8 | 71.1 | 62.8 | 50.0 |
| rd12 | Magma | 45.4 | 63.2 | 58.6 | 45.6 |
| openra | Espryt | 812.2 | 1067.5 | 953.8 | 529.2 |
| openra | Magma | 356.3 | 446.0 | 394.4 | 412.1 |
So over the pass, rd12 went from 69.8 to 91.1 fps on Espryt monolith (+31%) and from 45.4 to 83.8 on
Magma monolith (+85%). Split arms: inproc +57% / +74%, shm +40% / +63%, tcp +37% / +53%. On openra,
Magma monolith went 356 -> 747 and inproc 446 -> 1052.
## Against the three goals
| goal | rd12 | openra |
|---|---|---|
| push = pull (monolith vs dev) | Espryt 91.1 vs 115.2 (-21%); Magma 83.8 vs 87.0 (-4%) | Espryt 832 vs 889 (-6%); Magma 747 vs 293 (2.5x dev) |
| inproc beats pull | Espryt 111.9 vs 115.2 (-3%); Magma 110.0 vs 87.0 (+26%) | Espryt 1074 vs 889 (+21%); Magma 1052 vs 293 (3.6x) |
| backend parity (feat) | monolith 91.1 vs 83.8; inproc 111.9 vs 110.0 | monolith 832 vs 747; inproc 1074 vs 1052 |
## CPU- or GPU-bound
Every arm on both workloads is CPU-bound: GPU busy is 19-39 % everywhere (`GPU busy` column of the
report), and the arm's top thread runs at 94-99 % of a core on the monolith and inproc arms
(97-99 % for the render thread, 94-95 % for `mgl-srv-apply` where it is the top thread).
- monolith: one thread, `MobileGLTraceRe`, at 99 % on all four cells.
- inproc / shm: two threads at about the same load (client 8.5-8.7, apply 8.5-8.6 ms/frame on rd12),
so the split arms are bound on whichever of the two is ahead; on rd12 Espryt that is the client
(harness parse + client half), on rd12 Magma the apply thread.
- tcp: 70-80 % top-thread utilisation, the rest is the socket ping-pong (credit-1 waits), as in P10.
- `dev` Magma on openra is the one cell below 95 %: 63 % utilisation, wide spread (13.6 %) - dev's
Magma waits on the GPU there, which the feat build no longer does (see "Magma upload wait").
Backend parity therefore means "both CPU-bound at the same CPU cost". On the feat build the two
backends are within 2-11 % of each other on every arm, and the remaining gap is CPU debt (rd12
monolith: Magma backend + driver 3.93 vs Espryt 2.76 ms/frame, client half 1.91 vs 1.78).
## Layer split, rd12 monolith (render thread, ms/frame, simpleperf at 2 kHz, steady tail)
| layer | Espryt feat | Espryt dev | delta | Magma feat | Magma dev | delta |
|---|---|---|---|---|---|---|
| driver | 1.69 | 1.36 | +0.33 | 2.12 | 1.88 | +0.24 |
| backend | 1.07 | 0.62 | +0.45 | 1.81 | 2.56 | -0.75 |
| applier | 0.07 | 0 | +0.07 | 0.13 | 0 | +0.13 |
| client half | 1.78 | 0.39 | +1.39 | 1.91 | 0.70 | +1.21 |
| harness (trace parse) | 6.94 | 6.70 | +0.24 (same code) | 6.79 | 6.86 | -0.07 |
| total CPU | 11.56 | 9.07 | +2.49 | 12.77 | 12.01 | +0.76 |
About 6.7-6.9 ms of every rd12 frame is the replay harness, so the MobileGL part of an Espryt frame
is 4.6 ms on feat against 2.4 ms on dev, and of a Magma frame 6.0 against 5.1.
### What the Espryt monolith gap is made of (as % of an 11.2 ms frame)
- Validate point, 0.92 ms (none of it exists in pull): EmitGlobalConstants 0.17 (1.5 %),
EmitVertexElements 0.13 (1.1 %, includes the inline apply of the bind), Tracker::Update 0.12
(1.1 %), validate self 0.10 (0.8 %), CopyField 0.10 (0.9 %), EmitVertexBuffers 0.07 (0.6 %),
thread-local and atomic overhead about 0.07.
- Verb port, about 0.2 ms (1.8 %): EmitDrawRecord, ReadDrawBindings 0.06, MarkGpuWritesForDraw 0.05.
- Backend +0.45: SyncNeccessaryTextures 0.23 vs 0.10, SyncNeccessaryBuffers 0.16 vs 0.08,
AdoptUnchangedBufferSets 0.09, BindCurrentProgramWithResources 0.18 vs 0.11,
SyncCurrentProgramByHandle 0.09 vs 0.04.
- Driver +0.33 (2.9 %): inside glDrawElements itself, with the same GL call mix as dev. A
`simpleperf stat` over the whole run (render thread) has feat executing 16.6 % more instructions
than dev with 56 % more L1D and 23 % more L2D refills (IPC 3.73 vs 3.86): the record arm's larger
per-draw working set (records, applier state, slot tables, twins) evicting the driver's state is
the best reading, inferred from those counters, not measured inside the driver.
Each item is under 2 % of the frame except the driver term. Why the sum is structural on the record
arm: pull's backend reads frontend objects by pointer at draw time; push has to detect what moved
(tracker), serialise it (emitters, uniform spans), apply it (applier), then let the backend resolve
it again by handle through slot tables, with per-bound-object clean checks because the backend no
longer sees the frontend's dirty bits. Each step is tens of ns per draw on cache lines no other
step shares, times 1471 draws a frame.
Options left (none below the 2 % bar on its own, or each with a correctness risk the bar does not
pay for):
- a frontend mutation epoch (bumped in every `DECLARE_GL_FUNCTION_END` entry except draws,
glBindVertexArray and float glUniform*) so a validate with only VAO and uniform movement skips
the context-global shutters (at most 0.07 ms) and, after an audit of which residual fields
depend on the VAO/program bindings, the residual copies (0.15-0.25 ms together, 1.5-2 %);
- a per-(VAO, program) draw packet that keeps tracker, emitter, applier and backend state for a
draw on a few cache lines, keyed on lifetime ids - the structural answer to the cache-pressure
term, a redesign rather than a pass item;
- native TLS (minSdk / ANDROID_PLATFORM 29 instead of 26, the user's call): measured +1.7 %
Espryt, within noise on Magma (rd12 monolith, best of 3; whole APK, harness included).
## Pass findings worth keeping
- **Magma upload wait (553eb86d).** Every texture level upload flushed the recording and then
waited on the CPU for every submission to complete (`FlushWirePendingCommandsForTextureUpdate`,
added in P7 B4). The upload batch is submitted on the same queue after the flush and its first
barrier takes its source scope from the image's tracked layout, so the wait bought nothing; on
rd12 it was 1.8 ms/frame of fence stall (Minecraft updates its lightmap every frame). Removing it:
rd12 Magma monolith 70.4 -> 83.4, openra 367 -> 745. The preserve path keeps its wait. Proof: a
new between-draws upload scenario under the synchronization validator (red on a broken barrier),
sync validation of the texture/remint/copy scenarios (no new hazards), SSIM 16/16.
- **Sync validation needs submit-time validation on.** The Khronos layer (1.4.350) does not report
a hazard between two queue submissions unless `VK_KHRONOS_VALIDATION_SYNCVAL_SUBMIT_TIME_VALIDATION`
is set; the run-ahead and cache gates now set it (781a7446), and the run-ahead gate goes red on
the broken-barrier build where it used to stay green.
- **GPU-fault latch regression (549a34bb -> fixed in c24b3428).** Polling the reset status every
256 records let a record published right after the previous acknowledgement run unpolled on a
guilty context. The drain now polls before any record the last poll could not have seen
(e058a450 hands the apply loop's poll to the drain so a woken drain does not poll twice).
- **Record-publish batching, measured and dropped.** Publishing an unbarriered validate's state
records with the draw record behind them was within +-2 % on every arm and opens a deadlock class
(a staging-retirement wait on a record that is written but not published - it hung rd12 Espryt
split until the stage wait published owed records). Not worth the surface; the patch and its
stress scenario are not in the tree.
## W9 (cache capacities), closed with readings
`MOBILEGL_PIPE_STATS` on rd12 and openra: the render-state CSO cache mints 0 per frame in the steady
state, and Magma's pipeline memo hits 99.0 % on rd12 and 92.7 % on openra. No cache thrashes; the
P8 capacities stand.
## FCL (Minecraft in world, the representative app check)
FCL's own build of the library (plugin flavor, no POISON), swapped into FCL. Readings come from
MobileGL's fps log: mean of two launches × three steady windows. Clocks pinned. Runs:
`.trace-work/perf-p14/fcl-final/`.
| backend | dev (pull) | feat monolith | feat inproc | presenting-thread CPU, monolith feat / dev |
|---|---|---|---|---|
| Espryt | 228.6 | 198.3 (-13 %) | 241.5 (+6 %) | 2.43 / 1.85 ms/frame |
| Magma | 231.8 | 171.4 (-26 %) | 194.8 (-16 %) | 2.80 / 1.64 ms/frame |
FCL Magma does not match the trace: the presenting thread is busy only about 2.8 ms of a 5.8 ms
frame, so about 3 ms is a wait the trace does not have. That is the next phase's first item: an
off-CPU profile, the method that found the trace's upload stall.
+79
View File
@@ -0,0 +1,79 @@
# P14 perf pass: estimate from the code (written before the benchmark)
Date: 2026-10-07. Code: `feat/disaggregated` at `f8a9466b` ("feat") against `dev` at `08124c99` ("dev",
the pull monolith, the "before" baseline per DEBTS.md). Device for the measurement that follows:
Lenovo Y700 (TB321FU, Snapdragon 8 Gen 3, Adreno 750), rooted, clocks pinned.
**Disclosure.** Before writing this I ran one unpinned pipeline-validation smoke of the new harness
(Magma only, one rep, dev / feat monolith / feat inproc): openra 247 / 178 / 146 fps, rd12
127 / 46 / 33 fps. Those numbers are bigger than the logged 16 % (DEBTS ID-P13-2) and I had seen
them when I wrote the Magma rows below, so the Magma magnitudes are not blind. The mechanism
attribution (which costs, on which thread) is from the code and is what the profile will test;
the Espryt, spawn and tcp rows were written before any run of those arms.
## 1. What each variant does per call and per draw
| | dev pull monolith | feat monolith (record arm) | feat inproc | feat spawn (shm) | feat tcp (localhost) |
|---|---|---|---|---|---|
| GL entry | frontend state only | + `GLStreamScope` (one call + branch) | + process-wide recursive mutex per GL call | same as inproc | same as inproc |
| state calls | frontend state | + mutation notes (`MGP_NOTE_*`), dirty bits | same as monolith | | |
| per draw, client side | none | `MGPipeValidateForVerb`: tracker update (18 shutters), up to 14 emitters (CSO cache probe 64-entry, sampler views/states walks with hash-map slot lookups + XXH64), **`EmitContextValues` + XXH64 every draw**, residual fill of up to 63 fields incl. 3 SharedPtr copies (atomics); verb port: `ReadDrawBindings` hash lookups, persistent-map tracker, `MarkGpuWritesForDraw` | same, plus each record memcpy'd into the SEG_CMD ring and blobs (UBO images, sub-data) into SEG_STAGE | same as inproc | + `sendmsg` flush at every wait / present / fence poll |
| per draw, server side | backend reads `pGLContext` | applier (`MGPipeApply*`, same thread) + backend reads applier records; one extra copy of each changed default-uniform block into the applier | decode + bounds checks + apply + backend on the apply thread; staged bytes copied into server storage | same as inproc, in a second process | + io thread `recv` into mirrors |
| Magma draw setup | `TrySetupDrawFastPath` (4 snapshots keyed on program lifetime id; skips render-pass, descriptor, texture-sync and pipeline work when nothing moved) | `SetupWireDraw` every draw: `PrepareWireTextureResources` (texture sync per sampled binding), attachments resolved twice, `BuildWireVertexInput` rebuilt (Vector pushes + 4 XXH64), descriptor writes rebuilt + FNV over every info + 4-entry reuse memo, ~10 heap Vectors for the render-pass key, `UnorderedMap` intern of the pass-compatibility key, pipeline memo probe, unconditional `vkCmdBindPipeline` | same backend work on the apply thread | | |
| Espryt draw setup | memos keyed on frontend pointers / versions | the same memos re-keyed on applier serials (texture sync list, unit-bindings epoch, FBO lists, resolved-buffers memo, cached `glBindBufferBase/Range`); per-unit record lookups in `ResolveAndBindUnitTextures` | same on the apply thread | | |
| threads that burn CPU | 1 | 1 | 2 (client + apply), run-ahead armed | 2 processes | 2 processes + io threads |
| sync points | none | none | `kWaitReply` / `kWaitApplied` rows (fence waits, queries, readbacks, mipmaps, copy-tex, make-current), present credit 1 | same + socket doorbell when parked | same + credit-1 ping-pong over the socket (P10: 11.2 of 19.4 ms/frame parked) |
The trace APK's library is the split shape (`MOBILEGL_BUILD_DISAGGREGATED=ON`), which also turns on
`MOBILEGL_PIPE_POISON` (a stamp per filled field and a freshness check per backend accessor read).
FCL's library is the push shape without it. So the trace APK's monolith arm carries a little more
than the FCL library does.
## 2. Predictions
Workloads: openra (light: 128 frames, ~28 draws and ~199 records per frame) and rd12 (Minecraft
1.21.4 RD12: 251 frames, ~7,500 records per frame, several hundred draws per frame). Both are light
on the GPU, so every arm should be **CPU-bound**: GPU busy well under 60 %, with one thread at or
near 100 % for monolith and dev.
Per-draw costs I expect (Y700 big core at ~2 GHz):
- feat validate point + verb port over dev: **~2-4 us per draw** (an XXH64 of the context values, a
few hash-map lookups, up to 63 field copies, the persistent-map and GPU-write walks).
- Magma `SetupWireDraw` over dev's fast path: **~8-15 us per draw** (about 10 heap allocations, a
texture sync per sampled texture, two attachment resolves, the vertex-input rebuild and the
descriptor rebuild). This dwarfs everything else in this table.
| workload / backend | dev | feat monolith | feat inproc | feat shm | feat tcp |
|---|---|---|---|---|---|
| rd12 Espryt | baseline, 1 thread saturated | **10-25 % slower** (validate point + verb port on a ~7-9 ms frame) | **0-20 % faster than feat monolith**, roughly at dev: client and apply threads overlap, total CPU +20-40 % | about inproc (P5/P8 history: shm about inproc) | **3-6x slower than shm**: credit-1 ping-pong and wake-ups, not bandwidth |
| rd12 Magma | baseline; dev Magma at or ahead of dev Espryt (fast path + earlier walk-skip work) | **1.6-2.5x the dev frame time**: SetupWireDraw dominates | apply-thread bound (the same SetupWireDraw), so **at most 10-25 % better than feat monolith and still far behind dev** | about inproc | as Espryt tcp, worse in absolute terms |
| openra Espryt | baseline | within 5 % (few draws; the fixed per-frame cost dominates) | **5-15 % slower** than monolith (handshake, no work to overlap) | 10-15 % slower than monolith | 20-30 % slower than monolith |
| openra Magma | baseline | 15-35 % slower (per-draw gap times only ~28 draws, plus per-frame wire costs) | slower than monolith, as Espryt | | |
## 3. What this says about the three beliefs (to be checked)
1. **"inproc is faster than the pull monolith."** Partly. Only on draw-heavy frames, only where the
apply thread's backend is cheap enough for the client half to be the critical path (Espryt),
and the gain is overlap on a second core, not less work: total CPU goes up. Light loads show
no gain and pay the handshake. Magma inproc stays behind dev until its draw setup is fixed.
2. **"push and pull are on par."** Wrong on Magma (the lost fast path) and wrong by a smaller margin
on draw-heavy Espryt (the validate point and verb port are per-draw costs pull never paid).
Close on light loads.
3. **"the two backends are on par."** On dev, roughly yes (Magma ahead on Minecraft-like frames).
On feat, no: Magma pays its draw setup in full. Both backends are CPU-bound on these scenes,
so parity means cutting CPU work until both feed the GPU.
## 4. Fix plan the estimate implies, in order
1. A Magma wire draw fast path: snapshot the resolved draw (program lifetime id, applier serials
for bound views, samplers, FBO and vertex elements, render-pass key, pipeline, descriptor
signature) and skip the re-resolution while nothing moved. Keys must be monotonic lifetime ids
or applier generations, never handles or pointers. Also cheaper on a miss: no per-draw heap
Vectors, one attachment resolve, a cached compatibility id, `vkCmdBindPipeline` deduplicated.
Expected: most of the Magma gap on rd12.
2. W9: read `MOBILEGL_PIPE_STATS` on both workloads (CSO mints vs binds, pipeline memo misses,
descriptor reuse misses) and resize where a cache thrashes. Expected: small (<5 %) unless a
cache is thrashing.
3. Then whatever simpleperf shows on the client half (validate point, `EmitContextValues`,
residual fill) and on the split arms (ring encode, mutex, waits).
+100
View File
@@ -0,0 +1,100 @@
# P14 perf pass: handoff (closed 2026-10-07)
The pass is closed at `781a7446`, plus this docs commit. Results are in [BENCH.md](BENCH.md) and
[README.md](README.md). The next phase is the per-(VAO, program) draw-packet cache
([NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md)). Its first item is an off-CPU profile of
FCL Magma monolith (README "Headline numbers").
## Where things stand (rd12, pinned, FINISH=false, best of 3)
| arm | Espryt | Magma | dev Espryt / Magma |
|---|---|---|---|
| monolith | 91.1 | 83.8 | 115.2 / 87.0 |
| inproc | 111.9 | 110.0 | |
| shm | 87.8 | 95.6 | |
| tcp | 68.7 | 69.9 | |
FCL MC in world:
- dev: Espryt 228.6, Magma 231.8;
- feat monolith: Espryt 198.3, Magma 171.4;
- feat inproc: Espryt 241.5, Magma 194.8.
All arms are CPU-bound (GPU 19-39 %).
## Open items (DEBTS.md has the rows)
1. FCL Magma monolith waits about 3 ms/frame. The trace does not show it. Profile it off-CPU
first. Both close-out attempts caught no game frames, because FCL start-up ran past the
60 s warm-up.
2. The draw-packet cache (design sketch). Prototype the client half on Espryt monolith first.
3. The Espryt driver +0.33 ms is a cache-pressure inference. Check it with L1D refills
before and after (2).
4. Native TLS, minSdk 29 vs 26, is the user's call: +1.7 % Espryt, noise on Magma.
## Harness
Session scratchpad scripts:
- `build_trace.sh W:/perf .p14X`: trace APK, package `top.mobilegl.plugin.p14X.trace`.
- `sign_install.sh <apk> <name>`.
- `devjob.sh <log> <cmd>`: runs a device job under a mkdir lock. Always use it; two concurrent
device jobs contaminate each other, and killing a job leaves its children running.
- `incl.py` / `children.py` / `lines.py` / `layers2.py` / `drvcall.py` / `offcpu.py`:
simpleperf analyses. The tail window comes from the render thread's `retraceCall` samples.
- `profile_offcpu.sh`: `--trace-offcpu` variant of `profile_p14.sh`.
- `stat_p14.sh`: `simpleperf stat` counters.
- `fcl_offcpu.sh`: FCL profile plus restore.
Then:
```
cd tools/device_bench/disagg
FIXTURES=<main checkout>/tools/trace_replay/fixtures PREP_PKGS=top.mobilegl.plugin.p14X.trace bash matrix_p14.sh prep
ARMS="monolith@p14a monolith@p14b" BACKENDS="DirectGLES DirectVulkan" WLS=rd12 REPS=3 TAG=ab bash matrix_p14.sh run
python p14_report.py ../../../.trace-work/perf-p14/ab
ARMS="monolith@p14X inproc@p14X shm@p14X tcp@p14X" REPS=1 TAG=ssim bash matrix_p14.sh ssim # SSIM gate
ARM=monolith BACKEND=DirectGLES WL=rd12 TAG=prof PKG_FEAT=... SYMDIR=<unstripped libs> bash profile_p14.sh
```
The FCL library comes from `:app:assemblePluginRelease`. Unzip `lib/arm64-v8a/libMobileGL.so` from
`app/build/outputs/apk/plugin/release/`; the unstripped copy is under
`intermediates/merged_native_libs/pluginRelease`. Run `fcl_p14.sh run`, then `restore`
(md5-checked).
Gradle's `packageTraceRelease` sometimes fails on the first try with
`IncrementalSplitterRunnable`; a rerun succeeds.
Host:
- Unit tests: WSL archlinux `~/mgl-perf-bld` (clang, split + inproc).
- ServerSpawnTest, FuzzArm2 and PeerLatchTest fail there with `Refuse{BuildFingerprint}`;
that is environmental.
- The google-benchmark targets don't build with clang 22; also environmental.
- CI-shaped integration tree: `~/mgl-perf-ci31`, configured with CMake 3.31.10 from the
`~/mgl-perf-tools` venv.
- The system CMake 4.x joins `gtest_discover_tests` ENVIRONMENT lists into one variable, and
every entry then skips. Use 3.31.10.
- lavapipe ICD; spawn/tcp lanes are environmental there.
- To run the syncval gates: `scripts/ci/magma_runahead_checks.py validation --build-dir ~/mgl-perf-ci31 --execute`,
with `GIT_DIR=.../.git/modules/MobileGL/worktrees/perf` set for the cache script.
- `~/mgl-perf-ci` is a CMake 4 tree. Use it only through the scratchpad's `runentry.sh`, which
splits the joined environment.
## Device state
- HA27Q3LQ CPU/GPU clocks are still PINNED: the next phase wants them pinned.
`bash pin_clocks.sh restore` undoes it; ask before unpinning.
- FCL is restored: lib md5 matches `/data/local/tmp/p14-fcl-orig.so`, config.json matches its
saved copy, and the `debug.mobilegl.*` properties are deleted.
- Trace APKs `top.mobilegl.plugin.p14*.trace` are installed; `p14fin` is the close-out build.
## CI
Dispatched runs (workflow_dispatch, not cancelled by later pushes):
- 86ab2835: Test 37702990844, APK 37702995519.
- 781a7446: Test 37707221661.
- The close-out docs head: see the final report.
Every `runs-on` in test.yml is GitHub-hosted `ubuntu-latest`, so long queueing is GitHub capacity /
max-parallel, not a runner of ours. `PeerLatch.SiteMap` (`scripts/ci/ph_latch_sites.py`) greps
ServerLoop.cpp's text: keep its patterns in step when DrainRing's shape changes (it caught the
`DrainRing(polledHead)` signature at a0f4aa83). Locally, filter environmental failures by exact
name - a `grep -v PeerLatch` hid it.
@@ -0,0 +1,150 @@
# Next phase: a per-(VAO, program) draw-packet cache (design sketch)
This is a starting point for the next phase, not an implementation. The numbers are from the P14
close-out (`BENCH.md`, rd12 monolith, Y700, pinned, render thread, ms/frame).
## 1. The problem
On the record arm, a draw crosses five layers, and each layer keeps its own per-object state:
| layer | Espryt feat | Espryt dev | delta | what it touches per draw |
|---|---|---|---|---|
| client half | 1.78 | 0.39 | +1.39 | tracker shutters (VAO, program, context gens), VaoMemo, emitter latches, slot allocator, residual block, verb port |
| applier | 0.07 | 0 | +0.07 | vertex-elements record, applier unit windows, global-constants block |
| backend | 1.07 | 0.62 | +0.45 | VAO twin, texture/sampler records + twins (clean checks), program twin, UBO ring |
| driver | 1.69 | 1.36 | +0.33 | the same GL calls as dev; slower, consistent with cache pressure (+56 % L1D refills, +16.6 % instructions, whole run) |
Minecraft rd12 issues 1471 draws a frame, about 3.3 GL calls per draw. Between two consecutive
draws, usually only the bound VAO and the program's default-uniform block change. Each layer still
re-derives "what moved" from its own structures, and each of those structures is a different cold
cache line per VAO: the frontend VAO, the VaoMemo entry, the emitter latch, the applier record, and
the backend twin. Pull touches one of them, the frontend VAO.
None of the individual items is above 2 % of the frame. The sum is the gap: on Espryt monolith,
91 vs 115 fps.
## 2. What one draw packet holds
There is one packet per (VAO lifetime id, program lifetime id) pair that has drawn. It lives in a
client-side table, and in the server's record space on the split arms. Its fields:
- **Identity:** VAO lifetime id + config version, program lifetime id + link version, and the
context serial.
- **Client side:**
- the vertex-elements handle and its published config version;
- the buffer-set and index handles and their hash, which the VaoMemo holds today;
- the shader CSO handle;
- the residual-field values this pair contributed (the bound-VAO and program pins);
- the context-global epoch it was validated at (see section 3).
- **Applier side:** a pointer, or slot index plus generation, to the vertex-elements record and the
stored buffer/index window, so that bind_vertex_elements applies by reference instead of copying.
- **Backend side (Espryt):**
- the VAO twin;
- the resolved draw buffers;
- the program twin;
- the sampled-texture list with each entry's synced serials;
- the synced attachment list;
- the UBO block size.
- **Backend side (Magma):** the wire vertex-input state, the pipeline handle, the descriptor-set
layout, and the pass-compatibility id.
- **A "packet valid at" stamp per layer:** the serials and epochs each layer checks to trust its
slice of the packet.
The goal is that a draw whose packet is valid at every layer touches the packet and the per-draw
inputs, and nothing else. The per-draw inputs are the uniform span, the draw arguments, and the UBO
ring slot.
## 3. Keys and invalidation sources
**Keys.** Lifetime ids only, never GL names or pointers. This is the ABA rule, already broken and
fixed three times: sampler, texture, program. The packet table key is (VAO lifetime id, program
lifetime id, context serial).
Invalidation sources, by layer:
- **Frontend context-global state.** This is everything the tracker reads that is neither VAO- nor
program-derived: render/pipeline state versions, texture content/params/bind generations,
framebuffer attachment and bind versions, buffer bind-point generations, TFB, pixel pack, patch,
vertex attribute defaults, and the program pipeline.
- This needs a context mutation epoch, bumped in every GL/EGL entry except the
whitelist: draws, glBindVertexArray, and float glUniform*.
- The bump fits in the `DECLARE_GL_FUNCTION_END` macros, with the whitelist as a constexpr name
test, plus the manual entries (glFinish/glFlush and the hand-written extension wrappers) and
the EGL entries.
- **Audit item:** every MOBILEGL_GL_API definition outside the macros.
- **VAO.** The config version (attribute pointers and enables), the index slot version, and buffer
respecification through `AnyVaoAttributeGeneration`. These are already lifetime- and
version-keyed.
- **Program.** Link version, backend state version (sampler units through glUniform1i), block
binding version, and image unit version. Float uniforms move only the UBO content version,
which is a per-draw input and not a packet invalidation.
- **Applier.** The record `ContentSerial`, `ContextSerial`, `MGPipeSlots().ResetEpoch()`, and the
vertex-buffers and index-buffer serials. These are already monotonic.
- **Backend.**
- Texture: record `Serial`, `ParamsSerial`, and `PendingUploads`; twin force-resync flags;
sampler CSO serial; shared-image state.
- Context generation.
- Magma only: texture image epoch, resource erase epoch, the image layout leaving GENERAL, and the
store destroy epoch. The P14 Magma memos already check these.
- The backend needs one "texture state moved" epoch, so that the per-bound-texture clean check
can be skipped when nothing in the texture family moved. This is the item the content-version
lesson warns about: glGenerateMipmap grew mips without a version bump. Every backend-side mutation
of a twin's synced state must bump the epoch too.
## 4. Split arms and the verify comparator
- **The packet is per role.** The client half of the packet (handles, latches, residual pins)
lives in the client. The applier and backend halves live wherever the applier runs: the same
thread on monolith, the apply thread on inproc, the server process on spawn and tcp.
- **Nothing in the packet crosses the wire.** The wire stays the record stream. A server-side
packet is keyed on the vertex-elements handle and the shader-CSO handle (slot + generation)
plus the applier context serial, because the server never sees frontend lifetime ids.
- **Run-ahead.** Client-side validity must not depend on server state. The client's half is
checked against client epochs only, exactly like today's latches.
- **The verify comparator (`MOBILEGL_PIPE_VERIFY`) compares the residual block against a live
snapshot at the verb.**
- A packet hit that skips residual copies would make the comparator diverge, unless the packet's
pins are what it compares.
- Proposal: under verify, take the slow path always and assert that the packet would have hit
with the same values. That gives a free cross-check of the epoch's coverage on every verify
lane.
- **POISON stamps (trace build only).** The stamp loop stays, or the packet records the stamp set
and applies it with one bulk write.
## 5. Risks
- **A missed epoch bump renders stale state.** This is the dominant risk. Mitigations:
- the verify-lane cross-check above;
- a negative control that removes the bump from one entry and must go red;
- SSIM 16/16 on every change;
- the existing retrace CI.
- **ABA on packet keys.** Lifetime ids only, plus context serial and slot reset epoch.
- **Memory.** One packet per drawn (VAO, program) pair. Minecraft has thousands of chunk VAOs ×
a few programs, so a bounded table with LRU is about 100-200 B × 10k ≈ 2 MB. The table must
not itself become the next cache-miss source: entries go in a flat array with an index stored
on the frontend VAO.
- **Backend-side invalidation completeness** (section 3, backend): the same class as the reverted
e8e1521.
- **Code size and review cost.** It touches every layer. It should land behind a runtime knob,
default off until the CI and SSIM gates and an FCL soak are green.
## 6. Rough gain estimate (rd12, ms/frame, Espryt monolith)
| layer | removable on a packet hit | why not all of the delta |
|---|---|---|
| client half (+1.39) | 0.6-0.9 | the uniform span, bind record and draw record still have to be produced |
| applier (+0.07) | 0.03 | bind by reference |
| backend (+0.45) | 0.25-0.35 | the UBO ring slot and the GL calls stay |
| driver (+0.33) | 0.1-0.3, if the cache-pressure reading is right | unproven; measure L1D refills before and after |
Total: about 1.0-1.6 ms of the 2.5 ms gap. That is roughly 91 -> 100-106 fps on Espryt monolith.
Magma's share is similar on the client half (+1.21). Its backend is already below dev, and its
driver term is +0.24.
Inproc gains the client-half part on the client thread: that is the bound thread on Espryt rd12,
so 112 -> above dev's 115. Magma inproc is bound on the apply thread, which gains only the backend
and applier part.
Prototype first, before the full design: the client-half packet (tracker + emitter + residual pins,
with the context epoch) on monolith Espryt only. Measure it against the 0.6-0.9 ms estimate, then
decide on the backend half.
+111
View File
@@ -0,0 +1,111 @@
# P14: the post-P13 performance pass
P14 measured the record (push) arm against the `dev` pull monolith and cut its CPU cost. It ran on
`feat/disaggregated` from `f8a9466b` (start) to `781a7446` (close-out), on a Lenovo Y700 (Adreno
750) with pinned clocks.
- [ESTIMATE.md](ESTIMATE.md): the code-based estimate, written before the first benchmark.
- [BENCH.md](BENCH.md): the final trace matrix, the layer split, CPU vs GPU binding, and what the
remaining gap is made of.
- [HANDOFF.md](HANDOFF.md): harness commands, device state, open items.
- [NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md): the design sketch for the next phase.
## Headline numbers
### Minecraft in FCL (the representative app check)
Setup:
- FCL Minecraft, in world.
- The library is swapped into FCL, which runs it as a plugin build without `MOBILEGL_PIPE_POISON`.
- Readings come from MobileGL's own fps log.
- Two launches × three steady windows of about 2 s each. Dev Espryt is three windows: its second
launch died at start-up, twice.
- Clocks are pinned. Runs: `.trace-work/perf-p14/fcl-final/`.
| backend | dev (pull) | feat monolith | feat inproc | feat monolith, presenting-thread CPU |
|---|---|---|---|---|
| Espryt | 228.6 | 198.3 (−13 %) | 241.5 (+6 %) | 2.43 ms/frame (dev 1.85) |
| Magma | 231.8 | 171.4 (−26 %) | 194.8 (−16 %) | 2.80 ms/frame (dev 1.64) |
Earlier in the pass, at `94705adc`, the same measurement read:
- Espryt: monolith about 192, inproc about 233;
- Magma: monolith about 169, inproc about 179;
- dev: 209-245.
FCL Magma monolith did not follow the trace's improvement. Its presenting thread is busy only
about 2.8 of the roughly 5.8 ms frame, so the rest is a wait. That is the first item for the next
phase: an off-CPU profile of FCL Magma monolith, the same method that found the upload stall in
the trace. Two attempts at it during close-out caught no game frames (FCL start-up).
### Trace replay, rd12 (Minecraft 1.21.4 RD12, 1471 draws/frame), best of 3, fps
| backend | dev (pull) | monolith | inproc | shm | tcp |
|---|---|---|---|---|---|
| Espryt | 115.2 | 91.1 (start 69.8) | 111.9 (71.1) | 87.8 (62.8) | 68.7 (50.0) |
| Magma | 87.0 | 83.8 (start 45.4) | 110.0 (63.2) | 95.6 (58.6) | 69.9 (45.6) |
openra is in [BENCH.md](BENCH.md). On openra, Magma monolith now runs 2.5x faster than dev
(747 vs 293), and inproc 3.6x (1052 vs 293).
Every arm is CPU-bound: GPU busy is 19-39 %, and the top thread sits at 94-99 % of a core
(70-80 % on tcp).
## What landed
The commits run from `f8a9466b` to `781a7446`. Per-commit numbers are in the commit subjects.
- **Magma wire draw:**
- no per-draw SPIR-V rehash or allocation;
- unit sampler memo; unchanged pipeline, vertex and index binds skipped;
- direct-mapped texture and buffer fronts;
- descriptor-set rebind with fresh offsets;
- pass-continuation memo;
- vertex-input layout kept on its record;
- **texture level uploads no longer wait for the GPU to drain** (`553eb86d`, the largest
single gain of the pass: rd12 70 -> 83, openra 367 -> 745);
- slices stamped from the owner's submit counter.
- **Espryt:**
- VAO twin re-points only the moved buffers;
- clean attachments skip resync;
- unchanged buffer sets are adopted;
- twin lookups without SharedPtr copies;
- one thread-local block;
- one UBO-ring lookup per draw.
- **MG_Pipe client half:**
- lifetime-id slot fronts;
- per-VAO memo;
- fill pins;
- kept VAO bindings on the server;
- uniform spans;
- context-values gate;
- inline process applier;
- precomputed residual walk, with split copy and stamp loops;
- memo-answered draw bindings.
- **MG_Remote:** fault-verdict polling restructured.
- It first went from every record to every 256 records (549a34bb).
- The drain now polls before any record the last poll could not have seen (`c24b3428`). This
fixes the regression where a guilty context ran one more record.
- The apply loop hands its poll to the drain (`e058a450`).
- **Magma hang watch:** a marker slot is reused only once its submission's fence completes
(`ec973fba`).
- **Tests and CI:**
- TextureUploadBetweenDrawsScenario on every arm, plus inproc sync validation (`67f4f311`,
`86ab2835`);
- submit-time synchronization validation in the run-ahead and cache gates (`781a7446`);
- ServerLoopEglLatchTest now holds the race window open.
- **Tools:**
- pinned-clock matrix with a per-thread CPU and GPU-busy sampler;
- layer-split and off-CPU profile helpers;
- FCL harness;
- opt-in fps log and transport override properties.
## Measured and not landed
- **Record-publish batching:** publishing a validate's state records with the draw record. It came
in within ±2 % on every arm, and it opens a deadlock class (a staging-retirement wait on a
written but unpublished record).
- **Native TLS (minSdk / ANDROID_PLATFORM 29):** +1.7 % on Espryt, noise on Magma (rd12 monolith).
This is the user's call.
- **Tracker / residual mutation-epoch fast path:** estimated at most 0.07 ms from the tracker alone,
0.15-0.25 ms with the residual copies. Each step is below the 2 % bar, and a missed epoch bump
renders stale. It is folded into the draw-packet design.