mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-07 10:03:23 +09:00
[Docs] (P14): close-out bench, README, next-phase draw-packet sketch, DEBTS rows and handoff (rd12 monolith Espryt 69.8 -> 91.1, Magma 45.4 -> 83.8 fps).
This commit is contained in:
@@ -0,0 +1,151 @@
|
||||
# P14 perf pass: final bench
|
||||
|
||||
**Headline:** FCL Minecraft gives Espryt monolith 198 vs dev 229 and inproc 242; Magma
|
||||
monolith 171 vs dev 232 and inproc 195. On the rd12 trace, Espryt monolith is 91 vs 115 and
|
||||
Magma monolith 84 vs 87; inproc is 112 (Espryt) / 110 (Magma). Details below.
|
||||
|
||||
Head `781a7446` on `feat/disaggregated` (trace APK `p14fin`) against `dev` at `08124c99` (the pull
|
||||
monolith, the baseline). Lenovo Y700 (Snapdragon 8 Gen 3, Adreno 750), rooted, CPU and GPU clocks
|
||||
pinned (`pin_clocks.sh`), no per-frame glFinish (`FINISH=false`), best of 3, steady frames only.
|
||||
Raw runs: `.trace-work/perf-p14/final-p14/` (`matrix_p14.sh run`, `p14_report.py`).
|
||||
|
||||
## Final matrix (fps, best of 3)
|
||||
|
||||
| workload | backend | dev (pull) | monolith | inproc | shm | tcp |
|
||||
|---|---|---|---|---|---|---|
|
||||
| rd12 | Espryt | 115.2 | 91.1 | 111.9 | 87.8 | 68.7 |
|
||||
| rd12 | Magma | 87.0 | 83.8 | 110.0 | 95.6 | 69.9 |
|
||||
| openra | Espryt | 889.2 | 832.5 | 1073.9 | 1063.1 | 602.0 |
|
||||
| openra | Magma | 293.1 | 747.4 | 1051.6 | 976.0 | 544.5 |
|
||||
|
||||
Start of the pass (same harness, `nofinish` tag, `f8a9466b`):
|
||||
|
||||
| workload | backend | monolith | inproc | shm | tcp |
|
||||
|---|---|---|---|---|---|
|
||||
| rd12 | Espryt | 69.8 | 71.1 | 62.8 | 50.0 |
|
||||
| rd12 | Magma | 45.4 | 63.2 | 58.6 | 45.6 |
|
||||
| openra | Espryt | 812.2 | 1067.5 | 953.8 | 529.2 |
|
||||
| openra | Magma | 356.3 | 446.0 | 394.4 | 412.1 |
|
||||
|
||||
So over the pass, rd12 went from 69.8 to 91.1 fps on Espryt monolith (+31%) and from 45.4 to 83.8 on
|
||||
Magma monolith (+85%). Split arms: inproc +57% / +74%, shm +40% / +63%, tcp +37% / +53%. On openra,
|
||||
Magma monolith went 356 -> 747 and inproc 446 -> 1052.
|
||||
|
||||
## Against the three goals
|
||||
|
||||
| goal | rd12 | openra |
|
||||
|---|---|---|
|
||||
| push = pull (monolith vs dev) | Espryt 91.1 vs 115.2 (-21%); Magma 83.8 vs 87.0 (-4%) | Espryt 832 vs 889 (-6%); Magma 747 vs 293 (2.5x dev) |
|
||||
| inproc beats pull | Espryt 111.9 vs 115.2 (-3%); Magma 110.0 vs 87.0 (+26%) | Espryt 1074 vs 889 (+21%); Magma 1052 vs 293 (3.6x) |
|
||||
| backend parity (feat) | monolith 91.1 vs 83.8; inproc 111.9 vs 110.0 | monolith 832 vs 747; inproc 1074 vs 1052 |
|
||||
|
||||
## CPU- or GPU-bound
|
||||
|
||||
Every arm on both workloads is CPU-bound: GPU busy is 19-39 % everywhere (`GPU busy` column of the
|
||||
report), and the arm's top thread runs at 94-99 % of a core on the monolith and inproc arms
|
||||
(97-99 % for the render thread, 94-95 % for `mgl-srv-apply` where it is the top thread).
|
||||
- monolith: one thread, `MobileGLTraceRe`, at 99 % on all four cells.
|
||||
- inproc / shm: two threads at about the same load (client 8.5-8.7, apply 8.5-8.6 ms/frame on rd12),
|
||||
so the split arms are bound on whichever of the two is ahead; on rd12 Espryt that is the client
|
||||
(harness parse + client half), on rd12 Magma the apply thread.
|
||||
- tcp: 70-80 % top-thread utilisation, the rest is the socket ping-pong (credit-1 waits), as in P10.
|
||||
- `dev` Magma on openra is the one cell below 95 %: 63 % utilisation, wide spread (13.6 %) - dev's
|
||||
Magma waits on the GPU there, which the feat build no longer does (see "Magma upload wait").
|
||||
|
||||
Backend parity therefore means "both CPU-bound at the same CPU cost". On the feat build the two
|
||||
backends are within 2-11 % of each other on every arm, and the remaining gap is CPU debt (rd12
|
||||
monolith: Magma backend + driver 3.93 vs Espryt 2.76 ms/frame, client half 1.91 vs 1.78).
|
||||
|
||||
## Layer split, rd12 monolith (render thread, ms/frame, simpleperf at 2 kHz, steady tail)
|
||||
|
||||
| layer | Espryt feat | Espryt dev | delta | Magma feat | Magma dev | delta |
|
||||
|---|---|---|---|---|---|---|
|
||||
| driver | 1.69 | 1.36 | +0.33 | 2.12 | 1.88 | +0.24 |
|
||||
| backend | 1.07 | 0.62 | +0.45 | 1.81 | 2.56 | -0.75 |
|
||||
| applier | 0.07 | 0 | +0.07 | 0.13 | 0 | +0.13 |
|
||||
| client half | 1.78 | 0.39 | +1.39 | 1.91 | 0.70 | +1.21 |
|
||||
| harness (trace parse) | 6.94 | 6.70 | +0.24 (same code) | 6.79 | 6.86 | -0.07 |
|
||||
| total CPU | 11.56 | 9.07 | +2.49 | 12.77 | 12.01 | +0.76 |
|
||||
|
||||
About 6.7-6.9 ms of every rd12 frame is the replay harness, so the MobileGL part of an Espryt frame
|
||||
is 4.6 ms on feat against 2.4 ms on dev, and of a Magma frame 6.0 against 5.1.
|
||||
|
||||
### What the Espryt monolith gap is made of (as % of an 11.2 ms frame)
|
||||
|
||||
- Validate point, 0.92 ms (none of it exists in pull): EmitGlobalConstants 0.17 (1.5 %),
|
||||
EmitVertexElements 0.13 (1.1 %, includes the inline apply of the bind), Tracker::Update 0.12
|
||||
(1.1 %), validate self 0.10 (0.8 %), CopyField 0.10 (0.9 %), EmitVertexBuffers 0.07 (0.6 %),
|
||||
thread-local and atomic overhead about 0.07.
|
||||
- Verb port, about 0.2 ms (1.8 %): EmitDrawRecord, ReadDrawBindings 0.06, MarkGpuWritesForDraw 0.05.
|
||||
- Backend +0.45: SyncNeccessaryTextures 0.23 vs 0.10, SyncNeccessaryBuffers 0.16 vs 0.08,
|
||||
AdoptUnchangedBufferSets 0.09, BindCurrentProgramWithResources 0.18 vs 0.11,
|
||||
SyncCurrentProgramByHandle 0.09 vs 0.04.
|
||||
- Driver +0.33 (2.9 %): inside glDrawElements itself, with the same GL call mix as dev. A
|
||||
`simpleperf stat` over the whole run (render thread) has feat executing 16.6 % more instructions
|
||||
than dev with 56 % more L1D and 23 % more L2D refills (IPC 3.73 vs 3.86): the record arm's larger
|
||||
per-draw working set (records, applier state, slot tables, twins) evicting the driver's state is
|
||||
the best reading, inferred from those counters, not measured inside the driver.
|
||||
|
||||
Each item is under 2 % of the frame except the driver term. Why the sum is structural on the record
|
||||
arm: pull's backend reads frontend objects by pointer at draw time; push has to detect what moved
|
||||
(tracker), serialise it (emitters, uniform spans), apply it (applier), then let the backend resolve
|
||||
it again by handle through slot tables, with per-bound-object clean checks because the backend no
|
||||
longer sees the frontend's dirty bits. Each step is tens of ns per draw on cache lines no other
|
||||
step shares, times 1471 draws a frame.
|
||||
|
||||
Options left (none below the 2 % bar on its own, or each with a correctness risk the bar does not
|
||||
pay for):
|
||||
- a frontend mutation epoch (bumped in every `DECLARE_GL_FUNCTION_END` entry except draws,
|
||||
glBindVertexArray and float glUniform*) so a validate with only VAO and uniform movement skips
|
||||
the context-global shutters (at most 0.07 ms) and, after an audit of which residual fields
|
||||
depend on the VAO/program bindings, the residual copies (0.15-0.25 ms together, 1.5-2 %);
|
||||
- a per-(VAO, program) draw packet that keeps tracker, emitter, applier and backend state for a
|
||||
draw on a few cache lines, keyed on lifetime ids - the structural answer to the cache-pressure
|
||||
term, a redesign rather than a pass item;
|
||||
- native TLS (minSdk / ANDROID_PLATFORM 29 instead of 26, the user's call): measured +1.7 %
|
||||
Espryt, within noise on Magma (rd12 monolith, best of 3; whole APK, harness included).
|
||||
|
||||
## Pass findings worth keeping
|
||||
|
||||
- **Magma upload wait (553eb86d).** Every texture level upload flushed the recording and then
|
||||
waited on the CPU for every submission to complete (`FlushWirePendingCommandsForTextureUpdate`,
|
||||
added in P7 B4). The upload batch is submitted on the same queue after the flush and its first
|
||||
barrier takes its source scope from the image's tracked layout, so the wait bought nothing; on
|
||||
rd12 it was 1.8 ms/frame of fence stall (Minecraft updates its lightmap every frame). Removing it:
|
||||
rd12 Magma monolith 70.4 -> 83.4, openra 367 -> 745. The preserve path keeps its wait. Proof: a
|
||||
new between-draws upload scenario under the synchronization validator (red on a broken barrier),
|
||||
sync validation of the texture/remint/copy scenarios (no new hazards), SSIM 16/16.
|
||||
- **Sync validation needs submit-time validation on.** The Khronos layer (1.4.350) does not report
|
||||
a hazard between two queue submissions unless `VK_KHRONOS_VALIDATION_SYNCVAL_SUBMIT_TIME_VALIDATION`
|
||||
is set; the run-ahead and cache gates now set it (781a7446), and the run-ahead gate goes red on
|
||||
the broken-barrier build where it used to stay green.
|
||||
- **GPU-fault latch regression (549a34bb -> fixed in c24b3428).** Polling the reset status every
|
||||
256 records let a record published right after the previous acknowledgement run unpolled on a
|
||||
guilty context. The drain now polls before any record the last poll could not have seen
|
||||
(e058a450 hands the apply loop's poll to the drain so a woken drain does not poll twice).
|
||||
- **Record-publish batching, measured and dropped.** Publishing an unbarriered validate's state
|
||||
records with the draw record behind them was within +-2 % on every arm and opens a deadlock class
|
||||
(a staging-retirement wait on a record that is written but not published - it hung rd12 Espryt
|
||||
split until the stage wait published owed records). Not worth the surface; the patch and its
|
||||
stress scenario are not in the tree.
|
||||
|
||||
## W9 (cache capacities), closed with readings
|
||||
|
||||
`MOBILEGL_PIPE_STATS` on rd12 and openra: the render-state CSO cache mints 0 per frame in the steady
|
||||
state, and Magma's pipeline memo hits 99.0 % on rd12 and 92.7 % on openra. No cache thrashes; the
|
||||
P8 capacities stand.
|
||||
|
||||
## FCL (Minecraft in world, the representative app check)
|
||||
|
||||
FCL's own build of the library (plugin flavor, no POISON), swapped into FCL. Readings come from
|
||||
MobileGL's fps log: mean of two launches × three steady windows. Clocks pinned. Runs:
|
||||
`.trace-work/perf-p14/fcl-final/`.
|
||||
|
||||
| backend | dev (pull) | feat monolith | feat inproc | presenting-thread CPU, monolith feat / dev |
|
||||
|---|---|---|---|---|
|
||||
| Espryt | 228.6 | 198.3 (-13 %) | 241.5 (+6 %) | 2.43 / 1.85 ms/frame |
|
||||
| Magma | 231.8 | 171.4 (-26 %) | 194.8 (-16 %) | 2.80 / 1.64 ms/frame |
|
||||
|
||||
FCL Magma does not match the trace: the presenting thread is busy only about 2.8 ms of a 5.8 ms
|
||||
frame, so about 3 ms is a wait the trace does not have. That is the next phase's first item: an
|
||||
off-CPU profile, the method that found the trace's upload stall.
|
||||
@@ -1,74 +1,50 @@
|
||||
# P14 perf pass: handoff (checkpoint 2026-10-07)
|
||||
# P14 perf pass: handoff (closed 2026-10-07)
|
||||
|
||||
Worktree `.claude/worktrees/perf` (branch `perf`, pushed to `origin/feat/disaggregated`). Estimate:
|
||||
[ESTIMATE.md](ESTIMATE.md). BENCH.md / README.md are not written yet (see "Next").
|
||||
The pass is closed at `781a7446`, plus this docs commit. Results are in [BENCH.md](BENCH.md) and
|
||||
[README.md](README.md). The next phase is the per-(VAO, program) draw-packet cache
|
||||
([NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md)). Its first item is an off-CPU profile of
|
||||
FCL Magma monolith (README "Headline numbers").
|
||||
|
||||
## Head and running totals
|
||||
## Where things stand (rd12, pinned, FINISH=false, best of 3)
|
||||
|
||||
Head: `94705adc` (plus this handoff commit). rd12, Lenovo Y700 (Adreno 750), clocks pinned, no
|
||||
per-frame glFinish (`FINISH=false`, the primary mode; finish=true serialises the split arms on one
|
||||
FenceWait round trip per frame), best of 3:
|
||||
|
||||
| arm | Espryt start -> now | Magma start -> now | dev (pull) Espryt / Magma |
|
||||
| arm | Espryt | Magma | dev Espryt / Magma |
|
||||
|---|---|---|---|
|
||||
| monolith | 69.8 -> 88.6 | 45.4 -> 69.5 | 115.1 / 86.1 |
|
||||
| inproc | 71.1 -> ~107 | 63.2 -> ~99 | |
|
||||
| shm | 62.8 -> ~82 | 58.6 -> ~85 | |
|
||||
| tcp | 50.0 -> ~64 | 45.6 -> ~61 | |
|
||||
| monolith | 91.1 | 83.8 | 115.2 / 87.0 |
|
||||
| inproc | 111.9 | 110.0 | |
|
||||
| shm | 87.8 | 95.6 | |
|
||||
| tcp | 68.7 | 69.9 | |
|
||||
|
||||
(split-arm "now" values are from the p14p A/B, before the last two monolith commits; re-measure.)
|
||||
GPU busy 22-33 % everywhere: every arm is CPU-bound on one thread.
|
||||
FCL MC in world:
|
||||
- dev: Espryt 228.6, Magma 231.8;
|
||||
- feat monolith: Espryt 198.3, Magma 171.4;
|
||||
- feat inproc: Espryt 241.5, Magma 194.8.
|
||||
|
||||
## Latest layer split (rd12 monolith, render thread ms/frame, simpleperf, head `94705adc` = p14r)
|
||||
All arms are CPU-bound (GPU 19-39 %).
|
||||
|
||||
| layer | Espryt feat | Espryt dev | Magma feat | Magma dev |
|
||||
|---|---|---|---|---|
|
||||
| driver | 1.56 | 1.40 | 2.15 | 1.98 |
|
||||
| backend | 1.21 | 0.59 | 1.95 | 2.12 |
|
||||
| applier | 0.07 | 0 | 0.13 | 0 |
|
||||
| client half | 1.96 | 0.36 | 2.01 | 0.69 |
|
||||
| harness | 6.46 | 6.27 | 6.50 | 6.25 |
|
||||
| total CPU | 11.27 | 8.62 | 12.77 | 11.05 |
|
||||
## Open items (DEBTS.md has the rows)
|
||||
|
||||
Client half top: validate self 0.22-0.26, verb port (MG_Record) 0.15-0.18, Tracker::Update 0.14-0.17,
|
||||
CopyField 0.07-0.09, VAO memo probe 0.07-0.09. Espryt backend top: PrepareForDraw 0.10,
|
||||
AdoptUnchangedBufferSets 0.10, PerNativeContext<...>::Current 0.084, SyncTextureToBackendByHandle
|
||||
0.066, IsBufferDrawCleanByHandle 0.058. Magma backend top: SetupWireDraw self 0.35,
|
||||
BindDescriptorSetDeduped 0.23 (dynamic UBO offset moves every chunk draw), AcquireWireSlice 0.19.
|
||||
|
||||
Per-draw counts (instrumented build, `p14count2.patch` in the scratchpad, not committed): rd12 1471
|
||||
draws/frame, 3.3 GL calls/draw; after the protocol change 66.8 B/validate (bind 16 B + uniform span
|
||||
~47 B), VertexBuffers/IndexBuffer 100 % no-op.
|
||||
|
||||
## Commits this pass (newest first)
|
||||
|
||||
94705adc inline applier / fill plan / word-wise spans / memo-answered draw bindings (+6 % Espryt, +3.7 % Magma);
|
||||
81bdfb1f Espryt twin lookups w/o SharedPtr copy + one TLS block (+3.8 %);
|
||||
032dcd24 Magma vertex-input layout kept on its record + slot-indexed buffer front;
|
||||
c1497bb8 VAO-kept bindings, uniform spans, context-values gate (split arms +5-10 %; stream span merge fixes a tcp regression);
|
||||
0158a115 Magma pass-continuation memo; 8883e496 fill pins + VAO hash memo; 80dc9d45 VAO memo;
|
||||
5788a602 Magma descriptor rebind; afcf8d96 Espryt AdoptUnchangedBufferSets; e6281b26, 4ef9705e Magma
|
||||
lookups/sampler memo; b556ec31, a9108e1a Espryt; 549a34bb server fault latch; 8346d809 slot allocator
|
||||
front; 267211e8 Magma hash memo + scratch.
|
||||
|
||||
## In flight / next
|
||||
|
||||
1. Espryt backend lookups: `PerNativeContext<T>::Current` (TLS cache per T + non-inline serial/epoch
|
||||
calls), `IsBufferDrawCleanByHandle`, `AdoptUnchangedBufferSets` (compares the window every draw
|
||||
because the bind re-apply bumps VertexBuffersSerial), `SyncTextureToBackendByHandle`.
|
||||
2. Tracker::Update: 18 shutters evaluated per draw, 4 fire. Needs a cheap "nothing but the VAO and
|
||||
the uniform moved" signal; no central frontend mutation epoch exists yet.
|
||||
3. Residual fill: now a precomputed walk, but still ~11 CopyField per draw.
|
||||
4. Magma: SetupWireDraw self, per-draw dynamic-offset descriptor bind.
|
||||
5. Final full matrix (dev + 4 arms x 2 backends x openra/rd12, REPS=3), then BENCH.md, README.md,
|
||||
DEBTS.md rows (Magma 16 %, W9: closed with readings - render-state CSO cache mints 0/frame, Magma
|
||||
pipeline memo 99.0 % hit on rd12, 92.7 % on openra), FCL re-run on the final lib.
|
||||
1. FCL Magma monolith waits about 3 ms/frame. The trace does not show it. Profile it off-CPU
|
||||
first. Both close-out attempts caught no game frames, because FCL start-up ran past the
|
||||
60 s warm-up.
|
||||
2. The draw-packet cache (design sketch). Prototype the client half on Espryt monolith first.
|
||||
3. The Espryt driver +0.33 ms is a cache-pressure inference. Check it with L1D refills
|
||||
before and after (2).
|
||||
4. Native TLS, minSdk 29 vs 26, is the user's call: +1.7 % Espryt, noise on Magma.
|
||||
|
||||
## Harness
|
||||
|
||||
Scratchpad scripts (session-local): `build_trace.sh W:/perf .p14X` (trace APK, suffix = package
|
||||
`top.mobilegl.plugin.p14X.trace`), `sign_install.sh <apk> <name>`. W: is `subst` of
|
||||
`.claude/worktrees` (MAX_PATH). Then:
|
||||
Session scratchpad scripts:
|
||||
- `build_trace.sh W:/perf .p14X`: trace APK, package `top.mobilegl.plugin.p14X.trace`.
|
||||
- `sign_install.sh <apk> <name>`.
|
||||
- `devjob.sh <log> <cmd>`: runs a device job under a mkdir lock. Always use it; two concurrent
|
||||
device jobs contaminate each other, and killing a job leaves its children running.
|
||||
- `incl.py` / `children.py` / `lines.py` / `layers2.py` / `drvcall.py` / `offcpu.py`:
|
||||
simpleperf analyses. The tail window comes from the render thread's `retraceCall` samples.
|
||||
- `profile_offcpu.sh`: `--trace-offcpu` variant of `profile_p14.sh`.
|
||||
- `stat_p14.sh`: `simpleperf stat` counters.
|
||||
- `fcl_offcpu.sh`: FCL profile plus restore.
|
||||
|
||||
Then:
|
||||
|
||||
```
|
||||
cd tools/device_bench/disagg
|
||||
@@ -77,21 +53,44 @@ ARMS="monolith@p14a monolith@p14b" BACKENDS="DirectGLES DirectVulkan" WLS=rd12 R
|
||||
python p14_report.py ../../../.trace-work/perf-p14/ab
|
||||
ARMS="monolith@p14X inproc@p14X shm@p14X tcp@p14X" REPS=1 TAG=ssim bash matrix_p14.sh ssim # SSIM gate
|
||||
ARM=monolith BACKEND=DirectGLES WL=rd12 TAG=prof PKG_FEAT=... SYMDIR=<unstripped libs> bash profile_p14.sh
|
||||
python layers.py <perf.data> <binary_cache> MobileGLTraceRe <cpu ms/frame> --tail-ms=N [--detail=backend]
|
||||
```
|
||||
|
||||
FIXTURES must point at a checkout with real LFS goldens (worktrees have pointer files). Host tests:
|
||||
WSL archlinux `~/mgl-perf-bld` (clang, split+inproc, tests on); ServerSpawnTest / FuzzArm2 /
|
||||
PeerLatchTest fail there with Refuse{BuildFingerprint} (no build stamp in that tree), environmental.
|
||||
The FCL library comes from `:app:assemblePluginRelease`. Unzip `lib/arm64-v8a/libMobileGL.so` from
|
||||
`app/build/outputs/apk/plugin/release/`; the unstripped copy is under
|
||||
`intermediates/merged_native_libs/pluginRelease`. Run `fcl_p14.sh run`, then `restore`
|
||||
(md5-checked).
|
||||
|
||||
Gradle's `packageTraceRelease` sometimes fails on the first try with
|
||||
`IncrementalSplitterRunnable`; a rerun succeeds.
|
||||
|
||||
Host:
|
||||
- Unit tests: WSL archlinux `~/mgl-perf-bld` (clang, split + inproc).
|
||||
- ServerSpawnTest, FuzzArm2 and PeerLatchTest fail there with `Refuse{BuildFingerprint}`;
|
||||
that is environmental.
|
||||
- The google-benchmark targets don't build with clang 22; also environmental.
|
||||
- CI-shaped integration tree: `~/mgl-perf-ci31`, configured with CMake 3.31.10 from the
|
||||
`~/mgl-perf-tools` venv.
|
||||
- The system CMake 4.x joins `gtest_discover_tests` ENVIRONMENT lists into one variable, and
|
||||
every entry then skips. Use 3.31.10.
|
||||
- lavapipe ICD; spawn/tcp lanes are environmental there.
|
||||
- To run the syncval gates: `scripts/ci/magma_runahead_checks.py validation --build-dir ~/mgl-perf-ci31 --execute`,
|
||||
with `GIT_DIR=.../.git/modules/MobileGL/worktrees/perf` set for the cache script.
|
||||
- `~/mgl-perf-ci` is a CMake 4 tree. Use it only through the scratchpad's `runentry.sh`, which
|
||||
splits the joined environment.
|
||||
|
||||
## Device state
|
||||
|
||||
HA27Q3LQ: CPU/GPU clocks PINNED (`bash pin_clocks.sh restore` to undo). FCL lib/config/props restored
|
||||
(md5 checked; `debug.mobilegl.*` props unset). Trace APKs `top.mobilegl.plugin.p14*.trace` installed.
|
||||
- HA27Q3LQ CPU/GPU clocks are still PINNED: the next phase wants them pinned.
|
||||
`bash pin_clocks.sh restore` undoes it; ask before unpinning.
|
||||
- FCL is restored: lib md5 matches `/data/local/tmp/p14-fcl-orig.so`, config.json matches its
|
||||
saved copy, and the `debug.mobilegl.*` properties are deleted.
|
||||
- Trace APKs `top.mobilegl.plugin.p14*.trace` are installed; `p14fin` is the close-out build.
|
||||
|
||||
## CI
|
||||
|
||||
No run has completed since f8a9466b (each push cancelled the previous). workflow_dispatch runs on
|
||||
the checkpoint head: see the milestone message / below.
|
||||
Dispatched runs (workflow_dispatch, not cancelled by later pushes):
|
||||
- 86ab2835: Test 37702990844, APK 37702995519.
|
||||
- 781a7446: Test 37707221661.
|
||||
- The close-out docs head: see the final report.
|
||||
|
||||
Dispatched on ff53ba3b: Test run 37674322398, MobileGL APK run 37674328683 (workflow_dispatch, not cancelled by later pushes).
|
||||
There is only one self-hosted GPU runner (`minipc-8845-arch-wsl-gpu`), so GPU jobs queue behind it.
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# Next phase: a per-(VAO, program) draw-packet cache (design sketch)
|
||||
|
||||
This is a starting point for the next phase, not an implementation. The numbers are from the P14
|
||||
close-out (`BENCH.md`, rd12 monolith, Y700, pinned, render thread, ms/frame).
|
||||
|
||||
## 1. The problem
|
||||
|
||||
On the record arm, a draw crosses five layers, and each layer keeps its own per-object state:
|
||||
|
||||
| layer | Espryt feat | Espryt dev | delta | what it touches per draw |
|
||||
|---|---|---|---|---|
|
||||
| client half | 1.78 | 0.39 | +1.39 | tracker shutters (VAO, program, context gens), VaoMemo, emitter latches, slot allocator, residual block, verb port |
|
||||
| applier | 0.07 | 0 | +0.07 | vertex-elements record, applier unit windows, global-constants block |
|
||||
| backend | 1.07 | 0.62 | +0.45 | VAO twin, texture/sampler records + twins (clean checks), program twin, UBO ring |
|
||||
| driver | 1.69 | 1.36 | +0.33 | the same GL calls as dev; slower, consistent with cache pressure (+56 % L1D refills, +16.6 % instructions, whole run) |
|
||||
|
||||
Minecraft rd12 issues 1471 draws a frame, about 3.3 GL calls per draw. Between two consecutive
|
||||
draws, usually only the bound VAO and the program's default-uniform block change. Each layer still
|
||||
re-derives "what moved" from its own structures, and each of those structures is a different cold
|
||||
cache line per VAO: the frontend VAO, the VaoMemo entry, the emitter latch, the applier record, and
|
||||
the backend twin. Pull touches one of them, the frontend VAO.
|
||||
|
||||
None of the individual items is above 2 % of the frame. The sum is the gap: on Espryt monolith,
|
||||
91 vs 115 fps.
|
||||
|
||||
## 2. What one draw packet holds
|
||||
|
||||
There is one packet per (VAO lifetime id, program lifetime id) pair that has drawn. It lives in a
|
||||
client-side table, and in the server's record space on the split arms. Its fields:
|
||||
|
||||
- **Identity:** VAO lifetime id + config version, program lifetime id + link version, and the
|
||||
context serial.
|
||||
- **Client side:**
|
||||
- the vertex-elements handle and its published config version;
|
||||
- the buffer-set and index handles and their hash, which the VaoMemo holds today;
|
||||
- the shader CSO handle;
|
||||
- the residual-field values this pair contributed (the bound-VAO and program pins);
|
||||
- the context-global epoch it was validated at (see section 3).
|
||||
- **Applier side:** a pointer, or slot index plus generation, to the vertex-elements record and the
|
||||
stored buffer/index window, so that bind_vertex_elements applies by reference instead of copying.
|
||||
- **Backend side (Espryt):**
|
||||
- the VAO twin;
|
||||
- the resolved draw buffers;
|
||||
- the program twin;
|
||||
- the sampled-texture list with each entry's synced serials;
|
||||
- the synced attachment list;
|
||||
- the UBO block size.
|
||||
- **Backend side (Magma):** the wire vertex-input state, the pipeline handle, the descriptor-set
|
||||
layout, and the pass-compatibility id.
|
||||
- **A "packet valid at" stamp per layer:** the serials and epochs each layer checks to trust its
|
||||
slice of the packet.
|
||||
|
||||
The goal is that a draw whose packet is valid at every layer touches the packet and the per-draw
|
||||
inputs, and nothing else. The per-draw inputs are the uniform span, the draw arguments, and the UBO
|
||||
ring slot.
|
||||
|
||||
## 3. Keys and invalidation sources
|
||||
|
||||
**Keys.** Lifetime ids only, never GL names or pointers. This is the ABA rule, already broken and
|
||||
fixed three times: sampler, texture, program. The packet table key is (VAO lifetime id, program
|
||||
lifetime id, context serial).
|
||||
|
||||
Invalidation sources, by layer:
|
||||
|
||||
- **Frontend context-global state.** This is everything the tracker reads that is neither VAO- nor
|
||||
program-derived: render/pipeline state versions, texture content/params/bind generations,
|
||||
framebuffer attachment and bind versions, buffer bind-point generations, TFB, pixel pack, patch,
|
||||
vertex attribute defaults, and the program pipeline.
|
||||
- This needs a context mutation epoch, bumped in every GL/EGL entry except the
|
||||
whitelist: draws, glBindVertexArray, and float glUniform*.
|
||||
- The bump fits in the `DECLARE_GL_FUNCTION_END` macros, with the whitelist as a constexpr name
|
||||
test, plus the manual entries (glFinish/glFlush and the hand-written extension wrappers) and
|
||||
the EGL entries.
|
||||
- **Audit item:** every MOBILEGL_GL_API definition outside the macros.
|
||||
- **VAO.** The config version (attribute pointers and enables), the index slot version, and buffer
|
||||
respecification through `AnyVaoAttributeGeneration`. These are already lifetime- and
|
||||
version-keyed.
|
||||
- **Program.** Link version, backend state version (sampler units through glUniform1i), block
|
||||
binding version, and image unit version. Float uniforms move only the UBO content version,
|
||||
which is a per-draw input and not a packet invalidation.
|
||||
- **Applier.** The record `ContentSerial`, `ContextSerial`, `MGPipeSlots().ResetEpoch()`, and the
|
||||
vertex-buffers and index-buffer serials. These are already monotonic.
|
||||
- **Backend.**
|
||||
- Texture: record `Serial`, `ParamsSerial`, and `PendingUploads`; twin force-resync flags;
|
||||
sampler CSO serial; shared-image state.
|
||||
- Context generation.
|
||||
- Magma only: texture image epoch, resource erase epoch, the image layout leaving GENERAL, and the
|
||||
store destroy epoch. The P14 Magma memos already check these.
|
||||
- The backend needs one "texture state moved" epoch, so that the per-bound-texture clean check
|
||||
can be skipped when nothing in the texture family moved. This is the item the content-version
|
||||
lesson warns about: glGenerateMipmap grew mips without a version bump. Every backend-side mutation
|
||||
of a twin's synced state must bump the epoch too.
|
||||
|
||||
## 4. Split arms and the verify comparator
|
||||
|
||||
- **The packet is per role.** The client half of the packet (handles, latches, residual pins)
|
||||
lives in the client. The applier and backend halves live wherever the applier runs: the same
|
||||
thread on monolith, the apply thread on inproc, the server process on spawn and tcp.
|
||||
- **Nothing in the packet crosses the wire.** The wire stays the record stream. A server-side
|
||||
packet is keyed on the vertex-elements handle and the shader-CSO handle (slot + generation)
|
||||
plus the applier context serial, because the server never sees frontend lifetime ids.
|
||||
- **Run-ahead.** Client-side validity must not depend on server state. The client's half is
|
||||
checked against client epochs only, exactly like today's latches.
|
||||
- **The verify comparator (`MOBILEGL_PIPE_VERIFY`) compares the residual block against a live
|
||||
snapshot at the verb.**
|
||||
- A packet hit that skips residual copies would make the comparator diverge, unless the packet's
|
||||
pins are what it compares.
|
||||
- Proposal: under verify, take the slow path always and assert that the packet would have hit
|
||||
with the same values. That gives a free cross-check of the epoch's coverage on every verify
|
||||
lane.
|
||||
- **POISON stamps (trace build only).** The stamp loop stays, or the packet records the stamp set
|
||||
and applies it with one bulk write.
|
||||
|
||||
## 5. Risks
|
||||
|
||||
- **A missed epoch bump renders stale state.** This is the dominant risk. Mitigations:
|
||||
- the verify-lane cross-check above;
|
||||
- a negative control that removes the bump from one entry and must go red;
|
||||
- SSIM 16/16 on every change;
|
||||
- the existing retrace CI.
|
||||
- **ABA on packet keys.** Lifetime ids only, plus context serial and slot reset epoch.
|
||||
- **Memory.** One packet per drawn (VAO, program) pair. Minecraft has thousands of chunk VAOs ×
|
||||
a few programs, so a bounded table with LRU is about 100-200 B × 10k ≈ 2 MB. The table must
|
||||
not itself become the next cache-miss source: entries go in a flat array with an index stored
|
||||
on the frontend VAO.
|
||||
- **Backend-side invalidation completeness** (section 3, backend): the same class as the reverted
|
||||
e8e1521.
|
||||
- **Code size and review cost.** It touches every layer. It should land behind a runtime knob,
|
||||
default off until the CI and SSIM gates and an FCL soak are green.
|
||||
|
||||
## 6. Rough gain estimate (rd12, ms/frame, Espryt monolith)
|
||||
|
||||
| layer | removable on a packet hit | why not all of the delta |
|
||||
|---|---|---|
|
||||
| client half (+1.39) | 0.6-0.9 | the uniform span, bind record and draw record still have to be produced |
|
||||
| applier (+0.07) | 0.03 | bind by reference |
|
||||
| backend (+0.45) | 0.25-0.35 | the UBO ring slot and the GL calls stay |
|
||||
| driver (+0.33) | 0.1-0.3, if the cache-pressure reading is right | unproven; measure L1D refills before and after |
|
||||
|
||||
Total: about 1.0-1.6 ms of the 2.5 ms gap. That is roughly 91 -> 100-106 fps on Espryt monolith.
|
||||
Magma's share is similar on the client half (+1.21). Its backend is already below dev, and its
|
||||
driver term is +0.24.
|
||||
|
||||
Inproc gains the client-half part on the client thread: that is the bound thread on Espryt rd12,
|
||||
so 112 -> above dev's 115. Magma inproc is bound on the apply thread, which gains only the backend
|
||||
and applier part.
|
||||
|
||||
Prototype first, before the full design: the client-half packet (tracker + emitter + residual pins,
|
||||
with the context epoch) on monolith Espryt only. Measure it against the 0.6-0.9 ms estimate, then
|
||||
decide on the backend half.
|
||||
@@ -0,0 +1,111 @@
|
||||
# P14: the post-P13 performance pass
|
||||
|
||||
P14 measured the record (push) arm against the `dev` pull monolith and cut its CPU cost. It ran on
|
||||
`feat/disaggregated` from `f8a9466b` (start) to `781a7446` (close-out), on a Lenovo Y700 (Adreno
|
||||
750) with pinned clocks.
|
||||
|
||||
- [ESTIMATE.md](ESTIMATE.md): the code-based estimate, written before the first benchmark.
|
||||
- [BENCH.md](BENCH.md): the final trace matrix, the layer split, CPU vs GPU binding, and what the
|
||||
remaining gap is made of.
|
||||
- [HANDOFF.md](HANDOFF.md): harness commands, device state, open items.
|
||||
- [NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md): the design sketch for the next phase.
|
||||
|
||||
## Headline numbers
|
||||
|
||||
### Minecraft in FCL (the representative app check)
|
||||
|
||||
Setup:
|
||||
- FCL Minecraft, in world.
|
||||
- The library is swapped into FCL, which runs it as a plugin build without `MOBILEGL_PIPE_POISON`.
|
||||
- Readings come from MobileGL's own fps log.
|
||||
- Two launches × three steady windows of about 2 s each. Dev Espryt is three windows: its second
|
||||
launch died at start-up, twice.
|
||||
- Clocks are pinned. Runs: `.trace-work/perf-p14/fcl-final/`.
|
||||
|
||||
| backend | dev (pull) | feat monolith | feat inproc | feat monolith, presenting-thread CPU |
|
||||
|---|---|---|---|---|
|
||||
| Espryt | 228.6 | 198.3 (−13 %) | 241.5 (+6 %) | 2.43 ms/frame (dev 1.85) |
|
||||
| Magma | 231.8 | 171.4 (−26 %) | 194.8 (−16 %) | 2.80 ms/frame (dev 1.64) |
|
||||
|
||||
Earlier in the pass, at `94705adc`, the same measurement read:
|
||||
- Espryt: monolith about 192, inproc about 233;
|
||||
- Magma: monolith about 169, inproc about 179;
|
||||
- dev: 209-245.
|
||||
|
||||
FCL Magma monolith did not follow the trace's improvement. Its presenting thread is busy only
|
||||
about 2.8 of the roughly 5.8 ms frame, so the rest is a wait. That is the first item for the next
|
||||
phase: an off-CPU profile of FCL Magma monolith, the same method that found the upload stall in
|
||||
the trace. Two attempts at it during close-out caught no game frames (FCL start-up).
|
||||
|
||||
### Trace replay, rd12 (Minecraft 1.21.4 RD12, 1471 draws/frame), best of 3, fps
|
||||
|
||||
| backend | dev (pull) | monolith | inproc | shm | tcp |
|
||||
|---|---|---|---|---|---|
|
||||
| Espryt | 115.2 | 91.1 (start 69.8) | 111.9 (71.1) | 87.8 (62.8) | 68.7 (50.0) |
|
||||
| Magma | 87.0 | 83.8 (start 45.4) | 110.0 (63.2) | 95.6 (58.6) | 69.9 (45.6) |
|
||||
|
||||
openra is in [BENCH.md](BENCH.md). On openra, Magma monolith now runs 2.5x faster than dev
|
||||
(747 vs 293), and inproc 3.6x (1052 vs 293).
|
||||
|
||||
Every arm is CPU-bound: GPU busy is 19-39 %, and the top thread sits at 94-99 % of a core
|
||||
(70-80 % on tcp).
|
||||
|
||||
## What landed
|
||||
|
||||
The commits run from `f8a9466b` to `781a7446`. Per-commit numbers are in the commit subjects.
|
||||
|
||||
- **Magma wire draw:**
|
||||
- no per-draw SPIR-V rehash or allocation;
|
||||
- unit sampler memo; unchanged pipeline, vertex and index binds skipped;
|
||||
- direct-mapped texture and buffer fronts;
|
||||
- descriptor-set rebind with fresh offsets;
|
||||
- pass-continuation memo;
|
||||
- vertex-input layout kept on its record;
|
||||
- **texture level uploads no longer wait for the GPU to drain** (`553eb86d`, the largest
|
||||
single gain of the pass: rd12 70 -> 83, openra 367 -> 745);
|
||||
- slices stamped from the owner's submit counter.
|
||||
- **Espryt:**
|
||||
- VAO twin re-points only the moved buffers;
|
||||
- clean attachments skip resync;
|
||||
- unchanged buffer sets are adopted;
|
||||
- twin lookups without SharedPtr copies;
|
||||
- one thread-local block;
|
||||
- one UBO-ring lookup per draw.
|
||||
- **MG_Pipe client half:**
|
||||
- lifetime-id slot fronts;
|
||||
- per-VAO memo;
|
||||
- fill pins;
|
||||
- kept VAO bindings on the server;
|
||||
- uniform spans;
|
||||
- context-values gate;
|
||||
- inline process applier;
|
||||
- precomputed residual walk, with split copy and stamp loops;
|
||||
- memo-answered draw bindings.
|
||||
- **MG_Remote:** fault-verdict polling restructured.
|
||||
- It first went from every record to every 256 records (549a34bb).
|
||||
- The drain now polls before any record the last poll could not have seen (`c24b3428`). This
|
||||
fixes the regression where a guilty context ran one more record.
|
||||
- The apply loop hands its poll to the drain (`e058a450`).
|
||||
- **Magma hang watch:** a marker slot is reused only once its submission's fence completes
|
||||
(`ec973fba`).
|
||||
- **Tests and CI:**
|
||||
- TextureUploadBetweenDrawsScenario on every arm, plus inproc sync validation (`67f4f311`,
|
||||
`86ab2835`);
|
||||
- submit-time synchronization validation in the run-ahead and cache gates (`781a7446`);
|
||||
- ServerLoopEglLatchTest now holds the race window open.
|
||||
- **Tools:**
|
||||
- pinned-clock matrix with a per-thread CPU and GPU-busy sampler;
|
||||
- layer-split and off-CPU profile helpers;
|
||||
- FCL harness;
|
||||
- opt-in fps log and transport override properties.
|
||||
|
||||
## Measured and not landed
|
||||
|
||||
- **Record-publish batching:** publishing a validate's state records with the draw record. It came
|
||||
in within ±2 % on every arm, and it opens a deadlock class (a staging-retirement wait on a
|
||||
written but unpublished record).
|
||||
- **Native TLS (minSdk / ANDROID_PLATFORM 29):** +1.7 % on Espryt, noise on Magma (rd12 monolith).
|
||||
This is the user's call.
|
||||
- **Tracker / residual mutation-epoch fast path:** estimated at most 0.07 ms from the tracker alone,
|
||||
0.15-0.25 ms with the residual copies. Each step is below the 2 % bar, and a missed epoch bump
|
||||
renders stale. It is folded into the draw-packet design.
|
||||
Reference in New Issue
Block a user