mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-18 07:23:24 +09:00
[Docs] (.claude/skills, .github/workflows, CMakeLists.txt, docs/archive, docs/Disaggregated, docs/MobileVA, docs/Performance, MG_Backend, MG_Backend/DirectGLES, MG_Backend/DirectVulkan, MG_Backend/Record, MG_Gbm, MG_Impl, MG_Impl/EGLImpl, MG_Impl/GLImpl, MG_Impl/GLXImpl, MG_Impl/Pipe, MG_IntegrationTest, MG_IntegrationTest/Harness, MG_IntegrationTest/Scenarios, MG_Pipe, MG_Pipe/generated, MG_Remote, MG_Remote/Client, MG_Remote/Server, MG_Remote/Transport, MG_Remote/Wire, MG_State/GLState, MG_Test/Pipe, MG_Test/Util, MG_Test/Wire, MobileGL, scripts, scripts/data, tools/device_bench, userguide): archive the planning notes and agent contracts under docs/archive
- DEBTS.md and OPEN-QUESTIONS.md stay maintained, now in docs/Disaggregated.
This commit is contained in:
@@ -0,0 +1,151 @@
|
||||
# P14 perf pass: final bench
|
||||
|
||||
**Headline:** FCL Minecraft gives Espryt monolith 198 vs dev 229 and inproc 242; Magma
|
||||
monolith 171 vs dev 232 and inproc 195. On the rd12 trace, Espryt monolith is 91 vs 115 and
|
||||
Magma monolith 84 vs 87; inproc is 112 (Espryt) / 110 (Magma). Details below.
|
||||
|
||||
Head `781a7446` on `feat/disaggregated` (trace APK `p14fin`) against `dev` at `08124c99` (the pull
|
||||
monolith, the baseline). Lenovo Y700 (Snapdragon 8 Gen 3, Adreno 750), rooted, CPU and GPU clocks
|
||||
pinned (`pin_clocks.sh`), no per-frame glFinish (`FINISH=false`), best of 3, steady frames only.
|
||||
Raw runs: `.trace-work/perf-p14/final-p14/` (`matrix_p14.sh run`, `p14_report.py`).
|
||||
|
||||
## Final matrix (fps, best of 3)
|
||||
|
||||
| workload | backend | dev (pull) | monolith | inproc | shm | tcp |
|
||||
|---|---|---|---|---|---|---|
|
||||
| rd12 | Espryt | 115.2 | 91.1 | 111.9 | 87.8 | 68.7 |
|
||||
| rd12 | Magma | 87.0 | 83.8 | 110.0 | 95.6 | 69.9 |
|
||||
| openra | Espryt | 889.2 | 832.5 | 1073.9 | 1063.1 | 602.0 |
|
||||
| openra | Magma | 293.1 | 747.4 | 1051.6 | 976.0 | 544.5 |
|
||||
|
||||
Start of the pass (same harness, `nofinish` tag, `f8a9466b`):
|
||||
|
||||
| workload | backend | monolith | inproc | shm | tcp |
|
||||
|---|---|---|---|---|---|
|
||||
| rd12 | Espryt | 69.8 | 71.1 | 62.8 | 50.0 |
|
||||
| rd12 | Magma | 45.4 | 63.2 | 58.6 | 45.6 |
|
||||
| openra | Espryt | 812.2 | 1067.5 | 953.8 | 529.2 |
|
||||
| openra | Magma | 356.3 | 446.0 | 394.4 | 412.1 |
|
||||
|
||||
So over the pass, rd12 went from 69.8 to 91.1 fps on Espryt monolith (+31%) and from 45.4 to 83.8 on
|
||||
Magma monolith (+85%). Split arms: inproc +57% / +74%, shm +40% / +63%, tcp +37% / +53%. On openra,
|
||||
Magma monolith went 356 -> 747 and inproc 446 -> 1052.
|
||||
|
||||
## Against the three goals
|
||||
|
||||
| goal | rd12 | openra |
|
||||
|---|---|---|
|
||||
| push = pull (monolith vs dev) | Espryt 91.1 vs 115.2 (-21%); Magma 83.8 vs 87.0 (-4%) | Espryt 832 vs 889 (-6%); Magma 747 vs 293 (2.5x dev) |
|
||||
| inproc beats pull | Espryt 111.9 vs 115.2 (-3%); Magma 110.0 vs 87.0 (+26%) | Espryt 1074 vs 889 (+21%); Magma 1052 vs 293 (3.6x) |
|
||||
| backend parity (feat) | monolith 91.1 vs 83.8; inproc 111.9 vs 110.0 | monolith 832 vs 747; inproc 1074 vs 1052 |
|
||||
|
||||
## CPU- or GPU-bound
|
||||
|
||||
Every arm on both workloads is CPU-bound: GPU busy is 19-39 % everywhere (`GPU busy` column of the
|
||||
report), and the arm's top thread runs at 94-99 % of a core on the monolith and inproc arms
|
||||
(97-99 % for the render thread, 94-95 % for `mgl-srv-apply` where it is the top thread).
|
||||
- monolith: one thread, `MobileGLTraceRe`, at 99 % on all four cells.
|
||||
- inproc / shm: two threads at about the same load (client 8.5-8.7, apply 8.5-8.6 ms/frame on rd12),
|
||||
so the split arms are bound on whichever of the two is ahead; on rd12 Espryt that is the client
|
||||
(harness parse + client half), on rd12 Magma the apply thread.
|
||||
- tcp: 70-80 % top-thread utilisation, the rest is the socket ping-pong (credit-1 waits), as in P10.
|
||||
- `dev` Magma on openra is the one cell below 95 %: 63 % utilisation, wide spread (13.6 %) - dev's
|
||||
Magma waits on the GPU there, which the feat build no longer does (see "Magma upload wait").
|
||||
|
||||
Backend parity therefore means "both CPU-bound at the same CPU cost". On the feat build the two
|
||||
backends are within 2-11 % of each other on every arm, and the remaining gap is CPU debt (rd12
|
||||
monolith: Magma backend + driver 3.93 vs Espryt 2.76 ms/frame, client half 1.91 vs 1.78).
|
||||
|
||||
## Layer split, rd12 monolith (render thread, ms/frame, simpleperf at 2 kHz, steady tail)
|
||||
|
||||
| layer | Espryt feat | Espryt dev | delta | Magma feat | Magma dev | delta |
|
||||
|---|---|---|---|---|---|---|
|
||||
| driver | 1.69 | 1.36 | +0.33 | 2.12 | 1.88 | +0.24 |
|
||||
| backend | 1.07 | 0.62 | +0.45 | 1.81 | 2.56 | -0.75 |
|
||||
| applier | 0.07 | 0 | +0.07 | 0.13 | 0 | +0.13 |
|
||||
| client half | 1.78 | 0.39 | +1.39 | 1.91 | 0.70 | +1.21 |
|
||||
| harness (trace parse) | 6.94 | 6.70 | +0.24 (same code) | 6.79 | 6.86 | -0.07 |
|
||||
| total CPU | 11.56 | 9.07 | +2.49 | 12.77 | 12.01 | +0.76 |
|
||||
|
||||
About 6.7-6.9 ms of every rd12 frame is the replay harness, so the MobileGL part of an Espryt frame
|
||||
is 4.6 ms on feat against 2.4 ms on dev, and of a Magma frame 6.0 against 5.1.
|
||||
|
||||
### What the Espryt monolith gap is made of (as % of an 11.2 ms frame)
|
||||
|
||||
- Validate point, 0.92 ms (none of it exists in pull): EmitGlobalConstants 0.17 (1.5 %),
|
||||
EmitVertexElements 0.13 (1.1 %, includes the inline apply of the bind), Tracker::Update 0.12
|
||||
(1.1 %), validate self 0.10 (0.8 %), CopyField 0.10 (0.9 %), EmitVertexBuffers 0.07 (0.6 %),
|
||||
thread-local and atomic overhead about 0.07.
|
||||
- Verb port, about 0.2 ms (1.8 %): EmitDrawRecord, ReadDrawBindings 0.06, MarkGpuWritesForDraw 0.05.
|
||||
- Backend +0.45: SyncNeccessaryTextures 0.23 vs 0.10, SyncNeccessaryBuffers 0.16 vs 0.08,
|
||||
AdoptUnchangedBufferSets 0.09, BindCurrentProgramWithResources 0.18 vs 0.11,
|
||||
SyncCurrentProgramByHandle 0.09 vs 0.04.
|
||||
- Driver +0.33 (2.9 %): inside glDrawElements itself, with the same GL call mix as dev. A
|
||||
`simpleperf stat` over the whole run (render thread) has feat executing 16.6 % more instructions
|
||||
than dev with 56 % more L1D and 23 % more L2D refills (IPC 3.73 vs 3.86): the record arm's larger
|
||||
per-draw working set (records, applier state, slot tables, twins) evicting the driver's state is
|
||||
the best reading, inferred from those counters, not measured inside the driver.
|
||||
|
||||
Each item is under 2 % of the frame except the driver term. Why the sum is structural on the record
|
||||
arm: pull's backend reads frontend objects by pointer at draw time; push has to detect what moved
|
||||
(tracker), serialise it (emitters, uniform spans), apply it (applier), then let the backend resolve
|
||||
it again by handle through slot tables, with per-bound-object clean checks because the backend no
|
||||
longer sees the frontend's dirty bits. Each step is tens of ns per draw on cache lines no other
|
||||
step shares, times 1471 draws a frame.
|
||||
|
||||
Options left (none below the 2 % bar on its own, or each with a correctness risk the bar does not
|
||||
pay for):
|
||||
- a frontend mutation epoch (bumped in every `DECLARE_GL_FUNCTION_END` entry except draws,
|
||||
glBindVertexArray and float glUniform*) so a validate with only VAO and uniform movement skips
|
||||
the context-global shutters (at most 0.07 ms) and, after an audit of which residual fields
|
||||
depend on the VAO/program bindings, the residual copies (0.15-0.25 ms together, 1.5-2 %);
|
||||
- a per-(VAO, program) draw packet that keeps tracker, emitter, applier and backend state for a
|
||||
draw on a few cache lines, keyed on lifetime ids - the structural answer to the cache-pressure
|
||||
term, a redesign rather than a pass item;
|
||||
- native TLS (minSdk / ANDROID_PLATFORM 29 instead of 26, the user's call): measured +1.7 %
|
||||
Espryt, within noise on Magma (rd12 monolith, best of 3; whole APK, harness included).
|
||||
|
||||
## Pass findings worth keeping
|
||||
|
||||
- **Magma upload wait (553eb86d).** Every texture level upload flushed the recording and then
|
||||
waited on the CPU for every submission to complete (`FlushWirePendingCommandsForTextureUpdate`,
|
||||
added in P7 B4). The upload batch is submitted on the same queue after the flush and its first
|
||||
barrier takes its source scope from the image's tracked layout, so the wait bought nothing; on
|
||||
rd12 it was 1.8 ms/frame of fence stall (Minecraft updates its lightmap every frame). Removing it:
|
||||
rd12 Magma monolith 70.4 -> 83.4, openra 367 -> 745. The preserve path keeps its wait. Proof: a
|
||||
new between-draws upload scenario under the synchronization validator (red on a broken barrier),
|
||||
sync validation of the texture/remint/copy scenarios (no new hazards), SSIM 16/16.
|
||||
- **Sync validation needs submit-time validation on.** The Khronos layer (1.4.350) does not report
|
||||
a hazard between two queue submissions unless `VK_KHRONOS_VALIDATION_SYNCVAL_SUBMIT_TIME_VALIDATION`
|
||||
is set; the run-ahead and cache gates now set it (781a7446), and the run-ahead gate goes red on
|
||||
the broken-barrier build where it used to stay green.
|
||||
- **GPU-fault latch regression (549a34bb -> fixed in c24b3428).** Polling the reset status every
|
||||
256 records let a record published right after the previous acknowledgement run unpolled on a
|
||||
guilty context. The drain now polls before any record the last poll could not have seen
|
||||
(e058a450 hands the apply loop's poll to the drain so a woken drain does not poll twice).
|
||||
- **Record-publish batching, measured and dropped.** Publishing an unbarriered validate's state
|
||||
records with the draw record behind them was within +-2 % on every arm and opens a deadlock class
|
||||
(a staging-retirement wait on a record that is written but not published - it hung rd12 Espryt
|
||||
split until the stage wait published owed records). Not worth the surface; the patch and its
|
||||
stress scenario are not in the tree.
|
||||
|
||||
## W9 (cache capacities), closed with readings
|
||||
|
||||
`MOBILEGL_PIPE_STATS` on rd12 and openra: the render-state CSO cache mints 0 per frame in the steady
|
||||
state, and Magma's pipeline memo hits 99.0 % on rd12 and 92.7 % on openra. No cache thrashes; the
|
||||
P8 capacities stand.
|
||||
|
||||
## FCL (Minecraft in world, the representative app check)
|
||||
|
||||
FCL's own build of the library (plugin flavor, no POISON), swapped into FCL. Readings come from
|
||||
MobileGL's fps log: mean of two launches × three steady windows. Clocks pinned. Runs:
|
||||
`.trace-work/perf-p14/fcl-final/`.
|
||||
|
||||
| backend | dev (pull) | feat monolith | feat inproc | presenting-thread CPU, monolith feat / dev |
|
||||
|---|---|---|---|---|
|
||||
| Espryt | 228.6 | 198.3 (-13 %) | 241.5 (+6 %) | 2.43 / 1.85 ms/frame |
|
||||
| Magma | 231.8 | 171.4 (-26 %) | 194.8 (-16 %) | 2.80 / 1.64 ms/frame |
|
||||
|
||||
FCL Magma does not match the trace: the presenting thread is busy only about 2.8 ms of a 5.8 ms
|
||||
frame, so about 3 ms is a wait the trace does not have. That is the next phase's first item: an
|
||||
off-CPU profile, the method that found the trace's upload stall.
|
||||
@@ -0,0 +1,79 @@
|
||||
# P14 perf pass: estimate from the code (written before the benchmark)
|
||||
|
||||
Date: 2026-10-07. Code: `feat/disaggregated` at `f8a9466b` ("feat") against `dev` at `08124c99` ("dev",
|
||||
the pull monolith, the "before" baseline per DEBTS.md). Device for the measurement that follows:
|
||||
Lenovo Y700 (TB321FU, Snapdragon 8 Gen 3, Adreno 750), rooted, clocks pinned.
|
||||
|
||||
**Disclosure.** Before writing this I ran one unpinned pipeline-validation smoke of the new harness
|
||||
(Magma only, one rep, dev / feat monolith / feat inproc): openra 247 / 178 / 146 fps, rd12
|
||||
127 / 46 / 33 fps. Those numbers are bigger than the logged 16 % (DEBTS ID-P13-2) and I had seen
|
||||
them when I wrote the Magma rows below, so the Magma magnitudes are not blind. The mechanism
|
||||
attribution (which costs, on which thread) is from the code and is what the profile will test;
|
||||
the Espryt, spawn and tcp rows were written before any run of those arms.
|
||||
|
||||
## 1. What each variant does per call and per draw
|
||||
|
||||
| | dev pull monolith | feat monolith (record arm) | feat inproc | feat spawn (shm) | feat tcp (localhost) |
|
||||
|---|---|---|---|---|---|
|
||||
| GL entry | frontend state only | + `GLStreamScope` (one call + branch) | + process-wide recursive mutex per GL call | same as inproc | same as inproc |
|
||||
| state calls | frontend state | + mutation notes (`MGP_NOTE_*`), dirty bits | same as monolith | | |
|
||||
| per draw, client side | none | `MGPipeValidateForVerb`: tracker update (18 shutters), up to 14 emitters (CSO cache probe 64-entry, sampler views/states walks with hash-map slot lookups + XXH64), **`EmitContextValues` + XXH64 every draw**, residual fill of up to 63 fields incl. 3 SharedPtr copies (atomics); verb port: `ReadDrawBindings` hash lookups, persistent-map tracker, `MarkGpuWritesForDraw` | same, plus each record memcpy'd into the SEG_CMD ring and blobs (UBO images, sub-data) into SEG_STAGE | same as inproc | + `sendmsg` flush at every wait / present / fence poll |
|
||||
| per draw, server side | backend reads `pGLContext` | applier (`MGPipeApply*`, same thread) + backend reads applier records; one extra copy of each changed default-uniform block into the applier | decode + bounds checks + apply + backend on the apply thread; staged bytes copied into server storage | same as inproc, in a second process | + io thread `recv` into mirrors |
|
||||
| Magma draw setup | `TrySetupDrawFastPath` (4 snapshots keyed on program lifetime id; skips render-pass, descriptor, texture-sync and pipeline work when nothing moved) | `SetupWireDraw` every draw: `PrepareWireTextureResources` (texture sync per sampled binding), attachments resolved twice, `BuildWireVertexInput` rebuilt (Vector pushes + 4 XXH64), descriptor writes rebuilt + FNV over every info + 4-entry reuse memo, ~10 heap Vectors for the render-pass key, `UnorderedMap` intern of the pass-compatibility key, pipeline memo probe, unconditional `vkCmdBindPipeline` | same backend work on the apply thread | | |
|
||||
| Espryt draw setup | memos keyed on frontend pointers / versions | the same memos re-keyed on applier serials (texture sync list, unit-bindings epoch, FBO lists, resolved-buffers memo, cached `glBindBufferBase/Range`); per-unit record lookups in `ResolveAndBindUnitTextures` | same on the apply thread | | |
|
||||
| threads that burn CPU | 1 | 1 | 2 (client + apply), run-ahead armed | 2 processes | 2 processes + io threads |
|
||||
| sync points | none | none | `kWaitReply` / `kWaitApplied` rows (fence waits, queries, readbacks, mipmaps, copy-tex, make-current), present credit 1 | same + socket doorbell when parked | same + credit-1 ping-pong over the socket (P10: 11.2 of 19.4 ms/frame parked) |
|
||||
|
||||
The trace APK's library is the split shape (`MOBILEGL_BUILD_DISAGGREGATED=ON`), which also turns on
|
||||
`MOBILEGL_PIPE_POISON` (a stamp per filled field and a freshness check per backend accessor read).
|
||||
FCL's library is the push shape without it. So the trace APK's monolith arm carries a little more
|
||||
than the FCL library does.
|
||||
|
||||
## 2. Predictions
|
||||
|
||||
Workloads: openra (light: 128 frames, ~28 draws and ~199 records per frame) and rd12 (Minecraft
|
||||
1.21.4 RD12: 251 frames, ~7,500 records per frame, several hundred draws per frame). Both are light
|
||||
on the GPU, so every arm should be **CPU-bound**: GPU busy well under 60 %, with one thread at or
|
||||
near 100 % for monolith and dev.
|
||||
|
||||
Per-draw costs I expect (Y700 big core at ~2 GHz):
|
||||
|
||||
- feat validate point + verb port over dev: **~2-4 us per draw** (an XXH64 of the context values, a
|
||||
few hash-map lookups, up to 63 field copies, the persistent-map and GPU-write walks).
|
||||
- Magma `SetupWireDraw` over dev's fast path: **~8-15 us per draw** (about 10 heap allocations, a
|
||||
texture sync per sampled texture, two attachment resolves, the vertex-input rebuild and the
|
||||
descriptor rebuild). This dwarfs everything else in this table.
|
||||
|
||||
| workload / backend | dev | feat monolith | feat inproc | feat shm | feat tcp |
|
||||
|---|---|---|---|---|---|
|
||||
| rd12 Espryt | baseline, 1 thread saturated | **10-25 % slower** (validate point + verb port on a ~7-9 ms frame) | **0-20 % faster than feat monolith**, roughly at dev: client and apply threads overlap, total CPU +20-40 % | about inproc (P5/P8 history: shm about inproc) | **3-6x slower than shm**: credit-1 ping-pong and wake-ups, not bandwidth |
|
||||
| rd12 Magma | baseline; dev Magma at or ahead of dev Espryt (fast path + earlier walk-skip work) | **1.6-2.5x the dev frame time**: SetupWireDraw dominates | apply-thread bound (the same SetupWireDraw), so **at most 10-25 % better than feat monolith and still far behind dev** | about inproc | as Espryt tcp, worse in absolute terms |
|
||||
| openra Espryt | baseline | within 5 % (few draws; the fixed per-frame cost dominates) | **5-15 % slower** than monolith (handshake, no work to overlap) | 10-15 % slower than monolith | 20-30 % slower than monolith |
|
||||
| openra Magma | baseline | 15-35 % slower (per-draw gap times only ~28 draws, plus per-frame wire costs) | slower than monolith, as Espryt | | |
|
||||
|
||||
## 3. What this says about the three beliefs (to be checked)
|
||||
|
||||
1. **"inproc is faster than the pull monolith."** Partly. Only on draw-heavy frames, only where the
|
||||
apply thread's backend is cheap enough for the client half to be the critical path (Espryt),
|
||||
and the gain is overlap on a second core, not less work: total CPU goes up. Light loads show
|
||||
no gain and pay the handshake. Magma inproc stays behind dev until its draw setup is fixed.
|
||||
2. **"push and pull are on par."** Wrong on Magma (the lost fast path) and wrong by a smaller margin
|
||||
on draw-heavy Espryt (the validate point and verb port are per-draw costs pull never paid).
|
||||
Close on light loads.
|
||||
3. **"the two backends are on par."** On dev, roughly yes (Magma ahead on Minecraft-like frames).
|
||||
On feat, no: Magma pays its draw setup in full. Both backends are CPU-bound on these scenes,
|
||||
so parity means cutting CPU work until both feed the GPU.
|
||||
|
||||
## 4. Fix plan the estimate implies, in order
|
||||
|
||||
1. A Magma wire draw fast path: snapshot the resolved draw (program lifetime id, applier serials
|
||||
for bound views, samplers, FBO and vertex elements, render-pass key, pipeline, descriptor
|
||||
signature) and skip the re-resolution while nothing moved. Keys must be monotonic lifetime ids
|
||||
or applier generations, never handles or pointers. Also cheaper on a miss: no per-draw heap
|
||||
Vectors, one attachment resolve, a cached compatibility id, `vkCmdBindPipeline` deduplicated.
|
||||
Expected: most of the Magma gap on rd12.
|
||||
2. W9: read `MOBILEGL_PIPE_STATS` on both workloads (CSO mints vs binds, pipeline memo misses,
|
||||
descriptor reuse misses) and resize where a cache thrashes. Expected: small (<5 %) unless a
|
||||
cache is thrashing.
|
||||
3. Then whatever simpleperf shows on the client half (validate point, `EmitContextValues`,
|
||||
residual fill) and on the split arms (ring encode, mutex, waits).
|
||||
@@ -0,0 +1,100 @@
|
||||
# P14 perf pass: handoff (closed 2026-10-07)
|
||||
|
||||
The pass is closed at `781a7446`, plus this docs commit. Results are in [BENCH.md](BENCH.md) and
|
||||
[README.md](README.md). The next phase is the per-(VAO, program) draw-packet cache
|
||||
([NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md)). Its first item is an off-CPU profile of
|
||||
FCL Magma monolith (README "Headline numbers").
|
||||
|
||||
## Where things stand (rd12, pinned, FINISH=false, best of 3)
|
||||
|
||||
| arm | Espryt | Magma | dev Espryt / Magma |
|
||||
|---|---|---|---|
|
||||
| monolith | 91.1 | 83.8 | 115.2 / 87.0 |
|
||||
| inproc | 111.9 | 110.0 | |
|
||||
| shm | 87.8 | 95.6 | |
|
||||
| tcp | 68.7 | 69.9 | |
|
||||
|
||||
FCL MC in world:
|
||||
- dev: Espryt 228.6, Magma 231.8;
|
||||
- feat monolith: Espryt 198.3, Magma 171.4;
|
||||
- feat inproc: Espryt 241.5, Magma 194.8.
|
||||
|
||||
All arms are CPU-bound (GPU 19-39 %).
|
||||
|
||||
## Open items (DEBTS.md has the rows)
|
||||
|
||||
1. FCL Magma monolith waits about 3 ms/frame. The trace does not show it. Profile it off-CPU
|
||||
first. Both close-out attempts caught no game frames, because FCL start-up ran past the
|
||||
60 s warm-up.
|
||||
2. The draw-packet cache (design sketch). Prototype the client half on Espryt monolith first.
|
||||
3. The Espryt driver +0.33 ms is a cache-pressure inference. Check it with L1D refills
|
||||
before and after (2).
|
||||
4. Native TLS, minSdk 29 vs 26, is the user's call: +1.7 % Espryt, noise on Magma.
|
||||
|
||||
## Harness
|
||||
|
||||
Session scratchpad scripts:
|
||||
- `build_trace.sh W:/perf .p14X`: trace APK, package `top.mobilegl.plugin.p14X.trace`.
|
||||
- `sign_install.sh <apk> <name>`.
|
||||
- `devjob.sh <log> <cmd>`: runs a device job under a mkdir lock. Always use it; two concurrent
|
||||
device jobs contaminate each other, and killing a job leaves its children running.
|
||||
- `incl.py` / `children.py` / `lines.py` / `layers2.py` / `drvcall.py` / `offcpu.py`:
|
||||
simpleperf analyses. The tail window comes from the render thread's `retraceCall` samples.
|
||||
- `profile_offcpu.sh`: `--trace-offcpu` variant of `profile_p14.sh`.
|
||||
- `stat_p14.sh`: `simpleperf stat` counters.
|
||||
- `fcl_offcpu.sh`: FCL profile plus restore.
|
||||
|
||||
Then:
|
||||
|
||||
```
|
||||
cd tools/device_bench/disagg
|
||||
FIXTURES=<main checkout>/tools/trace_replay/fixtures PREP_PKGS=top.mobilegl.plugin.p14X.trace bash matrix_p14.sh prep
|
||||
ARMS="monolith@p14a monolith@p14b" BACKENDS="DirectGLES DirectVulkan" WLS=rd12 REPS=3 TAG=ab bash matrix_p14.sh run
|
||||
python p14_report.py ../../../.trace-work/perf-p14/ab
|
||||
ARMS="monolith@p14X inproc@p14X shm@p14X tcp@p14X" REPS=1 TAG=ssim bash matrix_p14.sh ssim # SSIM gate
|
||||
ARM=monolith BACKEND=DirectGLES WL=rd12 TAG=prof PKG_FEAT=... SYMDIR=<unstripped libs> bash profile_p14.sh
|
||||
```
|
||||
|
||||
The FCL library comes from `:app:assemblePluginRelease`. Unzip `lib/arm64-v8a/libMobileGL.so` from
|
||||
`app/build/outputs/apk/plugin/release/`; the unstripped copy is under
|
||||
`intermediates/merged_native_libs/pluginRelease`. Run `fcl_p14.sh run`, then `restore`
|
||||
(md5-checked).
|
||||
|
||||
Gradle's `packageTraceRelease` sometimes fails on the first try with
|
||||
`IncrementalSplitterRunnable`; a rerun succeeds.
|
||||
|
||||
Host:
|
||||
- Unit tests: WSL archlinux `~/mgl-perf-bld` (clang, split + inproc).
|
||||
- ServerSpawnTest, FuzzArm2 and PeerLatchTest fail there with `Refuse{BuildFingerprint}`;
|
||||
that is environmental.
|
||||
- The google-benchmark targets don't build with clang 22; also environmental.
|
||||
- CI-shaped integration tree: `~/mgl-perf-ci31`, configured with CMake 3.31.10 from the
|
||||
`~/mgl-perf-tools` venv.
|
||||
- The system CMake 4.x joins `gtest_discover_tests` ENVIRONMENT lists into one variable, and
|
||||
every entry then skips. Use 3.31.10.
|
||||
- lavapipe ICD; spawn/tcp lanes are environmental there.
|
||||
- To run the syncval gates: `scripts/ci/magma_runahead_checks.py validation --build-dir ~/mgl-perf-ci31 --execute`,
|
||||
with `GIT_DIR=.../.git/modules/MobileGL/worktrees/perf` set for the cache script.
|
||||
- `~/mgl-perf-ci` is a CMake 4 tree. Use it only through the scratchpad's `runentry.sh`, which
|
||||
splits the joined environment.
|
||||
|
||||
## Device state
|
||||
|
||||
- HA27Q3LQ CPU/GPU clocks are still PINNED: the next phase wants them pinned.
|
||||
`bash pin_clocks.sh restore` undoes it; ask before unpinning.
|
||||
- FCL is restored: lib md5 matches `/data/local/tmp/p14-fcl-orig.so`, config.json matches its
|
||||
saved copy, and the `debug.mobilegl.*` properties are deleted.
|
||||
- Trace APKs `top.mobilegl.plugin.p14*.trace` are installed; `p14fin` is the close-out build.
|
||||
|
||||
## CI
|
||||
|
||||
Dispatched runs (workflow_dispatch, not cancelled by later pushes):
|
||||
- 86ab2835: Test 37702990844, APK 37702995519.
|
||||
- 781a7446: Test 37707221661.
|
||||
- The close-out docs head: see the final report.
|
||||
|
||||
Every `runs-on` in test.yml is GitHub-hosted `ubuntu-latest`, so long queueing is GitHub capacity /
|
||||
max-parallel, not a runner of ours. `PeerLatch.SiteMap` (`scripts/ci/ph_latch_sites.py`) greps
|
||||
ServerLoop.cpp's text: keep its patterns in step when DrainRing's shape changes (it caught the
|
||||
`DrainRing(polledHead)` signature at a0f4aa83). Locally, filter environmental failures by exact
|
||||
name - a `grep -v PeerLatch` hid it.
|
||||
@@ -0,0 +1,150 @@
|
||||
# Next phase: a per-(VAO, program) draw-packet cache (design sketch)
|
||||
|
||||
This is a starting point for the next phase, not an implementation. The numbers are from the P14
|
||||
close-out (`BENCH.md`, rd12 monolith, Y700, pinned, render thread, ms/frame).
|
||||
|
||||
## 1. The problem
|
||||
|
||||
On the record arm, a draw crosses five layers, and each layer keeps its own per-object state:
|
||||
|
||||
| layer | Espryt feat | Espryt dev | delta | what it touches per draw |
|
||||
|---|---|---|---|---|
|
||||
| client half | 1.78 | 0.39 | +1.39 | tracker shutters (VAO, program, context gens), VaoMemo, emitter latches, slot allocator, residual block, verb port |
|
||||
| applier | 0.07 | 0 | +0.07 | vertex-elements record, applier unit windows, global-constants block |
|
||||
| backend | 1.07 | 0.62 | +0.45 | VAO twin, texture/sampler records + twins (clean checks), program twin, UBO ring |
|
||||
| driver | 1.69 | 1.36 | +0.33 | the same GL calls as dev; slower, consistent with cache pressure (+56 % L1D refills, +16.6 % instructions, whole run) |
|
||||
|
||||
Minecraft rd12 issues 1471 draws a frame, about 3.3 GL calls per draw. Between two consecutive
|
||||
draws, usually only the bound VAO and the program's default-uniform block change. Each layer still
|
||||
re-derives "what moved" from its own structures, and each of those structures is a different cold
|
||||
cache line per VAO: the frontend VAO, the VaoMemo entry, the emitter latch, the applier record, and
|
||||
the backend twin. Pull touches one of them, the frontend VAO.
|
||||
|
||||
None of the individual items is above 2 % of the frame. The sum is the gap: on Espryt monolith,
|
||||
91 vs 115 fps.
|
||||
|
||||
## 2. What one draw packet holds
|
||||
|
||||
There is one packet per (VAO lifetime id, program lifetime id) pair that has drawn. It lives in a
|
||||
client-side table, and in the server's record space on the split arms. Its fields:
|
||||
|
||||
- **Identity:** VAO lifetime id + config version, program lifetime id + link version, and the
|
||||
context serial.
|
||||
- **Client side:**
|
||||
- the vertex-elements handle and its published config version;
|
||||
- the buffer-set and index handles and their hash, which the VaoMemo holds today;
|
||||
- the shader CSO handle;
|
||||
- the residual-field values this pair contributed (the bound-VAO and program pins);
|
||||
- the context-global epoch it was validated at (see section 3).
|
||||
- **Applier side:** a pointer, or slot index plus generation, to the vertex-elements record and the
|
||||
stored buffer/index window, so that bind_vertex_elements applies by reference instead of copying.
|
||||
- **Backend side (Espryt):**
|
||||
- the VAO twin;
|
||||
- the resolved draw buffers;
|
||||
- the program twin;
|
||||
- the sampled-texture list with each entry's synced serials;
|
||||
- the synced attachment list;
|
||||
- the UBO block size.
|
||||
- **Backend side (Magma):** the wire vertex-input state, the pipeline handle, the descriptor-set
|
||||
layout, and the pass-compatibility id.
|
||||
- **A "packet valid at" stamp per layer:** the serials and epochs each layer checks to trust its
|
||||
slice of the packet.
|
||||
|
||||
The goal is that a draw whose packet is valid at every layer touches the packet and the per-draw
|
||||
inputs, and nothing else. The per-draw inputs are the uniform span, the draw arguments, and the UBO
|
||||
ring slot.
|
||||
|
||||
## 3. Keys and invalidation sources
|
||||
|
||||
**Keys.** Lifetime ids only, never GL names or pointers. This is the ABA rule, already broken and
|
||||
fixed three times: sampler, texture, program. The packet table key is (VAO lifetime id, program
|
||||
lifetime id, context serial).
|
||||
|
||||
Invalidation sources, by layer:
|
||||
|
||||
- **Frontend context-global state.** This is everything the tracker reads that is neither VAO- nor
|
||||
program-derived: render/pipeline state versions, texture content/params/bind generations,
|
||||
framebuffer attachment and bind versions, buffer bind-point generations, TFB, pixel pack, patch,
|
||||
vertex attribute defaults, and the program pipeline.
|
||||
- This needs a context mutation epoch, bumped in every GL/EGL entry except the
|
||||
whitelist: draws, glBindVertexArray, and float glUniform*.
|
||||
- The bump fits in the `DECLARE_GL_FUNCTION_END` macros, with the whitelist as a constexpr name
|
||||
test, plus the manual entries (glFinish/glFlush and the hand-written extension wrappers) and
|
||||
the EGL entries.
|
||||
- **Audit item:** every MOBILEGL_GL_API definition outside the macros.
|
||||
- **VAO.** The config version (attribute pointers and enables), the index slot version, and buffer
|
||||
respecification through `AnyVaoAttributeGeneration`. These are already lifetime- and
|
||||
version-keyed.
|
||||
- **Program.** Link version, backend state version (sampler units through glUniform1i), block
|
||||
binding version, and image unit version. Float uniforms move only the UBO content version,
|
||||
which is a per-draw input and not a packet invalidation.
|
||||
- **Applier.** The record `ContentSerial`, `ContextSerial`, `MGPipeSlots().ResetEpoch()`, and the
|
||||
vertex-buffers and index-buffer serials. These are already monotonic.
|
||||
- **Backend.**
|
||||
- Texture: record `Serial`, `ParamsSerial`, and `PendingUploads`; twin force-resync flags;
|
||||
sampler CSO serial; shared-image state.
|
||||
- Context generation.
|
||||
- Magma only: texture image epoch, resource erase epoch, the image layout leaving GENERAL, and the
|
||||
store destroy epoch. The P14 Magma memos already check these.
|
||||
- The backend needs one "texture state moved" epoch, so that the per-bound-texture clean check
|
||||
can be skipped when nothing in the texture family moved. This is the item the content-version
|
||||
lesson warns about: glGenerateMipmap grew mips without a version bump. Every backend-side mutation
|
||||
of a twin's synced state must bump the epoch too.
|
||||
|
||||
## 4. Split arms and the verify comparator
|
||||
|
||||
- **The packet is per role.** The client half of the packet (handles, latches, residual pins)
|
||||
lives in the client. The applier and backend halves live wherever the applier runs: the same
|
||||
thread on monolith, the apply thread on inproc, the server process on spawn and tcp.
|
||||
- **Nothing in the packet crosses the wire.** The wire stays the record stream. A server-side
|
||||
packet is keyed on the vertex-elements handle and the shader-CSO handle (slot + generation)
|
||||
plus the applier context serial, because the server never sees frontend lifetime ids.
|
||||
- **Run-ahead.** Client-side validity must not depend on server state. The client's half is
|
||||
checked against client epochs only, exactly like today's latches.
|
||||
- **The verify comparator (`MOBILEGL_PIPE_VERIFY`) compares the residual block against a live
|
||||
snapshot at the verb.**
|
||||
- A packet hit that skips residual copies would make the comparator diverge, unless the packet's
|
||||
pins are what it compares.
|
||||
- Proposal: under verify, take the slow path always and assert that the packet would have hit
|
||||
with the same values. That gives a free cross-check of the epoch's coverage on every verify
|
||||
lane.
|
||||
- **POISON stamps (trace build only).** The stamp loop stays, or the packet records the stamp set
|
||||
and applies it with one bulk write.
|
||||
|
||||
## 5. Risks
|
||||
|
||||
- **A missed epoch bump renders stale state.** This is the dominant risk. Mitigations:
|
||||
- the verify-lane cross-check above;
|
||||
- a negative control that removes the bump from one entry and must go red;
|
||||
- SSIM 16/16 on every change;
|
||||
- the existing retrace CI.
|
||||
- **ABA on packet keys.** Lifetime ids only, plus context serial and slot reset epoch.
|
||||
- **Memory.** One packet per drawn (VAO, program) pair. Minecraft has thousands of chunk VAOs ×
|
||||
a few programs, so a bounded table with LRU is about 100-200 B × 10k ≈ 2 MB. The table must
|
||||
not itself become the next cache-miss source: entries go in a flat array with an index stored
|
||||
on the frontend VAO.
|
||||
- **Backend-side invalidation completeness** (section 3, backend): the same class as the reverted
|
||||
e8e1521.
|
||||
- **Code size and review cost.** It touches every layer. It should land behind a runtime knob,
|
||||
default off until the CI and SSIM gates and an FCL soak are green.
|
||||
|
||||
## 6. Rough gain estimate (rd12, ms/frame, Espryt monolith)
|
||||
|
||||
| layer | removable on a packet hit | why not all of the delta |
|
||||
|---|---|---|
|
||||
| client half (+1.39) | 0.6-0.9 | the uniform span, bind record and draw record still have to be produced |
|
||||
| applier (+0.07) | 0.03 | bind by reference |
|
||||
| backend (+0.45) | 0.25-0.35 | the UBO ring slot and the GL calls stay |
|
||||
| driver (+0.33) | 0.1-0.3, if the cache-pressure reading is right | unproven; measure L1D refills before and after |
|
||||
|
||||
Total: about 1.0-1.6 ms of the 2.5 ms gap. That is roughly 91 -> 100-106 fps on Espryt monolith.
|
||||
Magma's share is similar on the client half (+1.21). Its backend is already below dev, and its
|
||||
driver term is +0.24.
|
||||
|
||||
Inproc gains the client-half part on the client thread: that is the bound thread on Espryt rd12,
|
||||
so 112 -> above dev's 115. Magma inproc is bound on the apply thread, which gains only the backend
|
||||
and applier part.
|
||||
|
||||
Prototype first, before the full design: the client-half packet (tracker + emitter + residual pins,
|
||||
with the context epoch) on monolith Espryt only. Measure it against the 0.6-0.9 ms estimate, then
|
||||
decide on the backend half.
|
||||
@@ -0,0 +1,111 @@
|
||||
# P14: the post-P13 performance pass
|
||||
|
||||
P14 measured the record (push) arm against the `dev` pull monolith and cut its CPU cost. It ran on
|
||||
`feat/disaggregated` from `f8a9466b` (start) to `781a7446` (close-out), on a Lenovo Y700 (Adreno
|
||||
750) with pinned clocks.
|
||||
|
||||
- [ESTIMATE.md](ESTIMATE.md): the code-based estimate, written before the first benchmark.
|
||||
- [BENCH.md](BENCH.md): the final trace matrix, the layer split, CPU vs GPU binding, and what the
|
||||
remaining gap is made of.
|
||||
- [HANDOFF.md](HANDOFF.md): harness commands, device state, open items.
|
||||
- [NEXT-draw-packet-cache.md](NEXT-draw-packet-cache.md): the design sketch for the next phase.
|
||||
|
||||
## Headline numbers
|
||||
|
||||
### Minecraft in FCL (the representative app check)
|
||||
|
||||
Setup:
|
||||
- FCL Minecraft, in world.
|
||||
- The library is swapped into FCL, which runs it as a plugin build without `MOBILEGL_PIPE_POISON`.
|
||||
- Readings come from MobileGL's own fps log.
|
||||
- Two launches × three steady windows of about 2 s each. Dev Espryt is three windows: its second
|
||||
launch died at start-up, twice.
|
||||
- Clocks are pinned. Runs: `.trace-work/perf-p14/fcl-final/`.
|
||||
|
||||
| backend | dev (pull) | feat monolith | feat inproc | feat monolith, presenting-thread CPU |
|
||||
|---|---|---|---|---|
|
||||
| Espryt | 228.6 | 198.3 (−13 %) | 241.5 (+6 %) | 2.43 ms/frame (dev 1.85) |
|
||||
| Magma | 231.8 | 171.4 (−26 %) | 194.8 (−16 %) | 2.80 ms/frame (dev 1.64) |
|
||||
|
||||
Earlier in the pass, at `94705adc`, the same measurement read:
|
||||
- Espryt: monolith about 192, inproc about 233;
|
||||
- Magma: monolith about 169, inproc about 179;
|
||||
- dev: 209-245.
|
||||
|
||||
FCL Magma monolith did not follow the trace's improvement. Its presenting thread is busy only
|
||||
about 2.8 of the roughly 5.8 ms frame, so the rest is a wait. That is the first item for the next
|
||||
phase: an off-CPU profile of FCL Magma monolith, the same method that found the upload stall in
|
||||
the trace. Two attempts at it during close-out caught no game frames (FCL start-up).
|
||||
|
||||
### Trace replay, rd12 (Minecraft 1.21.4 RD12, 1471 draws/frame), best of 3, fps
|
||||
|
||||
| backend | dev (pull) | monolith | inproc | shm | tcp |
|
||||
|---|---|---|---|---|---|
|
||||
| Espryt | 115.2 | 91.1 (start 69.8) | 111.9 (71.1) | 87.8 (62.8) | 68.7 (50.0) |
|
||||
| Magma | 87.0 | 83.8 (start 45.4) | 110.0 (63.2) | 95.6 (58.6) | 69.9 (45.6) |
|
||||
|
||||
openra is in [BENCH.md](BENCH.md). On openra, Magma monolith now runs 2.5x faster than dev
|
||||
(747 vs 293), and inproc 3.6x (1052 vs 293).
|
||||
|
||||
Every arm is CPU-bound: GPU busy is 19-39 %, and the top thread sits at 94-99 % of a core
|
||||
(70-80 % on tcp).
|
||||
|
||||
## What landed
|
||||
|
||||
The commits run from `f8a9466b` to `781a7446`. Per-commit numbers are in the commit subjects.
|
||||
|
||||
- **Magma wire draw:**
|
||||
- no per-draw SPIR-V rehash or allocation;
|
||||
- unit sampler memo; unchanged pipeline, vertex and index binds skipped;
|
||||
- direct-mapped texture and buffer fronts;
|
||||
- descriptor-set rebind with fresh offsets;
|
||||
- pass-continuation memo;
|
||||
- vertex-input layout kept on its record;
|
||||
- **texture level uploads no longer wait for the GPU to drain** (`553eb86d`, the largest
|
||||
single gain of the pass: rd12 70 -> 83, openra 367 -> 745);
|
||||
- slices stamped from the owner's submit counter.
|
||||
- **Espryt:**
|
||||
- VAO twin re-points only the moved buffers;
|
||||
- clean attachments skip resync;
|
||||
- unchanged buffer sets are adopted;
|
||||
- twin lookups without SharedPtr copies;
|
||||
- one thread-local block;
|
||||
- one UBO-ring lookup per draw.
|
||||
- **MG_Pipe client half:**
|
||||
- lifetime-id slot fronts;
|
||||
- per-VAO memo;
|
||||
- fill pins;
|
||||
- kept VAO bindings on the server;
|
||||
- uniform spans;
|
||||
- context-values gate;
|
||||
- inline process applier;
|
||||
- precomputed residual walk, with split copy and stamp loops;
|
||||
- memo-answered draw bindings.
|
||||
- **MG_Remote:** fault-verdict polling restructured.
|
||||
- It first went from every record to every 256 records (549a34bb).
|
||||
- The drain now polls before any record the last poll could not have seen (`c24b3428`). This
|
||||
fixes the regression where a guilty context ran one more record.
|
||||
- The apply loop hands its poll to the drain (`e058a450`).
|
||||
- **Magma hang watch:** a marker slot is reused only once its submission's fence completes
|
||||
(`ec973fba`).
|
||||
- **Tests and CI:**
|
||||
- TextureUploadBetweenDrawsScenario on every arm, plus inproc sync validation (`67f4f311`,
|
||||
`86ab2835`);
|
||||
- submit-time synchronization validation in the run-ahead and cache gates (`781a7446`);
|
||||
- ServerLoopEglLatchTest now holds the race window open.
|
||||
- **Tools:**
|
||||
- pinned-clock matrix with a per-thread CPU and GPU-busy sampler;
|
||||
- layer-split and off-CPU profile helpers;
|
||||
- FCL harness;
|
||||
- opt-in fps log and transport override properties.
|
||||
|
||||
## Measured and not landed
|
||||
|
||||
- **Record-publish batching:** publishing a validate's state records with the draw record. It came
|
||||
in within ±2 % on every arm, and it opens a deadlock class (a staging-retirement wait on a
|
||||
written but unpublished record).
|
||||
- **Native TLS (minSdk / ANDROID_PLATFORM 29):** +1.7 % on Espryt, noise on Magma (rd12 monolith).
|
||||
This is the user's call.
|
||||
- **Tracker / residual mutation-epoch fast path:** estimated at most 0.07 ms from the tracker alone,
|
||||
0.15-0.25 ms with the residual copies. Each step is below the 2 % bar, and a missed epoch bump
|
||||
renders stale. It is folded into the draw-packet design.
|
||||
Reference in New Issue
Block a user