Files
MobileGL/docs/Disaggregated/notes/perf-p15/PLAN-P15.md
T

814 lines
47 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# P15 plan: the FCL gap first, then a draw-delta fast path
Written 2026-10-07 on `feat/disaggregated` at `a0f4aa83`. Device: Lenovo Y700 (Adreno 750,
`HA27Q3LQ`), rooted, clocks pinned. Baseline: `dev` at `08124c99` (the pull monolith).
This is the plan, plus the measurements it rests on (section 2). Nothing here is implemented yet.
## 0. Summary
1. **FCL's "unexplained ~3 ms/frame wait" is display pacing, and dev has it too.**
- On every build, the presenting thread spends about 2.0 ms sleeping and 0.4 ms runnable per
frame. That holds for dev and feat, on both backends.
- The sleep is in `queueBuffer`. There the render thread waits for the `BLASTBufferQueue`
mutex, which the app's binder thread holds across a synchronous `setTransactionState`. That
call is phase-locked to the 165 Hz display.
- The whole FCL fps gap vs dev is presenting-thread CPU:
- Magma: 3.25 vs 1.85 ms/frame;
- Espryt: 2.7 vs 2.05 ms/frame.
2. **FCL's Magma CPU gap is mostly non-draw verbs on the wire arm.** These are mechanisms dev's
frontend arm had and the wire arm lost:
- every `glClear` builds and destroys its own image view, render pass and framebuffer;
- every blit creates a descriptor pool;
- `glBufferSubData` takes a slow staged path;
- the retired-object garbage collection and command-buffer reset grow to match.
This explains about 1.0 of the 1.4 ms. rd12 Magma pays the clear and churn part too, about
0.45 ms/frame.
3. **The (VAO, program) draw-packet table as sketched would never hit on rd12.**
- rd12 draws 1471 times a frame, with 1466 distinct (VAO, program) pairs per frame. A pair
repeats only across frames, and something outside the whitelist always runs between frames.
So a packet stamped with a context epoch hits 0.0 % of the time.
- What does hold on rd12 is the **draw-to-draw delta**: before 99.0 % of draws, the only calls
since the previous draw are `glBindVertexArray` and a float uniform.
- P15 therefore builds a draw-delta fast path:
- "nothing context-global moved since the last draw", checked once per draw against two
epochs (a client one and a server one);
- per-VAO and per-program slices that persist across frames.
It does not build a pair-keyed packet table.
4. **Swap interval 0 does not unlimit the frame rate today, and interval ≥1 vsyncs correctly.**
- MC 1.21.5 in FCL runs with vsync off (interval 0) and maxFps 260, which is MC's "unlimited".
- At interval 1, every build locks to 165.0 fps.
- At interval 0, every build is still partly display-coupled: the GL thread blocks ~2 ms/frame
in `queueBuffer`. That includes dev and the MobileGlues plugin.
- The cause is on the platform side. The app's own buffer-queue binder thread holds the queue
mutex across a SurfaceFlinger call, and SurfaceFlinger in turn waits synchronously on the
composer HAL.
- Stage P moves the present off the GL thread to a present worker, so interval 0 is CPU-bound
and interval ≥1 is still paced to the display (section 2.4).
Stage order:
- 0: parity audit + wire non-draw fast path;
- 0b: an FCL 1.21.5 trace fixture;
- 1: client draw-delta prototype;
- 2: per-VAO / per-program slices;
- 3: applier + backend draw-delta;
- 4: split arms, verify, default-on;
- P (swap-interval-correct presentation): **not needed for the target; dropped unless the user asks**
(2026-10-08). MobileGlues pays the same ~2 ms present wait, so matching it does not need P.
Decisions taken with the user (2026-10-08):
- Presenting-thread CPU ms/frame is the primary FCL metric, with fps beside it.
- Swap interval must be honoured properly. This is stage P.
- minSdk / native TLS is the user's own change; P15 does not touch it.
- The probe tools are committed under `tools/device_bench/disagg/`.
- No Espryt deferred-clear port: its clear and blit costs match dev's.
## 0.1 Target and stopping rule (user, 2026-10-08)
**Target:** "We only need to match MobileGlues' performance."
**Success criterion:** both backends reach the MobileGlues plugin (`com.fcl.plugin.mobileglues`)
in FCL MC 1.21.5, on the same pinned device, same world, same swap interval. Its presenting-thread
CPU there is ~1.7 ms/frame (~248 fps) at interval 0 and ~1.9 ms/frame at interval 1.
| build | interval 0 CPU ms/frame | needed |
|---|---|---|
| feat Magma | 3.12 | −1.4 |
| feat Espryt | 2.73 | −1.0 |
| dev Magma | 1.76 | (reference) |
| dev Espryt | 2.14 | (reference) |
Espryt has to go below dev here.
**Success criterion, revised by the user after the S0 milestone (2026-10-08): two levels on CPU,
MobileGlues on GPU.**
- **Floor (must meet): match dev** - the CPU bullet below.
- **Goal (design intent):**
- **Monolith:** presenting-thread CPU on par with or below MobileGlues. MobileGL's own overhead
is paid back by driver work it removes: the frontend tracks state, so the host driver
receives fewer calls, none of them redundant. Measure host-driver calls per frame and the
redundant share (same value re-set, already-bound object re-bound, unchanged uniform data) on
both backends, against MobileGlues' pass-through.
- **Inproc:** clearly faster than MobileGlues, because the GL thread only records and the apply
thread drives the backend and driver. Measure FCL inproc on both backends against MobileGlues,
with the time of both threads. If inproc is not faster, find out whether the cause is the GL
thread's own cost, the handoff/wake-up, or the apply thread being the bottleneck.
- **CPU-bound scenes:** at or below dev's monolith presenting-thread CPU in FCL MC 1.21.5 vanilla
(Magma <= 1.79, Espryt <= 2.14 ms/frame at interval 0), with dev measured in the same session and
~5 % noise tolerance.
- **GPU-bound shaderpack scenes (Iris + BSL):** within ~5 % of MobileGlues fps.
- **Why the CPU bar moved:** the S0 profiles show MobileGlues is a near pass-through, with 13 us/frame
of its own code against libMobileGL's 0.82 (Espryt) / ~1.0 ms (Magma). MobileGL's frontend state
machine and its record/validate/apply layers cost more than S1-S3 can recover, and even dev
monolith was 2.14 ms on Espryt (HANDOFF, S0 milestone).
- **Kept in scope:** MC's own code costs +0.17 ms/frame on MobileGL. MobileGlues advertises GL 4.0
and MobileGL 3.3 core; if MC takes a call-heavier path because of what we advertise, that
matters for both targets. Check extension strings and per-frame call counts.
**Stopping rule:** P15 ends when FCL presenting-thread CPU on both backends is at or below dev's
(within ~5 % noise, same session, same interval) and the BSL scene is within ~5 % of MobileGlues
fps on both backends. It also ends if a remaining gap is shown to be irreducible, with the reason
written down.
**Reference rows in every FCL milestone,** measured in the same session:
- MobileGlues (primary);
- dev (second).
MobileGlues also runs on the trace fixtures where the harness can load it.
## 1. Measurement rules for P15
- **FCL:**
- The primary metric is the presenting-thread CPU in ms/frame, which MobileGL's fps log already
prints. Report fps beside it, never alone.
- Until stage P lands, at interval 0 and above ~165 fps, frame time ≈ CPU + ~2.4 ms of
display-coupled wait (section 2.1). So fps ratios understate CPU ratios and are not linear in
CPU saved.
- Record the swap interval of every FCL reading (MC's `enableVsync` in
`/sdcard/FCL/.minecraft/versions/1.21.5/options.txt`; the default bench is `false`). Back the
file up before editing and restore it md5-checked.
- At interval ≥1 the per-frame CPU of every build rises by 0.3-0.7 ms (section 2.4), because the
thread idles between frames. Compare CPU only within one interval setting.
- Take readings only from in-world steady windows. The probe waits for MC's `Time elapsed:`
line and then four fps-log windows of more than 150 frames each.
- **GPU-bound scene (user, 2026-10-08):**
- FCL `1.21.4-Fabric` with Iris and `BSL_v10.1.3`, at the same world and interval, measured live
with `fcl_scene.sh bsl` and restored afterwards.
- The BSL fixture `minecraft-1.21.4-fabric-iris-bsl-in-world` joins the SSIM gate. It is a
single-frame trim, so it is a correctness check, not a benchmark.
- Show the scene is GPU-bound, with evidence:
- kgsl busy near 100 % (`gpu_clock_stats`, kgsl cmdbatch union from ftrace);
- the presenting thread waiting in fences or present;
- the `gsl_syncobj_wait` share.
- On GPU-bound runs, fps is the metric.
- A GPU regression on BSL blocks a change even if CPU improved.
- **GPU metrics in every milestone:**
- kgsl GPU busy % for every FCL run (`fclsum.py`);
- for GPU-bound runs, fps against dev and MobileGlues.
- **Load and store ops:** check every pass change in GPU terms. Prefer LOAD_OP_CLEAR or DONT_CARE
over loading and clearing on the tiler, and use DONT_CARE stores for depth/stencil that nothing
reads after the pass.
- **Trace (rd12, openra, and the FCL 1.21.5 fixture from stage 0b):**
- fps, best of 3.
- Per-layer CPU in ms/frame from `profile_p14.sh` / `layers.py`. Remember that rd12 carries
~6.9 ms/frame of harness parse.
- **Method:**
- `FINISH=false`, pinned clocks, best of 3.
- Re-run any surprising result before believing it.
- Every probe needs proof it engaged: a marker line, a counter, or a knob that visibly changes
behaviour.
- Every optimisation needs a negative control that can go red.
- **Clocks and validity (user, 2026-10-08; supersedes "pinned clocks" above):**
- **Pin:** CPU 2035.2 MHz on policies 2/5/7 and 1574.4 MHz on policy0; GPU 680 MHz
(`pin_clocks.sh` defaults). The device maxima (3.3 GHz, 903 MHz GPU) are not sustainable. The
point is kept only after a soak: about 10 min BSL plus 10 min FCL vanilla with the sampler, no
cap, temperatures well below the trip point. Otherwise step down one OPP (GPU 629/578, CPU 1920)
and soak again.
- **Profiles (user, 2026-10-08):** pick one per job (`PROFILE=<name>`, read by `pin_clocks.sh`) and
compare numbers only within one profile.
| profile | CPU big/prime | CPU little | GPU | use |
|---|---|---|---|---|
| `sustained` (default) | 2035.2 MHz | 1574.4 MHz | 680 MHz | all milestone and reference numbers |
| `gpumax` | 2035.2 | 1574.4 | 903 | show whether a scene is GPU-bound (not sustainable) |
| `gpulow` | 2035.2 | 1574.4 | 422 | amplify GPU cost |
| `cpulow` | 1286.4 | 1017.6 | 680 | amplify CPU differences |
| `cpuhunt` | <=1.5 GHz OPP | <=1.25 GHz OPP | 903 | CPU-overhead hunting (vanilla/rd12 gap, driver-call redundancy, S1-S3): cleanly CPU-bound |
| `gpuhunt` | <=2.4 GHz OPP | <=1.8 GHz OPP | 578 | GPU-side hunting (Espryt BSL gap, pass structure, load/store): cleanly GPU-bound |
Choose per question (user, 2026-10-08): CPU questions on a slow CPU with a fast GPU, GPU
questions on a fast CPU with a slow GPU. Show the binding evidence for each run (GPU busy %,
main-thread on-CPU share). A high profile only has to hold for one short, cool-started run, and
the validity check proves it did.
- **Every reported number carries its device state.** `devstate.sh` writes one line per run
(`state.txt`), and `fclsum.py` prints it under the results:
`STATE profile=<name>(cpu/little/gpu MHz) VALID|INVALID <observed min/max freq, peak temperatures>
daemons=<stopped(n)|running> mc=<version> swap=<enableVsync> anland=<on|off>`.
Invalid runs never enter a number.
- **Session (`bench_session.sh`, run by `devjob.sh` around every device job):**
- It records, then stops, the services that move frequency caps: `thermal-engine`,
`perf2-hal-1-0`, `vendor.perfservice`, `performance` and `hyperschedule_hal_service`.
- It releases their leftover requests in `/sys/kernel/msm_performance/parameters/cpu_{max,min}_freq`
and re-pins.
- On exit it starts every service that was running and verifies it is running again. The exit
runs on interrupt too (trap). Kernel trip points stay armed; if one engages (frequency
collapse or shutdown), benchmarking stops and it is reported.
- Why the requests matter: measured 2026-10-08, the perf stack's game-mode levels left caps of
787 / 1286 / 1075 / 1248 MHz in `msm_performance` that clamped `scaling_max_freq` under the
pin for about 50 min, while `pin_clocks.sh show` looked pinned.
- **Sampler:** `freq_sampler.sh` runs on the device every 2 s for the whole measurement window of
every probe. It records per-policy cur/max frequency, GPU devfreq cur and gpuclk, kgsl
`thermal_pwrlevel` / `throttling`, GPU busy, and the hottest CPU, GPU and skin zones.
- **Validity:** `freqcheck.py` decides. A run is valid only if every sample shows each policy at
the INTENDED pin (one OPP step of tolerance), no `scaling_max` cap, the GPU at its pin whenever
busy, and no kgsl thermal power level. Invalid runs are discarded and re-run after cool-down.
Every reported number carries its run's min/max frequency and peak temperatures
(`valid.txt`).
- **Before this rule:** P15 numbers up to the S0 milestone had no sampler. They ran at the
2035 MHz CPU / 903 MHz GPU pin as far as `time_in_state` shows, but that is not proof per run.
The references (dev, MobileGlues, current feat; vanilla and BSL; both backends; monolith and
inproc) are re-measured at the new clocks under this rule before any decision relies on them.
- **Device hygiene:**
- One device job at a time, under the `devjob.sh` lock.
- Restore FCL's library, config and properties afterwards, md5-checked (`fcl_p14.sh restore`).
- Stop the anland desktop during runs.
- FCL's SIGRTMIN+2 start-up death is known (DEBTS). The probe relaunches up to 4 times.
- **Gates for each landing:**
- SSIM 16/16 (`matrix_p14.sh ssim`);
- the retrace CI;
- the run-ahead and syncval gates on any Magma change that touches barriers or pass structure;
- an FCL soak before default-on.
## 2. Measurements behind the plan (2026-10-07)
Tools are in `tools/device_bench/disagg/` (README section "P15 FCL probes"):
- `fcl_probe.sh`: waits for in-world frames, then runs 4 s of ftrace (sched, kgsl cmdbatch, atrace
gfx) and 6 s of `simpleperf --trace-offcpu -p`;
- `offwait.py`: off-CPU time by open atrace slice and waker;
- `sfalign.py`: SurfaceFlinger transaction phase;
- `fcl_split.py` / `fcl_incl.py` / `fcl_drvcaller.py` / `fcl_stacks.py`: layer split, inclusive
functions, driver time by caller, and stacks;
- `fcl_count.sh` + `p15count.patch` (measurement-only): per-draw pair and gap statistics.
### 2.1 Item 1: where FCL's wait is
FCL MC 1.21.5, in world, view distance 2, monolith. The figures for the presenting thread
(`Thread-14`) come from the ftrace window. The off-CPU breakdown comes from the simpleperf window.
| build | frame ms | CPU ms | sleep ms | runnable ms | biggest sleep |
|---|---|---|---|---|---|
| dev Magma (×2) | 4.27 / 4.46 | 1.77 | 2.05 / 2.07 | 0.35 / 0.41 | QueuePresentKHR > queueBuffer, woken by the app's binder thread |
| feat Magma (×2) | 5.67 / 5.79 | 3.43 | 1.98 / 2.10 | 0.44 / 0.47 | same |
| dev Espryt | 4.54 | 2.10 | 1.92 | 0.40 | eglSwapBuffers > queueBuffer, same waker |
| feat Espryt | 5.03 | 2.73 | 1.81 | 0.39 | same |
- **The off-CPU stack is the same on all four builds:**
- The render thread: `BLASTBufferQueue::onFrameAvailable` → `std::mutex::lock`, 1.4-1.7
ms/frame. A further 0.2-0.34 ms/frame goes to `Fence::waitForever` in `queueBuffer`.
- The lock holder is the app binder thread: `releaseBufferCallback` →
`acquireNextBufferLocked` → `Transaction::apply` → `BpSurfaceComposer::setTransactionState`,
synchronous and under the BBQ mutex.
- **Inside SurfaceFlinger, `setTransactionState` has a median of 3.4 ms** (p10-p90 2.7-3.8). It
returns **4.95 ms after SF's `flushTransactions`** (p10-p90 4.78-5.2). The flush period is 6.1
ms (165 Hz, `NextFrameInterval 165_Hz`), with about one call per refresh. So the BBQ binder thread is
paced by the display. Above refresh rate, the render thread's present waits for it through the
mutex.
- **Other checks:**
- GPU busy is ~14 % (union of kgsl `adreno_cmdbatch_retired`).
- The swapchain is MAILBOX with 6 images, so present mode and image count are not the cause.
- **SurfaceFlinger's side:** its binder thread sleeps inside `setTransactionState` until it is
woken by `binder:1555_x`, a thread of `vendor.qti.hardware.display.composer-service`. That
thread is busy with its `HWCDisplay::CommitOrPrepare` cycle. So the composer HAL's display
cycle sets the phase.
- **Consequence:** this is not a MobileGL defect and not a feat regression. The fix for interval 0
is stage P (section 2.4).
### 2.2 FCL CPU by function (render thread, on-CPU ms/frame, inclusive)
**Magma monolith: feat 3.43 vs dev 1.77.** Repeat run: 3.25 vs 1.85 by the fps log.
| item | feat | dev | note |
|---|---|---|---|
| glClear → ClearWireFramebuffer | 0.42-0.46 | ~0.05 (Clear + ClearAttachmentsOnActiveRenderPass) | per clear: RetireWireDrawPass, vkCreateImageView + vkCreateRenderPass + vkCreateFramebuffer, its own pass, retire. 6 clears/frame, ~70 µs each |
| glBufferSubData → WriteWireBuffer / StagedWireRangeCopy | 0.28 | ~0 | 11 calls/frame, ~25 µs each; check whether it also ends the pass |
| CollectWireObjects (vkDestroyFramebuffer / DescriptorPool / ImageView) | 0.18 | none | the GC of the objects above |
| glBlitNamedFramebuffer → BlitWireColorImage | 0.13 | 0.02 | vkCreateDescriptorPool per blit (0.08) |
| TransitionWireImage | 0.14 | small | |
| BeginCommandRecording (vkResetCommandBuffer) + EndCommandRecording | 0.24 | 0.08 | larger, fragmented command buffers |
| glDrawElements (whole draw) | 0.93 | 0.56 | +0.37: client validate/emit ~0.27; SetupWireDraw 0.55 vs dev SetupDraw 0.54 |
| Present | 0.37 | 0.25 | |
| driver layer total | 1.46 | 0.49 | almost all the create/destroy above; not cache pressure |
**Espryt monolith: feat 2.73 vs dev 2.10.** The +0.62 splits as:
- client half +0.39;
- backend +0.09;
- driver +0.04;
- applier +0.035.
The clear/blit/sub-data path costs about the same as dev's: clear 0.16 vs 0.13, blit 0.125 vs
0.11, sub-data 0.024 vs 0.007. So Espryt has no churn of this kind (at most 0.07 ms/frame). Its
gap is the draw path: glDrawElements 1.01 vs 0.60, which is +4.7 µs per draw over 87 draws.
**rd12 Magma** (P14 `prof-fin` profiles, scaled to ms/frame):
- ClearWireFramebuffer ≈ 0.27, CollectWireObjects ≈ 0.18, BeginCommandRecording 0.12 (dev 0.02),
blit 0.09 (dev 0.06).
- So stage 0 also pays on rd12: about 0.4-0.5 ms/frame.
- Dev's deferred-clear lookups cost about 0.6 ms/frame on rd12: `VkClearManager::HasPendingClear`
is probed per draw, plus the key find. A naive port would be a regression there. Pending clears
must be resolved once per pass begin, never per draw.
### 2.3 Draw-pair and gap statistics (P15Count, frontend entry level)
What each column measures:
- **pairs:** distinct (VAO lifetime id, program lifetime id) pairs per frame.
- **same as prev:** the previous draw had the same pair.
- **pair epoch hit:** the pair has drawn before, and no non-whitelisted GL entry has run since its
last draw. This is the packet-table hit rate.
- **prev-draw gap clean:** no non-whitelisted entry since the previous draw, whichever pair it
had. This is the draw-delta hit rate.
The whitelist is draws, `glBindVertexArray`, float uniforms and `glUseProgram`. "+buffer" adds
buffer binds and vertex-buffer binds.
| workload | draws/frame | GL calls/frame | pairs/frame (ever) | same as prev | pair epoch hit, whitelist / +buffer | prev-draw gap clean, whitelist / +buffer |
|---|---|---|---|---|---|---|
| rd12 (steady window) | 1471 | 4856 | 1466 (1473) | 0.3 % | 0.0 % / 0.0 % | **99.0 %** / 99.0 % |
| FCL MC 1.21.5 (10 windows, both backends) | 88 | 873 | 15.1 (27) | 82.9 % | 0.0 % / 80.6 % | 0.0 % / **80.6 %** |
| openra | 29 | 593 | 2 (2) | 93 % | 0 % / 0 % | 0 % / 0 % |
Engagement and controls:
- Allowing every category gives 99.9-100 %, which is the positive control.
- The FCL numbers are identical on Espryt and Magma, as expected for frontend counts.
What sits between draws:
- **rd12:** per draw, just BindVertexArray + Uniform3fv + DrawElements. The ~1 % of other gaps
carry Enable/Disable, BlendFunc, BindTexture, Uniform1i and similar.
- **FCL:**
- `BindBuffer` comes before 98.9 % of draws and `BindVertexBuffer` before 88.5 %. Both are
VAO-scoped state: the element buffer and the vertex binding.
- 19 % of draws also follow `BindFramebuffer` / `Viewport` / `PolygonMode`.
- MC issues 96 `BindFramebuffer` calls per frame, mostly redundant.
- **openra:** every draw changes blend, texture, attrib pointers and sub-data. No per-draw
shortcut applies, and none is needed: openra is not where the gap is.
### 2.4 Swap interval: what is requested, what each backend does, what happens
**What MC requests:**
- The bench world runs with `enableVsync:false` and `maxFps:260` (MC's "unlimited"). So the app
asks for interval 0 through FCL's GLFW → `eglSwapInterval`.
- That interval 0 reaches Espryt's host EGL is shown by behaviour: the host default of 1 would cap
at 165 fps, but Espryt runs at 194-225.
- For Magma, `enableVsync:true` logs `Picked present mode: VK_PRESENT_MODE_FIFO_KHR` and locks to
165 fps, so the interval is plumbed.
**How each backend honours it today:**
- **Espryt:**
- `EGLImpl::SwapInterval` → `BackendObject::SetEGLSwapInterval` → `DirectGLES::SetSwapInterval`
→ host `eglSwapInterval`. On Android that is `ANativeWindow` `setSwapInterval`; 0 is the
async queue mode.
- The interval is re-applied when the window surface is recreated.
- **Magma:**
- `DirectVulkan::SetSwapInterval` → `VulkanRenderer::SetSwapInterval`.
- A changed interval recreates the swapchain between frames.
- `ChooseSwapchainPresentMode`: 0 → IMMEDIATE, then MAILBOX, then FIFO; ≥1 → FIFO; negative →
FIFO_RELAXED, then FIFO; no request → MAILBOX first.
- Android offers no IMMEDIATE here, so interval 0 is MAILBOX with 6 images.
- Magma creates one queue. `m_presentQueue` is the same queue as `m_graphicsQueue` when the
families match.
**Measured, FCL, monolith, same session, mean of 4 fps-log windows** (presenting-thread CPU in
ms/frame):
| build | interval 1: fps | interval 1: CPU | interval 0: fps | interval 0: CPU | interval 0: GL-thread sleep | interval 0: runnable |
|---|---|---|---|---|---|---|
| dev Magma | 164.9 | 2.43 | 235.5 | 1.76 | 2.02 | 0.35 |
| feat Magma | 163.2 | 2.92 | 172.8 | 3.12 | 2.02 | 0.44 |
| dev Espryt | 164.9 | 2.37 | 216.5 | 2.14 | 1.94 | 0.36 |
| feat Espryt | 163.4 | 2.72 | 194.3 | 2.73 | 1.87 | 0.41 |
| MobileGlues plugin (control) | 164.9 | ~1.9 | 248.4 | ~1.7 | 1.98 | 0.34 |
- The MobileGlues CPU figures come from schedstat (the plugin has no fps log).
- **Interval 1 is correct everywhere:** FIFO, or host interval 1. The waits sit in
`dequeueBuffer` / `AcquireNextImageKHR` and in the render thread's own `setTransactionState`.
- **Interval 0 is wrong everywhere,** including dev and the control. The GL thread still spends
~2 ms/frame in `queueBuffer` → `BLASTBufferQueue::onFrameAvailable`, waiting for the mutex that
the app's binder thread holds across `setTransactionState`, and that call is phase-locked to the
composer HAL (section 2.1).
- **The async queue mode does not help.** It changes what happens to the buffers, not who holds the
mutex.
## 3. Stages
### Stage 0: parity audit, then the wire non-draw fast path (Magma first)
**0.1 Parity audit (first step, judgement work).**
- **What to list:** every performance mechanism of dev's frontend arm that the record and wire
arms lost when P13 W4 moved monolith onto the wire arm and W6e / `de6ab167` deleted the old
code as dead. The split arm (the anland desktop) has always paid for them.
- **References:** `git show origin/dev:<path>` and the pre-W6 feat tree `88a11b81`.
- **Seed inventory:** [PARITY-SEED.md](PARITY-SEED.md) lists the definitions in OLD but not NEW
for 17 backend files (regex-extracted, approximate). The audit's verdicts go in
`perf-p15/PARITY-AUDIT.md`.
- **For each mechanism, record:**
- what it did and its key;
- whether the wire/record arm has an equivalent, and where;
- whether FCL or rd12 shows its absence (section 2.2 numbers);
- porting cost and correctness risk;
- a verdict.
Seed rows (to be verified by the audit):
| mechanism (dev / 88a11b81) | wire/record arm today | shows in profiles | first estimate |
|---|---|---|---|
| `VkClearManager`: glClear queues a pending clear per attachment (texture + lifetime id + level/layer); the next pass on that attachment starts with LOAD_OP_CLEAR; materialised before reads, blits and copies | none: ClearWireFramebuffer runs a standalone pass with fresh objects | FCL 0.42, rd12 0.27 ms | **port, per pass begin**: dev's per-draw probe cost 0.6 ms on rd12 |
| `VkRenderPassManager::GetOrCreateRenderPass` hash cache of render pass + framebuffer, with OnPresent GC | partial: wire pass continuation, RenderPassEntry; framebuffers still created in SetupWireDraw (0.04) and in clears | FCL 0.04-0.19 (create) + 0.12 (destroy) | port a cache keyed on attachment lifetime ids + image epoch |
| shader blit to default FB (`TryBlitToDefaultFramebufferWithShader`), persistent blit resources (`InitializeBlitResources`) | BlitWireColorImage creates a descriptor pool per blit | FCL 0.08 + GC | persistent pool / set cache per (pipeline, source view) |
| `VkBufferManager::OnSubData` / `OnResidentSubData` (writes into resident or streamed slices) | WriteWireBuffer + StagedWireRangeCopy | FCL 0.28 | find out why it is ~25 µs per call; port the resident fast path; check for pass breaks |
| `TrySetupDrawFastPath` / `SetupDrawSnapshot` (4 snapshots keyed on program lifetime id) | P14 memos (pass continuation, unit samplers, pipeline/vertex/index bind skips) | SetupWireDraw ≈ dev SetupDraw on FCL | audit for leftovers only |
| `UniformManager` `FastRebindMemo`, `SampledBindingsUnchanged` | P14 descriptor-set rebind, walk-skip | small | audit |
| `VertexInputStateFactory` per-VAO memos (`VaoBackendMemos`) | vertex-input layout kept on its record (P14) | small | audit |
| `VkTextureManager` `SyncedTextureMemoEntry` / `DrawSyncScope` | P14 direct-mapped texture fronts | small | audit |
| Espryt `StateBackendObjectRegistry` (pointer-keyed twins), `SyncCurrentProgram`, `SyncVaoAttributeBuffersByHandle`, `BackendFramebufferObject::SyncToBackend` | twins keyed on applier serials (P14) | Espryt backend +0.09 on FCL | audit; low priority |
**0.2 Port, in order of measured value:**
1. **Deferred clears on the wire arm.**
- Key: image lifetime id + level/layer range + aspect, never a VkImage or a GL name.
- A clear records a pending entry. The next wire pass that targets the image takes it as
LOAD_OP_CLEAR.
- A clear inside an open pass on the same attachments becomes `vkCmdClearAttachments`, without
ending the pass.
- Materialise pending clears before any read: sampling, readback, blit source, copy, mipmap
generation, shared-image export and present. The cost is one check per pass begin and one
per read verb; no per-draw probe.
- The clear-mask cases (partial colour mask, stencil write mask, scissor) keep the current path.
2. **A render-pass + framebuffer cache** for wire passes and clears.
- Keyed on attachment lifetime ids and the image epoch.
- Retired when an attachment's lifetime ends: the store destroy epoch.
3. **Blit descriptor reuse:** a persistent pool, sets keyed on (source view lifetime id, sampler).
4. **The `glBufferSubData` path:** find the ~25 µs per call, then port the resident-slice write
and check whether it ends the pass.
5. **Command-buffer churn:** once 1-4 land, re-measure `vkResetCommandBuffer` /
`EndCommandRecording`. They should fall with the pass count.
**Espryt:** no churn of this kind measured (≤0.07 ms/frame on FCL). The audit should still cover
Espryt's deleted mechanisms. Port only those that show in profiles.
**Correctness:**
- Deferred clears are the e8e1521 class of risk: a read that skips materialisation renders stale
content.
- Tests:
- an integration scenario per read path (sample after clear; readback; blit source; copy; mipmap;
present), with negative controls that drop materialisation for one read kind and must go red;
- SSIM 16/16;
- the syncval gates (a LOAD_OP_CLEAR changes the pass's first access).
- Runtime knob, e.g. `MOBILEGL_MAGMA_DEFERRED_CLEARS=0`, kept for one release.
**Expected gain:**
- FCL Magma: 0.8-1.0 of the 1.4 ms presenting-thread CPU gap. Medium-high confidence: the items
are measured, the removal fraction is estimated.
- rd12 Magma: 0.3-0.5 ms/frame, about +3-5 % fps. Medium confidence.
### Stage 0b: an FCL MC 1.21.5 trace fixture
- **Why:** rd12 (MC 1.21.4) never exercised the per-frame clear, blit and BufferSubData pattern
that FCL's MC 1.21.5 runs. The trace matrix must cover what FCL runs.
- **Capture:**
- with the on-device FCL capture wrapper (`tools/trace_replay/skills/trace-fixture-authoring-on-android-fcl`);
- the same world and view distance as the FCL bench, from the in-world steady state;
- trace mode forces 854×480.
- **Trim:** a multi-frame window, like rd12 (`gltrim -f`, about 250 frames), so the benchmark
measures steady frames. The golden is the final frame.
- **Golden:**
- follow `trace-fixture-authoring/SKILL.md`: generate it on a real GPU (Windows `glretrace` on
the RTX 3080, or a known-good Android replay); never on llvmpipe for anything heavy;
- check its content against a live-run screenshot;
- remove alpha; `repack --brotli`; register with `ci: false` like rd12.
- **Use:**
- add to `matrix_p14.sh` `wl_args`;
- A/B every stage on rd12, openra and this fixture;
- check that P15Count on the fixture reproduces section 2.3's FCL row (88 draws, 15 pairs); if
not, the fixture does not represent FCL.
- **Order:** best done before stage 0.2 lands, so stage 0 can be A/B'd on a trace as well as on FCL.
### Stage P: swap-interval-correct presentation (both backends)
**Status: not needed for the target; dropped unless the user asks (2026-10-08).** MobileGlues
pays the same ~2 ms present wait, so the target (section 0.1) does not need this stage.
**Goal:**
- With interval 0, presentation never paces the app: FCL fps becomes 1 / presenting-thread CPU.
- With interval ≥1 the app is still paced to the display: 165 fps ±1 % on this device, with no
extra latency beyond one queued frame.
**Design: a present worker that owns the blocking present.**
- The GL thread finishes a frame and hands it to the worker through a bounded queue (depth 1 at
interval ≥1, depth 2 at 0). It returns at once.
- The worker performs the present call that may block.
- At interval ≥1, backpressure comes from the full queue: the GL thread waits for the worker,
which waits for the display. At interval 0 the queue never fills, because the worker's blocked
windows (≤3.4 ms, about once per refresh) are absorbed by the swapchain's spare images.
- The worker exists only for window surfaces. Pbuffer and offscreen surfaces keep the inline path.
**Magma:**
- The GL thread keeps `vkAcquireNextImageKHR` and the submit, which signals the image's
render-done semaphore.
- The worker calls `vkQueuePresentKHR`.
- Queue access must be externally synchronised, and the WSI's `queueBuffer` runs inside
`vkQueuePresentKHR`. So the worker needs **its own VkQueue**: a second queue of the graphics
family, or a present-capable family.
- Check the Adreno queue count first.
- If only one queue exists, the fallback is a queue mutex released before the WSI blocks. That
is not possible from outside the loader, so a single-queue device keeps the inline present
and logs it once.
- Swapchain recreation (resize, interval change, `OUT_OF_DATE`) drains the worker first.
- The frame-in-flight fences already gate the reuse of images.
**Espryt:** `eglSwapBuffers` is bound to the context's thread, so the prototype compares two routes:
- **(a) Own the native window.**
- Dequeue `ANativeWindowBuffer`s on the GL thread (or worker) and wrap each as an EGLImage
(`EGL_NATIVE_BUFFER_ANDROID`, cached by buffer identity: generation + id, never a pointer).
- Render the default framebuffer into it.
- The worker queues it with an `EGL_ANDROID_native_fence_sync` fd.
- This route also needs buffer count, transform / pre-rotation, damage, buffer age and
`setSwapInterval` on the window itself.
- No extra blit. The anland shared-image route already does part of this.
- **(b) A present thread with a shared context.**
- The default framebuffer becomes an offscreen FBO.
- The present thread waits on a fence, blits to its window surface and calls `eglSwapBuffers`.
- Costs one full-screen blit per frame (800×1280; small on Adreno 750), a second context, and
redirecting default-framebuffer reads.
- Prototype (a) first.
**Split arms:**
- The server presents on the apply thread (inproc) or in the server process (spawn/tcp). The
worker sits there.
- The present credit is returned when the frame is handed to the worker, not when `queueBuffer`
returns.
**Tests:**
- interval 0 → fps bounded by CPU, and the GL thread never sleeps in `queueBuffer`
(`offwait.py` shows no `queueBuffer` sleep on the GL thread);
- interval 1 → 165 fps ±1 %, and so is interval 2 → 82.5;
- interval changes at runtime (MC's vsync toggle);
- surface resize / rotation; eglMakeCurrent away and back with frames queued;
- teardown and device loss with a frame in the worker;
- the anland desktop (a split-arm present on the same device);
- negative control: a knob that makes the worker present inline must bring the ~2 ms
`queueBuffer` sleep back.
- Kill switch: `MOBILEGL_PRESENT_WORKER=0`.
**Expected at FCL interval 0** (presenting-thread CPU is unchanged; fps approaches 1/CPU):
| build | today | after stage P | after stage 0 as well |
|---|---|---|---|
| dev Magma | ~236 | ~450-550 | |
| feat Magma | ~173 | ~300-330 | ~450 |
| feat Espryt | ~194 | ~350 | |
At interval ≥1 nothing changes. Confidence: medium. It depends on the second queue (Magma) and on
route (a) working (Espryt).
### Stage 1: client draw-delta prototype (Espryt monolith)
This is the client half of the old sketch, re-keyed on what section 2.3 shows.
- **Client context epoch:**
- one `Uint64` per context;
- bumped by every GL/EGL entry except a whitelist;
- share-group-scoped objects (textures, buffers, programs, renderbuffers, samplers) bump a
share-group epoch that every context in the group checks.
- **Whitelist classes,** decided per entry in a checked-in table (section 4.4):
- **free:** draws, `glBindVertexArray`, float `glUniform*` / `glProgramUniform*`, `glUseProgram`
(the program is a key, not context state);
- **VAO-scoped:** `glBindVertexBuffer`, `glVertexAttrib*Pointer`, `glEnable/DisableVertexAttribArray`,
`glBindBuffer(GL_ELEMENT_ARRAY_BUFFER)` and the VAO DSA calls. These bump the bound VAO's own
versions, which the per-VAO slice checks. `glBindBuffer(GL_ARRAY_BUFFER)` changes nothing a draw
reads;
- **program-scoped:** integer uniforms and block bindings. These bump the program's backend-state
/ block-binding version;
- **everything else** bumps the epoch.
- **Redundant binds** (same object, same value) may skip the bump only where the entry already
compares. MC issues 96 BindFramebuffer calls per frame and 19 % of draws follow one.
- **At the validate point:**
- if (epoch, share-group epoch, context serial, slot reset epoch) are unchanged since the
previous draw, skip `Tracker::Update`'s context-global shutters, the residual copy and the
context-value emit;
- run only the VAO-derived shutters (through the per-VAO slice once stage 2 lands), the
program-derived ones, and the uniform span;
- also skip `ReadDrawBindings` / `MarkGpuWritesForDraw` when no persistent map is live and no
image or SSBO write is bound. The persistent-map re-sync stays a per-draw input otherwise.
- **Kill switch:** `MOBILEGL_PIPE_DRAW_DELTA=0`, default off until stage 4.
- **Measure** on rd12, FCL and the stage 0b fixture against P14's per-item numbers: Tracker::Update
0.12, CopyField 0.10, validate self 0.10, ReadDrawBindings 0.06, MarkGpuWritesForDraw 0.05,
thread-local/atomic 0.07 ms/frame.
- **Expected:** 0.3-0.5 ms/frame on rd12 Espryt monolith (91 → ~94-96 fps), and 0.1-0.25 ms on FCL
Espryt. Medium confidence.
- **Decide** on stages 2-3 from this.
### Stage 2: per-VAO and per-program slices
- **Per-VAO slice.** This is what the old "packet" becomes. It extends `VaoMemo`, which is already
keyed on VAO lifetime id + slot reset epoch.
- It holds the vertex-elements handle + published config version, the buffer set + index
handles + hash, the stored-on-server mirrors, and the backend's per-VAO answers (stage 3).
- It is reached by an index stored on the frontend VAO object, not a hash probe.
- rd12 rebinds a different VAO for every draw, but each VAO is seen every frame and its config
hardly moves. So the slice turns EmitVertexElements (0.13) / EmitVertexBuffers (0.07) into
a version compare plus a handle reuse.
- **Per-program slice:** the shader CSO handle, link and backend-state versions, the
uniform-span layout and the UBO block size.
- **Expected:**
- rd12: 0.1-0.2 ms on the client half;
- FCL: small, because only 27 pairs exist. But each FCL draw also changes the vertex buffer
binding, which is now a VAO-version bump that the slice re-derives cheaply.
### Stage 3: applier and backend draw-delta
- **Server epoch, generated from `PipeCalls.def`:**
- The applier bumps an epoch for every applied record except a whitelist: draw,
bind_vertex_elements, the uniform span, and the per-VAO / per-program record kinds.
- A constexpr table is generated from the record opcode list. A test fails if a record kind is
missing from it. This is the server-side twin of section 4.4.
- **Backend-internal mutations also bump a backend epoch.** These are:
- texture `Serial` / `ParamsSerial` / `PendingUploads`;
- twin force-resync flags;
- the sampler CSO serial;
- shared-image state;
- GPU-written marks;
- on Magma: the image epoch, the resource-erase epoch, an image layout leaving GENERAL, and the
store-destroy epoch.
The rule from the content-version lesson applies: every place that changes a twin's synced
state bumps it, including `glGenerateMipmap` growing mips. P14's Magma memos already check most of these.
- **When both epochs are unchanged since the previous draw,** the backend skips the
context-global walks:
- Espryt: `SyncNeccessaryTextures` / `SyncNeccessaryBuffers`, `AdoptUnchangedBufferSets`,
`BindCurrentProgramWithResources`;
- Magma: whatever of SetupWireDraw is not VAO- or program-derived.
- **Expected:**
- rd12 Espryt backend: 0.15-0.3 ms of the +0.45;
- rd12 Magma: 0.1-0.3 ms (its backend is already below dev);
- driver: 0-0.3 ms, if the cache-pressure reading is right. Measure L1D refills before and
after with `stat_p14.sh`.
### Stage 4: split arms, verify, default-on
- **Inproc / spawn / tcp:**
- the client epoch and the per-VAO/program client slices live in the client;
- the server epoch and the backend slices live where the applier runs: the same thread on
monolith, the apply thread on inproc, the server process on spawn and tcp.
- Nothing new crosses the wire.
- Under run-ahead, client validity depends only on client epochs. The server computes its own
epoch from the records it applies, so the two cannot disagree about what moved.
- **Default-on** after:
- CI and SSIM green on all arms;
- verify lanes green with the shadow check (section 4.5);
- an FCL soak on both backends.
Keep the knob for one release.
- **Expected inproc:**
- Espryt rd12 is client-bound, so stages 1-2 give 112 → ~116-119, above dev's 115.
- Magma inproc is apply-bound, so it gains from stages 0 and 3.
## 4. Design details
### 4.1 Scope and non-goals
**In scope:**
- the stage 0 wire non-draw fast path (Magma);
- the client and server draw-delta epochs;
- per-VAO and per-program slices;
- the backend draw-delta.
**Non-goals:**
- a (VAO, program) pair table: 0 % hit on rd12, and FCL's hits are covered by the draw delta;
- record batching (measured and dropped in P14);
- protocol changes;
- native TLS / minSdk (the user's own change);
- openra, whose gaps change state on every draw.
### 4.2 Per-layer state for one draw on the fast path
| layer | read per draw on a hit | kept in |
|---|---|---|
| tracker | client epoch + share-group epoch + context serial + slot reset epoch (4 compares); VAO identity (lifetime id + config + attribute generation); program identity (lifetime id + link + backend-state + block-binding versions) | context, per-VAO slice, per-program slice |
| emitter | per-VAO slice: elements / buffers / index handles + versions; uniform span; draw record | per-VAO slice |
| applier | server epoch; vertex-elements record by slot + generation (bind by reference) | applier context |
| backend | backend epoch; Espryt VAO twin + program twin pointers; Magma vertex-input id, pipeline handle, descriptor-set layout, pass-compatibility id | per-VAO / per-program backend slices |
### 4.3 Keys
- Use monotonic lifetime ids only: VAO, program, buffer and texture lifetime ids; slot +
generation on the server side. Never a GL name or a pointer. This is the ABA rule, broken three
times so far (sampler, texture, program).
- Slices are invalidated by:
- the context serial;
- `MGPipeSlots().ResetEpoch()`;
- the store-destroy epoch.
- A slice index stored on a frontend object is valid only together with the object's lifetime id.
### 4.4 Invalidation sources, and how to prove none is missed
**Client entry-point table** (`MG_Pipe/DrawDeltaClass.def`, new):
- one row per exported GL entry, giving its class (free / VAO-scoped / program-scoped /
share-group / context);
- `DECLARE_GL_FUNCTION_END*` passes `#name` to `GLStreamScope`, which looks up the class at
compile time;
- the hand-written `MOBILEGL_GL_API` definitions (~170 in `Definitions.cpp`, mostly EXT/ARB
aliases plus `glFinish`/`glFlush`) and the EGL entries use the same scope helper.
**Completeness tests:**
1. A CTest scans `Definitions.cpp` and the EGL exports. It fails if:
- any exported entry is missing from the table;
- a table row names a non-exported entry;
- any `MOBILEGL_GL_API` body neither constructs the scope nor forwards to a macro entry.
2. A runtime self-test: in a debug build, every entry asserts it consulted the table. A fuzz
driver walks all entries.
3. **Negative controls:** a test-only knob `MOBILEGL_PIPE_DRAW_DELTA_DROP=<entry>` removes one
entry's bump. Run for a representative entry per class:
- `glBindTexture`;
- `glTexParameteri`;
- `glBindFramebuffer`;
- `glEnable`;
- `glBindBufferBase`;
- `glUniform1i` (program-scoped);
- `glBindVertexBuffer` (VAO-scoped);
- an EXT alias;
- an EGL entry.
Each must turn the shadow check (4.5) and a scenario red.
**Mutations that bypass GL entries:**
- persistent-mapped writes stay a per-draw input;
- async link completion (link version);
- `eglMakeCurrent` (context serial);
- another context in the share group (share-group epoch);
- server-side backend mutations (backend epoch, 3.3).
**Server side:**
- the generated table from `PipeCalls.def`, with the same three tests;
- for the backend epoch, a list of every writer of the synced-state fields, enforced by making
those fields private behind a setter that bumps. This is the same chokepoint design as the
Magma layout setter in the whole-draw fast-path note.
### 4.5 The verify comparator
`MOBILEGL_PIPE_VERIFY` compares the residual block with a live snapshot at the verb.
Under verify:
- the fast path is armed but always takes the slow path;
- it asserts that the fast path would have hit with byte-identical residual, record and
vertex-element output (a **shadow check**).
So every verify lane cross-checks epoch coverage for free, and the negative controls in 4.4
redden it.
A non-verify build has a sampled shadow mode, `MOBILEGL_PIPE_DRAW_DELTA_SHADOW=N`, which checks
every Nth hit. It is for FCL soaks.
POISON stamps (trace builds): on a hit, the stamp set recorded at the last slow draw is applied in
one bulk write. Freshness checks then keep working.
### 4.6 Tests
- Unit tests for the epoch classes; the table tests (4.4).
- Integration scenarios:
- draw-delta hit/miss per class;
- share-group mutation from a second context;
- persistent-map write between two hit draws;
- async link completing between draws;
- VAO rebind with the same lifetime id after delete + gen (ABA);
- context switch;
- slot reset.
Each runs with a negative control.
- Stage 0: deferred-clear read-path scenarios with drop-materialisation controls; syncval.
- All: SSIM 16/16, the retrace CI, FCL soak.
## 5. Expected gains
ms/frame of CPU on the bound thread.
| stage | rd12 Espryt mono | rd12 Magma mono | FCL Espryt mono (presenting CPU) | FCL Magma mono (presenting CPU) | confidence |
|---|---|---|---|---|---|
| 0 wire non-draw | 0-0.07 | 0.3-0.5 | ≤0.07 | 0.8-1.0 | medium-high (FCL), medium (rd12) |
| 1 client draw-delta | 0.3-0.5 | 0.3-0.5 | 0.1-0.25 | 0.1-0.25 | medium |
| 2 slices | 0.1-0.2 | 0.1-0.2 | ~0.05 | ~0.05 | low-medium |
| 3 backend draw-delta | 0.15-0.3 (+ driver 0-0.3) | 0.1-0.3 | ~0.05 | 0.05-0.1 | low-medium; driver unproven |
| total | 0.55-1.0 (+driver), gap 2.5 | 0.8-1.5, gap 0.76 | 0.25-0.4, gap 0.62 | 1.0-1.4, gap 1.4 | |
What the totals mean:
- **rd12 Espryt monolith:** 91 → roughly 96-101 fps. Short of dev's 115 unless the driver term
moves.
- **rd12 Magma monolith:** 84 → about dev's 87 or above.
- **FCL Magma:** CPU gap mostly closed. Espryt about halved.
- **Inproc:** above dev on Espryt rd12. Magma inproc gains stages 0 + 3.
- **Revision:** these are below the old sketch's 1.0-1.6 ms because the pair table does not hit on
rd12.
## 6. Risks
- **A missed epoch bump renders stale state.** This is the dominant risk. Mitigations:
- the generated tables and their completeness tests;
- the verify shadow check;
- per-class negative controls;
- SSIM and the retrace CI;
- default-off until stage 4.
- **Deferred clears skipping a materialisation.** Same class as the reverted e8e1521. Mitigations:
per-read-path scenarios with drop controls, syncval, SSIM.
- **ABA on slice keys.** Mitigations: lifetime ids, slot + generation, the reset and destroy
epochs.
- **Share groups and multiple sessions:** per-context epochs miss a mutation from another context
in the group. Mitigation: the share-group epoch. The multi-session server hazards also apply on
the split arms.
- **Gains below estimate:** the stage 1 prototype is the go/no-go for stages 2-3.
- **FCL fixture fidelity:** stage 0b's P15Count check.
## 7. Questions resolved with the user (2026-10-08)
1. FCL metric: presenting-thread CPU ms/frame is primary, with fps beside it.
2. Swap interval: interval 0 must unlimit the frame rate, and interval ≥1 must vsync. This is
stage P.
3. minSdk 29 / native TLS: the user decides and changes it. P15 does not touch it.
4. The probe tools are committed under `tools/device_bench/disagg/`.
5. No Espryt deferred-clear port: Espryt's clear and blit costs match dev's.
Still open (for stage P): does Adreno 750 expose a second graphics-family queue for the Magma
present worker? Which Espryt route, (a) or (b)?