[Docs] (Disaggregated): P6's plan and contract draft; the working notes move from a local directory into the repository

Two things land together: the plan and contract draft P6 needs before it starts, and the
working notes that until now existed on exactly one machine.

1. P6's plan and contract draft (neither started; not one line of code)

     docs/Disaggregated/P6-SPAWN-PLAN.md      the package plan
     docs/Disaggregated/P6-CONTRACT-DRAFT.md  the contract draft (moves to
                                              MG_Remote/CONTRACT-P6.md when c6 lands)

   The three findings that mattered, all written into the documents:

   a. P6 is SMALLER than its roadmap cell suggests. The transport primitives are already in
      the tree and already tested: SocketDoorbell including hangup detection and the two
      lost-wakeup fences (Doorbell.cpp:211-360, four cases in FdPassingTest), SCM_RIGHTS fd
      passing, ShmSegment::Adopt, and the SurfaceOp / WindowKind schema already defined in
      protocol.fbs. inproc uses a real second shm mapping rather than an alias, and
      SessionRings.h:22-28 says that was done so today is not the first time it runs. P6 is
      assembly plus a control plane plus a process, not "write a socket transport".

   b. The evidence for "P6 is only a transport swap" has EXPIRED. It was bought by P5c, whose
      audit found 59 direct memory accesses in a tree that already looked clean. P5e changed
      the tree again. So the first package, a6, is a read-only audit that writes no code, and
      every contract row that depends on it is marked PENDING a6 rather than decided in
      advance.

   c. P6 does not get a game running on the phone. ANativeWindow* is meaningless in the child,
      and protocol.fbs:186-196's own comment says Android transfers the window out of band -
      that band is P12. P6 lands pbuffer / surfaceless / offscreen, which is what HeadlessGL,
      the split-equivalent lane and trace replay need, and which is exactly what P6's own exit
      gate measures. A real window reaching the server is a named refusal. This gets its own
      section because "P6 is done" would otherwise read as "the game runs now".

   The contract adds rule G: a role may not name a process-local handle. It extends P5c's
   rule E (a role may not name the other role's memory) from memory to identity - EGLDisplay /
   EGLSurface / EGLContext / ANativeWindow* / fd number / pid / mapped address either cross as
   a token this contract defines or they do not cross, and a descriptor crosses only through
   SCM_RIGHTS, where the kernel renumbers it.

   What P5e left for P6 gets its own section. The latch semantics are already designed
   (ARCHITECTURE.md:504) and are copied unchanged. What is new is that after run-ahead the
   client no longer waits while the peer process can DIE: the client holds records that were
   published and will never be applied, and Android's freezer suspends the child without
   closing anything. So slow and dead must be distinguishable WITHOUT a timeout threshold.
   That is P6's only new design; everything else is carriage. It shares a predicate with E1's
   outstanding re-specification (ID-122), and the two should be designed together.

   Two gates carried over from P5e's lessons: G2 requires integration-spawn's test-name set to
   match integration-split's name for name, because a differing set means the lane is quietly
   running less; and the arm-proof gate requires every entry to record the child pid and
   transport=spawn in its OWN private log - ID-124 established that this kind of false green
   really happens, and a spawn lane that actually ran monolith would pass every other gate.
   S2's shape must be settled before it is written, and the document says why: E1 is the
   cautionary case.

2. The working notes move into the repository (docs/Disaggregated/notes/, 415 .md files)

   ROADMAP / CURRENT_STAGE_PROGRESS / MEASUREMENTS cite ruling numbers, audit line numbers and
   the provenance of gate figures from ~/w7/notes/, a directory that exists on one machine and
   is under no version control. A record cited hundreds of times that lives only locally is not
   a record, so it moves in whole, with its directory structure intact.

   Only .md moves. Scripts, run logs, census and gate result JSON, patches, APKs and nm dumps
   stay where they are: they are evidence DATA rather than documents - large, regenerated per
   run, and mostly carrying absolute paths and device serials that mean something only on the
   machine that produced them. The mapping and this split are written in
   docs/Disaggregated/notes/README.md.

   Two exceptions are recorded there too: P6's BRIEF has been promoted to a first-class
   document and keeps no copy under notes/, so the two cannot drift apart; and recovered/ holds
   material salvaged from failed workflow runs, which may duplicate the like-named phase
   directories and is kept because a few of those were the only complete version at the time.

   CI's citation lint takes docs/Disaggregated/*.md only (non-recursive), and notes/ is
   deliberately outside it: those notes' file:line citations were written against their own
   code heads, and resolving them against today's HEAD would fail wholesale - which is not an
   error, it is their correct state as a historical record. To check one, use the revision it
   declares (--rev).

3. Inconsistencies fixed in passing

   - README.md's status line still said P5e was in progress while the stage table already said
     closed.
   - CURRENT_STAGE_PROGRESS's next-step item 1 still listed the exit gate as pending, though
     (a) and (b) were run in the previous commit; rewritten to closed plus the one item that
     actually remains (E1).
   - ROADMAP's P6 row still had the pre-phase plan in three of its cells; it now carries the
     package order, the scope boundary and the two new gates.

Checked with check_doc_citations.py: 125 citations across docs/Disaggregated/*.md, 0 problems.
This commit is contained in:
2026-09-18 23:31:39 -04:00
parent edd0fef5cd
commit aba329cc53
420 changed files with 131235 additions and 6 deletions
@@ -0,0 +1,237 @@
# P5d round 3 brief: inproc performance, the profile-directed cuts
Repo: MobileGL (branch `feat/disaggregated`, head `85362a85`). You work in an isolated git worktree
of it (your cwd). Read `docs/Disaggregated/P5D-INPROC-PERFORMANCE.md` (rounds 1-2), the relevant
parts of `MobileGL/MG_Remote/CONTRACT-P5C.md` (role guards, rule E) and `docs/Disaggregated/ROADMAP.md`
"通用纪律" before touching code.
## Why this round exists (measured, not guessed)
Workload: FCL + Minecraft 26.3-rc-3, view distance 12, Redmi 2f7cbe2e (Adreno 830), DirectGLES.
Split (`MOBILEGL_TRANSPORT=inproc`) 63-65 fps vs monolith 110-118 fps, ~852 draws/frame.
simpleperf (cpu-cycles, whole process, 15 s in world) at head 56a77348, symbolized:
Process split: GL/client thread (`Thread-17`) 39.5% of cycles, apply thread (`mgl-srv-apply`) 36.9%.
Monolith: GL thread 52.6% of a 39% smaller total.
### Apply thread (mgl-srv-apply), self %
- `std::mutex::unlock` 13.6, `std::mutex::lock` 12.4, `pthread_mutex_lock` 12.1, `pthread_mutex_unlock` 11.0
-> ~49% of the thread. Call graph: 58-60% of `mutex::lock/unlock` samples have
`ServerLoop::ApplyThreadMain()` as the DIRECT caller; 99.8% of `pthread_mutex_lock` via `mutex::lock`
is ApplyThreadMain. These are NOT the backend `pendingMutex` sites the round-2 report guessed.
They are the idle poll: `for(;;){ PumpControlRequest(); DrainRing(); if(ready()) continue; bell.Wait(...) }`
where `PumpControlRequest()` takes `m_controlMutex` every iteration and the `ready` lambda calls
`ControlIsPending()` which ALSO takes `m_controlMutex` (ServerLoop.cpp:371-384, 452-475).
- `ApplyThreadMain` self 7.95, `@plt` 6.1 (31% of plt from ApplyThreadMain), `__emutls_get_address` 7.4
- Inclusive: `ApplyThreadMain` 78.4%, `DrainRing` only 29.0% (= the real apply work: OnDrawVbo 22.7%,
PrepareForDraw 16.3%, SyncNeccessaryTextures 8.6%, SyncMipmapsToBackend 3.3%, HandleOf<ITextureObject> 3.0%,
`MGPipeRefuseAllocatorFromApplyThread` 2.4% incl.)
- `MGPipeServerStampVerbBoundary` 1.1 self (63-field loop per verb, PipeInputs.cpp:103-130)
- `__aarch64_swp2_rel`/`cas2_acq`/`ldadd8` ~2.5 (outlined LSE atomics; leave alone)
### Client thread (Thread-17), self % / inclusive %
- `SessionProducer::WaitForApplied` 27.9 self / 34.2 incl (the R-1 barrier spin; `EmitAndWaitTails` 36.3 incl)
- persistent-map push: `PushDrawConsumers` 1.96 self / 22.5 incl; `PushBlocksForChecked` 5.57 self / 21.5 incl;
`XXH3_hashLong_64b_default` 6.51 + `XXH3_64bits` 2.02 (callers: `PushBlocksForChecked` via PushDrawConsumers 98%);
`BufferObject::MappedData()` 1.16 self / 3.86 incl; `IsMapped` 0.58; `GetMappedRange` 0.43; `PushIsArmed` 0.47
- `MGPipeValidateForVerb` 4.11 self / 10.9 incl (65% called from `DrawElementsInstancedBaseVertex_Backend`);
inside it `MGPipeTracker::Update` 1.46, `MGPipeFillAccess::CopyField` 0.64 self / 2.54 incl
(45% of `BufferState::GetBindingSlot` 1.73 comes from CopyField in the draw fill)
- clock: `steady_clock::now()` 2.34 self / 5.91 incl, `__kernel_clock_gettime` 2.37, `clock_gettime` 1.94
-> ~6.6% of the thread is reading the clock. `Doorbell::Wait` (Doorbell.h:123-178) reads the clock at
least twice per wait (start + spinEnd) and once per 64 spins; ~900+ waits/frame at 64 fps.
- `ServerLoop::OnApplyThread()` 2.66 self / 3.64 incl on the CLIENT thread (role guards on hot paths:
`RefuseLegacyBufferArmFromApplyThread` in `BufferObject::MappedData/IsMapped/...`,
`PersistentMapTracker::OnServerRole`, `RefusePipeInputsTouchWhileApplierOwnsIt`, `RunsAsTheServerRole`,
`MGPipeRefuseAllocatorFromApplyThread`). `OnApplyThread` = `ServerLoopInstance()` (function-static guard)
+ `m_running` atomic + `m_applyThreadId` atomic<std::thread::id> + `std::this_thread::get_id()` +
thread::id compare (pthread_equal call). `pthread_self` shows 0.56 self.
- `MarkWritableImageBufferTextures` 2.15 self / 3.59 incl (GpuWritePending.cpp:85-101: sweeps all
`MAX_TEXTURE_IMAGE_UNITS` image units per draw; `GetImageTextureBinding` 1.37)
- `__emutls_get_address` 0.46 client; on MONOLITH's GL thread it is 7.7% (top symbol!) with callers
`MGPipeFrontendKeyedRegistryScope` ctor 18.75% / dtor 13.95%, `MGPipeReverseAnnouncementScope` ctor/dtor 8.4%
(SlotAllocator.cpp:41-75: `thread_local Uint32` depth counters; the scope is constructed at ~20 sites in
DirectGLES.cpp, e.g. inside `StateBackendObjectRegistry::HandleOf`). On the apply thread emutls is 7.4%.
### The shape of the frame (why the client-side cuts matter)
Lockstep (R-1): every kCtxVerb (draw/clear/blit/dispatch/readback) and every reply-slot row waits for
`appliedSeq == seq`. Frame time ~= client work + server apply time (waited for) + handoff latency.
Nothing in this round removes the barrier (that is P11's versioned inputs / P3b-P4b twin tables). This
round removes overhead on BOTH threads: every cycle cut on the apply thread's real work shortens the
client's wait one-for-one; every cycle cut on the client thread shortens the frame directly.
## Packages (one agent each, independent, non-overlapping files where possible)
### T - transport: apply loop idle poll, wait spin, park counters
Files: `MobileGL/MG_Remote/Server/ServerLoop.{h,cpp}`, `MobileGL/MG_Remote/Transport/Doorbell.{h,cpp}`,
`MobileGL/MG_Remote/Transport/Ring.cpp`, `SessionRings.h`, `MobileGL/MG_Remote/Client/ClientSession.cpp`,
`MobileGL/MG_Util/PipeStats*` (find the "MGPipe stats:" line producer).
1. `ControlIsPending()` and `PumpControlRequest()` must not take `m_controlMutex` when nothing is posted:
add `std::atomic<Bool> m_controlPosted` (set release under the lock by the poster in
`RunOnApplyThread`, cleared by the pump under the lock when it takes the request). The mailbox
handshake semantics (C2 critical section, NOT_INITIALIZED answer at exit, kMaxControlFramesPerPump)
must stay exactly as documented in the file. The `ready` lambda reads only atomics.
2. `DrainRing()` on an empty ring must be a couple of loads, no lock, no clock. Check.
3. `Doorbell::Wait`: the spin phase must not read the clock at all in the common case. Replace
"spin until steady_clock passes spinEnd, clock read every 64 iterations" with a calibrated iteration
budget: once per process (or per Doorbell), measure how many `ready()`+`CpuRelax()` iterations fit in
`spinUs` (or simply how many CpuRelax fit in 1 us) and spin that many iterations clock-free; only the
park phase (rare) computes deadlines. Keep `timeoutMs`/`kWaitForever` semantics, the parked-flag fence
protocol and the Dead() handling byte-for-byte in meaning. Document the overshoot bound.
`WaitForApplied`'s common path (server not yet there) currently pays 2+ clock reads per wait; after
this it pays zero unless it parks.
4. Counters for the measurement: server-side `m_parks` already exists (`ParkCount()`); add a client-side
park counter (SessionProducer::Park -> count parks that reach `Doorbell::Park`) and a wait counter,
and print both (server parks, client parks, waits) in the periodic `MGPipe stats:` line (find how the
line is assembled; keep it one line; add a `wait[...]` group). If the stats line producer lives in a
place that cannot see MG_Remote symbols, expose the counters through a small accessor registered by
the client session (no new include of MG_Remote into MG_Util if that violates the include-closure
gate `scripts/check_include_closure.py` - read its rules first).
Tests: existing `MG_Test/Wire/ServerLoopTest.cpp`, `DoorbellTest`/`RingTest` must keep passing by
inspection; add one small test for "a posted control request is visible to ControlIsPending without
the pump" and one for the calibrated spin (e.g. Wait returns true once ready flips within the spin
budget, and parks otherwise) if the existing fixtures make that cheap. Do not build.
### D - role guards and TLS: make every hot-path guard nearly free
Files: `MobileGL/MG_Remote/Server/ServerLoop.{h,cpp}` (OnApplyThread only - coordinate: package T also
edits ServerLoop.cpp; keep your hunk confined to `OnApplyThread`, the thread-identity member(s) and
`ApplyThreadMain`'s first lines where the identity is stored), `MobileGL/MG_Impl/Pipe/SlotAllocator.cpp`
(+ its header), `MobileGL/MG_State/GLState/BufferState/BufferObject.cpp` (RefuseLegacyBufferArmFromApplyThread),
`MobileGL/MG_Remote/Client/PersistentMapTracker.cpp` (OnServerRole / IsLivePersistentMap guard order),
`MobileGL/MG_Remote/Client/ClientSession.cpp` (RefusePipeInputsTouchWhileApplierOwnsIt: order the
early-outs cheapest-first: Transport==Monolith, BatchWaits!=0, then the rest),
`MobileGL/MG_Remote/Client/WireTables.cpp` (RunsAsTheServerRole), `MobileGL/MG_State/GLState/TextureState/MipmapStorage.cpp`.
1. `ServerLoop::OnApplyThread()`: store the apply thread's identity as a plain integer key readable
with a relaxed atomic load and compare it against a register-cheap identity of the calling thread:
on aarch64 and x86_64 with clang/gcc use `__builtin_thread_pointer()` (verify availability per
target with `__has_builtin`; NDK clang 18/19 supports aarch64; check x86_64 support and fall back to
`pthread_self()` (POSIX) or `std::this_thread::get_id()` (MSVC) where the builtin is unavailable).
Keep the semantics: false when no apply thread is running (m_running false / identity 0), true only
on the running apply thread. The function-static `ServerLoopInstance()` guard on every call must go
(a namespace-scope atomic key is enough; the ServerLoop object still owns it logically).
2. `MGPipeFrontendKeyedRegistryScope` / `MGPipeReverseAnnouncementScope` (SlotAllocator.cpp:39-75):
the depth counters are only ever READ by `MGPipeRefuseAllocatorFromApplyThread`, which returns early
unless `OnApplyThread()`. So the counters only matter on the apply thread. Replace the `thread_local`
with a plain (non-TLS) counter incremented/decremented only when `OnApplyThread()` is true (cheap after
item 1); on the GL thread the ctor/dtor become one predicate call. Prove in a comment why no data race
exists (only the apply thread writes and reads; monolith never reads). Keep `Active()` semantics on the
apply thread exact. If you find a reader other than the apply-thread guard, stop and report instead of
changing semantics.
3. `RefuseLegacyBufferArmFromApplyThread` (BufferObject.cpp:42-52) and `PersistentMapTracker::OnServerRole`,
`IsLivePersistentMap`: with item 1 these are ~free; make sure `PushIsArmed()` (a config read) is not
itself expensive (it reads `MG_Config::Transport` - fine) and that no guard does more than one
OnApplyThread call per accessor.
4. Look for any other per-draw `thread_local` reads on the client and apply paths (`t_activeDecoder` in
PipeWireCodec.cpp:359, `g_reply` in PipeRoute.cpp:61, `g_inBarrierWait` in ClientSession.cpp:117,
`g_textureLegacyArmScopeDepth` MipmapStorage.cpp:73, `tl_initializing`) and report which are on the
hot path; fix the cheap ones the same way ONLY if the single-writer argument holds; otherwise report.
Monolith equivalence: the pull build (`MOBILEGL_BUILD_DISAGGREGATED=OFF`) must compile and its behaviour
must be identical; G1 (pull-build symbol/.text identity) changes must be named in your report.
### B - persistent-map push: no edge hashing, page-granular shadows
Files: `MobileGL/MG_State/GLState/BufferState/PipeResource.h` (the aligned allocator), the shadow
container in `BufferObject.{h,cpp}` / `PipeResource.*`, `MobileGL/MG_Remote/Client/PersistentMapTracker.{h,cpp}`
(TrackWriteMap ~259-330, the SIGSEGV handler ~200-240, PushBlocksForChecked 492-685, PushDrawConsumers 704-818).
Today: the shadow allocator gives 4096-byte ALIGNMENT but not a page-multiple SIZE, so the last page of
a shadow can hold foreign bytes; TrackWriteMap therefore aligns the protected interior INWARD and the
partial-page edges (head edge when the mapped offset is not page-aligned; tail edge for every buffer
whose mapped end is not page-aligned - i.e. nearly every buffer) are hashed with XXH3 on EVERY draw for
every live map with edges (the epoch-skip fast path pushes edges of ALL live maps, not just consumers).
That is 8.5% of the client thread in XXH3 plus the MappedData/IsMapped/GetMappedRange guards around it.
Goal: zero per-draw hashing in steady state.
1. Make every shadow allocation page-granular: round the allocation SIZE up to a multiple of
SHADOW_ALLOCATION_ALIGNMENT in the allocator (verify the container: what `Bytes()` returns, whether
capacity vs size matters, whether any shadow can be backed by something else - adopted GPU stores are
excluded from tracking by IsLivePersistentMap; confirm). Then every byte of
`[base, base + RoundUp(size, 4096))` belongs to this shadow and no other allocation.
2. TrackWriteMap: when the shadow base is page-aligned and the allocation is known page-granular, align
the protected range OUTWARD to page boundaries (clamped to the shadow's own page-rounded extent), so
there are no edges and `hasEdges` is false; sub-page maps become one protected page. Keep the inward
arm as the fallback for any shadow that fails the page-granular test (and keep the hash arm for
untracked maps). The SIGSEGV handler's ownership tests (SEGV_ACCERR, inside a live tracked range,
bitmap-bit replay discrimination) must hold for the widened range: bytes outside the mapped range but
inside the shadow are written by the client's own API paths (SubData/FlushMappedRange memcpy into the
shadow) - a fault there is answered like any interior fault; make sure the push clamps to [begin,end)
as today and that nothing else (e.g. readback writeback into the shadow on the client thread,
`Forget`, resize/respecify that reallocates the shadow, `NoteMapStateChanged`) breaks the
"protected pages are re-armed / released" invariant. Re-read the round-1 commit message of cb06538c
(git show cb06538c) for the reasons behind inward alignment before you widen anything.
3. The epoch-skip fast path (PushDrawConsumers 727-737) then does nothing per member; keep the loop or
drop it - measure nothing, reason. First push still ships everything once.
4. Report: a table of the per-draw work that remains in PushDrawConsumers when the epoch has not moved.
Tests: `MG_Test/Buffer/SplitBufferTest.cpp` and `MG_IntegrationTest/Scenarios/PersistentCoherentMapScenario.cpp`
exercise this; add/adjust a unit case for the outward alignment (a sub-page map gets a protected page;
an unaligned base falls back to inward+edges). Do not build.
Danger notes from cb06538c: a signal-blocked driver/runtime worker thread faulting on a foreign byte
inside a protected page kills the process (that was the ctest SmallRing segfault) - the whole point of
item 1 is that no foreign byte can be inside a protected page any more; state that argument precisely.
### C - client verb overhead: validate/fill, image-unit sweep, server stamp loop
Files: `MobileGL/MG_Impl/Pipe/PipeFill.cpp` (MGPipeValidateForVerb 2781+, the residual fill / CopyField),
`MobileGL/MG_Pipe/PipeInputs*`/`MobileGL/MG_Backend/MGPipe/PipeInputs.cpp` (MGPipeServerStampVerbBoundary),
`MobileGL/MG_Remote/Client/GpuWritePending.cpp`, `MobileGL/MG_Remote/Client/EmitTables.cpp` (draw emit path),
`MobileGL/MG_State/GLState/TextureState/*` only if an image-unit high-water mark is needed.
1. `MarkWritableImageBufferTextures`: replace the 32-unit sweep per draw with a high-water mark
(the GLContext already keeps touched-unit marks for texture units and buffer binding points; add the
same for image units at the BindImageTexture site if none exists - note this is MG_State (shared with
monolith) so keep it minimal and name the G1 impact; or, if `ImageEmit`/the tracker already knows
whether any writable image-buffer unit is bound, use that latch).
2. `MGPipeValidateForVerb` draw path: find what the 10.9% inclusive is made of beyond tracker.Update
(1.5%) and CopyField (2.5%): the residual fill's per-verb copies (which fields does the draw class
copy, and which of those are already record-supplied under a live wire - P5c rv skips the value-class
rows; check the object-class rows' copy cost, e.g. `GetBindingSlot` from CopyField), the
`RefusePipeInputsTouchWhileApplierOwnsIt` guard order (package D reorders it; don't duplicate),
`ContextValuesWireLive()` / `RunsAsTheServerRole()` per verb (cache per session if legal),
`ParsePoisonOmissionKnob()` per verb (should be once). Cut what is provably redundant; leave the
emitters' semantics alone.
3. `MGPipeServerStampVerbBoundary`: the 63-field loop per verb on the apply thread computes
`answerable(field, verb)` from constant tables - precompute per verb once (a static table of
which fields are answerable per verb, built lazily or constexpr) so the per-verb work is a
table-driven fill. Keep the withdrawal semantics exactly (FilledGen = serial or 0).
4. Anything else on the draw emit path in EmitTables.cpp that is per-draw and redundant (e.g. repeated
session lookups, ReadDrawBindings re-reading slots the fill already read). Report what you found and
what you left.
Monolith: PipeFill.cpp is compiled into the monolith push build too; keep monolith behaviour identical
(G1 for the pull build; push-build equivalence by inspection).
### E - research only (no code): can the draw barrier be deferred by one verb?
Deliverable: a design note, not a patch. Question: the client today fills the residual object-class
rows of `gPipeInputs` (FieldOwnership.def BARRIER_PULLED: GetBoundVertexArray, GetBufferBindingSlot,
GetBufferBindingPoint, GetFramebufferBindingSlot, GetImageTextureBinding, GetTextureUnitObject,
GetProgramForDraw, GetProgramForDispatch, GetTransformFeedbackProgram, GetBufferBindingPointCount,
GetProgramObject, GetTextureObject, HasOpenTransformFeedbackSpan, ValidateProgramName, RecordError),
publishes the draw, and WAITS. Map, for the draw/clear/blit verbs on DirectGLES under inproc:
(a) every server-side read that reaches client-owned memory through those rows (what does the server
dereference through each pointer: VAO attribute array? texture unit binding slots? UBO binding point
table entries + ranges? program object internals?), including the two exemption scopes
(`MGPipeFrontendKeyedRegistryScope` sites in DirectGLES.cpp, `MGPipeReverseAnnouncementScope`);
(b) which of those reads would be UNSAFE if the client continued past the publish and executed the
next draw's GL calls (glBindBufferRange / glBindTexture / glUseProgram / glVertexAttribPointer / ...)
before the server applied this draw;
(c) the minimal "snapshot at fill" scheme: what would have to be copied per draw into a per-verb
snapshot the server reads instead (count the bytes/refcount ops per draw at 852 draws/frame), and
where the client would then wait ("before the next fill" instead of "after this publish"), i.e.
one-verb pipelining that overlaps the client's inter-draw work with the server's apply;
(d) what the existing pushed records already cover (SetSamplerViews / VertexElements CSO /
set_vertex_buffers / set_context_values) such that the server COULD stop reading the live tables;
(e) a cost/benefit estimate and a recommendation: P5d round 4, or leave to P11 / P3b-P4b.
Write it to the report path given below (markdown, <= 400 lines, with file:line citations at head 85362a85).
## Rules for every package
- Do NOT build (no gradle, no cmake, no ndk), do NOT touch adb/devices, do NOT push, do NOT commit
in any shared tree. Work only in your isolated worktree; deliver `git diff HEAD -- MobileGL` as a
patch FILE at the path given below, plus a short report file. Both must be written before your
final message.
- No `Co-Authored-By`, no attribution lines anywhere.
- No hot-path instrumentation left in (counters that print per frame are fine when behind the
existing `MG_Util::PipeStats::Enabled()` / stats period; no per-draw logging).
- Prefer changes inside `#if MOBILEGL_BUILD_DISAGGREGATED` or in MG_Remote (split-only). When a change
touches code compiled into the pull build (MOBILEGL_BUILD_DISAGGREGATED=OFF, MOBILEGL_PIPE_PUSH=OFF),
say so in the report (G1 names every symbol/.text delta) and keep behaviour identical.
- Every correctness-bearing change gets a red-once argument: name the existing test (or the one you
add) that goes red if the change is reverted or broken. Keep tests light; do not add a test for a
refactor with no semantic change.
- Keep the file's own comment discipline: explain WHY in the code comment, in the voice of the
surrounding code, and do not delete the reasoning comments you replace - update them.
- Report format (markdown): what changed (per file), why (tie to the profile numbers above), the
red-once test, G1/monolith impact, risks and what you did not do. <= 120 lines.
@@ -0,0 +1,66 @@
# P5d round 3 - device results log (Redmi 2f7cbe2e, FCL + MC 26.3-rc-3 world "test", VD12, DirectGLES)
Protocol: CPU pinned big 1958400 / little 1555200 (pin_device.sh), fan level 2 (~14.2k rpm), screen on,
cpuss-0-0 50-54 C during runs. GPU: the vendor holds thermal_pwrlevel=1 and resets min_pwrlevel to 12 at
every app start, so the GPU is GOVERNED, not pinned (gpuclk 342 MHz in every inproc arm = GPU idle,
832 MHz in monolith). All inproc arms are CPU-bound on the client thread; monolith may be slightly GPU-
limited. fps = frames/second from the `MGPipe stats:` lines over a 30 s window after 20 s settle (world
loads in ~15 s on this device); ms/frame = per-thread utime+stime over the window / frames.
Thread affinity: `sched_setaffinity`/`taskset` are accepted and silently ignored by this kernel for
app threads (RESOLVED mask 0xff; taskset "new affinity mask: ff"), so core placement is the scheduler's.
## Session base0 (unpatched head 85362a85 lib, APK with mg_env.txt plumbing) - 2026-09-18 03:18-03:26
| arm | env | fps mean / p50 / min | client ms/frame (core) | apply ms/frame (core) |
|---|---|---|---|---|
| base0-inproc | - | 64.8 / 64.3 / 60.9 | 15.03 (cpu5, mid) | 12.42 (cpu7, big) |
| base0-mono | monolith | 129.2 / 116.5 / 112.0 | 5.85 (cpu7, big) | - |
| spin2000 | MOBILEGL_IPC_SPIN_US=2000 | 72.7 / 72.6 / 68.5 | 14.00 (cpu2) | 13.64 (cpu7) |
| nobarrier | MOBILEGL_IPC_VERB_BARRIER=0 | aborted: Fatal{UnmigratedPipeInput, "GetFramebufferBindingSlot@FenceSync"} at the first race - the negative control cannot run a real workload | | |
| affoff | MOBILEGL_IPC_SERVER_AFFINITY=off | 65.9 / 65.9 / 57.7 | 14.20 (cpu5) | 12.21 (cpu7) |
## Session place1 (same lib) - 03:27-03:35
| arm | env | fps mean / p50 / min | client ms/frame (core) | apply ms/frame (core) |
|---|---|---|---|---|
| bigclient | taskset c0 on GL thread (ignored) | 64.9 / 65.0 / 57.4 | 15.05 (cpu4) | 12.75 (cpu7) |
| bigclient-spin2000 | SPIN_US=2000 (+ignored taskset) | 69.7 / 68.9 / 65.0 | 13.94 (cpu4) | n/a |
| bigboth-spin2000 | SPIN_US=2000 (+ignored taskset) | 68.3 / 68.3 / 65.2 | 14.62 (cpu4) | 14.31 (cpu7) |
| spin10000 | SPIN_US=10000 | 62.5 / 63.2 / 59.1 | 15.49 (cpu2) | 15.41 (cpu6) |
Readings:
- Split/monolith p50 ratio at the head: 64.3 / 116.5 = 1:1.81 (report said ~1:1.7).
- The split's client thread is the critical path: ~15 ms CPU per 15.5 ms frame (97% busy, spin-waits
included), and the scheduler puts it on a MID core (cpu2/4/5 at 1555 MHz) while the always-spinning
apply thread holds a big core; monolith's render thread sits on cpu7 (big).
- Spin 2000 us: +4..+8 fps (3 arms 68-73 vs 64-66) - park/unpark latency is a real, second-order cost.
Spin 10000 us: worse (63) - both threads then burn full cores and the client lands on cpu2.
- Server affinity knob: no effect (it never took on this kernel).
## Session r3a (round-3 PRE-review build, patches T/D/B/C as first written) - 03:42-03:49
| arm | fps mean / p50 / min | client ms/frame (core) | apply ms/frame (core) | wait[] (run totals at window end) |
|---|---|---|---|---|
| r3-inproc | 112.5 / 114.0 / 88.8 | 8.68 (cpu4) | 6.93 (cpu7) | srv=7.56M srvpark=76k cli=6.70M clipark=17k |
| r3-spin2000 | 113.0 / 112.3 / 106.1 | 8.88 (cpu5) | 8.88 (cpu7) | srvpark=1.6k clipark=286 |
| r3-mono | 115.4 / 115.7 / 108.4 | 5.70 (cpu5) | - | capped |
| r3-spin1000 | 107.6 / 108.5 / 98.0 | 9.27 (cpu5) | 8.63 (cpu7) | |
## Session r3b/r3c/r3d (round-3 POST-review build = commit 1f8de61b) - 04:04-04:32
| arm | fps mean / p50 / min / max | client ms/frame (core) | apply ms/frame (core) | note |
|---|---|---|---|---|
| r3b-inproc | (false "app died": one pidof hiccup; fixed in bench_fcl3+) | | | 14 stats lines, 855 draws/f, no Fatal |
| r3b-mono | 246.9 / 266.1 / 123 / 298 | 3.9 (cpu7, 97% busy) | - | UNCAPPED run - vsync not enforced this time |
| r3b-inproc2 | 107.4 / 106.7 / 98 / 120 | 9.3 (cpu5) | 7.1 (cpu7) | |
| r3b-spin2000 | 109.1 / 109.6 / 102 / 113 | 9.1 (cpu4) | 8.7 (cpu7) | presented (SF timestats) 113 fps |
| r3c-mono1 | 166.1 / 205.8 / 110 / 280 | 5.0 (cpu7) | - | uncapped, swinging 110-280 inside the window |
| r3c-inproc1 | 103.8 / 106.1 / 91 / 112 | 9.24 (cpu4) | 7.0 (cpu7) | |
| r3c-mono2 | 114.6 / 115.3 / 108 / 117 | 6.2 (cpu7) | - | capped at 120 Hz |
| r3c-inproc2 | 101.6 / 103.3 / 82 / 111 | 9.2 (cpu5) | 7.4 (cpu7) | |
| r3c-inproc3 (csbig spec ignored by v3 driver) | 100.9 / 101.9 / 83 / 109 | 9.1 (cpu5) | 7.3 (cpu7) | |
| r3c-mono3 | 114.9 / 115.1 / 109 / 117 | 6.1 (cpu7) | - | capped |
| r3d-inproc-csbig (cpuset 6-7 written for the GL tid) | 112.5 / 115.3 / 92 / 122 | 8.63 (cpu5) | 6.98 (cpu7) | thread still on cpu5: cpuset ignored too |
pmap wire bytes identical pre/post review (364.7 KB/frame); post-review inproc 103-107 vs pre-review 114 is inside run-to-run drift (body 50.8 vs 53-54 C at window start, mid-core placement).
Reading: monolith fps is bimodal on this device (115 = 120 Hz vsync cap; 206-266 = uncapped runs); per-thread CPU ms/frame is the comparable metric. Client thread 15.0 -> 8.7-9.2 ms/frame (mid core), apply 12.4 -> 7.0-7.4 (mostly spin), monolith render thread 5.0-6.2 (big core).
+188
View File
@@ -0,0 +1,188 @@
# P5d round 3, package B: persistent-map push - page-granular shadows, no edge hashing
Worktree `MobileGL-p5d-B` (branch `p5d-B`, base `85362a85`). Patch: `patches/B.patch` (5 files,
+416/-91). Not built, not committed.
## What changed (per file)
**`MG_State/GLState/BufferState/PipeResource.h`**
- `ShadowAllocationBytesFor(count)` (split-only): rounds an allocation SIZE up to a multiple of
`SHADOW_ALLOCATION_ALIGNMENT` (4096). `MapAlignedAllocator::allocate` asks the heap for that many
bytes under `#if MOBILEGL_BUILD_DISAGGREGATED`; the pull-build arm of `allocate` is the old text,
token for token. Round 2 gave shadows a page-aligned BASE only; the last page could still end
mid-way with a neighbour allocation in the remainder.
- `PipeResource::ShadowAllocationBytes()` (split-only): `ShadowAllocationBytesFor(m_shadow->capacity())`,
0 for an adopted store or an unsized shadow. `capacity()` is exactly the count `std::vector` passed to
`allocate()` (libc++/libstdc++/MSVC all record it), and the rounding is the allocator's own function,
so the extent the tracker widens into can never exceed what was allocated.
- Confirmed: a live member's `MappedData()` is always `m_shadow->data()` - `Bytes()` returns the GPU map
only when `m_gpuMapped != nullptr`, and `IsLivePersistentMap`'s second row (`IsBackendPersistentMapped`)
excludes exactly that; nothing else backs a shadow. `ResizeShadow` is reserve(bit_ceil)+resize, so the
allocation is `capacity()` bytes and `[0, size) ⊆ [0, capacity)`; the shadow is never reallocated while a
map is live (Respecify/AllocateImmutableStorage call `ReleaseMemory(false)` first, which untracks;
EnsureGpuResidentStorage/TryAdoptLargeStorage refuse while mapped).
**`MG_State/GLState/BufferState/BufferObject.h`**: `ShadowAllocationBytes()` accessor inside the existing
`#if MOBILEGL_BUILD_DISAGGREGATED` block (pull build: no text, no layout change).
**`MG_Remote/Client/PersistentMapTracker.cpp`**
- `TrackWriteMap(..., shadowExtent)`: **outward** alignment when the shadow is page-granular (base
page-aligned, extent != 0, extent a page multiple, `rangeEnd <= extent`): `base = RoundDown(begin)`,
`end = RoundUp(end)` clamped to `shadow + extent`; `hasEdges = base > rawBegin || end < rawEnd` is then
false and a sub-page map is one protected page. Anything failing the test keeps the **inward** arm
(fallback, comment rewritten to state both ownership arguments; the round-1 reasoning is kept, not
deleted). `static_assert(SHADOW_ALLOCATION_ALIGNMENT % kPageBytes == 0)` pins the precondition.
- `InstallWriteFaultHandler`: refuses (once, `g_pageSizeRefused`) when `sysconf(_SC_PAGESIZE) != 4096`.
Every ownership claim is made at 4 KB; on a 16 KB-page kernel (Android 15 admits them) a 4 KB
`mprotect` covers three foreign quarters - the process-killer. Pre-existing hazard, made deterministic:
hash arm on such kernels, never an unsafe protect.
- `PushBlocksForChecked`: both XXH3 edge hashes are gated on `tracked->hasEdges`; the `interiorBegin/End`
comments no longer claim `>= begin / <= end`; the page loop's clamp to `[begin, end)` is unchanged and is
what keeps a head/tail page that starts before `begin` or runs past `end` inside the mapped range.
- `PushDrawConsumers` epoch skip: walks the new `m_edgedMembers` set (members whose slot has unprotected
edges - inward fallback only) instead of the whole membership; empty in steady state. Keys copied out
before the walk because `PushBlocksForChecked` may `Forget()`.
- `NoteMapStateChanged` passes `buffer.ShadowAllocationBytes()` and maintains `m_edgedMembers` beside
`m_untrackedMembers` (erase on re-note, Forget, ClearForTest).
- Test hooks: `MprotectArmAvailableForTest`, `TrackedSlotForTest`, `TrackForTest`, `UntrackForTest`,
`FaultEpochForTest`.
- SIGSEGV handler: logic untouched. Comment on the every-overlapping-slot loop updated (no two tracked
ranges can share a page any more; loop kept as the checkable form).
**`MG_Remote/Client/PersistentMapTracker.h`**: `TrackedWriteMap` edge-field comments (edges exist only on
the inward fallback), `m_edgedMembers`, the five test hooks.
**`MG_Test/Buffer/SplitBufferTest.cpp`**: two cases + two skip stubs (G2/G14 name-set rule, "eighteen").
## Why (profile)
Client thread at 56a77348: `XXH3_hashLong_64b_default` 6.51 + `XXH3_64bits` 2.02 = 8.5%, 98% via
`PushBlocksForChecked` from `PushDrawConsumers`; plus `PushBlocksForChecked` 5.57 self, `MappedData` 1.16,
`IsMapped` 0.58, `GetMappedRange` 0.43 around it. Cause: nearly every mapped end is not page-aligned, so
nearly every tracked map had `hasEdges` and the epoch-skip fast path hashed its edges for EVERY live map
on EVERY draw. With page-granular shadows there are no edges, the fast path touches no member, and steady
state has zero per-draw hashing and zero `MappedData/IsMapped/GetMappedRange` calls from this module.
## The ownership argument, precisely (cb06538c's danger note)
A page may be protected only if every byte of it belongs to this shadow. Outward: the heap handed the
shadow out as ONE block of `extent` bytes at a 4096-aligned base (`operator new(extent, align_val_t{4096})`,
extent a multiple of 4096); allocator metadata lives before the block, never inside; therefore
`[shadow, shadow + extent)` contains no byte of any other allocation, and every page intersecting
`[shadow+begin, shadow+end)` with `end <= extent` lies inside it. A signal-blocked driver/runtime worker
never writes a client shadow (rule E: the server reads its own staged copy). Bytes of a head/tail page
outside the mapped range are written only by the client's API paths (SubData/FlushMappedRange/Respecify
memcpy, the readback writeback landing in `WaitForApplied`'s event drain on the client thread) - threads
that run the handler; such a fault is answered like an interior one, the push clamps to `[begin, end)`,
and the byte itself crossed by its own record. `Forget`/unmap/re-note all route through `UntrackWriteMap`
(restore writable, then retire) before any reallocation, unchanged.
## Per-draw work remaining in PushDrawConsumers, epoch unmoved
| step | cost |
|---|---|
| `PersistentMapTracker::Instance()` (3 call sites in EmitTables) | function-static guard load |
| `PushIsArmed()` | one `MG_Config::Transport` load + compare |
| `OnServerRole()` | `ServerLoop::OnApplyThread()` (package D makes it a relaxed load + compare) |
| `m_livePersistentMaps.empty()` | one load |
| `g_faultEpoch.load(acquire)` + compare with `m_lastFaultEpoch` | one atomic load |
| `m_untrackedMembers.empty()`, `m_edgedMembers.empty()` | two loads |
| per member | nothing |
First push after a map still ships everything once (all bits set at registration; first fault moves the
epoch, the walk pushes every set bit). The loop was kept (over `m_edgedMembers`, not the membership) because
the inward fallback still exists; it is not entered in steady state.
## Red-once
- `SplitBufferSet.ASubPageMapIsOneProtectedPageAndAWriteThroughItFaults`: reverting the outward alignment
gives `pageCount == 0`, `hasEdges == true`, and a write through the pointer that does not move the fault
epoch - three red assertions. Reverting the allocator rounding alone: `ShadowAllocationBytes() % 4096`
fails for a 100-byte buffer (capacity 128).
- `SplitBufferSet.AShadowThatFailsThePageGranularTestKeepsTheInwardAlignmentAndItsEdges`: an unaligned base,
an unknown extent (0) and a range past its extent each register inward (1 page, edges); the true extent
widens to 3 pages, no edges, end == extent. Widening without the extent test goes red on the third
registration.
- Existing: `ThePushCutsTheMappedSpanIntoBlocksAndMovesPmap`, `TheLastBlockIsTheRemainderAndNotAWholeBlock`
(64K+7: now one page-rounded span, still 2 blocks / kSize bytes by the clamp), `PersistentCoherentMapScenario`
(pixels through the push) keep passing by inspection.
## G1 / monolith
Pull build (`MOBILEGL_BUILD_DISAGGREGATED=OFF`): zero delta. `ShadowAllocationBytesFor`,
`PipeResource::ShadowAllocationBytes`, `BufferObject::ShadowAllocationBytes` exist only under the option;
`allocate()`'s pull arm is the previous text verbatim. Push monolith build with the option ON but
`Transport=Monolith`: allocations of shadows/staging stores round up to 4096 multiples (invisible to the
application; capacities >= 4096 are already powers of two, so only sub-4K stores grow), tracker inert as
before (`PushIsArmed` false).
## Risks / not done
- Not built (per rules). Watch for: `sysconf`/`unistd.h` already included; `uintptr_t` in the test via
`<cstdint>` through the tracker header; `flat_hash_map::value_type::first` for `m_edgedMembers`.
- The 16 KB-page refusal is new behaviour on such kernels (hash arm instead of mprotect); on the Redmi
(4 KB) it is a no-op. It is the honest answer, but it is a performance cliff there rather than a fix.
- Two tracked shadows can no longer share a page; the handler still scans every slot per fault (unchanged).
- Not touched: role guards (`RefuseLegacyBufferArmFromApplyThread`, `OnServerRole`) - package D; the hash
arm's `MOBILEGL_IPC_PERSISTENT_HASH_SUPPRESS` XXH3 for untracked members - still exists for table-full /
handler-unavailable / >256 MB maps; docs (`P5D-INPROC-PERFORMANCE.md`) not updated - integrator's row.
## Fix round
Patch regenerated from `MobileGL-p5d-B` (6 files, +654/-149; BufferObject.cpp is new to the set). Not built.
**Blocker (both reviews): the epoch skip lost the drain the old edge service did by accident.**
Agreed, and the diagnosis is exact: `m_lastFaultEpoch = epoch` before a walk that pushed only
consumers left a faulted non-consumer marked, writable and unable to move the epoch again. Fix, in
`PersistentMapTracker.cpp`:
- The walk now DRAINS every tracked member with a marked page, consumer or not, before the binding
walk (`AnyPageMarked`, one word load per 64 pages, walk-time only). The skip's proof then holds:
the epoch moves only on a mark, and the last walk drained every mark there was. A drained push is
bitmap-driven (faulted blocks only), so nothing pays the whole-range rescan the consumer filter
exists to avoid. `m_lastFaultEpoch` is still taken before the drain, so a fault landing mid-drain
forces the next draw to walk again.
- `TrackWriteMap` bumps the epoch after publishing a slot: a fresh slot has every page marked, which
is a fault of every page as far as the push is concerned. This closes the second review's
"registered and drawn without a fault is never shipped" hole - "first push ships everything once"
is now true of the draw path, not only of the read-only verbs' whole push.
- The handler's `g_faultEpoch.fetch_add` is now `release` (was relaxed) so the walk's existing
`acquire` load actually orders the bit before the epoch across threads (chunk builders write
arenas off the GL thread). Cheap: handler-only.
- Skip path unchanged in cost: one atomic load, two empties, nothing per member. The per-draw table
above still stands. The walk (epoch moved) gained one word-scan per tracked member.
- The binding walk moved under `if (ctx != nullptr)` so the drain runs without a context too; the
walk body is re-indented, otherwise unchanged.
**Major (review 1): no red-once for the skip semantic.** Added
`SplitBufferSet.AFaultedMapTheDrawDidNotBindStillShipsBeforeTheDrawThatDoes` (+ skip stub; name
sets 20/20, "TWENTY"). Two one-page maps A and B, B bound to SSBO point 0, driven through
`PushDrawConsumers` itself: draw 1 ships both (2 blocks: fresh A ships although unbound - the
registration bump), draw 2 ships nothing, a write through A's pointer faults (asserted via the
epoch), draw 3 with only B bound must ship A's block (3), draw 4 binding A ships nothing more (3,
`BytesPushed == 3 * 4096`). Reverting the drain: draw 3 stays at 2 and draw 4 skips - red at the
"3u" assertion. Reverting the registration bump alone: draw 1 stays at 0 - red at "2u".
**Minor (review 1): `g_segvInstalled` published before the page-size check.** The check is now
`KernelPageIsTheTrackerPage()`, a function-static memo asked BEFORE the CAS; `g_pageSizeRefused`
is gone. No caller can pass `TrackWriteMap`'s installed gate while the size is undecided.
**Minor (review 1): `ShadowAllocationBytes()` unguarded.** Now out of line in `BufferObject.cpp`
(under the option) with `RefuseLegacyBufferArmFromApplyThread("ShadowAllocationBytes")`, beside
`MappedData`. Pull build: no text (declaration and definition are both under the option).
**Minor (review 1): outward can exceed `kMaxTrackedPages` by one page where inward fit.** Widening
never demotes: if the outward count exceeds the capacity, `TrackWriteMap` takes the inward arm for
that map (edges, `m_edgedMembers`) instead of refusing it onto the hash arm, so no map that was
tracked before this round becomes an untracked member that vetoes the skip process-wide. Only a
~256 MB map on the boundary pays the old edge-hash cost; no test (it would need a 256 MB shadow).
**Minor (review 2): extent inferred from `capacity()`.** `PipeResource` now records
`m_shadowExtent = ShadowAllocationBytesFor(reserved)` in `ResizeShadow` (the count it asked the
allocator for) and zeroes it in `AdoptPersistentMap`; `ShadowAllocationBytes()` returns it (0 while
GPU-resident). When the reserve did not reallocate, the block behind `Bytes()` is a larger earlier
reserve's, so the recorded value is at most the true extent in every case; `size <= reserved`
keeps `rangeEnd <= extent`. `Shadow()` has no callers, and every shadow allocation goes through
`ResizeShadow`, so nothing else can move the block. Split-only member (G1 layout unchanged).
**Minor (review 2): the `hasEdges &&` conjuncts were dead.** Dropped both; the comment now says
the two geometric conditions are self-gating under outward alignment and that this, not a flag,
is what removed the per-draw XXH3. `hasEdges` stays for `m_edgedMembers` (header comment updated).
**Not changed:** the 16 KB refusal, the outward arithmetic, the allocator rounding, the SIGSEGV
handler's ownership tests, `PushAllMembers`. Risks: not built - watch `if (const auto& ctx =
MG_State::pGLContext)` (UniquePtr in an if-declaration, same shape as the `vao` line below it),
`entry.first` on `UnorderedMap<Uint64, MemberEntry>`, and `MGLOG_W` inside the function-static
lambda.
+250
View File
@@ -0,0 +1,250 @@
# P5d round 3 - package C: client verb overhead (image-unit sweep, validate/fill, server stamp)
Worktree `MobileGL-p5d-C`, branch `p5d-C`, base `85362a85`. Not built, not committed.
Patch: `.../scratchpad/p5d/patches/C.patch` (865 lines, 11 files, after the fix round below).
## What changed, per file
### 1. The 32-unit image sweep per draw (brief item 1)
- `MG_State/GLState/TextureState/TextureState.h` - new `NoteImageUnitTouched(Int)` /
`GetMaxTouchedImageUnit()` and an `Int m_maxTouchedImageUnit = -1` member, all inside
`#if MOBILEGL_PIPE_PUSH` (the private member joins the two push-only generations already there).
A **second** mark, not a widening of `m_maxTouchedUnit` (image units are their own array). It
only ever grows, so a walk bounded by it over-approximates the units that hold a binding - the
only direction a conservative GPU-write set may fail in.
- `MG_State/GLState/Core.h` - the two `GLContext` pass-throughs, same guard.
- `MG_Impl/GLImpl/Texture/GL_Texture.cpp` - `BindImageTexture` feeds the mark, same guard.
- `MG_Remote/Client/GpuWritePending.cpp` - `MarkWritableImageBufferTextures` walks
`unit <= GetMaxTouchedImageUnit()` instead of `unit < MAX_TEXTURE_IMAGE_UNITS`. On the profiled
workload (no image binding anywhere) the mark is -1, so the walk is one compare.
Profile tie-in: 2.15% self / 3.59% inclusive of the GL thread, `GetImageTextureBinding` 1.37 of it.
- `MG_Impl/Pipe/ImageEmit.h` - its comment claimed no such mark could exist without resizing the
pull build's object. Updated, not deleted: the mark is push-only, and the emitter deliberately
keeps its program-derived window (the narrower "which units can a *shader* read").
### 2. `MGPipeValidateForVerb` (brief item 2)
- `MG_Impl/Pipe/PipeFill.cpp` - step 4's seven-term `supplied` conjunction was evaluated 63 times
per verb; it reads nothing about the verb. It is now a memo (`SuppliedFieldMask`) keyed on the
four facts it does read: the push mask, `ApplierDerivesRenderStateFields()`,
`contextValuesWireLive` (passed in, so P5c rv's emission half and fill half still read the ONE
answer), and `P4aFamilyHasItsConsumer(kMGPipeP4aFamilySubsystems)` - all four families ride that
one signal by that function's own rule. The walk becomes one bit test per field; the expression
is moved, not copied.
Profile tie-in: `MGPipeValidateForVerb` 4.11% self / 10.9% incl.; `P4aFamilyHasItsConsumer` alone
is a `CapsMirror` read taken 63x/verb for one answer under split.
- `ParsePoisonOmissionKnob` - the per-verb `String == String` (out-of-line even for two empty
strings, which is every shipping configuration) is now a size compare inline, contents only when
a non-empty value is involved. Re-parse condition unchanged.
- `MG_Impl/Pipe/PipeFill.h` - `MGPipeResidualFillSuppliesField(field, pushMask, applierDerives,
contextValuesWireLive)`, exported for the unit gate only, in `MGPipeP4aFamilyEmits`' shape.
### 3. `MGPipeServerStampVerbBoundary` (brief item 3)
- `MG_Backend/MGPipe/PipeInputs.cpp` - the per-verb 63-field `answerable` derivation is now a
`constexpr` per-verb-class mask table (`AnswerableMaskForClass` / `kAnswerableByClass`), built
from the same expression. The loop body is `FilledGen[i] = serial & -bit` - no lookups, no calls,
no branch. Withdrawal semantics byte-identical, including the out-of-range verb
(`kNoFieldIsAnswerable`, all 63 withdrawn), which is what `FieldIsInVerbClass` answered for it;
that helper stays, `MGPipeInputUnfreshRead` still uses it. Profile: 1.1% self of the apply
thread, and on a lockstep frame that is client wait time one-for-one.
## Red-once
| change | test |
|---|---|
| stamp table | `FieldOwnershipTest.AServerStampMakesRecordSuppliedFieldsFreshAndWithdrawsTheRest` (existing) - it recomputes `answerable` independently per field and compares. `TheStickyExemptionIsCancelledByTheServerStamp` covers the withdrawal of the seven forwards. |
| supplied-set memo | `FieldOwnershipTest.TheResidualFillsSuppliedMemoReKeysOnEveryInputThatMovesAnAnswer` (**added**; renamed and extended in the fix round below, which restates this row) - moves each of the three key inputs that can move an answer on its own and reads the answer back in between, in an order where a memo missing that input must return the stale answer. This is the only thing a memo can get wrong that the expression could not; which fields the fill copies has no other observable, because a push build routes every emission straight into the applier, which writes the same storage. |
| image-unit bound | `SplitBufferSet.Row2TheSweepIsBoundedByTheImageUnitHighWaterMark` (**added**) - pins both ends of the bound (mark at -1 walks nothing; a unit at the mark is found). Revert the bound and its second half goes red. |
| the mark's producer | `ImageEmit.TheBindFeedsTheImageUnitHighWaterMark` (**added**) - `GL::BindImageTexture` must move the mark, and an unbind must not shrink it. Drop `NoteImageUnitTouched` from `GL_Texture.cpp` and this goes red at the entry point instead of as a missed GPU write. |
| (adjusted) | `SplitBufferSet.Row2AWritableImageBufferTextureIsMarkedByADraw` now notes the unit, because it writes the binding straight into the slot and never goes through `glBindImageTexture`. |
G2/G14 name parity kept: the two new split/push-only names have pull-lane skip twins
(`SplitBufferTest.cpp`'s `MGL_SPLIT_ONLY_OR_SKIP` list, `ImageEmitTest.cpp`'s
`MGL_IMAGE_EMIT_TEST_LIST`, `FieldOwnershipTest.cpp`'s `#if !MOBILEGL_PIPE_PUSH` block).
## G1 / monolith impact
- **Pull build (`MOBILEGL_PIPE_PUSH=OFF`): no change at all.** `PipeFill.{h,cpp}` is appended to
`SOURCE_FILES` only under `MOBILEGL_PIPE_PUSH` (CMakeLists.txt:519-523); `PipeInputs.cpp`'s stamp
is inside `#if MOBILEGL_BUILD_DISAGGREGATED`; `GpuWritePending.cpp` is MG_Remote. The two MG_State
files and the one MG_Impl call site are each `#if MOBILEGL_PIPE_PUSH`. G1 stays 0/0/0/0.
- **Push monolith:** `TextureState` grows 4 bytes (`m_maxTouchedImageUnit`) and `glBindImageTexture`
gains one compare. Behaviour identical - the member has no reader outside MG_Remote.
`PipeFill.cpp`'s memo and the knob fast path are behaviour-identical by inspection.
- **Behavioural delta worth naming:** `MG_Remote::Client::ConsumerRefusals()` will count fewer
refusals, because the residual fill asks `P4aFamilyHasItsConsumer` once per environment instead of
63 times per verb. It is a diagnostic counter; the one-time `MGLOG_W` per refused subsystem still
fires, and no test asserts an exact value that the fill contributes to
(`SanityTest.…CountsTheRefusal` asserts a delta `> 0` from the mint path;
`RemoteClientTest`/`CapsMirrorTest` drive `ServerConsumes` directly).
## What remains per draw in the validate point / draw emit (brief item 4, surveyed not cut)
- `wants()` still calls `P4aFamilyHasItsConsumer` per emitted gate (~10/verb) **and tests the dirty
bit LAST**, so a clean bit still pays the consumer read. Reordering the dirty test first is a pure
win *except* that `ServerConsumes` counts and logs - "a refusal that happened cannot fail to be
counted" is R-8's stated rule. **Left alone deliberately**; it needs R-8's owner to say whether
the counter may move.
- `ContextValuesWireLive()` per verb = three loads (`g_active`, `m_started`, one acquire load). Left.
- `RunsAsTheServerRole()` at PipeFill.cpp:3037 is inside the `tracker.FreshlyPrimed()` arm - a
make-current edge, not per draw. Not hot. (PipeFill.cpp:992's is the respecify path, also not
per draw.) Package D owns the guard itself.
- `EmitTables.cpp`: `ReadDrawBindings()` re-resolves the element buffer's handle every draw
(`MGPipeResourceTrackerInstance().Find` -> `MGPipeSlots().FindByLifetimeId`, which under split also
pays `MGPipeRefuseAllocatorFromApplyThread` - package D's guard). `VertexInputEmit.h:257-274`
computed the same handle at validate, but **only when bit `NewIndexBuffer` was dirty**, so there is
no already-computed value to reuse on a clean-index draw; a latch keyed on the VAO's index slot +
the tracker generation would be needed. **Left** - it is a new latch with its own invalidation
story and belongs with package B/D's tracker work, not in a cut described as "provably redundant".
`RequireSession` and the `g_dropDrawEmission` ordering are one global read each and E2's control
order respectively; nothing else on that path is redundant.
## Risks / not done
- **Not built.** No compiler has seen any of this. The constexpr stamp table
(`MakeAnswerableMaskTable`, a `constexpr` function with loops over `inline constexpr` generated
tables) is the most likely place for a toolchain complaint.
- The memo is a file-scope global, like `g_residualDue` / `g_omission` / the verify latches beside
it; what keeps a single writer on all of them is the verb barrier. If a future change lets two
roles fill concurrently, this global joins the list that has to move - stated in the code comment.
- The image-unit mark's safety rests on `glBindImageTexture` being the only writer of
`m_imageTextureBindings` that can *add* a binding. I checked: the other writer is
`TextureState::MarkTextureObjectForDeletion`, which only unbinds. Tests that poke
`GetImageTextureBinding(u)` directly must note the unit - one such test existed
(`SplitBufferTest`) and was updated; `MG_IntegrationTest` has none.
- Brief item 2 also asked about the residual fill's *object-class* copy cost (`GetBindingSlot` from
`CopyField`, 1.73 self / 45% of it from the draw fill). That cost is the copy itself, not the
predicate: those rows are BARRIER-PULLED and must be copied until P3b/P4b/P7/P8 give them twins.
Nothing cut there; it is package E's question.
## Fix round
Same worktree (`MobileGL-p5d-C`), still uncommitted, still not built. Patch regenerated:
11 files, 587/64. Every blocker and major addressed; all seven minors addressed too (six were
one- or two-line corrections, the seventh is the by-value return).
### MAJOR - the memo's fourth key input is untested (`FieldOwnershipTest.cpp:405`)
**The finding is right, and it is stronger than stated: that input is not merely untested, it is
UNTESTABLE through this observable, and the suggested fix cannot work as written.**
The reviewer proposed flipping the consumer signal around `GetFramebufferBindingSlot`,
`GetTextureUnitObject`, `GetProgramForDraw` or `GetImageTextureBinding`. Those are exactly the
five fields whose emitter belongs to a P4a family - and every one of them is a field
`EmittedCallSuppliesTheWholeField()` answers **false** for (PipeFill.cpp:2276-2290: their storage
is a frontend heap reference - a `BindingSlot<FramebufferObject>`, an `ImageTextureBinding`, a
`TextureUnit`, two `SharedPtr<ProgramObject>` - that no payload can carry, so the fill pulls them
whatever the consumer says). So `supplied` is `false` for all five in BOTH consumer states: the
consumer conjunct is **dominated**, an assertion pair over those fields would read
`EXPECT_FALSE` / `EXPECT_FALSE`, and a memo whose key dropped the signal would pass it - vacuous
in the other direction, which is no better.
What went in instead, in three parts:
1. **`PipeFill.cpp` - a compile-time trip wire.** `NoP4aFamilyFieldIsWhollySupplied()`, a
`constexpr` walk of `kMGPipeFieldEmittedBy` through `SubsystemForEmitter` and
`EmittedCallSuppliesTheWholeField`, with a `static_assert` over it. It fires on the day
P3b/P4b/P7/P8 give one of those rows a twin the applier can write - which is the day the
consumer signal becomes observable - and its message says to go write the pair. This is the
durable guard; it sits next to the predicate and cannot rot.
2. **`FieldOwnershipTest.cpp` - step 4, which MOVES THE SIGNAL FOR REAL.** Under monolith the
signal is `MGPipeGetResourceOps() != nullptr` (PipeFill.cpp:1156-1171), and
`MG_Config::Transport` is a `constexpr` Monolith in every lane but a split one, so
`MGPipeSetResourceOps(&ops)` / `(nullptr)` moves it in the lane this case runs in, with no new
include and no MG_Remote dependency. The case collects the WHOLE 63-field answer set in each
state at `kEverySubsystem` (the only mask at which the fill reads the signal at all), asserts
the signal really moved, names `GetFramebufferBindingSlot` both ways, and asserts the two
answer sets are identical - with a failure message saying that a difference is *correct
behaviour* and the instruction to rewrite the step into the move-and-read-back pair, naming
`CapsMirrorInstance().Adopt()` and the SanityTest case that already drives it for the split
arm. It restores the previous op table on the way out.
3. **The name.** `...ReKeysOnEveryInput` becomes **`...ReKeysOnEveryInputThatMovesAnAnswer`**, in
both the push case and its pull-lane skip twin (G2/G14 parity kept). The finding that the old
name and the red-once table oversold the coverage is accepted.
Red-once table correction for that row: the memo's case covers **three** of the four key inputs
by moving them; the fourth is covered by a domination assertion plus a `static_assert`, because
no observable distinguishes it today.
### MINOR - the consumer read moved to every verb, counters move the wrong way (both reviewers)
Accepted, and fixed rather than merely documented. `SuppliedFieldMask` now reads the signal only
where the walk could reach it:
```cpp
const Bool p4aConsumer = (pushMask & kMGPipeP4aFamilySubsystems) != 0 &&
P4aFamilyHasItsConsumer(kMGPipeP4aFamilySubsystems);
```
At a mask with no P4a bit, `(pushMask & subsystem) != 0` short-circuits every P4a field in the
rebuild and `P4aFamilyHasItsConsumer` returns a constant `true` for every other subsystem, so no
field can reach the mirror - the guard is exact, not conservative. The key stays complete because
`pushMask` is itself in the key, so a mask that gains a P4a bit re-keys on the mask. Net effect:
`ConsumerRefusals()` and `LastRefusedSubsystem()` are now unmoved at the A/B lanes and in the
bring-up window (CallMask 0) where the baseline never touched the mirror, and strictly *fewer*
refusals elsewhere - which makes the report's "will count fewer refusals" line true in both
directions rather than only one. The reasoning is in the code comment at the read.
### MINOR - `SuppliedFieldMask` returned a reference into a memo it rebuilds in place
Accepted. It now returns `MGPipeFieldMask` **by value**; the rebuild fills a local and publishes
it into `g_fillPlan` in one assignment at the end, so no caller can observe a partially rebuilt
mask and no rebuild can zero storage a caller holds. The call site at the walk takes
`const MGPipeFieldMask supplied` (a copy, 16 bytes, read 63 times - cheaper than the indirection
it replaces). The comment that cited `g_residualDue` as precedent now says why that precedent
covers the file-scope-ness but not the width.
### MINOR - the stale-memo failure mode is stated wrong (`PipeFill.h:78`, test header)
Accepted; the reviewer is right. The walk stamps `FilledGen[i] = filled.CurrentVerbSerial`
whether or not it copied, so a wrongly-skipped field reads **FRESH with the previous verb's
value** - a silent stale binding slot, caught only by the verify lane, not
`Fatal{UnmigratedPipeInput}` (poison catches an *unstamped* read, not a stamped-but-uncopied
one). Corrected in the test case header and stated explicitly in `PipeFill.h`, where it is now
the reason the key case is the only guard.
### MINOR - "32 slots" / "32 units wide" (`TextureState.h`, `GpuWritePending.cpp`)
Accepted, in both places. `MAX_TEXTURE_IMAGE_UNITS` is **192**: the replaced sweep was 192
`GetImageTextureBinding` reads per draw, so the measured win is 6x larger than the comment
claimed, not smaller. `GpuWritePending.cpp` now names the constant and the count;
`TextureState.h`'s folding argument now uses real image-unit numbers (a bind of unit 7 pushing
the texture scan out to eight slots, the last advertised unit pushing it to 191) instead of the
invented 32. The report's own prose never quoted 32, so nothing there needed the same fix.
### MINOR - the G2 parity count comment is off by one (`SplitBufferTest.cpp:552`)
Accepted; the block declares eighteen names in each lane (diffed, 18 vs 18). Rather than write
`EIGHTEEN` and let it rot a third time - it read `SIXTEEN` against seventeen before this commit
and `SEVENTEEN` against eighteen after - the sentence now drops the count, says the list below is
the authority, and records that the number has never once matched.
### MINOR - the new producer case binds image unit 4 (`ImageEmitTest.cpp:582`)
Accepted. The case now binds **unit 1**, like every neighbouring case, and - because even that
depends on the backend advertising two image units - it `ASSERT`s that the binding actually
landed (`Ctx().GetImageTextureBinding(1).Texture != nullptr`) before it reads the mark. A bind
refused at `unit >= GetAdvertisedImageUnitCount()` (GL_Texture.cpp:6694) now says so by name
instead of reading as a missing `NoteImageUnitTouched`.
### Not changed, and why
Nothing was disputed on the merits. The reviewers' "checked and cleared" sections (stamp-table
equivalence, memo key completeness term by term, single-writer / no new concurrency hazard, the
image-unit mark's writers, G1 = 0 in the pull build, the new tests' typing,
`ParsePoisonOmissionKnob`'s equivalence) are unaffected by this round: the stamp table, the knob
fast path, the image-unit mark and every `#if` guard are byte-identical to the reviewed patch.
The only disagreement is the one recorded under the major, and it is a disagreement about the
proposed FIX, not about the finding.
### Risks, restated
- **Still not built.** This round adds one more `constexpr` function with a loop
(`NoP4aFamilyFieldIsWhollySupplied`) beside the stamp table's `MakeAnswerableMaskTable`, in the
same standard-conforming shape; no toolchain has seen either.
- The new test step depends on `MG_Config::Transport == Monolith` to move the consumer signal and
skips its body otherwise, so in a split-transport lane step 4 does not run. That is the right
trade today: the assertion it would make there is the same dominated one, and the
`static_assert` covers every lane.
+222
View File
@@ -0,0 +1,222 @@
# P5d round 3, package D - role guards and TLS made nearly free
Worktree `MobileGL-p5d-D`, branch `p5d-D`, base `85362a85`. Not built, not committed, not pushed.
Patch `scratchpad/p5d/patches/D.patch` (618 lines, 11 files). **What changed, per file:**
### `MG_Remote/Server/ServerLoop.{h,cpp}` - item 1, the point of the package
- New `Server::Detail::g_applyThreadKey` (`inline std::atomic<Uint64>`, header) + `Detail::CurrentThreadKey()`.
On aarch64 (`__has_builtin(__builtin_thread_pointer)`) the key is the thread pointer - one
`mrs TPIDR_EL0`, no call; elsewhere `std::this_thread::get_id()` memcpy'd into a `Uint64`. x86_64
deliberately stays on the fallback: the builtin only grew x86 support in clang 17 / GCC 11, and older
toolchains accept `__has_builtin` for it and then fail in codegen - a build break on the desktop
ctest lane for a target nobody profiles.
- `OnApplyThread()` is now **inline in the header**: `key != 0 && key == CurrentThreadKey()`. Gone:
`ServerLoopInstance()`'s function-static guard byte, the acquire loads of `m_running` and of
`atomic<std::thread::id>`, `std::this_thread::get_id()`, the out-of-line `pthread_equal`, and the
cross-library PLT hop for every caller that includes the header.
- `m_applyThreadId` is **deleted**. The key is published as `ApplyThreadMain`'s first statement and
cleared **inside the same `m_controlMutex` critical section that clears `m_running`** (the C2 exit
block), so the two move together and the "live but exiting apply thread" window is byte-for-byte the
old one. `Stop()` no longer clears an identity after the join.
- Relaxed on both sides. The one dangerous shape - a new thread reusing the exited apply thread's TLS
block (hence its thread pointer) while still reading the pre-clear key - cannot happen: the clear is
sequenced before that thread's exit, the exit releases the TLS block under libc's own lock and the
receiving thread takes the same lock, a happens-before chain a relaxed load must respect by
coherence. A stale *zero* is harmless. The argument is written out at the declaration.
### `MG_Impl/Pipe/SlotAllocator.{h,cpp}` and `MG_State/.../MipmapStorage.{h,cpp}` - item 2
`g_reverseAnnouncementScopeDepth`, `g_frontendKeyedRegistryScopeDepth` and
`g_textureLegacyArmScopeDepth` stop being `thread_local`. The three scopes increment/decrement only
when `OnApplyThread()` is true, each keeping that decision in a new `Bool m_counted` so the destructor
undoes exactly what the constructor did (re-asking would leak a count across the apply thread's exit).
**Single-writer proof, in the files:** each counter's only reader in the tree is its apply-thread
guard - `MGPipeRefuseAllocatorFromApplyThread` (`SlotAllocator.cpp:30-31`),
`RefuseLegacyTextureArmFromApplyThread` (`MipmapStorage.cpp:44`) - and both return before they look
unless the transport is active **and** `OnApplyThread()`; I grepped every `::Active()` call site and
those three are all of them. So the apply thread is sole writer and sole reader, and a monolith process
never reads them. `Active()` on the apply thread is unchanged; off it, it is now always false, and both
headers say a future reader has to account for that.
### `MG_State/GLState/BufferState/BufferObject.cpp` - item 3
The Fatal body is split out as `FatalLegacyBufferArmFromApplyThread` (`[[noreturn]]`).
`SyncPersistentMappedRange` called `RefuseLegacyBufferArmFromApplyThread` and then re-asked
`PushIsArmed()`/`OnServerRole()` for an early-out whose condition was *identical* - so its `return` was
unreachable, the guard having already aborted. It now asks once; no accessor makes more than one
`OnApplyThread()` call. `PushIsArmed()` is a read of `MG_Config::Transport`; left alone.
### `MG_Remote/Client/ClientSession.cpp` - item 3
`RefusePipeInputsTouchWhileApplierOwnsIt` reordered cheapest-first: `Transport == Monolith`,
`Ipc.BatchWaits != 0`, then `OnApplyThread()`, `Started()`, `BarrierArmed()`,
`ApplyThreadIsInsideApplier()`, and the `thread_local` `InBarrierWait()` **last**. The chain is a pure
conjunction (every row returns "no violation"), so which touches abort is unchanged; only the cost of
answering "no" moves. With the shipping default `MOBILEGL_IPC_BATCH_WAITS=1` it is two loads on a path
`MGPipeValidateForVerb` reaches per verb (PipeFill.cpp:542, :1951, :2789).
### `MG_Remote/Client/WireTables.cpp`, `.../PersistentMapTracker.cpp`
Comments only. `RunsAsTheServerRole()` and `PersistentMapTracker::OnServerRole()` stay out of line
**on purpose**: inlining them would put a server header inside `WireTables.h` (whose comment says it
exists so PipeFill need not include one) and inside `PersistentMapTracker.h` (which MG_State's
`BufferObject.cpp` includes). One PLT hop is that boundary's price; the four-part predicate behind it
- the measured cost - is gone either way. `IsLivePersistentMap`'s row order is *not* touched: it is
contractually the same chain as `SyncPersistentMappedRange`'s, and item 1 already made its one guarded
accessor (`IsMapped`) cheaper than the reorder would have.
## Why (profile at head 56a77348)
`OnApplyThread()` 2.66% self / 3.64% incl **on the client thread** for an answer that is "no" every
time, plus `@plt` 6.1% self on the apply thread (item 1). `__emutls_get_address` 7.7% and the **top
symbol** on the monolith GL thread, 7.4% on the apply thread, 32.7% of it exactly these scopes
(FrontendKeyedRegistry ctor 18.75 + dtor 13.95, ReverseAnnouncement 8.4), built at ~20 sites in
DirectGLES.cpp including inside `StateBackendObjectRegistry::HandleOf` per draw (item 2). Every cycle
cut on the apply thread shortens the client's R-1 wait one-for-one, so both pay on both threads.
## Red-once
- **Item 1**: `MG_Test/Wire/ServerLoopTest.cpp:291-324` - a posted control request records
`OnApplyThread()` inside itself (`EXPECT_TRUE`) and the test thread asserts `EXPECT_FALSE` after;
publish the key late, clear it early or compare the wrong identity and it goes red. Backed by the 5
`MGL_BUFFER_GUARD_TEST` + 9 `MGL_TEXTURE_GUARD_TEST` cases (`RemoteClientTest.cpp:1841-1912`), each
needing `OnApplyThread()` true inside `RunOnApplyThread`.
- **Item 2**: three cases **added** to `RemoteClientTest.cpp` after the buffer guard block:
`AnAllocatorProbeFromTheApplyThreadOutsideEveryScopeIsFatalByName` (the control),
`AnAllocatorProbeInsideAnExemptionScopeOnTheApplyThreadIsAllowed` (red if the increment is dropped)
and `AnExemptionScopeHeldOnTheGLThreadDoesNotExemptTheApplyThread` - **the one that pins the
single-writer argument**: the scope is open on the GL thread across the posted request and the apply
thread's probe must still abort; make the counter unconditional again and it goes green while a real
violation is silently exempted.
- **Item 3, ClientSession**: pure reorder, no new test; `ClientPipeInputsFillWhileTheApplierOwnsIt
IsFatalByName` (batch off -> Fatal), `...IsAllowedWhenBatchingIsOn` and
`...WithTheApplierIdleIsAllowed` already drive both sides of the row I moved to the front.
**BufferObject**: `BufferSyncPersistentMappedRangeFromTheApplyThreadIsFatalByName` still demands
`Fatal{RoleViolation, "buffer-legacy-arm"}` from that path.
## G1 / monolith impact
**G1 = 0/0/0/0.** Nothing reaches the pull build: MG_Remote and `MG_Test/Wire` are not compiled there
(`MG_Test/CMakeLists.txt:105`), `SlotAllocator.cpp` is a `MOBILEGL_PIPE_PUSH` file (disagg forces push
on, `CMakeLists.txt:496-500`), and every hunk in `BufferObject.cpp`, `MipmapStorage.h` and
`MipmapStorage.cpp` - the new `m_counted` member included - sits inside `#if MOBILEGL_BUILD_DISAGGREGATED`.
**Push build under `Transport=Monolith`**: identical by inspection - no `ServerLoop` runs, so the key
is 0 and `OnApplyThread()` is false exactly as the old predicate was; the scopes count nothing and
`Active()` is false, which the only reader never reaches (first line returns on Monolith). G14: three
names added, none removed. G2: the added names sit in a disagg-only suite beside names already there.
## Risks
1. `__builtin_thread_pointer` is the one new intrinsic; if unavailable the `#if` falls back and only
the speedup is lost. **Not compiled** (no-build rule), so this is inspection only.
2. The relaxed-load recycling argument leans on libc synchronising TLS-block reuse; it is written out
at the declaration so a reviewer can reject it, and an acquire load would cost one `ldar` and keep
every other saving. 3. `Active()` is now always false off the apply thread - harmless today (one
reader, proven), latent for a future second reader; both headers say so.
## What I did not do (item 4 inventory)
- `g_inBarrierWait` (`ClientSession.cpp:117`) - ~900+ set/clear pairs per frame. **Must stay TLS** (the
file's note: a second GL thread would read the first one's wait as its own), so single-writer fails;
only mitigated by moving its read to the end of the guard chain.
- `t_activeDecoder` (`PipeWireCodec.cpp:359`; read :1289/:1292, saved/restored :1385-1393) - per
applied record, so it *is* hot. **Not changed**: its comment says a unit suite drives several
decoders from several test threads, so the apply thread is not its only writer.
- `g_reply` (`PipeRoute.cpp:61`) - the monolith adapter's one-deep reply mailbox, per reply-slot row
rather than per draw, and per-thread by design (R-1 bounds the in-flight depth per thread). **Kept.**
- Cold, off the per-draw path: `tl_initializing` (`Init.cpp:31`), `tl_isPoolThread`
(`ShaderCompilePool.cpp:30`), `t_current` (GLX/WGL), the `static thread_local Vector<GLuint> names`
scratch buffers (`GL_ProgramPipeline.cpp:74,83`, `GL_Sampler.cpp:309,342`), `paddedUbo`
(`UniformManager.cpp:2072`, DirectVulkan); `DirectGLES.cpp:12828` already says why the owner stamp is
not thread_local.
- Also not done: inlining `RunsAsTheServerRole` / `OnServerRole` (layering), reordering
`IsLivePersistentMap` (contract) - both argued in the code comments. No build, no ctest, no device
run (the brief forbids all three): every claim is by inspection, no number was re-measured.
## Fix round
Both reviewers returned `apply-with-minors`, no blockers. Every major and every minor below is
addressed in code except the two that are explicitly integration notes. Still not built, not
committed, not pushed; no attribution lines. Patch regenerated (770 lines, 12 files).
### MAJOR (both reviewers) - the key clear at the `ServerLoop.cpp` C2 exit had no red-once. **Fixed.**
Both reviewers made the same argument and it is correct: the old predicate was
`m_running && id == mine`, so an uncleared identity was harmless by construction; the new one has no
second term, and delete-the-store left every test in the tree green. New case in
`MG_Test/Wire/ServerLoopTest.cpp:829` (right after the C2 `AForwarderCallAfterTheLoopStopped...`
case), `AThreadCreatedAfterTheLoopStoppedIsNotMistakenForTheApplyThread`, in three parts:
- `ASSERT_NE(Detail::g_applyThreadKey, 0)` while the loop is parked, so the post-Stop `EXPECT_EQ`
cannot pass for the wrong reason (a key that was never published is also 0);
- `EXPECT_EQ(Detail::g_applyThreadKey, 0)` after `fixture.Stop()` - reviewer 1's deterministic half;
it pins the **presence** of the store on every platform;
- reviewer 2's half, which pins **why it matters**: 8 sequentially created-and-joined
`std::thread`s, each asserting `ServerLoop::OnApplyThread()` false. With the store removed the
first thread to inherit the exited apply thread's TLS block (hence its thread pointer) answers
true and the case goes red for its own reason. It is second, not alone, because TLS-block reuse is
likely, not guaranteed - the case says so in its own comment.
**Position** (reviewer 1's second half). I did not add a `Detach()`-time hook: it would put a third D
hunk into `ServerLoop.cpp` and deepen the overlap with package T, for a property no unit case can
reach anyway. Instead the C2 comment now names the hazard concretely (`ServerLoop.cpp:458-472`):
`ServerLoopTest` has no GL context, so `CreateBackend` never runs and the
`~BackendObject_DirectGLES` -> `ReleaseEGLResources` self-post does not exist here; move the store
above `m_backend.reset()` and that post misses `RunOnApplyThread`'s re-entrancy arm
(`ServerLoop.cpp:576`, `if (OnApplyThread()) return work(user);`) while `m_running` is still true,
publishes into the one-slot mailbox and waits on `m_controlDone` for an answer only that thread could
give. The comment says in as many words: do not move this on the strength of a green ctest.
### MINOR (both) - the `std::thread::id` fallback copies an object representation with no padding check. **Fixed.**
`ServerLoop.h:178`: added `static_assert(std::has_unique_object_representations_v<std::thread::id>)`
beside the size assert, plus `#include <type_traits>`. I chose this over
`std::hash<std::thread::id>{}(id) | 1u`: the hash is well-defined, but `| 1` is injective only
because today's `pthread_t` happens to be pointer-aligned, and on MSVC (an FNV hash of the id) it
would fold two distinct threads onto one key with no diagnostic - a guard answering **true off the
apply thread**, which is worse than the failure being fixed. A padding implementation is now a build
error, which is the loud-not-silent direction CONTRACT-P5C rule E asks for. The comment beside it
names the concrete consequence both reviewers derived (lost re-entrancy shortcut -> `m_controlDone`
deadlock; exemption scopes stop counting -> `MGPipeRefuseAllocatorFromApplyThread` aborts at the
per-draw `HandleOf` probes).
### MINOR (reviewer 2) - the selector macro is unguarded, never `#undef`'d and not forceable from the build. **Fixed.**
`MOBILEGL_SERVER_APPLY_THREAD_POINTER` -> `MOBILEGL_DETAIL_APPLY_THREAD_POINTER`
(`ServerLoop.h:150-161`): `#ifndef`-guarded, always **defined** (to 0 when unavailable) and tested
with `#if`, not `#ifdef`. `-DMOBILEGL_DETAIL_APPLY_THREAD_POINTER=0` now really disables the builtin
path - the escape hatch for risk 1 the package named itself - and no longer collides with the
header's own define. The nested `#if defined(__aarch64__) && defined(__has_builtin)` /
`#if __has_builtin(...)` shape is kept (GCC < 10 must not evaluate the inner line).
### MINOR (reviewer 2) - `Active()` is silently thread-conditional in public headers. **Fixed by renaming.**
`MGPipeReverseAnnouncementScope::Active()`, `MGPipeFrontendKeyedRegistryScope::Active()` and
`MGPipeTextureLegacyArmScope::Active()` are now `ActiveOnApplyThread()` (declarations
`SlotAllocator.h:205`, `:229`, `MipmapStorage.h:70`; definitions `SlotAllocator.cpp:83`, `:101`,
`MipmapStorage.cpp:94`; the three readers `SlotAllocator.cpp:28-29`, `MipmapStorage.cpp:44`; the
comment at `RemoteClientTest.cpp:1861`). I re-grepped the tree myself: those are all of them, so this
is a closed rename with no external caller. Chose the rename over an `MGLOG_E_ONCE`/assert inside the
query because the guard reads it on the apply thread on the per-draw path, where a diagnostic is
either free and useless or costly and hot; a name that carries the precondition costs nothing and
cannot be ignored by the future P4b/P7 reader the finding is about. Both headers' comments now say
why the name is spelled that way.
### MINOR (reviewer 1) - `docs/Disaggregated/ROADMAP.md:69`'s G4 citation went stale. **Fixed, but OUTSIDE `D.patch`.**
Confirmed: `MobileGLResult ServerLoop::RunOnApplyThread(` is at `:543` at baseline `85362a85` and at
`:570` in this worktree (the fix round's C2 comment moved it a further 15 lines), and the old `:598`
end maps line-for-line to `:625`. The cell now reads `ServerLoop.cpp:570-625`. **`ROADMAP.md` is not under `MobileGL/`**, so the mandated
`git diff HEAD -- MobileGL` cannot carry it; it is delivered as a separate one-line patch beside it,
`scratchpad/p5d/patches/D-docs.patch`. Integrator: apply both.
`python scripts/check_doc_citations.py docs/Disaggregated/ROADMAP.md` -> 56 citations, 0 problems.
Note for the integrator: package T also moves `RunOnApplyThread`, so this cell needs re-checking once
T lands - the gate will not catch it (it only resolves the file and counts lines).
### MINOR (both) - D's `ServerLoop.cpp` hunk overlaps the C2 block package T rewrites. **No code change; integration note, as both reviewers asked.**
Accepted as stated, and I deliberately did **not** shrink the hunk: the key clear has to be inside
the same `lock_guard(m_controlMutex)` critical section as `m_running.store(false)`, or the window in
which a live-but-exiting apply thread answers true moves - which is the property item 1's equivalence
argument rests on. Land **D before T**, or rebase T onto D, and at the merge check that T's
`m_controlPosted` clear and D's `Detail::g_applyThreadKey.store(0)` both end up inside that same
section beside `m_running.store(false)`. The new ServerLoopTest case above is what makes a merge that
drops the clear fail loudly instead of landing green. The `ServerLoop.h` overlap (D prepends the key
block, T adds park counters) is textual only - different regions of the file.
### Findings I did not act on
None outstanding, and nothing in either review was disputed. The two items left uncoded are the two
both reviewers themselves scoped as "flag it to the integrator" (overlap ordering) and the one the
patch boundary cannot carry (ROADMAP, delivered separately).
### Re-verification after the fix round
- `python scripts/check_include_closure.py` -> 4 probes, 0 skipped, 0 problems (unchanged; the rename
and the `<type_traits>` include add no header->header edge).
- `python scripts/check_doc_citations.py docs/Disaggregated/ROADMAP.md` -> 0 problems.
- Doc-citation sweep over every file this package touches: the four `ClientSession.cpp` citations in
`ROADMAP.md` / `CONTRACT-P5C.md` are all at or below `:880` and both D hunks start at `:996`, so
none moved. No document names `Scope::Active()`, so the rename breaks no citation.
- G1 unchanged and still 0/0/0/0: the new test is in `MG_Test/Wire` (not in the pull build), the
rename and the new comments are all inside `#if MOBILEGL_BUILD_DISAGGREGATED`, and `<type_traits>`
enters only `MG_Remote/Server/ServerLoop.h`, which the pull build does not compile.
- Still no build, no ctest, no device run (the brief forbids all three). Every claim here is by
inspection or by a read-only script.
@@ -0,0 +1,288 @@
# E - can the draw barrier be deferred by one verb? (research note, no code)
Head `85362a85`, worktree `MobileGL-p5d-E`, DirectGLES, `MOBILEGL_TRANSPORT=inproc`, push mask
`0x1fff` (every subsystem bit on, so every record arm in the backend is the arm that runs).
Every `file:line` below is at that head. Paths are under `MobileGL/` unless they start with `docs/`.
## 0. The lockstep as it is wired today (the thing (c) would move)
1. `MG_Impl/GLImpl/Drawing/GL_Drawing.cpp:544-550` - `DrawElements_Backend` runs `MGP_FILL(DrawElements)`
(= `MGPipeValidateForVerb`, `MG_Impl/Pipe/PipeFill.cpp:2781`) and only then calls the backend table
slot, which under a transport is the client emit table.
2. `PipeFill.cpp:2789-2790` (`RefusePipeInputsTouchWhileApplierOwnsIt`), `:2814` (withdraw the server
stamp), `:3059-3104` step 4: for every field in the kDraw mask not supplied by a record,
`MGPipeFillAccess::CopyField` (`:88-330`). The object-class rows copy either a **SharedPtr**
(VAO `:113`, dispatch program `:216`, draw program `:219`, XFB program `:262`) or a **raw pointer
into the live GLContext arrays** (15 buffer slots `:119-121`, 15 binding-point row bases `:127-129`,
2 framebuffer slots `:174-178`, image-unit base `:183`, texture-unit base `:246`). Storage:
`MG_Backend/MGPipe/PipeInputs.h:807-815`.
3. `MG_Remote/Client/EmitTables.cpp:529-553` `EmitIndexedDraw` -> `ReadDrawBindings` (`:459-482`, reads
the VAO element slot + the two indirect slots on the GL thread) -> `EmitDrawRecord` (`:484-525`):
`BeforeDrawVerb` (`:222-226`: `PushDrawConsumers`, deferred-destroy drain, `MarkGpuWritesForDraw`)
then `EmitAndWaitTails`.
4. `MG_Remote/Client/ClientSession.cpp:814-990`: encode, `PublishAndNotify` (`:871`); DrawVbo is
`kCtxVerb`, so the BatchWaits early return at `:899-905` does not apply; `WaitForAppliedBudget`
(`:916-917`), then `DrainEventRing` (`:951` - "the one instant at which the apply thread is known
not to be inside the applier").
5. Server: `MG_Remote/Server/PipeApplier.cpp:1043-1080` `ApplyOne` = `ScopedApplierEntry` (`:1057`),
`StampVerbBoundary` (`:1061` -> `PipeInputs.cpp:103-130`: every BARRIER_PULLED field's `FilledGen`
is set to 0, so each server read of one is counted as `rsp` by `MGPipeInputUnfreshRead`
`:137-170`), `DecodeAndApply` -> `OnDrawVbo` (`:430`) -> `gl.DrawElements` ->
`MG_Backend/DirectGLES/DirectGLES.cpp:6705-6716` -> `PrepareForDraw` (`:4762-4830`), then
`LeaveApplier`. `MG_Remote/Transport/SessionRings.h:378-394`: apply -> `++appliedSeq` ->
`PublishApplied` -> ring. The client is runnable again only after step 5 has fully returned.
What the barrier protects is stated at `PipeInputs.h:904-910`: `m_filled` (`FilledGen[63]`,
`CurrentVerbSerial`), `m_serverStampedVerb` and the `rsp` counter are plain, not atomic, because
"the verb barrier leaves at most one of {GL thread, apply thread} runnable". `CONTRACT-P5C.md:35-43`
(rule E) says "the barrier held" is no longer an accepted mechanism - and then `:157-201` (§3.1) and
`:410-437` (§5.4) grant exactly the object-class family and the two scopes a named exemption from it.
## (a) Every server-side read that reaches client-owned memory through the object rows
Legend: **live** = a dereference of the frontend array/object the row points at; **id** = a
`GetLifetimeId()` read plus a probe of the CLIENT slot allocator (`MGPipeSlots().FindByLifetimeId`,
`DirectGLES/SlotTables.h:436-449`, inside `MGPipeFrontendKeyedRegistryScope` or
`MGPipeReverseAnnouncementScope`; `MG_Impl/Pipe/SlotAllocator.cpp:22-38` is the guard the scope
disarms); **mint** = `GetOrCreate(const StatePtr&)` -> `MGPipeSlots().Acquire` (`SlotTables.h:244-248`),
a WRITE into the client allocator; **rec** = the same site has a record arm that reads no frontend
memory and is the arm that runs on this workload.
### a.1 `GetBoundVertexArray` (SharedPtr copy, `PrepareForDraw` `:4770`)
| site | what is dereferenced | kind |
|---|---|---|
| `ResolveVaoTwin` `:1602-1633` (scope `:1609`) | `g_backendVertexArrayObjects.Find(vao.get())` -> `SlotTables.h:411-414` -> `HandleOf` `:436-449`: `vao->GetLifetimeId()` + allocator probe; `GetOrCreate(vao)` on a miss `:1618` | id, mint |
| `:4776` | `currentVAO->GetConfigVersion()` | live |
| `SyncVaoAttributeBuffersByHandle` `:699-790` | memo hit `:719-737`: per entry `IsBufferDrawCleanByHandle(entry.handle, entry.resource, entry.frontend)` (`Managers.cpp:3109-3153`; under a transport `:3143-3145` does NOT touch `frontend`), dirty repair `:731` `currentVAOObject->GetAttribute(i).Buffer` -> `EnsureBufferResource`; memo miss `:741-780`: `GetAllAttributes()` (32 x `{Enabled, Buffer SharedPtr, ...}`), `attrib.Buffer.get()`, `EnsureBufferResource(bufferObject)` (`Managers.cpp:3448-3455` -> `HandleOfBuffer` `:2922-2936`, scope `:2933`: `GetLifetimeId` + allocator probe), `HandleOfBuffer(bufferKey)` `:767` | live, id |
| IBO arm `:823-861` | `currentVAOObject->GetIndexBufferBindingSlot().GetBoundObject()` (SharedPtr in the VAO), `HandleOfBuffer(possibleIBO.get())`, `EnsureBufferResource(possibleIBO)` | live, id |
| `SyncCurrentVAO` `:1637-1649` -> `SyncToBackend` (`Managers.cpp:4881-4891`) -> `SyncToBackendFromApplier` (`:5160+`) | nothing: "the frontend VAO is not read at all"; the index arm resolves `st.IndexBuffer.Res` only (`:5320-5331`) | rec |
| `SyncCurrentVertexAttributeValues` `:1659-1700` | `GetBoundVertexArray()` again `:1669`, `vao->GetConfigVersion()` `:1682`, memo miss `vao->GetAttribute(location).Enabled` `:1686` | live |
| `DrawArrays` `:6718-6738` (scope `:6729`), `MultiDrawArrays` `:6775` | `Find(currentVAO.get())` + `SyncClientSideAttributesForDrawArrays(currentVAO, ...)` | id, live |
| `EnsureBufferResourceForHandle` `Managers.cpp:3193-3446` | under a transport the object is HELD, not read: base `:3207-3222` = `resource->hostBytes`, `SyncPersistentMappedRange` skipped `:3329-3341`, content bit from the descriptor `:3376-3381`; it still takes the SharedPtr by reference | held only |
### a.2 `GetProgramForDraw` (SharedPtr copy, `:4781`; the client-side `GetProgramForDraw` join is `MG_State/GLState/Core.cpp:609-628` and runs at fill, not on the server)
| site | what is dereferenced | kind |
|---|---|---|
| `SyncCurrentProgram` `:4321-4400` (scope `:4329`) | `GetLinkStatus()/GetSpirvStatus()` `:4339`; `g_backendProgramObjects` probes; broadcast count from the framebuffer record when `g_fboRecordsTrusted` `:4350-4372`, else `GetFramebufferBindingSlotChecked(Draw)` + `drawFBO->GetObjectVersion()/GetDrawBuffers()` `:4384-4395` | live, id, rec |
| `BindCurrentTextures` `:5245-5305` | `currentProgram.get()` as key, `GetLifetimeId()`, `GetBackendStateVersion()`, `GetLinkStatus()` `:5279-5282`; miss -> `ResolveAndBindUnitTextures` `:4836+`: `GetMaxUniformLocation/GetUniformSamplerOrImageUnitIndex/GetUniformType` `:4845-4855` (only on an alias conflict) | live |
| `BindCurrentProgramWithResources` `:5317-5600` (scope `:5328`) | `GetLinkStatus/GetSpirvStatus` `:5320`, registry `Find` fallback `:5336`, `GetUBOSize` `:5347,:5371`, `MapUBO()`/`GetUBOContentVersion()` ONLY when no ShaderCso record `:5364-5387` (record arm `:5364-5368`), `GetUniformBlockBinding(i)` `:5463`, `GetBackendStateVersion` `:5514`, sampler pass miss `GetUniformSamplerOrImageUnitIndex` `:5537-5539` | live, rec (UBO bytes) |
| `SyncAtomicCounterBuffers` `:600-660` | `GetBufferBindingPointCount` (sticky forward -> `PipeFill.cpp:1861-1865`, live context) - only for programs with counters | live |
### a.3 `GetTextureUnitObject` (raw base of `Array<TextureUnit,192>`; a unit is 11 `BindingSlot<ITextureObject>` + one `SharedPtr<SamplerObject>`, `MG_State/GLState/TextureState/TextureUnit.h:15-26`, `MG_Util/Types.h:181-201`)
| site | what is dereferenced | kind |
|---|---|---|
| `CaptureDrawTextureSyncKeys` `:2188-2211` -> `CurrentUnitBindingsEpoch` `:2044-2098` | record arm `:2048-2056` reads two applier serials (`SamplerViewsSerial`, `SamplerStatesSerial`); walk arm `:2073-2097` -> `CaptureUnitBindings/UnitBindingsUnchanged` `:1867-1920`: per unit `GetAllBindingSlots()[i].GetBoundObject()->GetLifetimeId()`, `GetSamplerObject()` | rec; walk = live |
| `SyncNeccessaryTextures` `:2333-2480`, memo HIT `:2342-2362` | `PairingsIntact` `:2134-2139`: `entry.slot->get()` where `entry.slot` is `const SharedPtr<ITextureObject>*` INTO the frontend unit's slot (`:2127-2131`, borrowed at `:2374`); then `entry.backend->IsDrawSyncClean(entry.slot->get(), ...)` `:2356` - the texture object's versions. **This runs on every draw even when every memo hits.** | live |
| same, memo MISS `:2372-2385` | `GetTextureUnitObject(index)` per unit, every slot's `GetBoundObject()`, `IsUndefinedDefaultTexture`, `SyncTextureObjectToBackend(textureObject)` (`:1725-1760`, scope `:1734`: `Find` id / `GetOrCreate` mint, then params/sampler/mips sync of the object) | live, id, mint |
| `ResolveAndBindUnitTextures` `:4861-4880` (memo miss of `BindCurrentTextures`) | per unit `GetAllBindingSlots()`, `GetBoundObject()`, twin `Find` (scope `:4926`) | live, id |
| sampler pass `:5562-5582` | `GetTextureUnitObject(unit).GetSamplerObject()`, `GetBindingSlot(Texture2D).GetBoundObject()`, `samplerObject->GetLodBias()`, `texture2D->GetFormat()/GetSamplerObject()` | live |
| `BindCurrentUnitSamplers` `:5051-5170` | record arm `:5111-5150` reads `st.BoundSamplerStates[]` only; decline arm `:5157` `GetTextureUnitObject(unit).GetSamplerObject()` | rec |
### a.4 `GetImageTextureBinding` (raw base of `Array<ImageTextureBinding,192>`, `TextureState.h:18-27`: SharedPtr + 5 scalars)
| site | what is dereferenced | kind |
|---|---|---|
| `SyncImageTextureBindingsForDraw` gate `:2870-2900+` | `g_imageUnitHighWaterMark == 0` early-out (`:2893`): zero reads on a draw with no image unit ever bound | - |
| `SyncImageTextureBinding` `:2636-2748` | `imageBinding.Texture` (SharedPtr) `:2650,:2660`, `ResolveShaderImageRecord` (`:2582-2635`, scope `:2588`; identity test `view.Res != HandleOf(boundTexture)` `:2626`), Level/Layer/Layered/Format from the record when it matches `:2670-2676`, **`imageBinding.Access` always frontend** `:2745` (`:2663-2669`: the record's `Access` byte has no written encoding) | live, id, rec (4 of 5 fields) |
| `MarkWritableImageBufferTexturesGpuWritten` `:2764-2785` | gated `g_writableImageBufferUnitCount == 0`; else `imageBinding.Texture.get()` -> `TextureObjectBuffer::GetBufferBindingSlot().GetBoundObject()` -> `MarkBufferGpuWritten` (`Managers.cpp:2947`, id) | live, id |
### a.5 `GetFramebufferBindingSlot` (raw pointers to the two `BindingSlot<FramebufferObject>`)
| site | what is dereferenced | kind |
|---|---|---|
| `SyncCurrentFBO` `:3277-3330` | `SyncCurrentFBOByRecord` first `:3292`; decline (scope `:3303`): slot `:3305`, `GetBoundObject()/GetVersion()`, `currentFBO->GetObjectVersion()`, attachment walk | rec |
| `BindCurrentFBO` `:4514-4600` | record arm `:4527-4560` (`FindByHandle(record.Fbo)`); decline `:4568-4590`: slot, `GetBoundObject()`, compare with `pDefaultFramebufferInfo->defaultFBO` `:4577`, `Find(currentFBO.get())` (scope `:4585`) | rec |
| `SyncNeccessaryTextures` draw-FBO list `:2405-2470` and `SyncReadFramebufferTextureAttachments` `:2246-2331` | **always** `GetFramebufferBindingSlotChecked(target)`, `GetBoundObject()`, `GetVersion()`, `currentFBO->GetObjectVersion()`, and on a list rebuild `currentFBO->GetAllAttachmentObjects()` / `attachment.GetTexture()` (borrowed SharedPtr slots); the record only replaces the VERSION half of the key, "THE IDENTITY HALF IS STILL A POINTER COMPARE" `:2159-2168` | live |
| `Clear` `:6180-6190`, `BlitFramebuffer` bound arm `:8380-8418` | the three functions above, plus `GetFramebufferBindingSlot(Read/Draw).GetBoundObject()` `:8407-8410` for the layered-blit workaround; the named arm `:8330-8378` reads handles and the twin table's `StateForHandle` note | live / rec |
### a.6 `GetBufferBindingSlot`, `GetBufferBindingPoint`, the rest
- `GetBufferBindingSlot` (15 pointers): on the inproc DRAW path there is **no dereference** - the
indirect arm reads `MGPipeApplier().VerbIndirectBuffer` (`:941-960`; P5c hd §3.5), the
`:962` slot read is the monolith `else`. Readers are readback (`:10232` etc.) and the monolith
`ReadIndirectCommandBytes` `:401`. The row is still FILLED every draw (15 x `GetBufferBindingSlot`
in `CopyField` `:119-121` - the `GetBindingSlot` 1.73% in the profile).
- `GetBufferBindingPoint` (15 row bases into `Array<BindingSlotRange1D,84>`): `SyncBufferBindingPoints
(ShaderStorage)` `:477-521` and `MarkShaderStorageBuffersGpuWritten` `:585-598` (count is
record-supplied; loop is over 0 entries on Minecraft), the UBO loop `:5463-5480` (1-3 points per
program on Minecraft: `point.GetBoundObject()`, `point.GetRange()` - which itself reads
`object->GetSize()` when no explicit range, `Types.h:218-224` -, `GetBufferResource`,
`IsBufferDrawClean` -> `HandleOfBuffer` id, `EnsureBufferResource`), atomic counters `:628`, XFB
`:1407-1420,:569` (only with a pending span). All **live + id**.
- `GetProgramForDispatch`: `PrepareForCompute` `:6126` (dispatch only; same shape as a.2).
- `GetTransformFeedbackProgram`: `StartPendingTransformFeedback` `:1381-1420` at the tail of every
draw (`:4829`) but returns at `:1386` unless a span is pending.
- Sticky forwards on the draw path: `GetBufferBindingPointCount` (`:621`, atomic counters only).
`GetProgramObject/ValidateProgramName` (`:6152-6157`), `GetTextureObject` (CopyTex),
`HasOpenTransformFeedbackSpan`, `RecordError` (`:8804`) are off the steady draw path.
### a.7 The two exemption scopes
`MGPipeFrontendKeyedRegistryScope` (20 sites: `:1609, :1734, :2588, :3303, :4236, :4329, :4585,
:4926, :4993, :5328, :5689, :6166, :6729, :6775, :8837, :9161, :9216, :9695, :10029, :11884`) wraps a
`Find` / `HandleOf` / `GetOrCreate(const StatePtr&)` on one of the five frontend-keyed twin tables;
each is `stateObj->GetLifetimeId()` + `MGPipeSlots().FindByLifetimeId` (`SlotTables.h:436-449`),
and a miss on `GetOrCreate` is `MGPipeSlots().Acquire` (`:244-248`). `MGPipeReverseAnnouncementScope`
(`Managers.cpp:2933`) wraps the same probe for every buffer (`HandleOfBuffer`), reached from every
`EnsureBufferResource` / `IsBufferDrawClean` / `MarkBufferGpuWritten`. Both counters are
`thread_local` (`SlotAllocator.cpp:41-70`) - the 7.4% apply-thread emutls and 3.0% `HandleOf<ITextureObject>`
in the profile are these probes. On the steady Minecraft draw (memos hitting) the scope sites that
still execute per draw are `:1609` (VAO twin), `:4329` (program), `:5328` (program resources),
`:2933` for each UBO point, and `:1734`/`:4926` only on memo misses.
## (b) Which of those reads are UNSAFE if the client runs draw N+1's GL calls before the server applied draw N
Between `PublishAndNotify` for N and `MGPipeValidateForVerb` for N+1 the GL thread runs arbitrary
entry points: `glBindVertexArray`, `glBindBuffer(Base/Range)`, `glActiveTexture`/`glBindTexture`,
`glBindSampler`, `glUseProgram`, `glBindFramebuffer`, `glVertexAttribPointer`, `glEnable/DisableVertexAttribArray`,
`glBufferData/SubData`, `glTexParameter*`, `glUniform*`, `glLinkProgram`, `glDelete*`, plus the client's
own pre-verb work (`PushDrawConsumers` `PersistentMapTracker.cpp:704-745`, `MarkGpuWritesForDraw`
`GpuWritePending.cpp:126-138`, `ReadDrawBindings`). Four classes of hazard:
1. **The raw-pointer rows point INTO live arrays, so every deref reads draw N+1's binding, and the
read races the client's write.** `BindingSlot::Bind` is a non-atomic `shared_ptr` move plus
`++m_version` (`Types.h:189-195`); `BindingSlotRange1D::SetRange` is a two-word store (`:229-232`).
Concretely unsafe: `PairingsIntact` / `IsDrawSyncClean(entry.slot->get())` (`:2136, :2356` - on
EVERY draw, memo hit or not), the unit walks (`:1871, :2374, :4861, :5562`), the UBO point reads
(`:5464-5466`, incl. `GetRange()` -> `obj->GetSize()` while `glBufferData` may resize), the
framebuffer slot reads (`:2405-2410, :2246, :3305, :4568`) and `GetAllAttachmentObjects()` while
`glFramebufferTexture` mutates the container, image bindings (`:2650-2660, :2745`). A torn
`shared_ptr` read is UB; a texture the client just `glDeleteTextures`'d while the server holds
only a borrowed slot pointer is a use-after-free (the borrow rules at `:2103-2126` are written
on the barrier). **Safe**: `GetBufferBindingSlot` (no draw-path reader under a transport).
2. **The SharedPtr rows pin identity but not internals.** VAO: `GetConfigVersion` (`:4776,:1682`),
`GetAllAttributes()/GetAttribute(i).Buffer` (`:731, :743-760, :1686`), the element slot (`:824`)
move under `glVertexAttribPointer`/`glBindBuffer(ELEMENT_ARRAY)` on the same VAO - wrong draw AND a
`shared_ptr` race. Program: `MapUBO()` (`:5385`, only without a ShaderCso record) copies the
scratch `glUniform*` writes into - a torn UBO upload; `GetUniformSamplerOrImageUnitIndex`,
`GetUniformBlockBinding`, `GetBackendStateVersion`, link/SPIR-V status move under
`glUniform1i`/`glLinkProgram` (`Core.cpp:609-628` joins the pending link on the CLIENT at fill,
so a relink issued after publish N is exactly the window the join exists to close). Objects reached
through the rows: `GetSize` (`glBufferData`), texture params/shape for `IsDrawSyncClean` and
`SyncTextureParamsToBackend` (`glTexParameter`), `GetLodBias` (`glSamplerParameter`), FBO draw
buffers/attachments. Only `GetLifetimeId()` is immutable and safe.
3. **The client allocator is client memory, mutated by every create/delete.** Every `HandleOf` /
`Find` in both scopes (`SlotTables.h:445-446`, `Managers.cpp:2935`) probes `MGPipeSlots()`; a
`GetOrCreate` miss WRITES it (`:248`). `glGen*`/`glDelete*` between two draws make this a data
race on a hash container whatever draw the probe "means" - and it is unsolved by any snapshot of
the pointer rows. This is rule E's own sentence at `SlotAllocator.cpp:30-35`, currently
suspended by the two scopes on the strength of the barrier.
4. **`gPipeInputs` itself is one block.** N+1's fill rewrites the pointer rows, `CurrentVerbSerial`
and `FilledGen[]` (`PipeFill.cpp:2806, :3068-3103`) while N's stamp (`PipeInputs.cpp:110-127`)
and N's reads are in flight; `RefusePipeInputsTouchWhileApplierOwnsIt` (`ClientSession.cpp:998-1022`)
is disarmed under `BatchWaits=1` on the argument that "the pull-verbs' own wait keeps their
pulled reads fenced" (`:1006-1012`, `:889-898`) - deferring the draw wait removes precisely that
fence. `DrainEventRing` (`:951`) and the `OnBufferWriteback`/`OnGpuWritten` callbacks that write
frontend objects also rest on it.
Net: on the inproc draw path, every object-class read except `GetBufferBindingSlot` is unsafe under
one-verb pipelining; the record arms (a.1 `SyncToBackendFromApplier`, a.3 epoch + sampler bind,
a.5 `SyncCurrentFBOByRecord` + `BindCurrentFBO`) are the only parts already safe.
## (c) The minimal "snapshot at fill" scheme
Two layers have to be snapshotted, not one:
**Layer 1 - the block.** `gPipeInputs` becomes a ring of >= 2 `PipeInputs` (63 fields; the value rows
are record-supplied and need no copy, but `m_filled`, `m_currentVerb`, `m_serverStampedVerb` are per
slot). The three doors (`MGPipeFillAccess` `PipeFill.cpp:69`, `MGPipeStampAccess` `PipeInputs.cpp:95`,
`MGPipeApplyAccess`), the generated poison (`PipeFilled.inc`), the verify comparator
(`PipeFill.cpp:470-510`) and the stamp all address ONE block today. Cost per draw: 4 SharedPtr copies
(8 atomic RMW) + 34 pointers (272 B) - trivial - but the pointers are useless as a snapshot: they
name live arrays (b.1).
**Layer 2 - what the pointers point at.** The closure the server dereferences per draw, if copied
by value (upper bound, everything a.1-a.6 reaches on a memo miss):
| family | per draw | bytes | refcount ops (inc+dec) |
|---|---|---|---|
| texture units 0..maxTouchedUnit (Minecraft ~4-8) | 12 SharedPtr per unit (11 slots + sampler) + versions | 8 x ~320 B = ~2.5 KB | ~96-192 |
| image units 0..highWaterMark | `ImageTextureBinding` x n (0 on Minecraft) | 0-40 B x n | 0-2n |
| UBO points per program block (1-3) | SharedPtr + `Range1D` + explicit flag | ~40 B each | 2-6 |
| SSBO / atomic / XFB points | 0 on Minecraft | 0 | 0 |
| FBO slots x2 + attachment SharedPtrs | 2 SharedPtr + versions (+ n attachments for Iris) | ~64 B (+24n) | 4 (+2n) |
| VAO: identity + config version + (memo miss) 32 attributes + IBO | 1 SharedPtr + 4 B; miss: 32 x ~48 B + 1 | ~24 B / ~1.6 KB | 2 / 2+2k |
| program: identity + 6-10 scalars, `MapUBO` bytes only without ShaderCso | 1 SharedPtr + ~64 B | ~80 B | 2 |
| **total, steady (memos hit)** | | **~3 KB** | **~110-210** |
At 852 draws/frame: ~2.5 MB/frame of memcpy (same order as the round-1 persistent-map push that
was worth cutting) and ~95k-180k lock-prefixed RMW/frame; at 20-40 ns each that is **2-4 ms per
15.6 ms frame on the GL thread** - comparable to the wait it would remove - and it does NOT fix
(b.2) (object internals still live), (b.3) (allocator probes) or the borrowed-slot lists
(`:2127-2131`), which would have to be rebuilt to point at the snapshot rather than at the unit.
A **handle-only snapshot** (lifetime id / `{slot,gen}` + version per entry, no SharedPtr) is
~1.5 KB/draw, zero refcount ops, ~1 us/draw. But the server can only consume it if every twin
lookup is BY HANDLE (`FindByHandle`, `SlotTables.h:420-425`) and `IsDrawSyncClean` compares
server-owned serials - which is the G6 rekey (ROADMAP.md:71) plus the `set_shader_buffers` emission
(CONTRACT-P5C.md:169-176). So the only snapshot that is both minimal and SAFE is not a snapshot:
it is the retirement of the frontend-keyed path.
**Where the client would wait.** Today: `ClientSession.cpp:916-917`, immediately after `:871`.
Under one-verb pipelining: `EmitDrawRecord` publishes N and returns; the GL thread runs its
inter-draw work; at the top of `MGPipeValidateForVerb(N+1)` (`PipeFill.cpp:2781-2790`, where the
`RefusePipeInputsTouch...` guard already stands) it waits `appliedSeq >= seq(N)` BEFORE the residual
fill writes the next block (or before it recycles slot N of the ring), then drains events. The
reply-slot rows (`:876-879`) and `PushDrawConsumers`' `resource_subdata` records (ordered before
N+1 on the ring, `EmitTables.cpp:215-219`) keep their shape. Overlap gained = min(client inter-draw
work, server apply of N).
## (d) What the pushed records already cover, so the server COULD stop reading the live tables
| record(s), applier state (`MG_Pipe/PipeApply.h`) | draw-path consumer that already uses it | live read that remains and what would replace it |
|---|---|---|
| `bind/create_vertex_elements` -> `VertexElementsCsos`, `BoundVertexElements` (`:487,:594`); `set_vertex_buffers` -> `VertexBuffers[32]{Res,...}`, `VertexBuffersSerial` (`:598-616`); `set_index_buffer` -> `IndexBuffer.Res` (`:620`) | `SyncToBackendFromApplier` (`Managers.cpp:5160+`, reads nothing frontend); memo KEY of `SyncVaoAttributeBuffersByHandle` (`:703-716`); index resolve `:5320-5331` | the ensure WALK (`:731, :741-780`) and the IBO arm (`:823-861`) still take the VAO's attribute/element SharedPtrs to reach `HandleOfBuffer`; under a transport `EnsureBufferResourceForHandle(nullptr, h)` needs no object (`:3193-3222`, the indirect arm at `:953` already calls it that way) - walk `st.VertexBuffers[Start..+Count].Res` and `st.IndexBuffer.Res` instead. `ResolveVaoTwin`'s twin is keyed by VAO lifetime id, the CSO handle is content-addressed (`SetHashSuppressor`) - the twin needs a rekey (P3b/P4b). `SyncCurrentVertexAttributeValues` can read `rec->Attributes[i].Enabled` |
| `set_sampler_views` -> `BoundSamplerViews[unit] = {View, Texture, Unit}` (`MGPipeTypes.h:856-861`), `SamplerViewsSerial`; `bind_sampler_states` -> `BoundSamplerStates[]` (`PipeApply.h:673-681`); tx staged shadow + `set_texture_params` | `CurrentUnitBindingsEpoch` record arm (`:2032-2056`), `BindCurrentUnitSamplers` record arm (`:5111-5150`) | `SyncNeccessaryTextures`' unit list (`:2372-2385`, and its per-draw `PairingsIntact`/`IsDrawSyncClean` on borrowed slots), `ResolveAndBindUnitTextures` (`:4861-4880`), the sampler pass (`:5562-5582`). One view per unit is already the RESOLVED sampled target, so the 11-slot alias resolution (`:4864-4880`) collapses; `IsDrawSyncClean` would key on the applier's texture serials. Caveat A8 (`:2816-2822, :5127-5133`): nothing pins the window to cover `[0, maxTouchedUnit]` |
| `set_shader_images` -> `BoundShaderImages[unit] = {Res, Unit, InternalFormat, Layer, Level, Layered, Access}` (`MGPipeTypes.h:877-886`) | `SyncImageTextureBinding` takes Level/Layer/Layered/Format from the record (`:2670-2676`); the window/mark union (`:2837-2842`) is server-owned | `Access` is carried but deliberately unread (`:2663-2669`: encoding undefined) -> define it; the identity test `HandleOf(boundTexture)` (`:2626`) -> `FindByHandle(view.Res)`; `MarkWritableImageBufferTexturesGpuWritten` needs the TexBuffer's backing-buffer handle (not in `MGPImageView`) |
| `set_framebuffer_state` -> `FramebufferRecords[h] = {Fbo, Color[8]/Depth/Stencil surfaces, DrawBuffers, IsDefault, ...}` (`MGPipeTypes.h:750-760`), `BoundFramebuffer[2]`, `FramebufferSerial` (`PipeApply.h:639-654`) | `SyncCurrentFBOByRecord` (`:3292`), `BindCurrentFBO` record arm (`:4527-4560`), broadcast count (`:4350-4372`), the ContentHash half of both texture-list keys | the identity half of `g_fboTextureSyncList` / read list (`:2405-2410, :2246-2258`) and their attachment borrows -> `FindByHandle(record.Fbo)` + the record's surface handles; `BlitLayeredDestinationAspects` (`:8407-8410`) -> the twins' `StateForHandle` note as the named arm already does (`:8355-8362`); `pDefaultFramebufferInfo` (G6) |
| `set_draw_program` -> `DrawProgram`, `bind_shader_state` -> `BoundShaderCso`, ShaderCso records with `GlobalConstants` bytes + version (`PipeApply.h:503,:689-693`) | UBO bytes and version (`:5364-5368`), program twin caches block indices / sampler bindings at link (`:5449, :5537`) | link/SPIR-V status, `GetLifetimeId`, `GetBackendStateVersion`, `GetUBOSize`, `GetUniformBlockBinding(i)`, `GetUniformSamplerOrImageUnitIndex(loc)` - the reflection/binding table (ROADMAP.md:32 "program registry 重键", ARCHITECTURE 3.2 "explicitly-not-ported") |
| `set_context_values` (P5c rv) -> the 8 value rows | `SyncBufferBindingPoints` count, `maxTouchedUnit`, XFB flags | - (RECORD_SUPPLIED) |
| `set_shader_buffers` / `set_stream_output_targets` (catalogued, **not emitted until P4b**, CONTRACT-P5C.md:169-176) | nothing | the 84-entry `GetBufferBindingPoint` rows: the UBO loop `:5463-5480` (1-3 per Minecraft draw), SSBO/atomic/XFB. **The one family with no producer at all** |
Also structurally: `EnsureBufferResourceForHandle` already reads nothing from the object under a
transport (a.1 last row) - the frontend argument survives only so `HandleOfBuffer(obj)` can find
the handle. Every consumer that has the handle in the applier (`VertexBuffers[].Res`,
`IndexBuffer.Res`, `VerbIndirectBuffer`) could pass `(nullptr, h)` today; UBO/SSBO/atomic/XFB
cannot until `set_shader_buffers` crosses.
So the records are ~80% there in SHAPE (vertex / sampler / image / framebuffer / program handles
all cross) and ~0% there in the IDENTITY path (every twin table is still frontend-keyed, every
buffer ensure still goes through `HandleOfBuffer(obj)`); plus one family (`GetBufferBindingPoint`)
has no wire form yet.
## (e) Cost / benefit and recommendation
**Benefit bound.** One-verb pipelining overlaps the client's non-wait work with the server's apply.
Profile at 56a77348: client thread 39.5% of cycles, `WaitForApplied` 27.9 self / 34.2 incl, so the
client's overlappable work is ~5-12% of process cycles; the apply thread's real work (`DrainRing`)
is 29%. Best case, with the frame server-bound: ~10-20% of the critical path, ~1.5-3 ms of a
15.6 ms frame, i.e. 64 -> ~72-80 fps - and only if the snapshot costs nothing. The gap to monolith
(110-118) is the apply thread's own work (49% of it is the idle-poll mutex, package T) and the
per-draw guard/TLS/validate overhead (D, C): those cut both sides one-for-one and carry no barrier
change. After T/D/C land the client's non-wait share should be re-measured; if it falls under ~10%
the pipelining gain is inside the noise of a paired A/B.
**Cost.** (1) The full-copy snapshot (c) costs 2-4 ms/frame at 852 draws in refcount traffic alone and
is still incorrect ((b.2), (b.3)); the handle-only snapshot is cheap but presupposes the G6 rekey.
(2) The block itself has to be versioned/double-buffered: every stamp, fill, poison and verify
site assumes one `gPipeInputs` - this is exactly what `P5D-INPROC-PERFORMANCE.md:103-106` assigns to
P11 ("`gPipeInputs` 版本化/双缓冲"); note ROADMAP.md:37's P11 row (persistent map / adoption) does
not yet list that item - a doc gap to close whichever way this goes. (3) The 20 frontend-keyed
scope sites + `HandleOfBuffer` + `set_shader_buffers` = ROADMAP.md:32's P3b/P4b row verbatim.
(4) R-1's contract text (`ClientSession.cpp:889-898`, `PipeInputs.h:904-910`, ARCHITECTURE.md:528-530),
the R-1 negative control, `RefusePipeInputsTouchWhileApplierOwnsIt`, the strict lane's expected-red
and `rsp` accounting all have to be re-derived. (5) With the registries still frontend-keyed, the
7.4% emutls / 3% `HandleOf` on the apply thread get worse, not better, under any snapshot.
**Recommendation.** Do not take the deferred draw barrier into P5d round 4. The reads in (a) show
that on the inproc draw path every object-class row except `GetBufferBindingSlot` is dereferenced
live, and (b) shows that most of those dereferences are data races the moment the client moves on -
the racy identity path (twin tables keyed by frontend lifetime id, `HandleOfBuffer` through the
client allocator) cannot be snapshotted, only retired, and one family (`GetBufferBindingPoint`) has
no wire form at all. The cheap-and-safe subset that round 4 CAN take, because it needs no barrier
change and is split-only: (i) under a transport, drive `SyncVaoAttributeBuffersByHandle`'s walk
and the IBO arm off `st.VertexBuffers[].Res` / `st.IndexBuffer.Res` with
`EnsureBufferResourceForHandle(nullptr, h)`, removing the per-draw VAO attribute reads and their
`HandleOfBuffer` probes (also cuts emutls); (ii) skip the 15-slot `GetBufferBindingSlot` copy in the
kDraw residual fill when the wire is live, since no draw-path site reads it under a transport
(package C's residual-fill territory - coordinate rather than duplicate); (iii) after T/D/C,
re-profile the client's non-wait share. Sequence the real thing as P3b/P4b (handle-keyed twins,
`set_shader_buffers`) followed by P11's versioned `gPipeInputs`; the deferred barrier is one
`WaitForApplied` move (`ClientSession.cpp:917` -> `PipeFill.cpp:2790`) once both are in, and is
not worth a package before then.
+286
View File
@@ -0,0 +1,286 @@
# P5d round 3, package T - transport: apply loop idle poll, clock-free spin, park counters
Worktree `MobileGL-p5d-T`, branch `p5d-T`, base `85362a85`. Not built, not committed. Patch:
`scratchpad/p5d/patches/T.patch` (13 files, +489/-25).
## What changed, per file
### `MG_Remote/Server/ServerLoop.{h,cpp}` - item 1, the idle poll's mutex
- New `std::atomic<Bool> m_controlPosted` replaces a `std::condition_variable` of the same name
that had **no waiter and no notifier anywhere in the tree** (`m_controlDone` carries the whole
handshake). `ControlIsPending()` is now one acquire load, no lock, and moved to the public
section (a test reads it - see red-once); `PumpControlRequest()` gained a fast negative on the
same load before it touches `m_controlMutex`.
- Protocol: the poster stores the shadow **last of the four fields, with release**, inside the
existing critical section in `RunOnApplyThread`; the pump clears it **under the lock, when it
takes the request**; the C2 exit block clears it under the same lock as the `m_running` clear.
The C2 critical section, the NOT_INITIALIZED answer at exit, publish-then-ring and the
re-entrant inline arm are unchanged in meaning; `ready` now reads only atomics.
- Why: simpleperf at `56a77348` put `mutex::lock` 12.4 + `mutex::unlock` 13.6 + the two
`pthread_mutex_*` halves at ~49% of `mgl-srv-apply`, with `ApplyThreadMain` the **direct** caller
of 58-60% and 99.8% of `pthread_mutex_lock`-via-`mutex::lock`. Those are these two call sites,
twice per idle iteration; most of `@plt` 6.1% goes with them.
- Item 2 (`DrainRing` on an empty ring): **audited, no code change**, finding written into the
function. `ApplyOne` -> `RingConsumer::Pop`, whose first act is one acquire load of `cmdHead`
against `m_localTail` (`Ring.cpp:222-232`); `applied == 0` skips the retire block. No mutex, no
clock, no allocation.
### `MG_Remote/Transport/Doorbell.{h,cpp}` - item 3, the clock-free spin
- `Detail::g_spinItersPerUs` + `Detail::CalibrateSpinItersPerUs()`, and an inline
`SpinItersPerUs()` that is a relaxed load and a predicted branch - deliberately **not** a
function-local static, whose guard-variable load on every `Wait` is the cost this removes.
`Wait()`'s spin phase is now `SpinItersPerUs() * spinUs` iterations of `ready()` + `CpuRelax()`
and reads **no clock at all**; the deadline is computed once, at the top of the park phase, and
`kWaitForever` reads no clock even there.
- Calibration: two passes of 8192 `load(acquire)` + `CpuRelax()` timed with `steady_clock`, faster
pass wins (a preempted pass reads absurdly slow, and under-spinning costs a park), clamped to
[4, 2000] iters/us with a 200 fallback for a clock too coarse to see the probe; a race between
two calibrators stores two legitimate budgets.
- **Overshoot bound** (documented in the file): the old loop overshot by at most one batch of 64
yields; this one by the calibration error - a DVFS drop or little-core migration stretches it by
the frequency ratio (~3-4x here, so 50 us can reach ~200 us), a preemption adds one slice as
before, and the probe omits the caller's `ready()`, so a real iteration is *dearer* than a
probed one and the spin ends early by that factor. Not a correctness bound: `ready()` is
re-tested every iteration and the deadline is still real-clock. Stated drift: the spin is no
longer charged against `timeoutMs`, so a wait is bounded by `timeoutMs + budget` - 50 us against
millisecond-scale timeouts, and erring long is the safe direction for a timeout.
- `Doorbell::ParkEntries()` + a private relaxed counter incremented at the one place a waiter
stops spinning and blocks - unconditional (one relaxed add on a path about to enter a condvar
or `poll()`), so a test can read it without `MOBILEGL_PIPE_STATS`.
- Why: `steady_clock::now()` 2.34 self / 5.91 incl + `__kernel_clock_gettime` 2.37 + `clock_gettime`
1.94 = ~6.6% of the GL thread at 900+ waits/frame, for a duration that is a hint.
`WaitForApplied`'s common path paid 2+ reads per wait; it now pays zero unless it parks.
### `MG_Remote/Transport/SessionRings.h`, `Transport/Ring.cpp` - item 4, client half
- `SessionProducer::Waits()` (relaxed atomic, incremented in `Park` *after* the two refusals that
never reach the bell) and `Parks()` (delegates to the self bell's `ParkEntries()` - only
`Doorbell::Wait` knows whether its budget ran out). Atomic because `ClientSession::Stop`'s drain
waits through this producer from the tearing-down thread; it is never copied (checked).
### `MG_Util/Metrics/PipeStats.{h,cpp}`, `Client/EmitTables.cpp`, `Client/ClientSession.cpp`
- Four gauges, **all inside existing `#if MOBILEGL_PIPE_PUSH` blocks**: `ServerWaits`,
`ServerParks`, `ClientWaits`, `ClientParks`. New `wait[srv= srvpark= cli= clipark=]` group on
the one `MGPipe stats:` line (it closes the `emit[` bracket the ring gauges were riding), same
four under long names in the JSON dump, run totals like the gauges above them. Both server
numbers, not one: `srv` is `ParkCount()`'s long-standing meaning (an intention, counted on the
way toward a park - ServerLoopTest's C10 note), `srvpark` is a real block.
- No new include edge into MG_Util: the existing `PublishGauge` pattern already crosses that way
(`EmitTables.cpp:1000`). The client pair is published from `EmitPresent`; the **server pair by
the apply thread from its own loop**, because CONTRACT-P5C rule E layer 2 forbids a GL-thread
read of `ServerLoop`'s memory - PipeStats is the meeting point precisely because it is below
both roles. Both publishes sit behind `PipeStats::Enabled()`.
- `ClientSession::LogWireLedger` also gained `cliwait=`/`clipark=`: that line is in **every** split
private log, and the round-3 profile runs did not set `MOBILEGL_PIPE_STATS`.
## Red-once
| change | goes red as |
|---|---|
| shadow **set** side dropped | `ServerLoopTest.AControlRequestRunsOnTheApplyThreadAndUnparksIt` (existing) - the predicate sees nothing, the poster blocks, the case says so within 5 s |
| shadow **clear** side dropped | `ServerLoopTest.TheControlShadowIsClearedSoTheLoopParksAgainAfterARequest` (**new**) - `ready` stays true, the loop spins for ever and never stores `consumerParked`, so the second `WaitUntilTrulyParked()` times out; plus a direct `ControlIsPending()` read |
| spin budget zero / calibration broken | `InProcessTransportTest.DoorbellSpinsWithoutParkingWhenReadyFlipsInsideTheBudget` (**new**) - `ParkEntries()` moves although the condition flipped on the 8th probe |
| spin never ends / park unreachable | `InProcessTransportTest.DoorbellStillParksWhenTheSpinBudgetRunsOut` (**new**) - `ParkEntries()` does not move |
| `wait[]` renamed or dropped | `PipeStatsTest.SummaryLineCarriesEveryClassAndGate` (extended) |
Kept passing by inspection: `DoorbellWakesAParkedWaiter`, `DoorbellReturnsImmediatelyWhenAlreadyReady`,
`DoorbellTimesOutWhenNothingHappens` (the uncharged spin only makes elapsed larger than its `>= 20 ms`),
`ShutdownUnparksAWaiterWithNoDeadline` (fence protocol and `Dead()` unchanged), `RingTest`, `SessionTest`.
## G1 / monolith impact
**None.** Every touched file is MG_Remote (built only under `MOBILEGL_BUILD_DISAGGREGATED`,
`CMakeLists.txt:547-578`), a test, or - for `PipeStats.{h,cpp}`, which *is* in the pull build -
edited exclusively inside pre-existing `#if MOBILEGL_PIPE_PUSH` regions (`PipeStats.h:186-236`,
`PipeStats.cpp:484-545`, `:600-622`), so `MOBILEGL_PIPE_PUSH=OFF` compiles no added token and the
pull build's text is unmoved. No include-closure probe is touched.
## Risks
1. **The spin's duration bound is weaker** (see above): it burns CPU before a park, cannot change
a result, and the [4, 2000] clamp plus probed-cheaper-than-real bounds both directions. The
probe reads the clock twice per pass, once per process (2-30 us) - no steady-state path.
2. `m_controlPosted` is a second copy of one mailbox bit. Disagreement is a performance fault in
both directions (a spin that never parks; a wakeup the doorbell's own re-test recovers), never
a lost request - the pump re-checks `m_controlPending` under the lock and clears a stale shadow.
3. `wait[` closes `emit[` earlier than before, but `PipeStatsWindow::CounterOrAbsent` matches
`" name="` / `"[name="`, so `fbe`/`sve`/`csom`/`mpr`/`rsp`/... are unaffected and the four new
names collide with nothing (`cli=` cannot match `clipark=`).
## What I did not do
- Did not build, run, commit or push. No `Co-Authored-By` anywhere.
- Did not hoist `ServerSession::Consumer()` / `Applier()` out of `DrainRing` - two out-of-line
calls per idle iteration, a small win, but caching them touches `ApplyThreadMain`'s opening
lines, which package D owns. Flagged for round 4 / D; nothing of `OnApplyThread`,
`m_applyThreadId` or those lines is touched here. Also left: the per-park `steady_clock::now()`
under `kWaitForever` (one read per real park), to keep the park phase structurally identical.
- Did not update `docs/Disaggregated/ARCHITECTURE.md:623`, which enumerates the summary line's
counters and now omits `wait[]` - left to the round's docs pass so four packages do not collide.
Same for stale `Doorbell.h:NNN` references nearby: already drifted at `85362a85`.
- The brief's "visible to `ControlIsPending` **without the pump**" is not directly constructible:
`m_callerMutex` allows one outstanding request and a live apply thread pumps every iteration, so
no window exists in which a posted-but-unpumped request is observable by a third thread. The new
test pins the clear side (which nothing else could see), the existing unpark test the set side.
---
# Fix round (after review)
Two reviews, no blockers, three distinct majors (two of them the same defect seen from two
sides) and five minors. Every one is addressed below; nothing was widened. Patch regenerated
from `MobileGL-p5d-T`: 16 files, +749/-40 (was 13 / +489/-25). Still not built, not committed.
## Majors
### M1 (review 1) - the clear side's red-once did not fire
**Upheld.** `PumpControlRequest` clears the shadow twice - at the take (`ServerLoop.cpp`, under
the lock, when the work is handed out) and defensively on the next pump when it finds the shadow
set with `m_controlPending` false. The old case asserted only the END STATE ("the shadow is clear
after the request was answered, and the loop parks again"), and the defensive clear supplies that
end state on the very next iteration, so deleting the take-clear left the case green. The
reviewer's trace is exact.
**Fix: observe the window, keep both clears.** `ServerLoopTest.TheControlShadowIsClearedSoTheLoop
ParksAgainAfterARequest` is replaced by
`ServerLoopTest.TheControlShadowIsClearedWhenThePumpTakesTheRequest`. The posted work now blocks
on a gate the test thread holds; the test waits until the work is really running and reads
`ControlIsPending()` in that window. There `m_controlPending` is still true and no pump is
running, so the ONLY thing that can make the accessor answer false is the clear at the take -
delete it and the read returns true on every run, deterministically. The end-state assertions
(answer returned, `WaitUntilTrulyParked()` again) are kept as the symptom an operator would see,
with their comment now saying which clear each one pins.
The defensive clear is kept: it is a genuine self-heal for a shadow that outlived its request,
and with the window observed directly it no longer masks anything.
### M2 (both reviews) - `Parks()` was not the subset of `Waits()` that everything claimed
**Upheld, in all three of its parts**, and the fix is one mechanism for all three.
1. *Counted per `Park()`, not per `Wait()`.* The increment sat inside the park-retry loop, so a
wait that parked, woke on a remembered `Notify`, re-tested false and parked again counted
twice - a routine path, exactly as review 2 describes.
2. *Counted for `Park(0)`.* When the remaining deadline rounds down to 0 ms the loop spins
through the counting site without ever blocking.
3. *Counted off a shared bell.* `SessionProducer::Parks()` returned `m_selfBell->ParkEntries()`,
and `ClientSession::Start` installs that same object as the encoder's stage-retirement bell
(`PipeWireCodec::SetStageRetirementDoorbell`). Those waits never touch `m_waits`, and with
`spinUs == 0` they now park unconditionally, so `clipark` could print larger than `cli` on
any lane with stage retirement (the SmallRing / E3(e) lane asserts `stageReclaimWaits >= 1`).
**Fix.**
- `Doorbell::Wait` gained a fifth, defaulted parameter `std::atomic<std::uint64_t>* parkTally`.
It is bumped at the same site as the bell's own counter, and both are now guarded by
`chunkMs != 0 && !countedPark`, where `countedPark` is a local hoisted above the park loop. One
park per `Wait`, and only when the call really blocks.
- `SessionProducer` owns `m_parks` and `SessionProducer::Park` passes `&m_parks`, so the client
pair counts the same events on both sides. `Parks()` reads it; `ParkEntries()` is no longer
consulted by any ledger.
- The server half got the same treatment even though its bell has no second waiter today:
`ServerLoop::m_parkBlocks` (+ `ParkBlockCount()`, reset in m-3's per-session block beside
`m_parks`) is passed into the loop's one `Wait`, so `srvpark <= srv` cannot be broken later by
anything else waiting on the consumer doorbell (`SessionConsumer::WaitForWork` waits on that
same bell).
- Comments corrected where they asserted the old, false relation: `Doorbell::ParkEntries()`'s
block now says it is per BELL and names the two waiters that share the client's,
`SessionRings.h`'s `Waits()`/`Parks()` block says why the tally is ours and not the bell's,
and `PipeStats.h` says the subset relation holds only because both halves are counted by the
same path.
**Red-once for the fix** (both new, both deterministic):
- `InProcessTransportTest.AWaitThatParksTwiceOnARememberedNotifyIsStillOnePark` - `Notify()` with
nobody waiting, then one `Wait` whose condition never flips: the first `Park` returns at once on
the remembered wakeup and the loop parks again. `ParkEntries()` must move by exactly 1. Revert
`countedPark` and it moves by 2.
- `SessionTest.TheProducersParkLedgerCountsItsOwnWaitsAndNotTheBellsOtherWaiters` - a foreign
`Wait` on `clientTransport->SelfDoorbell()` (spinUs 0, condition never true - the encoder's
exact shape) must move neither `producer.Waits()` nor `producer.Parks()`; the producer's own
20 ms `WaitForApplied` must move both by exactly one. Revert `Parks()` to `ParkEntries()` and
the first `EXPECT_EQ` on `Parks()` goes red.
## Minors
### m1 - `spinUs == 0` silently changed meaning (both reviews)
**Upheld; stated, and the probe no longer runs on that path.** The old batched loop ran 63
iterations before its first clock read could end it; the new budget is `itersPerUs * spinUs`, so
zero means zero. That is now an explicit `spinUs == 0 ? 0ull : ...` short-circuit with a comment
naming the delta, both callers that pass 0 (PipeWireCodec's SEG_STAGE retirement wait, and any
lane with `MOBILEGL_IPC_SPIN_US=0`), and the reason it is deliberate rather than accidental. The
short-circuit also keeps a zero-spin caller from paying the one-time calibration probe - review
1's second half of the finding. `PipeWireCodec.cpp`'s `0` is left as it is: the retirement wait is
rare, already slow, and wants the park; changing it is a codec decision, not this package's.
### m2 - `ControlIsPending()` no longer means what its name says
**Upheld; documented rather than renamed.** The brief names the accessor, and three call sites'
worth of churn buys nothing, so the header block now states the predicate literally - "posted and
not yet taken" - says that it answers false for the whole duration of `work(user)` while
`m_controlPending` is still true, says it is a PARK PREDICATE and not a liveness query, and notes
that the only in-tree reader besides the test is the apply thread's own `ready` lambda, which by
construction cannot be inside that window. The new test in M1 is built on exactly that window, so
the documented meaning is now the tested one.
### m3 - the calibration clamp's two opposite readings (review 2)
**Upheld.** `perUs` truncates to 0 for any pass longer than 8.192 ms, i.e. the heavily preempted
probe, and that landed on the same `best == 0` arm as "no usable clock" and installed 200 iters/us
- 50x the floor - on the machine least able to afford it. `CalibrateSpinItersPerUs` now carries
`sawUsableClock`, set whenever `elapsedNs > 0`: measured-but-slow clamps UP to
`kMinSpinItersPerUs`, and `kFallbackSpinItersPerUs` is reserved for the case where no pass was
measurable at all. The `Doorbell.h` paragraph already described this rule ("a probe that was
preempted cannot produce a budget smaller than kMinSpinItersPerUs"); it is now true.
### m4 - `Doorbell.h:NNN` citations invalidated by this package's insert (review 2)
**Upheld - and the first report's "already drifted at 85362a85" was wrong.** Checked at baseline:
`139-178` really was the spin loop through the `Dead()` arm, `186-193` really was the `};` plus
NotifyIfParked's PRECONDITION, `211-221` really was CondVarDoorbell/`Kill`. The +60 lines moved all
three, and `check_doc_citations.py` cannot see it because the file only grew. Every in-tree
citation is re-pointed to SYMBOL NAMES rather than to new line numbers, so the next insert cannot
break them again: `ServerLoop.cpp` (x3), `ServerLoop.h`, `SessionRings.h` (x3), `EventRing.h`,
`ClientSession.cpp`, `ClientSession.h` (x2) and `CONTRACT-P5.md:355`. The checker only resolves
`path:line` pairs, so dropping the numbers leaves every gate green.
### m5 (review 2's note, not filed as a finding) - `srv=0` under `spawn`
Not a defect of this patch, but the new `PipeStats.h` comment asserted "PipeStats is the meeting
point precisely because it is below both", which is true in one transport mode. The comment now
says so: under `spawn` the server pair is published into the server process's PipeStats and the
client's line prints `srv=0 srvpark=0` for the whole run - the same property `rsp` already has -
and the round this pair was added for is `inproc`.
## Red-once table, updated
| change | goes red as |
|---|---|
| shadow **set** side dropped | `ServerLoopTest.AControlRequestRunsOnTheApplyThreadAndUnparksIt` (existing) |
| shadow **take-clear** dropped | `ServerLoopTest.TheControlShadowIsClearedWhenThePumpTakesTheRequest` (**new, rewritten**) - reads `ControlIsPending()` while the work is still running, where the defensive clear cannot cover for it |
| park counted per `Park` instead of per `Wait` | `InProcessTransportTest.AWaitThatParksTwiceOnARememberedNotifyIsStillOnePark` (**new**) |
| client parks read off the shared bell again | `SessionTest.TheProducersParkLedgerCountsItsOwnWaitsAndNotTheBellsOtherWaiters` (**new**) |
| spin budget zero / calibration broken | `InProcessTransportTest.DoorbellSpinsWithoutParkingWhenReadyFlipsInsideTheBudget` |
| spin never ends / park unreachable | `InProcessTransportTest.DoorbellStillParksWhenTheSpinBudgetRunsOut` |
| `wait[]` renamed or dropped | `PipeStatsTest.SummaryLineCarriesEveryClassAndGate` (extended) |
## G1 / monolith impact
**Still none.** The fix round touched `Doorbell.{h,cpp}`, `Ring.cpp`, `SessionRings.h`,
`EventRing.h`, `ServerLoop.{h,cpp}`, `ClientSession.{h,cpp}`, `CONTRACT-P5.md` (all MG_Remote or
a doc), three test files, and `PipeStats.h` - the last one inside the same pre-existing
`#if MOBILEGL_PIPE_PUSH` region as before, comment text only. No new include edge; the new `Wait`
parameter is defaulted, so the two call sites that do not pass a tally (`PipeWireCodec.cpp`,
`SessionConsumer::Park` in `Ring.cpp`) are unchanged.
## Not changed, with reasons
- **The defensive clear in `PumpControlRequest` stays.** Review 1 offered "drop it so the
take-clear is load-bearing" as an alternative; dropping a self-heal to make a test work is the
wrong trade when the test can observe the real window instead.
- **`ControlIsPending` is not renamed** (see m2).
- **`PipeWireCodec.cpp`'s `spinUs == 0` is not changed** (see m1).
- **`ARCHITECTURE.md:623`'s summary-line enumeration** still omits `wait[]` - left to the round's
docs pass, as before, so four packages do not collide on one file. The `Doorbell.h` citations
were different: those were made wrong BY this patch, so this patch fixes them.
- Did not build, run, commit or push. No attribution lines anywhere.