MobileGL/docs/archive/notes/p5d/BRIEF-P5D-R3.md

P5d round 3 brief: inproc performance, the profile-directed cuts

Repo: MobileGL (branch feat/disaggregated, head 85362a85). You work in an isolated git worktree of it (your cwd). Read docs/Disaggregated/P5D-INPROC-PERFORMANCE.md (rounds 1-2), the relevant parts of MobileGL/MG_Remote/CONTRACT-P5C.md (role guards, rule E) and docs/Disaggregated/ROADMAP.md "通用纪律" before touching code.

Why this round exists (measured, not guessed)

Workload: FCL + Minecraft 26.3-rc-3, view distance 12, Redmi 2f7cbe2e (Adreno 830), DirectGLES. Split (MOBILEGL_TRANSPORT=inproc) 63-65 fps vs monolith 110-118 fps, ~852 draws/frame. simpleperf (cpu-cycles, whole process, 15 s in world) at head 56a77348, symbolized:

Process split: GL/client thread (Thread-17) 39.5% of cycles, apply thread (mgl-srv-apply) 36.9%. Monolith: GL thread 52.6% of a 39% smaller total.

Apply thread (mgl-srv-apply), self %

• std::mutex::unlock 13.6, std::mutex::lock 12.4, pthread_mutex_lock 12.1, pthread_mutex_unlock 11.0 -> ~49% of the thread. Call graph: 58-60% of mutex::lock/unlock samples have ServerLoop::ApplyThreadMain() as the DIRECT caller; 99.8% of pthread_mutex_lock via mutex::lock is ApplyThreadMain. These are NOT the backend pendingMutex sites the round-2 report guessed. They are the idle poll: for(;;){ PumpControlRequest(); DrainRing(); if(ready()) continue; bell.Wait(...) } where PumpControlRequest() takes m_controlMutex every iteration and the ready lambda calls ControlIsPending() which ALSO takes m_controlMutex (ServerLoop.cpp:371-384, 452-475).
• ApplyThreadMain self 7.95, @plt 6.1 (31% of plt from ApplyThreadMain), __emutls_get_address 7.4
• Inclusive: ApplyThreadMain 78.4%, DrainRing only 29.0% (= the real apply work: OnDrawVbo 22.7%, PrepareForDraw 16.3%, SyncNeccessaryTextures 8.6%, SyncMipmapsToBackend 3.3%, HandleOf 3.0%, MGPipeRefuseAllocatorFromApplyThread 2.4% incl.)
• MGPipeServerStampVerbBoundary 1.1 self (63-field loop per verb, PipeInputs.cpp:103-130)
• __aarch64_swp2_rel/cas2_acq/ldadd8 ~2.5 (outlined LSE atomics; leave alone)

Client thread (Thread-17), self % / inclusive %

• SessionProducer::WaitForApplied 27.9 self / 34.2 incl (the R-1 barrier spin; EmitAndWaitTails 36.3 incl)
• persistent-map push: PushDrawConsumers 1.96 self / 22.5 incl; PushBlocksForChecked 5.57 self / 21.5 incl; XXH3_hashLong_64b_default 6.51 + XXH3_64bits 2.02 (callers: PushBlocksForChecked via PushDrawConsumers 98%); BufferObject::MappedData() 1.16 self / 3.86 incl; IsMapped 0.58; GetMappedRange 0.43; PushIsArmed 0.47
• MGPipeValidateForVerb 4.11 self / 10.9 incl (65% called from DrawElementsInstancedBaseVertex_Backend); inside it MGPipeTracker::Update 1.46, MGPipeFillAccess::CopyField 0.64 self / 2.54 incl (45% of BufferState::GetBindingSlot 1.73 comes from CopyField in the draw fill)
• clock: steady_clock::now() 2.34 self / 5.91 incl, __kernel_clock_gettime 2.37, clock_gettime 1.94 -> ~6.6% of the thread is reading the clock. Doorbell::Wait (Doorbell.h:123-178) reads the clock at least twice per wait (start + spinEnd) and once per 64 spins; ~900+ waits/frame at 64 fps.
• ServerLoop::OnApplyThread() 2.66 self / 3.64 incl on the CLIENT thread (role guards on hot paths: RefuseLegacyBufferArmFromApplyThread in BufferObject::MappedData/IsMapped/..., PersistentMapTracker::OnServerRole, RefusePipeInputsTouchWhileApplierOwnsIt, RunsAsTheServerRole, MGPipeRefuseAllocatorFromApplyThread). OnApplyThread = ServerLoopInstance() (function-static guard)
  • m_running atomic + m_applyThreadId atomicstd::thread::id + std::this_thread::get_id() + thread::id compare (pthread_equal call). pthread_self shows 0.56 self.
• MarkWritableImageBufferTextures 2.15 self / 3.59 incl (GpuWritePending.cpp:85-101: sweeps all MAX_TEXTURE_IMAGE_UNITS image units per draw; GetImageTextureBinding 1.37)
• __emutls_get_address 0.46 client; on MONOLITH's GL thread it is 7.7% (top symbol!) with callers MGPipeFrontendKeyedRegistryScope ctor 18.75% / dtor 13.95%, MGPipeReverseAnnouncementScope ctor/dtor 8.4% (SlotAllocator.cpp:41-75: thread_local Uint32 depth counters; the scope is constructed at ~20 sites in DirectGLES.cpp, e.g. inside StateBackendObjectRegistry::HandleOf). On the apply thread emutls is 7.4%.

The shape of the frame (why the client-side cuts matter)

Lockstep (R-1): every kCtxVerb (draw/clear/blit/dispatch/readback) and every reply-slot row waits for appliedSeq == seq. Frame time ~= client work + server apply time (waited for) + handoff latency. Nothing in this round removes the barrier (that is P11's versioned inputs / P3b-P4b twin tables). This round removes overhead on BOTH threads: every cycle cut on the apply thread's real work shortens the client's wait one-for-one; every cycle cut on the client thread shortens the frame directly.

Packages (one agent each, independent, non-overlapping files where possible)

T - transport: apply loop idle poll, wait spin, park counters

Files: MobileGL/MG_Remote/Server/ServerLoop.{h,cpp}, MobileGL/MG_Remote/Transport/Doorbell.{h,cpp}, MobileGL/MG_Remote/Transport/Ring.cpp, SessionRings.h, MobileGL/MG_Remote/Client/ClientSession.cpp, MobileGL/MG_Util/PipeStats* (find the "MGPipe stats:" line producer).

1. ControlIsPending() and PumpControlRequest() must not take m_controlMutex when nothing is posted: add std::atomic<Bool> m_controlPosted (set release under the lock by the poster in RunOnApplyThread, cleared by the pump under the lock when it takes the request). The mailbox handshake semantics (C2 critical section, NOT_INITIALIZED answer at exit, kMaxControlFramesPerPump) must stay exactly as documented in the file. The ready lambda reads only atomics.
2. DrainRing() on an empty ring must be a couple of loads, no lock, no clock. Check.
3. Doorbell::Wait: the spin phase must not read the clock at all in the common case. Replace "spin until steady_clock passes spinEnd, clock read every 64 iterations" with a calibrated iteration budget: once per process (or per Doorbell), measure how many ready()+CpuRelax() iterations fit in spinUs (or simply how many CpuRelax fit in 1 us) and spin that many iterations clock-free; only the park phase (rare) computes deadlines. Keep timeoutMs/kWaitForever semantics, the parked-flag fence protocol and the Dead() handling byte-for-byte in meaning. Document the overshoot bound. WaitForApplied's common path (server not yet there) currently pays 2+ clock reads per wait; after this it pays zero unless it parks.
4. Counters for the measurement: server-side m_parks already exists (ParkCount()); add a client-side park counter (SessionProducer::Park -> count parks that reach Doorbell::Park) and a wait counter, and print both (server parks, client parks, waits) in the periodic MGPipe stats: line (find how the line is assembled; keep it one line; add a wait[...] group). If the stats line producer lives in a place that cannot see MG_Remote symbols, expose the counters through a small accessor registered by the client session (no new include of MG_Remote into MG_Util if that violates the include-closure gate scripts/check_include_closure.py - read its rules first). Tests: existing MG_Test/Wire/ServerLoopTest.cpp, DoorbellTest/RingTest must keep passing by inspection; add one small test for "a posted control request is visible to ControlIsPending without the pump" and one for the calibrated spin (e.g. Wait returns true once ready flips within the spin budget, and parks otherwise) if the existing fixtures make that cheap. Do not build.

D - role guards and TLS: make every hot-path guard nearly free

Files: MobileGL/MG_Remote/Server/ServerLoop.{h,cpp} (OnApplyThread only - coordinate: package T also edits ServerLoop.cpp; keep your hunk confined to OnApplyThread, the thread-identity member(s) and ApplyThreadMain's first lines where the identity is stored), MobileGL/MG_Impl/Pipe/SlotAllocator.cpp (+ its header), MobileGL/MG_State/GLState/BufferState/BufferObject.cpp (RefuseLegacyBufferArmFromApplyThread), MobileGL/MG_Remote/Client/PersistentMapTracker.cpp (OnServerRole / IsLivePersistentMap guard order), MobileGL/MG_Remote/Client/ClientSession.cpp (RefusePipeInputsTouchWhileApplierOwnsIt: order the early-outs cheapest-first: Transport==Monolith, BatchWaits!=0, then the rest), MobileGL/MG_Remote/Client/WireTables.cpp (RunsAsTheServerRole), MobileGL/MG_State/GLState/TextureState/MipmapStorage.cpp.

1. ServerLoop::OnApplyThread(): store the apply thread's identity as a plain integer key readable with a relaxed atomic load and compare it against a register-cheap identity of the calling thread: on aarch64 and x86_64 with clang/gcc use __builtin_thread_pointer() (verify availability per target with __has_builtin; NDK clang 18/19 supports aarch64; check x86_64 support and fall back to pthread_self() (POSIX) or std::this_thread::get_id() (MSVC) where the builtin is unavailable). Keep the semantics: false when no apply thread is running (m_running false / identity 0), true only on the running apply thread. The function-static ServerLoopInstance() guard on every call must go (a namespace-scope atomic key is enough; the ServerLoop object still owns it logically).
2. MGPipeFrontendKeyedRegistryScope / MGPipeReverseAnnouncementScope (SlotAllocator.cpp:39-75): the depth counters are only ever READ by MGPipeRefuseAllocatorFromApplyThread, which returns early unless OnApplyThread(). So the counters only matter on the apply thread. Replace the thread_local with a plain (non-TLS) counter incremented/decremented only when OnApplyThread() is true (cheap after item 1); on the GL thread the ctor/dtor become one predicate call. Prove in a comment why no data race exists (only the apply thread writes and reads; monolith never reads). Keep Active() semantics on the apply thread exact. If you find a reader other than the apply-thread guard, stop and report instead of changing semantics.
3. RefuseLegacyBufferArmFromApplyThread (BufferObject.cpp:42-52) and PersistentMapTracker::OnServerRole, IsLivePersistentMap: with item 1 these are ~free; make sure PushIsArmed() (a config read) is not itself expensive (it reads MG_Config::Transport - fine) and that no guard does more than one OnApplyThread call per accessor.
4. Look for any other per-draw thread_local reads on the client and apply paths (t_activeDecoder in PipeWireCodec.cpp:359, g_reply in PipeRoute.cpp:61, g_inBarrierWait in ClientSession.cpp:117, g_textureLegacyArmScopeDepth MipmapStorage.cpp:73, tl_initializing) and report which are on the hot path; fix the cheap ones the same way ONLY if the single-writer argument holds; otherwise report. Monolith equivalence: the pull build (MOBILEGL_BUILD_DISAGGREGATED=OFF) must compile and its behaviour must be identical; G1 (pull-build symbol/.text identity) changes must be named in your report.

B - persistent-map push: no edge hashing, page-granular shadows

Files: MobileGL/MG_State/GLState/BufferState/PipeResource.h (the aligned allocator), the shadow container in BufferObject.{h,cpp} / PipeResource.*, MobileGL/MG_Remote/Client/PersistentMapTracker.{h,cpp} (TrackWriteMap ~259-330, the SIGSEGV handler ~200-240, PushBlocksForChecked 492-685, PushDrawConsumers 704-818). Today: the shadow allocator gives 4096-byte ALIGNMENT but not a page-multiple SIZE, so the last page of a shadow can hold foreign bytes; TrackWriteMap therefore aligns the protected interior INWARD and the partial-page edges (head edge when the mapped offset is not page-aligned; tail edge for every buffer whose mapped end is not page-aligned - i.e. nearly every buffer) are hashed with XXH3 on EVERY draw for every live map with edges (the epoch-skip fast path pushes edges of ALL live maps, not just consumers). That is 8.5% of the client thread in XXH3 plus the MappedData/IsMapped/GetMappedRange guards around it. Goal: zero per-draw hashing in steady state.

1. Make every shadow allocation page-granular: round the allocation SIZE up to a multiple of SHADOW_ALLOCATION_ALIGNMENT in the allocator (verify the container: what Bytes() returns, whether capacity vs size matters, whether any shadow can be backed by something else - adopted GPU stores are excluded from tracking by IsLivePersistentMap; confirm). Then every byte of [base, base + RoundUp(size, 4096)) belongs to this shadow and no other allocation.
2. TrackWriteMap: when the shadow base is page-aligned and the allocation is known page-granular, align the protected range OUTWARD to page boundaries (clamped to the shadow's own page-rounded extent), so there are no edges and hasEdges is false; sub-page maps become one protected page. Keep the inward arm as the fallback for any shadow that fails the page-granular test (and keep the hash arm for untracked maps). The SIGSEGV handler's ownership tests (SEGV_ACCERR, inside a live tracked range, bitmap-bit replay discrimination) must hold for the widened range: bytes outside the mapped range but inside the shadow are written by the client's own API paths (SubData/FlushMappedRange memcpy into the shadow) - a fault there is answered like any interior fault; make sure the push clamps to [begin,end) as today and that nothing else (e.g. readback writeback into the shadow on the client thread, Forget, resize/respecify that reallocates the shadow, NoteMapStateChanged) breaks the "protected pages are re-armed / released" invariant. Re-read the round-1 commit message of cb06538c (git show cb06538c) for the reasons behind inward alignment before you widen anything.
3. The epoch-skip fast path (PushDrawConsumers 727-737) then does nothing per member; keep the loop or drop it - measure nothing, reason. First push still ships everything once.
4. Report: a table of the per-draw work that remains in PushDrawConsumers when the epoch has not moved. Tests: MG_Test/Buffer/SplitBufferTest.cpp and MG_IntegrationTest/Scenarios/PersistentCoherentMapScenario.cpp exercise this; add/adjust a unit case for the outward alignment (a sub-page map gets a protected page; an unaligned base falls back to inward+edges). Do not build. Danger notes from cb06538c: a signal-blocked driver/runtime worker thread faulting on a foreign byte inside a protected page kills the process (that was the ctest SmallRing segfault) - the whole point of item 1 is that no foreign byte can be inside a protected page any more; state that argument precisely.

C - client verb overhead: validate/fill, image-unit sweep, server stamp loop

Files: MobileGL/MG_Impl/Pipe/PipeFill.cpp (MGPipeValidateForVerb 2781+, the residual fill / CopyField), MobileGL/MG_Pipe/PipeInputs*/MobileGL/MG_Backend/MGPipe/PipeInputs.cpp (MGPipeServerStampVerbBoundary), MobileGL/MG_Remote/Client/GpuWritePending.cpp, MobileGL/MG_Remote/Client/EmitTables.cpp (draw emit path), MobileGL/MG_State/GLState/TextureState/* only if an image-unit high-water mark is needed.

1. MarkWritableImageBufferTextures: replace the 32-unit sweep per draw with a high-water mark (the GLContext already keeps touched-unit marks for texture units and buffer binding points; add the same for image units at the BindImageTexture site if none exists - note this is MG_State (shared with monolith) so keep it minimal and name the G1 impact; or, if ImageEmit/the tracker already knows whether any writable image-buffer unit is bound, use that latch).
2. MGPipeValidateForVerb draw path: find what the 10.9% inclusive is made of beyond tracker.Update (1.5%) and CopyField (2.5%): the residual fill's per-verb copies (which fields does the draw class copy, and which of those are already record-supplied under a live wire - P5c rv skips the value-class rows; check the object-class rows' copy cost, e.g. GetBindingSlot from CopyField), the RefusePipeInputsTouchWhileApplierOwnsIt guard order (package D reorders it; don't duplicate), ContextValuesWireLive() / RunsAsTheServerRole() per verb (cache per session if legal), ParsePoisonOmissionKnob() per verb (should be once). Cut what is provably redundant; leave the emitters' semantics alone.
3. MGPipeServerStampVerbBoundary: the 63-field loop per verb on the apply thread computes answerable(field, verb) from constant tables - precompute per verb once (a static table of which fields are answerable per verb, built lazily or constexpr) so the per-verb work is a table-driven fill. Keep the withdrawal semantics exactly (FilledGen = serial or 0).
4. Anything else on the draw emit path in EmitTables.cpp that is per-draw and redundant (e.g. repeated session lookups, ReadDrawBindings re-reading slots the fill already read). Report what you found and what you left. Monolith: PipeFill.cpp is compiled into the monolith push build too; keep monolith behaviour identical (G1 for the pull build; push-build equivalence by inspection).

E - research only (no code): can the draw barrier be deferred by one verb?

Deliverable: a design note, not a patch. Question: the client today fills the residual object-class rows of gPipeInputs (FieldOwnership.def BARRIER_PULLED: GetBoundVertexArray, GetBufferBindingSlot, GetBufferBindingPoint, GetFramebufferBindingSlot, GetImageTextureBinding, GetTextureUnitObject, GetProgramForDraw, GetProgramForDispatch, GetTransformFeedbackProgram, GetBufferBindingPointCount, GetProgramObject, GetTextureObject, HasOpenTransformFeedbackSpan, ValidateProgramName, RecordError), publishes the draw, and WAITS. Map, for the draw/clear/blit verbs on DirectGLES under inproc: (a) every server-side read that reaches client-owned memory through those rows (what does the server dereference through each pointer: VAO attribute array? texture unit binding slots? UBO binding point table entries + ranges? program object internals?), including the two exemption scopes (MGPipeFrontendKeyedRegistryScope sites in DirectGLES.cpp, MGPipeReverseAnnouncementScope); (b) which of those reads would be UNSAFE if the client continued past the publish and executed the next draw's GL calls (glBindBufferRange / glBindTexture / glUseProgram / glVertexAttribPointer / ...) before the server applied this draw; (c) the minimal "snapshot at fill" scheme: what would have to be copied per draw into a per-verb snapshot the server reads instead (count the bytes/refcount ops per draw at 852 draws/frame), and where the client would then wait ("before the next fill" instead of "after this publish"), i.e. one-verb pipelining that overlaps the client's inter-draw work with the server's apply; (d) what the existing pushed records already cover (SetSamplerViews / VertexElements CSO / set_vertex_buffers / set_context_values) such that the server COULD stop reading the live tables; (e) a cost/benefit estimate and a recommendation: P5d round 4, or leave to P11 / P3b-P4b. Write it to the report path given below (markdown, <= 400 lines, with file:line citations at head 85362a85).

Rules for every package

• Do NOT build (no gradle, no cmake, no ndk), do NOT touch adb/devices, do NOT push, do NOT commit in any shared tree. Work only in your isolated worktree; deliver git diff HEAD -- MobileGL as a patch FILE at the path given below, plus a short report file. Both must be written before your final message.
• No Co-Authored-By, no attribution lines anywhere.
• No hot-path instrumentation left in (counters that print per frame are fine when behind the existing MG_Util::PipeStats::Enabled() / stats period; no per-draw logging).
• Prefer changes inside #if MOBILEGL_BUILD_DISAGGREGATED or in MG_Remote (split-only). When a change touches code compiled into the pull build (MOBILEGL_BUILD_DISAGGREGATED=OFF, MOBILEGL_PIPE_PUSH=OFF), say so in the report (G1 names every symbol/.text delta) and keep behaviour identical.
• Every correctness-bearing change gets a red-once argument: name the existing test (or the one you add) that goes red if the change is reverted or broken. Keep tests light; do not add a test for a refactor with no semantic change.
• Keep the file's own comment discipline: explain WHY in the code comment, in the voice of the surrounding code, and do not delete the reasoning comments you replace - update them.
• Report format (markdown): what changed (per file), why (tie to the profile numbers above), the red-once test, G1/monolith impact, risks and what you did not do. <= 120 lines.