MobileGL/docs/Disaggregated/notes/p5e/reports/ra2-v1.md

ra2-v1 — make run-ahead actually correct

Tree ~/w7/p5e-ra2, branch p5e/flip, base 985ae62d, head efe58eac (three commits).

1 Verdict (TASK-ra2 §3.2)

Fixable inside P5e, and fixed. The P11 deferral was not needed for this defect. With the flip still thrown the lane that must be green is green — integration-split 179/179, from 112/181 on the base — and the two lanes that must be red are red for their named reasons. Every one of the four fields / eleven verbs / 70 markers ID-132 measured is gone, and no wait was added to the shipping path to get there.

1.1 The most reusable thing in this package

A wait added to satisfy a lane is a real cost on a real path. The check on any escalation is "which arm pays for it", not "which lane goes green".

ID-133 escalated a plain multi-draw so that a pull in Espryt's indirect multi-draw tier became legal. It turned 18 entries green and it charged every glMultiDraw* on the auto → ext default tier with a per-batch rendezvous — because there is one draw opcode and the tier is chosen server-side per batch, so the narrowest predicate both roles could compute could not name the arm that actually needed it. The lane could not see that; only asking who pays could. ID-136 then retired the pull instead (§4.1) and the escalation went away with it. Both halves landed in one commit, so neither intermediate state was ever a head.

That question generalises past this phase: every escalation is a rendezvous, every rendezvous is paid by some arm, and the lane that motivated it is usually not the arm that pays.

ID-132 was right about the object and wrong about the mechanism, and that distinction is the most useful thing in this package. The ruling suspected gPipeInputs' field values racing. They do not: the values are already partitioned — the client's fill writes only fields no record supplies (SuppliedFieldMask), the applier writes only record-supplied ones. What races is the metadata: the four shared scalars CurrentVerbSerial, FilledGen[63], m_currentVerb and m_serverStampedVerb, which both roles write and which no partition covers. That is why a P5e-sized fix existed at all, and it is the sentence P11's scope should be written in (§5).

Three things landed on top of the diagnosis: the fix (§3), the indirect-binding pull's retirement (§4.1, ID-133 escalated then ID-136 retired instead) and ID-134's lane (§4.2). Two of ID-133's premises did not survive measurement; they are corrected in §4.1 and they are what ID-136 acted on.

Commits: 2d03565f (the fix), f2865480 (ID-133 / ID-134 / the contract), efe58eac (ID-136).

2 The diagnosis, with evidence

2.1 What the marker's verb proved before any instrumentation

Of ID-132's 70 red entries, 54 abort with the plain poison (no [BARRIER-PULLED] bracket). For a kRecordSupplied / kApplierDerived field that branch of MGPipeInputUnfreshRead (MG_Backend/MGPipe/PipeInputs.cpp:311-345) is reachable only when serverStamped == false — with the stamp up those two ownerships take the Fatal{RoleViolation} arm instead, and GetTextureContextId's accessor (PipeInputs.h:524) short-circuits to MGPipeApplierContextSerial() before the check can fire. All 54 are on those two ownerships, so all 54 read the block with the server's own stamp withdrawn.

And the verb in the marker is the client's. The applier can only ever write a verb that has a row in MGP_VERB_OP_LIST (FieldOwnership.def:256-276). DrawElements, DrawElementsBaseVertex, DrawElementsIndirect, DrawElementsInstancedBaseVertex, MultiDrawElementsBaseVertex, GetIntegeri_v and GetSyncStatus have no row — and 40 of the 54 name one of those. Nothing but the GL thread can have put them there.

2.2 The interleaving, verbatim

Two temporary probes (fill site and MGPipeInputUnfreshRead; both removed before the commit), one run of DirectGLES.Split.MultiDrawTierDrawElements.MultiDrawScenario.PlainBatchMatchesUnrolledDraws, consecutive lines of ~/w7/p5e-logs/ra2-P/…PlainBatchMatchesUnrolledDraws.01.log:580-585:

[MobileGLIntegra/ERROR]: RA2-PROBE fill verb=DrawElements wireop=81 waitclass=-1 applierInside=1
[MobileGLIntegra/ERROR]: RA2-PROBE fill verb=DrawElements wireop=81 waitclass=-1 applierInside=1
[mgl-srv-apply/ERROR]:   RA2-PROBE unfresh field=IsCapabilityEnabled verb=DrawElements serverStamped=0 ownership=RECORD-SUPPLIED
[mgl-srv-apply/FATAL]:   MGPipe: Fatal{UnmigratedPipeInput, "IsCapabilityEnabled@DrawElements"}

• writer: the GL thread, in MGPipeValidateForVerb's residual fill, with applierInside=1 — the apply thread is inside PipeApplier::ApplyOne for an earlier record.
• the write that did it: MGPipeServerClearVerbBoundary() (PipeFill.cpp:3343), setting m_serverStampedVerb = false; hence serverStamped=0 at the reader. Two more in the same window: ++CurrentVerbSerial (un-freshens every stamp the applier made) and SetVerb (renames the verb the abort prints).
• reader: the apply thread, mid-DecodeAndApply of an unbarriered draw_vbo, reading IsCapabilityEnabled.
• why four fields over eleven verbs, flakily: the field is whichever the applier touches next after the window opens.

2.3 Why the existing guard never fired

RefusePipeInputsTouchWhileApplierOwnsIt's run-ahead arm was if (isBarrieredFill) return;. isBarrieredFill is the caller's own sentence "this touch is the residual fill of a record this thread is about to park behind" — a claim about the future. The order at the validate point is fill → emit → park, so at the instant of the write the applier is still draining the backlog. The exemption was unconditional, so the guard was unfireable on exactly the class it was written for. That is the answer to the brief's question.

2.4 A second, independent defect underneath it

ClientVerbIsBarriered inverts MGP_VERB_OP_LIST, which joins the whole draw family through one row (DrawVbo -> DrawArrays). So the inverse answers kOpCount for nineteen draw verbs, and the file's default — "an op with no verb row answers BARRIERED: fill it, wait for it" — filled for each. The fill happens and the wait does not: the wait is decided per record by MGPipeBarriered(op, …) at the publish, and the record is a draw_vbo (kWaitNone). wireop=81 waitclass=-1 in the probe above is that, measured.

3 The fix (commit 2d03565f)

file	hunks	what
MobileGL/MG_Impl/Pipe/PipeFill.cpp	397-429, 439-472, 706-713, 3392-3402, 3718-3731	the kDraw fallback in ClientVerbIsBarriered (+ a static_assert tying it to the generated row); new QuiesceApplierBeforeFill; its three call sites — validate-point phase 1, step 4, MGPipeNoteFrontendMutation.
MobileGL/MG_Remote/Client/ClientSession.cpp	1319-1352	the run-ahead arm becomes isBarrieredFill && !ApplyThreadIsInsideApplier(); the Fatal names which shape fired.
MobileGL/MG_Remote/Client/ClientSession.h	211-222	the block's rationale updated, not deleted.
MobileGL/MG_Test/Wire/RemoteClientTest.cpp	2300-2303, 2314-2376	the two regression cases and a note on the existing green control.

Two waits per filling verb, and that is the shape of the block, not belt-and-braces. The validate point writes gPipeInputs in two phases that straddle publication: the serial bump, MGPipeServerClearVerbBoundary() and SetVerb before the fourteen emitters, and step 4's 63-field walk after them — and the records those emitters published are records whose apply reads the block. One wait cannot cover both halves.

Everything added reduces to MG_Config::Transport != Monolith through WaitForApplyToCatchUp's own first test (m_runAheadArmed && m_started), stated rather than inferred (ID-81). Under lockstep, monolith and MOBILEGL_IPC_RUN_AHEAD=0 every added line is a call that returns; the refusal is a named Fatal{RoleViolation, "gPipeInputs"}, never a fall-back (ID-110).

3.1 Controls and red-onces

A/B control (TASK-ra2 §2). Same binary, same records, entry's own ctest ENVIRONMENT, knob layered on last (.p5e/one_entry.py), on base 985ae62d: default arm 2/6 runs red (Fatal{UnmigratedPipeInput, "GetRenderStateParametersVersion@DrawElements"}); MOBILEGL_IPC_RUN_AHEAD=0 0/8 red. The defect is in the wait rule, not in a record's content. (MOBILEGL_IPC_VERB_BARRIER=0 was not used — ID-114.)

Control C1 — which half is load-bearing. With the forced wait in place but the kDraw fallback reverted, isplit is 161/181 as well. So the wait is the correctness fix and the fallback is the performance half: without it every indexed draw takes a forced drain and run-ahead is lockstep by another name on the draw path. Measured the other way round — fallback alone, no wait — the lane is 133/181 (~/w7/p5e-logs/ra2-census-E1).

Red-once A (guard arm back to if (isBarrieredFill) return;), 1 of 48 fails: RemoteGuards.ClientBarrieredFillWhileTheApplierIsInsideUnderRunAheadIsFatalByName, Value of: DiedOfAbort(child) Actual: false Expected: true exited 0 — and the green control ClientPipeInputsTouchInsideABarrieredFillUnderRunAheadIsAllowed keeps passing, so the pair is a rule and not "abort unconditionally".

Red-once B (kDraw fallback deleted), 1 of 48 fails, signal 6: Fatal{RoleViolation, "gPipeInputs"} - the GL thread touched gPipeInputs (MGPipeValidateForVerb) on a RUN-AHEAD session in a barriered fill that did not first wait for the applier to catch up.

Where the regression test lives (TASK-ra2 §3.4): both cases are in MG_Test/Wire/RemoteClientTest.cpp, in the ordinary unit lane, and StartRunAheadSession arms run-ahead in the child — so they run with the flip's behaviour on any head, including heads where kMGPipeP5eRunAheadReady is false. The strict lane runs under lockstep and cannot see this class by construction. The probe is the guard call itself and the in-applier flag is raised on the client thread before it (ID-102).

4 The two rulings (commit f2865480)

4.1 ID-133 then ID-136 — escalated, then retired instead

ID-133 landed as ruled (commit f2865480): MGPipeBarriered gained a fourth clause — a draw_vbo with NumDraws > 1 that is not kDrawIsIndirect is barriered — with the client's verb-keyed half in ClientVerbIsBarriered and a row in contract §2.1. The 18 entries went green. ID-136 then withdrew all of it and retired the pull instead (commit efe58eac). Both halves of that withdrawal are in one commit, by ruling, so no head ever carried one without the other. What follows is why, because the two premise corrections are the substance.

First: I did NOT measure the default differently from mv. MultiDraw.cpp:391's kAutoLadder is {Ext, BaseVertex, MultiIndirect, Indirect, DrawElements}, first supported wins, and ResolveTierForBatch can only DEMOTE a batched tier toward BaseVertex/DrawElements — never promote into an indirect one. My own confirmation is in the gate itself: the un-prefixed MultiDrawScenario.* entries were green throughout, and only the MultiDrawTierIndirect. / MultiDrawTierMultiIndirect. lanes failed. auto → ext, and the default tier never reaches the read.

But there is no indirect multi-draw OP to escalate. All twenty draw entry points collapse onto one opcode, draw_vbo (MGPDrawInfo's own header says so; PipeCalls.def:261 is the single row), and the tier is resolved on the SERVER, per batch, from driver caps the client does not hold. So the ruling's granularity does not exist on the wire, and the narrowest predicate both roles can compute is NumDraws > 1 && !kDrawIsIndirect. Consequence the ruling's cost argument did not anticipate: this is not confined to the opt-in tier lanes. A plain glMultiDraw* now takes a wait on every tier including the auto → ext default. kDrawIsIndirect is excluded precisely so Sodium's glMultiDrawElementsIndirect — which resolves its buffer from the handle in the record's second tail and pulls nothing — keeps running ahead.

Second, and this is the one worth acting on: the pull is not a data dependency. Backtrace of the aborting thread (gdb --batch, entry run under its own ctest command and ENVIRONMENT):

#4  MG_Pipe::(anonymous namespace)::CountBarrierPull(...)
#5  MG_Backend::DirectGLES::MultiDrawImpl::(anonymous namespace)::RunIndirect(...)
#6  MG_Backend::DirectGLES::MultiDrawImpl::DrawElementsBatch(...)
#7  MG_Remote::Server::ServerVerbSink::OnDrawVbo(...)
#10 MG_Remote::Server::PipeApplier::ApplyOne(...)

and the site is MultiDraw.cpp:87:

Uint BoundDrawIndirectBufferId() {
    const auto& indirect = MGB_CTX->GetBufferBindingSlot(BufferTarget::DrawIndirect).GetBoundObject();
    ...
}

called once at :584 as const Uint previousIndirectBinding = BoundDrawIndirectBufferId();, immediately before the tier binds its own scratch buffer (BindBufferId(GL_DRAW_INDIRECT_BUFFER, g_indirectCommands.id)) and restored at :603. The tier does not want the client's indirect buffer; it wants to put back whatever GL name was bound. That is a question the server can answer from its own binding record, and it is the same question mv already retired one function lower — ResolveBoundIndexBuffer (:167) has a stated handle arm; BoundDrawIndirectBufferId, twenty lines above it, was missed.

ID-136: done, and the escalation withdrawn with it. BoundDrawIndirectBufferId now has the handle arm, in mv's own idiom rather than a second one beside it:

• the arm is MG_Config::Transport != MG_Config::TransportMode::Monolith, stated (ID-81) and never inferred from a null handle (ID-110);
• a null MGPipeApplier().VerbIndirectBuffer is "this verb bound no indirect buffer" — the same 0 the monolith arm returns for an empty slot, and explicitly not the arm test;
• a missing backend resource is a named refusal, Fatal{RoleViolation, "multidraw-indirect-buffer-arm"}, a sibling of mv's RefuseMissingIndexBufferRecord placed beside it, never a quiet fall-back to the frontend;
• the monolith body below is unchanged token for token, and the function moved down beside BoundIndexBufferId — the same question about the other target, now with the same two arms.

Why a handle arm and not a migration. The tier never reads a byte of the client's indirect buffer. With a transport, the only other writer of this process's GL_DRAW_INDIRECT_BUFFER is DirectGLES.cpp's DrawSyncBit::IndirectBuffer arm, which binds from VerbIndirectBuffer and nothing else — so "what was bound" is "what the verb's record named", and restoring it is a question the server answers about itself.

MG_Backend/DirectGLES/MultiDraw.cpp is mv's file per ID-113; I edited it under ID-136, which overrode that boundary for this one change so the two halves could not land apart. The commit message says so, and the boundary has not otherwise moved.

4.2 ID-134 — the client-array lane

DirectGLES.Split.ClientVertexArrayScenario.* now carry the single label integration-clientarrays-split (MG_IntegrationTest/CMakeLists.txt), and .github/workflows/test.yml gains an expected-red step shaped like gl's Magma one: it asserts the marker, not merely a failure, and it fails both when the lane goes green with run-ahead armed and when it goes red without Fatal{UnmigratedVerb, "(Multi)DrawArrays+CLIENT_ARRAYS"} in the entries' private logs (they already had private, distinct log paths via SplitLogPaths.cmake.in).

ID-131 heeded by counting, not by assuming, on the built tree:

ctest -N -L …	before	after
integration-split	181	179
integration-gpu	1359	1357
integration-clientarrays-split	—	2
integration-magma-split	2	2

integration-clientarrays-split contains neither integration-split nor integration-gpu as a substring, and the drop is exactly two on each.

One deliberate deviation from "an expected-red step", and it is not hedging. The step branches on whether run-ahead is actually ARMED, read from the client's own log line. kMGPipeP5eRunAheadReady is a build constant: on a head where it is false no environment can arm run-ahead, ID-82's refusal is unreachable, and the entries are legitimately green — they are still vi's positive control for the lockstep arm. A step that demanded red unconditionally would fail on feat/disaggregated today for the one reason that is not a defect. With the flip landed for good the green branch is dead and its comment says to delete it.

4.3 The contract

CONTRACT-P5E.md §3.5 amended to what the code does, with the reason the old formulation could not be checked (a guard whose exemption cannot be false on the class it exists for reads as coverage and is not); §2.1 carries escalation (iii) with the "why the record and not the tier" argument and the withdrawal condition. And §3.2 is marked UNLANDED — nobody had noticed: it says "the server's stamp is server-private … never into the shared block", which describes applier-owned storage, and MGPipeServerStampVerbBoundary still writes m_filled / m_currentVerb / m_serverStampedVerb into gPipeInputs itself. That gap is what made the race possible: the contract already specified the P11 shape and the phase shipped the shared one.

5 What P11 still owes, and why the halves cannot be split

The fix works by re-establishing the quiescent window the lockstep barrier used to give for free. It does not make the block safe to share: gPipeInputs is still one copy with unpartitioned metadata, and the price of the current answer is a forced drain at every filling verb.

Smallest sufficient form, both parts required:

• (a) per-role stamp state — the applier gets its own {CurrentVerbSerial, FilledGen[], m_currentVerb, m_serverStampedVerb}, i.e. contract §3.2 actually landed, so the GL thread cannot withdraw the server's stamp at all;
• (b) a generation on the ~15 BARRIER_PULLED values the applier still reads, keyed to the record's produce-generation.

(a) without (b) is worse than today, and this is the load-bearing paragraph. Today a read of a value the client has since moved aborts loudly on the withdrawn stamp. Give the applier its own stamp and that read succeeds — against whatever the client's latest fill happens to have left in the shared value. A loud Fatal{UnmigratedPipeInput} becomes a silent stale read, which is the failure mode this campaign has twice paid for. So: both, or (b) discharged by retirement (P7/P8/P9/P13) first and then (a) alone suffices. The values are already partitioned by SuppliedFieldMask, which is why (b) is small — it is 15 rows, not the 63-field block.

6 Gate (head efe58eac, flip thrown, .p5e/p5e_gate_frozen.sh — a frozen copy per ID-127)

step	base 985ae62d	head efe58eac
build	rc=0	rc=0
unit	2254/2254	2256/2256 (+2 new cases)
isplit (-L integration-split)	112/181	179/179 — hard green
gpu (-L integration-gpu)	1290/1359	1357/1357 — hard green
integration-clientarrays-split	—	2/2 red by design, Fatal{UnmigratedVerb, "DrawArrays+CLIENT_ARRAYS"} and "MultiDrawArrays+CLIENT_ARRAYS"
integration-magma-split	2 expected-red	unchanged, not touched
gens	rc=0	rc=0; doc citations rc=0 with 3 pre-existing problems, all in CONTRACT-P5.md, none introduced here
flavours	rc=0	rc=0 — pull 915 TUs (push=absent), push 929 TUs (push=1), effective-macro assertion held
census (.p5e/census_flip.py, per-entry own command + own ENVIRONMENT)	70 red / 17 pairs / 4 fields	179/179 green, 0 red, zero Fatal{UnmigratedPipeInput}

The two expected-red lanes, run as themselves:

integration-clientarrays-split   rc=8  0% passed, 2 failed of 2
  Fatal{UnmigratedVerb, "DrawArrays+CLIENT_ARRAYS"}
  Fatal{UnmigratedVerb, "MultiDrawArrays+CLIENT_ARRAYS"}
  run-ahead ARMED present in both private logs
integration-magma-split          rc=8  0% passed, 2 failed of 2
  Fatal{UnmigratedPipeInput, "GetFramebufferBindingSlot@Clear"}

The one Fatal{ the census still prints is RoleViolation{g_applier} at rc=0 — it is Ct.CtWireScenario.TheDirectApplierResetCallOnTheGLThreadIsRoleViolation, a death test asserting its own abort. The census reports markers from passing entries too, which is why it appears.

6.1 The admitted set is ID-128's eight rows — which is how ID-136 checks out

Under MOBILEGL_IPC_STRICT_ERRORS=1, with Fatal{ at zero:

n	pair	why admitted
140	GetFramebufferBindingSlot@ReadPixels	ID-116 disjunct 1
140	GetBufferBindingSlot@ReadPixels	ID-116 disjunct 1
11	GetBufferBindingPoint@DrawArrays	ID-125 disjunct 2
11	GetTransformFeedbackProgram@DrawArrays	ID-125 disjunct 2
6	GetTextureObject@CopyImageSubData	ID-125 disjunct 2 / ID-118
2	GetProgramObject@ShaderStorageBlockBinding	ID-125 disjunct 2
2	ValidateProgramName@ShaderStorageBlockBinding	ID-125 disjunct 2
2	GetTextureUnitObject@CopyTexImage2D	ID-116 disjunct 1

The ninth row ID-133 introduced is gone, and its absence is the test. Under the escalation the census showed a ninth pair, GetBufferBindingSlot@DrawArrays (18), tagged:

Admitted{UnmigratedPipeInput, "GetBufferBindingSlot@DrawArrays"} [BARRIER-PULLED, ADMITTED,
retires in P8 (indirect), P9 (readback), P13 (transfer)]

ADMITTED rather than ADMITTED-ESCALATED, i.e. the static table answered and no admission rule was widened (gen_pipe_field_ownership.py --self-test reports the same 174 derived pairs throughout). That row existed only because the escalation had moved the pull onto the barriered arm where the allowlist already covered it. With ID-136 the pull does not happen at all, so the row disappears rather than being forgiven — which is the difference between retiring a read and legalising it, and the reason to prefer the former.

G1: not verified locally (ID-123) — the gate compares no symbols. Every line added is inside #if MOBILEGL_BUILD_DISAGGREGATED or in a split-only translation unit, and both non-split flavours build clean with the effective-macro assertion held.

Kimi audit cross-check: ra2 owns no field family, so there are no kimi-audit.md rows to retire, barrier or report missing. The four fields ID-132 named are untouched; what moved is who may write the block and when, plus one wait class.

6.2 Red-once for both halves of ID-136

Retiring the pull alone leaves the lane green, so the escalation is provably unnecessary — that is this head: escalation (iii) is gone from MGPipeBarriered, from ClientVerbIsBarriered and from contract §2.1, and isplit is 179/179 with the strict census back to eight admitted rows (§6.1). Nothing on the lane needs it.

Reverting the handle arm puts the marker back, if (false && …) on the new arm, 3/3 runs, deterministic:

Fatal{UnmigratedPipeInput, "GetBufferBindingSlot@DrawArrays"} [BARRIER-PULLED, UNBARRIERED,
the client did not fill it, retires in P8 (indirect), P9 (readback), P13 (transfer)]

So the arm is necessary and, with the escalation withdrawn, sufficient.

7 What I did not do

• The device measurement of the forced wait's cost — the coordinator's, not mine, per their message. The lane runs in ~12 s either way and cannot see it. My falsifiable expectation, now that ID-136 has removed the multi-draw rendezvous: draws no longer fill, so the client's forced waits should fall to the Clear / readback / blit / XFB / query count and nothing on the steady draw path — including plain glMultiDraw*, which after ID-136 is an ordinary unbarriered draw again.
• Anything in P11's shape. §5 states what it must provide instead.
• Any other part of mv's file. ID-136's override covers BoundDrawIndirectBufferId and the refusal beside it; nothing else in MultiDraw.cpp was touched.

Artefacts: ~/w7/p5e-logs/ra2-{A,B,P,red1} (controls, probes, red-onces), ~/w7/p5e-logs/ra2-census-{E1,E2,E3,final,final-strict}, scripts in ~/w7/p5e-ra2/.p5e/.