The three things that made the deferral inert are all fixed, in the order they were
found: the reply ring could not hold an answer across the ~1,000 records between two
creates; a residency budget could not help because what displaces an answer is what
the SERVER sends, not what the client emits; and so the mechanism had to be retention.
WHAT LANDED
* ILink::DeclareReplyRead + LinkCapabilities::RetainsReplies. The wire declares a
seq will be read BEFORE it publishes, so the declaration cannot race its own
answer; a link with the capability false ignores it, and the window does not
defer on one (CreateWindowEffective() is gated on it).
* StreamLink's ring of the last 8 answers becomes a declared-answer store: a
monotone whitelist plus a cursor, answers kept until their reader TAKES them
(take-and-delete under one lock, bytes swapped into the scratch), and a capacity
of 24 that is FATAL to exceed rather than evicting. An evicted wanted answer is
unrecoverable - re-latching by sending another create wipes the server's record
(PipeApply.cpp:2033) and its PendingUploads (PipeApply.h:332) while the client
cleared its dirty flags at emission (PipeApply.h:298-304) - so the capacity is
sized so it cannot be reached and reaching it says its own name.
* An answer nobody declared is DROPPED, which is what keeps the store bounded by the
client's deferral depth instead of by the server's answer rate. Retain-on-arrival
grows ~11.3 entries per create against a server that ignores the no-reply bit.
* A read that fails after WaitForApplied has returned Reached is now Fatal{ReplyMissing}
naming the seq: on the stream arm the reply frame strictly precedes the progress
frame that publishes appliedSeq, so that combination is a broken invariant, and
retrying it (what the row used to do) wedges the window for the rest of the
process while leaving every deferred object latched on a provisional accept.
* The residency budget is deleted (WireTables.cpp keeps its history), the window
retirement branch drains once before falling back - it could otherwise strand the
answers it was holding - and Detach clears the store.
MEASURED ON THE DEVICE, load frame, both ends rebuilt, frame-sent identical per op
(create 4,536, records 103,097) throughout:
window 1 (R-5's shape) 4,536 takes 28.66 s
window 2 explicit 2,500 17.27 s
window 2 AS THE SHIPPED DEFAULT 2,563 16.52 s
window 4 1,644 11.02 s
window 2 against an OLD SERVER 2,663 16.38 s (zero Fatals)
The last row is the version-skew case the design review named as a must: a server
built before the no-reply bit answers ~55,903 records a frame, the whitelist drops
every answer nobody declared, and the deferral still reads its own back.
GATES (172 + 15 green), and the important one is proved to go red:
* ADeclaredAnswerSurvivesFarMoreInterveningAnswersThanAnyFixedBuffer - reverting the
store to "keep the last 8 arrivals" fails it immediately with "the declared answer
did not survive 64 intervening ones". That is the line between the ring-1024
stand-in and a bounded mechanism.
* AnAnswerNobodyDeclaredIsNotKeptAtAll - red if everything is retained.
* TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers - on the in-process (ShmLink)
arm the knob at 4 must still block every create, red if the capability gate goes.
* The refused-create case now reads the repair that exists on THAT arm (the latch),
and asserts the window's own books are empty there.
NOT DONE, and named: the window has no unit gate on the STREAM arm yet - only the
device runs. Three rounds of this feature were green in the lanes while inert on the
device because the lanes and the device run different transports; the transport-level
retention gates close that for the store, and the window-level one should be built on
StreamClientPublicationTest's real stream session next.
The second one-line experiment, and the one that decides whether the retention design
is worth building. Ring 1024 plus the residency budget DISABLED (its constant raised
out of reach), W=2 and W=4:
arm waits rtt_mean by_op 2 wall
W=1 (today's default) 4,543 5,672 us 4,536 28.66 s
budget on + ring 8 9,078 2,598 us 9,071 27.04 s
ring 1024 + budget 7 3,450 5,367 us 3,443 23.04 s
ring 1024 + budget OFF W=2 2,560 4,916 us 2,553 16.80 s
ring 1024 + budget OFF W=4 1,691 4,224 us 1,684 11.45 s
rtt_mean does NOT grow (4,916 / 4,224 us, both BELOW the 5,672 us baseline) while the
waits fall by 2x and 2.7x and the wall follows. So the ~5 ms is a WAKEUP round trip and
the server drains a batch per wakeup - which is exactly what makes W pay, and it is the
question the audit listed as undecidable from the repo and the watershed for whether
this is worth doing at all. It is.
AND IT BEATS THE HANDOFF'S OWN FLOOR: handoff-2 section 0.4 put the window's landing at
15-20 s, treating the client's ~13.4 s CPU as ADDITIVE with the waits. This frame turns
out to be wait-dominated, so amortising four creates per wakeup lands at 11.45 s, 3.5 s
below that estimate.
1024 IS A STAND-IN, NOT THE DESIGN: it works only because 1024 happens to be deep enough
for this workload, and 1024 retained answers is ~2 GiB worst case (a ReadPixels answer can
be 2 MiB). Retention gives the same property - the answer is still there - with a bound
tied to the client's own deferral depth, which is why the implementation the audit
specified is what gets built, not "make the ring 1024".
Also corrects the note's section 7 path 1 (audit F15): on the stream arm SEG_REPLY and the
Hello's ReplySlotCount have nothing to do with answers - they travel as Reply frames and
StreamLink::ReadReply reads the client's own store - so "deepen the pool" does not address
the arm that was measured, and should not be paid for.
Reverted (ring 8, budget 7, probes deleted); 172 + 13 green.
One-line experiment before writing the retention table: StreamLink's kReplyRing 8 -> 1024,
nothing else changed, same load frame on the same device.
ring = 8 (today) ring = 1024
readfail 8,872 0
taken 63 4,499 / 4,500
blocked (fell back) 4,435 0
by_op 2 9,071 3,443
wall 27.9 s 23.04 s
frame-sent create 4,536, records 103,097 identical per op
TWO CONCLUSIONS. First, eviction was the ONLY cause: with a buffer deep enough, every
deferred answer was read back and not one create fell through to the blocking path.
The wedge (want=310) and the useless budget (budget=4,451) were two faces of "eight
slots is not enough". Second, what brakes now is the BUDGET, not the buffer: by_op 2 =
3,443 matches the number of times the budget forced an early read (3,395), i.e. every
one of those is a real blocking wait. Its constant is 7, sized for the 8-slot pool, so
against a 1,024-deep buffer it became the only throttle and the wall only moved 27.9 ->
23.0 s.
WHAT IT MEANS FOR THE DESIGN. Once the answers are RETAINED until read, correctness no
longer depends on how long one survives in a bounded buffer, so the budget must be
sized to the RETENTION CAPACITY rather than to the pool - or be replaced by "retained
means safe". by_op 2 = records/W then puts W=2 at ~2,268 waits and the wall at ~15 s,
which is the goal's own success criterion.
Experiment reverted (ring back to 8, probes deleted); 172 + 13 + 22 + 103 green.
Steps 1 and 2 of the plan the design review set out, landed and measured on the
device. Step 1 answers the question step 2 depended on; step 2 answers the one the
whole line depended on, and the answer is that the window's shape is wrong.
STEP 1 - CAN AN ANSWER BE LOST IN DELIVERY? NO. ReplyPool now counts what the link
took (PostedReplies) and what it refused (FailedReplies), and a non-OK result is
NAMED instead of swallowed: it checked only BUFFER_TOO_SMALL and let
TRANSPORT_CLOSED vanish with no line and no counter, so "the server never sent it"
and "it was sent and the buffer overwrote it" were the same symptom with opposite
fixes. Measured: posted=4501 failed=0. The twenty-answer gap between the server's
tally and the client's was a sampling difference, not a drop - and the server's
m_answered counts ATTEMPTS (incremented before PostReply), which is why it could
never have settled it.
STEP 2 - DOES A RESIDENCY BUDGET FIX THE WINDOW? NO.
Wire_ResourceCreate now measures how many reply-owning records the client has
emitted since the oldest window entry was pushed (ClientSession::ReplyPostings, a
new clock) and takes the oldest answer back as soon as that approaches the reply
buffer's depth, instead of deferring past the point where the answer can still
exist. On the device:
P12win call=4501 cfg=2 eff=2 pending=2 taken=63 pushed=65 blocked=4435
budget=4451 readfail=8872 posts=54377
kind=frame-op frame=14 wait_replies=9078 by_op=2=9071 wall 27.87 s
budget=4451: it fires on essentially EVERY create, the window stops deferring
(pushed=65), and the drain still reads nothing back. THE REASON IS STRUCTURAL, not
a constant:
* the window only gets control WHEN A CREATE ARRIVES. Between two creates this
client emits on the order of a thousand records - at the FIRST failed read the
reply ring already spanned [1175..1504] while the head was seq 310 - and nothing
looks at the window in that gap;
* what displaces a deferred answer is not what this client emits, it is WHAT THE
SERVER ANSWERS. The server is the one behind (one answer per ~5 ms wakeup while
the client runs ahead), so the replies arriving during the gap are its LATE
ANSWERS TO THE EARLIER CREATES - they fill the 8 slots first;
* so the check fires too late by construction. Tightening it does not fix that; it
only removes the deferral entirely, which is what pushed=65 is.
WHAT THAT LEAVES. A deferral is readable only if the client KEEPS the answers it
deferred. The buffer is bounded by COUNT and the client cannot bound how many
answers the server will post between two of its creates, so hoping the buffer still
holds one is a race the client loses - which is also why the earlier two diagnoses
(reply-pool churn, then FIFO-with-no-way-out) were both describing symptoms of the
same thing. The shape that can work is retention: the reader keeps the answers for
the seqs still outstanding (a handful), drops them when read, and reports a LOSS
when a read asks for one it no longer has. That is a transport change - StreamLink's
ring becomes a retention map with a cap and a lost counter - and it is the next step.
BOTH STEPS STAY: step 1 is a net gain on its own, and step 2 is the safe direction
(it can only degrade the window to no-deferral, i.e. the pre-window shape) and its
clock is the one the retention design needs. MOBILEGL_IPC_CREATE_WINDOW stays at 1.
172/172 and 13/13 after the probes were removed. The measurements and the revised
conclusion are in notes/p12/CREATE-WINDOW-MEASURED.md section 9.
The create window could not collect its deferred answers. The previous commit
diagnosed that as the reply pool being 8 slots deep against 55,903 replies in one
frame, and left a no-reply bit as the way out. This lands that bit, measures it on
the device, and reports that the diagnosis was half wrong.
WHAT LANDED. ClientSession::EmitAndWaitTails takes a second question beside
wantReply: willReadReply. They are not the same question and the window is the case
that separates them - it emits with wantReply=false and READS the answer later, so
its records must still be answered. kRecNoReply (RingRecordFlags bit 5, which
aliases kOptional in the call-flag space; the flag-space asserts were extended and
the alias named) is stamped when willReadReply is false, and the decoder skips
PostReply while keeping every acceptance counter - "nobody reads the answer" is not
"nothing happened". The two illegal combinations are Fatal by name.
Three call sites say it: SetTextureParams for a confirmed object, the respecify
row's fire-and-forget half, and sub-data where MGPipeSubDataWantsItsReply decides.
MEASURED ON THE DEVICE (aikjs4s8cys8ba7l, rebuilt APK + client, both halves new):
server: skipped=50,001 answered=4,521 (a frame's replies: 55,903 -> 4,543)
client: Reply frames received seen=4,501, latest seq=97,181
frame-sent: identical per op, records untouched
create row: by_op 2 = 9,070 (was 4,536) - because the drain's blocking take is
now counted, so the row's number stays honest instead of reading as
"create stopped waiting" while the waiting merely moved
AND THE WINDOW STILL READ BACK NOTHING: readfail 8,951 -> 8,872, i.e. -0.9%.
THE REAL CAUSE, from the miss probe:
P12miss n=1 want=310 replySeq=1504 ring=[1175..1504]
P12miss n=9001 want=310 replySeq=102915 ring=[102866..102915]
want=310 every time. The drain is STRICTLY FIFO and a failed read has no way out -
it retries the same head for ever. Any reply buffer is bounded by design (8 pool
slots, or the 8-frame ring the stream link now keeps), so with 4,536 deferred
answers at least one is overwritten before it is read, and that one wedges the
window permanently. Cutting the churn by two orders of magnitude does not help: it
turns a certainty into a certainty. That is the constraint, not the depth.
THE FIX IS NAMED AND NOT TAKEN (the goal this work ran under stops here on purpose).
On a failed read of the head, abandon it: MGPipeNoteHandleUnpublished +
NoteCreateSuspect + pop - the DECLINE path reused for "unknown". The window then
self-heals through the blocking path and the object merely gets slower. It needs its
own gate: a failed read must let the window ADVANCE.
Gates, all green (172 + 13 + 22 + 103):
PipeWireCodecTest.TheNoReplyBitSkipsTheAnswerAndKeepsTheAcceptanceRecord - the skip,
the kept tally, the answered/skipped split, and the negative control that a record
WITHOUT the bit is still answered (a server that stopped answering everything
would otherwise pass)
RemoteClientControls.ACreateWindowDefersTheAnswersAndTheWindowOffDoesNot - now also
asserts the answers CAME BACK (answers_taken=11, outstanding=1, fallbacks=0 against
12 records). RED-ONCE PROVED: giving the window's creates the literal
"fire-and-forget => skip" predicate makes this case fail.
RingTest - the sixth and last flag alias, and the hand-written exhaustiveness list
extended (with a note that it is only as exhaustive as that list).
Server side: AnsweredRecords()/SkippedAnswers() - the number that made this
debuggable, since a wire change whose effect is invisible is undebuggable.
Client-first is safe: an old server ignores bit 5 and replies as before, so the two
cross-version directions degrade to today's behaviour - this is "both ends must
upgrade to take EFFECT", not "to avoid breaking". Hit that in practice first.
An independent review of the one design door the measurement left open (stop sending
the ~51,000 answers nobody reads, so the 8-slot pool stops churning and the create
window becomes collectable). Verdict: feasible, worth doing, and the proposal AS
WRITTEN would disable the thing it exists to enable.
THE CATCH. The windowed create is wantReply=false AND its answer is read -
WireTables.cpp's drain reads exactly those seqs. A bit keyed on "fire-and-forget"
therefore stops the server answering the creates the window is waiting for: the
window goes from "cannot be collected" to "there is nothing to collect", and the
readfail counter that is the only way to diagnose it disappears with it. It fails
silently, not loudly. The predicate has to be per-record "this answer will never be
read" - true of params / respecify-escape / sub-data, false of a windowed create -
carried as a separate input from wantReply (willReadReply), with
willReadReply && !rowCarriesReplySlot rejected rather than ignored.
VERIFIED HERE, not taken on trust (the reviewer's file:line claims, checked):
- the flag space is RingRecordFlags (Ring.h:281-285), bits 0-4 taken, BIT 5 FREE,
and the header's own table (PipeWire.inc:35-65) says "bit 5 kOptional vs (nothing)
no ring counterpart" - so a new ring bit aliases kOptional, which is exactly what
the assert at PipeWireCodec.cpp:142-145 refuses;
- the three static_asserts that will trip are real and deliberate
(PipeWireCodec.cpp:106, :142-145, RingTest.cpp:740);
- WireFingerprint mixes payload-layout, catalogue, render-state digests + OpCount,
none of which a ring bit moves - so no protocol revision bump is required, and
both cross-version directions degrade to today's behaviour. That makes this
"must upgrade both ends to take EFFECT", not "must upgrade both ends not to
BREAK" - one notch cheaper than handoff-2 section 3.2's protocol surface;
- StreamLink.cpp:564-571 is the independent progress flush (>= 64 records or
>= 1 ms) and :577 is PostReply piggybacking SendProgress on the reply - so
skipping replies coarsens progress granularity. Bounded, but unmeasured.
AND IT CORRECTED TWO OF MY OWN NUMBERS. The frame's reply count is 55,903, not
59,673: op 4 ResourceDestroy carries kNone in the catalogue (PipeWire.inc:179) and
PostReply Fatals on a row with no kReplySlot, so it never answers - I had added its
3,770 records to the total. The derived figures move with it (12.3 replies per create,
0.65 of a create of survival, 6,988 pool churns), and every copy is fixed here:
Config.h, ConfigLoader.cpp, WireTables.cpp's section comment, CONTRACT-P5E.md,
CREATE-WINDOW-MEASURED.md.
TWO MORE THINGS THE FOLLOW-UP MUST CARRY, both recorded in the note's section 7:
- LinkMetricsBeginReply counts ownsReplySlot, so once the flag lands by_op 2 falls
to ~0 - and section 1 of the note tells the reader to use by_op 2 to check whether
the window engaged. That check becomes a FALSE GREEN; the flag needs a
server-side "how many times did this op answer" counter beside it.
- the ~112,000 sendmsg/frame the flag removes are on the apply thread, so the
flag's own value is a measurable claim that nobody has measured.
172/172 and 13/13 after the comment corrections.
swung0x48
BZLZHH