The three things that made the deferral inert are all fixed, in the order they were
found: the reply ring could not hold an answer across the ~1,000 records between two
creates; a residency budget could not help because what displaces an answer is what
the SERVER sends, not what the client emits; and so the mechanism had to be retention.
WHAT LANDED
* ILink::DeclareReplyRead + LinkCapabilities::RetainsReplies. The wire declares a
seq will be read BEFORE it publishes, so the declaration cannot race its own
answer; a link with the capability false ignores it, and the window does not
defer on one (CreateWindowEffective() is gated on it).
* StreamLink's ring of the last 8 answers becomes a declared-answer store: a
monotone whitelist plus a cursor, answers kept until their reader TAKES them
(take-and-delete under one lock, bytes swapped into the scratch), and a capacity
of 24 that is FATAL to exceed rather than evicting. An evicted wanted answer is
unrecoverable - re-latching by sending another create wipes the server's record
(PipeApply.cpp:2033) and its PendingUploads (PipeApply.h:332) while the client
cleared its dirty flags at emission (PipeApply.h:298-304) - so the capacity is
sized so it cannot be reached and reaching it says its own name.
* An answer nobody declared is DROPPED, which is what keeps the store bounded by the
client's deferral depth instead of by the server's answer rate. Retain-on-arrival
grows ~11.3 entries per create against a server that ignores the no-reply bit.
* A read that fails after WaitForApplied has returned Reached is now Fatal{ReplyMissing}
naming the seq: on the stream arm the reply frame strictly precedes the progress
frame that publishes appliedSeq, so that combination is a broken invariant, and
retrying it (what the row used to do) wedges the window for the rest of the
process while leaving every deferred object latched on a provisional accept.
* The residency budget is deleted (WireTables.cpp keeps its history), the window
retirement branch drains once before falling back - it could otherwise strand the
answers it was holding - and Detach clears the store.
MEASURED ON THE DEVICE, load frame, both ends rebuilt, frame-sent identical per op
(create 4,536, records 103,097) throughout:
window 1 (R-5's shape) 4,536 takes 28.66 s
window 2 explicit 2,500 17.27 s
window 2 AS THE SHIPPED DEFAULT 2,563 16.52 s
window 4 1,644 11.02 s
window 2 against an OLD SERVER 2,663 16.38 s (zero Fatals)
The last row is the version-skew case the design review named as a must: a server
built before the no-reply bit answers ~55,903 records a frame, the whitelist drops
every answer nobody declared, and the deferral still reads its own back.
GATES (172 + 15 green), and the important one is proved to go red:
* ADeclaredAnswerSurvivesFarMoreInterveningAnswersThanAnyFixedBuffer - reverting the
store to "keep the last 8 arrivals" fails it immediately with "the declared answer
did not survive 64 intervening ones". That is the line between the ring-1024
stand-in and a bounded mechanism.
* AnAnswerNobodyDeclaredIsNotKeptAtAll - red if everything is retained.
* TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers - on the in-process (ShmLink)
arm the knob at 4 must still block every create, red if the capability gate goes.
* The refused-create case now reads the repair that exists on THAT arm (the latch),
and asserts the window's own books are empty there.
NOT DONE, and named: the window has no unit gate on the STREAM arm yet - only the
device runs. Three rounds of this feature were green in the lanes while inert on the
device because the lanes and the device run different transports; the transport-level
retention gates close that for the store, and the window-level one should be built on
StreamClientPublicationTest's real stream session next.
The second one-line experiment, and the one that decides whether the retention design
is worth building. Ring 1024 plus the residency budget DISABLED (its constant raised
out of reach), W=2 and W=4:
arm waits rtt_mean by_op 2 wall
W=1 (today's default) 4,543 5,672 us 4,536 28.66 s
budget on + ring 8 9,078 2,598 us 9,071 27.04 s
ring 1024 + budget 7 3,450 5,367 us 3,443 23.04 s
ring 1024 + budget OFF W=2 2,560 4,916 us 2,553 16.80 s
ring 1024 + budget OFF W=4 1,691 4,224 us 1,684 11.45 s
rtt_mean does NOT grow (4,916 / 4,224 us, both BELOW the 5,672 us baseline) while the
waits fall by 2x and 2.7x and the wall follows. So the ~5 ms is a WAKEUP round trip and
the server drains a batch per wakeup - which is exactly what makes W pay, and it is the
question the audit listed as undecidable from the repo and the watershed for whether
this is worth doing at all. It is.
AND IT BEATS THE HANDOFF'S OWN FLOOR: handoff-2 section 0.4 put the window's landing at
15-20 s, treating the client's ~13.4 s CPU as ADDITIVE with the waits. This frame turns
out to be wait-dominated, so amortising four creates per wakeup lands at 11.45 s, 3.5 s
below that estimate.
1024 IS A STAND-IN, NOT THE DESIGN: it works only because 1024 happens to be deep enough
for this workload, and 1024 retained answers is ~2 GiB worst case (a ReadPixels answer can
be 2 MiB). Retention gives the same property - the answer is still there - with a bound
tied to the client's own deferral depth, which is why the implementation the audit
specified is what gets built, not "make the ring 1024".
Also corrects the note's section 7 path 1 (audit F15): on the stream arm SEG_REPLY and the
Hello's ReplySlotCount have nothing to do with answers - they travel as Reply frames and
StreamLink::ReadReply reads the client's own store - so "deepen the pool" does not address
the arm that was measured, and should not be paid for.
Reverted (ring 8, budget 7, probes deleted); 172 + 13 green.
swung0x48
BZLZHH