Compare commits

..
Author SHA1 Message Date
swung0x48 a078b6bcee [Perf] (MG_Remote): The producer's transport park is timed as its own per-frame number, and a reducer pairs the two role logs into a frame-time breakdown
transport_wait_ns measures wall time in Ring::Wait's flush/spin/sleep, so
the client's parked time is read directly instead of inferred by
subtracting CPU clocks that run concurrently with the server.
frame_stats.py matches client and server present serials into per-frame
avg/p50/p99 splits; the README documents the MOBILEGL_PIPE_STATS=1
workflow.
2026-09-27 02:55:38 -04:00
BZLZHH 6bccf98d7f [Perf] (MG_Remote): Frame time is accounted on both sides, down to the socket reads and the client's waits. 2026-09-26 17:57:48 +08:00
BZLZHH 3a6c5c7f04 [Perf] (MG_Remote): The create window ships at 10, and the server now reports where its own frame time goes. 2026-09-26 15:30:48 +08:00
BZLZHH b017ec1c1f [Test] (MG_IntegrationTest/Scenarios): The P4a white-box applier reads fence on a readback. 2026-09-26 12:53:10 +08:00
BZLZHH 102f2ca7d5 [Perf] (MG_Remote/Client): The create window's shipped depth is 6, and its ceiling 8. 2026-09-26 11:26:54 +08:00
BZLZHH 1f1ece8a33 [Test] (MG_IntegrationTest/Scenarios): Fence the object-death gates on a readback instead of a clear. 2026-09-26 09:36:37 +08:00
BZLZHH 677689454a [Test] (P12): the window's gates move onto the arm it ships on - a real stream session with an answering peer, and it goes red by name when the store is a bounded ring
THE STRUCTURAL FIX, not just one more case. Three rounds of this feature were green in
the lanes while being inert on the device, because every window-level gate in the tree
runs in RemoteClientControls, whose session is TransportMode::InProcess - ShmLink - and
since P12 the window is OFF there by design. So the lanes and the device ran different
transports, and nothing in the tree could tell.

StreamClientPublicationTest.EveryDeferredCreateAnswerComesBackOnTheShippingArm runs on a
real StreamLink with a real ClientSession and a peer that ANSWERS: a scripted servo pops
each record, replies with its seq, advances and flushes the watermark - i.e. exactly what
the drain's WaitForApplied oracle needs, and nothing more (nothing applies a record).

IT IS PROVED TO GO RED, and by the right name. Reverting the store to "keep the last 8
arrivals" makes it die on the diagnostic added with the store:

    MGPipe: Fatal{ReplyMissing, "create window"} - seq 1 was declared and APPLIED, and
    its answer is no longer in the link's store; the window cannot advance past it and
    nothing recovers a lost answer

AND THE FIRST VERSION OF THIS GATE WAS WORTHLESS, which is the part worth keeping: with
24 creates and nothing else it passed against the ring as well, because between a push
and its read only a couple of answers arrive. What killed the device was TRAFFIC - the
server is behind and its late answers to earlier records fill a bounded buffer first - so
the case now emits twelve MORE records whose answers are wanted (declared
SetTextureParams rows) between the first create and the rest, waits for the peer to
answer them, and only then lets the drain come back for the create. Two things were
needed to make that deterministic: the settlement wait (otherwise it is a race the ring
can win by luck) and glFlush(), because a published record reaches the peer on a flush
and not on PublishAndNotify - the neighbouring case in the same file is built on that.

ALSO IN THIS COMMIT: the assertion is the CONSERVATION LAW, not "taken == creates".
Nothing drains the window when no further create arrives, so the last few answers stay
held - taken + pending == creates, with pending <= window, is the honest contract, and
taken == creates would be asserting a drain on a path the row does not have.

318 tests green across five lanes (172 + 15 + 22 + 103 + 6).
2026-09-26 03:41:48 +08:00
BZLZHH 3a2555805a [Perf] (P12): the create window lands as the default - declared-answer retention replaces the ring, and the load frame goes 28.66 s -> 16.52 s (11.02 s at window 4)
The three things that made the deferral inert are all fixed, in the order they were
found: the reply ring could not hold an answer across the ~1,000 records between two
creates; a residency budget could not help because what displaces an answer is what
the SERVER sends, not what the client emits; and so the mechanism had to be retention.

WHAT LANDED
  * ILink::DeclareReplyRead + LinkCapabilities::RetainsReplies. The wire declares a
    seq will be read BEFORE it publishes, so the declaration cannot race its own
    answer; a link with the capability false ignores it, and the window does not
    defer on one (CreateWindowEffective() is gated on it).
  * StreamLink's ring of the last 8 answers becomes a declared-answer store: a
    monotone whitelist plus a cursor, answers kept until their reader TAKES them
    (take-and-delete under one lock, bytes swapped into the scratch), and a capacity
    of 24 that is FATAL to exceed rather than evicting. An evicted wanted answer is
    unrecoverable - re-latching by sending another create wipes the server's record
    (PipeApply.cpp:2033) and its PendingUploads (PipeApply.h:332) while the client
    cleared its dirty flags at emission (PipeApply.h:298-304) - so the capacity is
    sized so it cannot be reached and reaching it says its own name.
  * An answer nobody declared is DROPPED, which is what keeps the store bounded by the
    client's deferral depth instead of by the server's answer rate. Retain-on-arrival
    grows ~11.3 entries per create against a server that ignores the no-reply bit.
  * A read that fails after WaitForApplied has returned Reached is now Fatal{ReplyMissing}
    naming the seq: on the stream arm the reply frame strictly precedes the progress
    frame that publishes appliedSeq, so that combination is a broken invariant, and
    retrying it (what the row used to do) wedges the window for the rest of the
    process while leaving every deferred object latched on a provisional accept.
  * The residency budget is deleted (WireTables.cpp keeps its history), the window
    retirement branch drains once before falling back - it could otherwise strand the
    answers it was holding - and Detach clears the store.

MEASURED ON THE DEVICE, load frame, both ends rebuilt, frame-sent identical per op
(create 4,536, records 103,097) throughout:

    window 1 (R-5's shape)              4,536 takes   28.66 s
    window 2 explicit                   2,500         17.27 s
    window 2 AS THE SHIPPED DEFAULT     2,563         16.52 s
    window 4                            1,644         11.02 s
    window 2 against an OLD SERVER      2,663         16.38 s   (zero Fatals)

The last row is the version-skew case the design review named as a must: a server
built before the no-reply bit answers ~55,903 records a frame, the whitelist drops
every answer nobody declared, and the deferral still reads its own back.

GATES (172 + 15 green), and the important one is proved to go red:
  * ADeclaredAnswerSurvivesFarMoreInterveningAnswersThanAnyFixedBuffer - reverting the
    store to "keep the last 8 arrivals" fails it immediately with "the declared answer
    did not survive 64 intervening ones". That is the line between the ring-1024
    stand-in and a bounded mechanism.
  * AnAnswerNobodyDeclaredIsNotKeptAtAll - red if everything is retained.
  * TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers - on the in-process (ShmLink)
    arm the knob at 4 must still block every create, red if the capability gate goes.
  * The refused-create case now reads the repair that exists on THAT arm (the latch),
    and asserts the window's own books are empty there.

NOT DONE, and named: the window has no unit gate on the STREAM arm yet - only the
device runs. Three rounds of this feature were green in the lanes while inert on the
device because the lanes and the device run different transports; the transport-level
retention gates close that for the store, and the window-level one should be built on
StreamClientPublicationTest's real stream session next.
2026-09-26 03:31:14 +08:00
BZLZHH 1ef7439625 [Perf] (P12): with the budget off and a deep reply buffer the window pays - 28.66 s -> 11.45 s at W=4, and the round trip does NOT grow, so the ~5 ms is a wakeup the server batches away
The second one-line experiment, and the one that decides whether the retention design
is worth building. Ring 1024 plus the residency budget DISABLED (its constant raised
out of reach), W=2 and W=4:

  arm                        waits   rtt_mean   by_op 2    wall
  W=1 (today's default)      4,543   5,672 us     4,536   28.66 s
  budget on + ring 8         9,078   2,598 us     9,071   27.04 s
  ring 1024 + budget 7       3,450   5,367 us     3,443   23.04 s
  ring 1024 + budget OFF W=2 2,560   4,916 us     2,553   16.80 s
  ring 1024 + budget OFF W=4 1,691   4,224 us     1,684   11.45 s

rtt_mean does NOT grow (4,916 / 4,224 us, both BELOW the 5,672 us baseline) while the
waits fall by 2x and 2.7x and the wall follows. So the ~5 ms is a WAKEUP round trip and
the server drains a batch per wakeup - which is exactly what makes W pay, and it is the
question the audit listed as undecidable from the repo and the watershed for whether
this is worth doing at all. It is.

AND IT BEATS THE HANDOFF'S OWN FLOOR: handoff-2 section 0.4 put the window's landing at
15-20 s, treating the client's ~13.4 s CPU as ADDITIVE with the waits. This frame turns
out to be wait-dominated, so amortising four creates per wakeup lands at 11.45 s, 3.5 s
below that estimate.

1024 IS A STAND-IN, NOT THE DESIGN: it works only because 1024 happens to be deep enough
for this workload, and 1024 retained answers is ~2 GiB worst case (a ReadPixels answer can
be 2 MiB). Retention gives the same property - the answer is still there - with a bound
tied to the client's own deferral depth, which is why the implementation the audit
specified is what gets built, not "make the ring 1024".

Also corrects the note's section 7 path 1 (audit F15): on the stream arm SEG_REPLY and the
Hello's ReplySlotCount have nothing to do with answers - they travel as Reply frames and
StreamLink::ReadReply reads the client's own store - so "deepen the pool" does not address
the arm that was measured, and should not be paid for.

Reverted (ring 8, budget 7, probes deleted); 172 + 13 green.
2026-09-26 03:01:20 +08:00
BZLZHH 82de3f6826 [Docs] (P12): the ring-1024 experiment - every deferred answer comes back (readfail 8,872 -> 0), so eviction was the whole cause and the residency budget is now the brake
One-line experiment before writing the retention table: StreamLink's kReplyRing 8 -> 1024,
nothing else changed, same load frame on the same device.

                       ring = 8 (today)   ring = 1024
    readfail                    8,872              0
    taken                          63          4,499 / 4,500
    blocked (fell back)         4,435              0
    by_op 2                     9,071          3,443
    wall                        27.9 s         23.04 s
    frame-sent        create 4,536, records 103,097   identical per op

TWO CONCLUSIONS. First, eviction was the ONLY cause: with a buffer deep enough, every
deferred answer was read back and not one create fell through to the blocking path.
The wedge (want=310) and the useless budget (budget=4,451) were two faces of "eight
slots is not enough". Second, what brakes now is the BUDGET, not the buffer: by_op 2 =
3,443 matches the number of times the budget forced an early read (3,395), i.e. every
one of those is a real blocking wait. Its constant is 7, sized for the 8-slot pool, so
against a 1,024-deep buffer it became the only throttle and the wall only moved 27.9 ->
23.0 s.

WHAT IT MEANS FOR THE DESIGN. Once the answers are RETAINED until read, correctness no
longer depends on how long one survives in a bounded buffer, so the budget must be
sized to the RETENTION CAPACITY rather than to the pool - or be replaced by "retained
means safe". by_op 2 = records/W then puts W=2 at ~2,268 waits and the wall at ~15 s,
which is the goal's own success criterion.

Experiment reverted (ring back to 8, probes deleted); 172 + 13 + 22 + 103 green.
2026-09-26 02:44:30 +08:00
BZLZHH b1ef484116 [Perf] (MG_Remote): the residency budget does not make the window usable either - the answer is gone before the client next looks, so a deferral needs RETENTION, not a budget
Steps 1 and 2 of the plan the design review set out, landed and measured on the
device. Step 1 answers the question step 2 depended on; step 2 answers the one the
whole line depended on, and the answer is that the window's shape is wrong.

STEP 1 - CAN AN ANSWER BE LOST IN DELIVERY? NO. ReplyPool now counts what the link
took (PostedReplies) and what it refused (FailedReplies), and a non-OK result is
NAMED instead of swallowed: it checked only BUFFER_TOO_SMALL and let
TRANSPORT_CLOSED vanish with no line and no counter, so "the server never sent it"
and "it was sent and the buffer overwrote it" were the same symptom with opposite
fixes. Measured: posted=4501 failed=0. The twenty-answer gap between the server's
tally and the client's was a sampling difference, not a drop - and the server's
m_answered counts ATTEMPTS (incremented before PostReply), which is why it could
never have settled it.

STEP 2 - DOES A RESIDENCY BUDGET FIX THE WINDOW? NO.
Wire_ResourceCreate now measures how many reply-owning records the client has
emitted since the oldest window entry was pushed (ClientSession::ReplyPostings, a
new clock) and takes the oldest answer back as soon as that approaches the reply
buffer's depth, instead of deferring past the point where the answer can still
exist. On the device:

    P12win call=4501 cfg=2 eff=2 pending=2 taken=63 pushed=65 blocked=4435
           budget=4451 readfail=8872 posts=54377
    kind=frame-op frame=14 wait_replies=9078 by_op=2=9071   wall 27.87 s

budget=4451: it fires on essentially EVERY create, the window stops deferring
(pushed=65), and the drain still reads nothing back. THE REASON IS STRUCTURAL, not
a constant:

  * the window only gets control WHEN A CREATE ARRIVES. Between two creates this
    client emits on the order of a thousand records - at the FIRST failed read the
    reply ring already spanned [1175..1504] while the head was seq 310 - and nothing
    looks at the window in that gap;
  * what displaces a deferred answer is not what this client emits, it is WHAT THE
    SERVER ANSWERS. The server is the one behind (one answer per ~5 ms wakeup while
    the client runs ahead), so the replies arriving during the gap are its LATE
    ANSWERS TO THE EARLIER CREATES - they fill the 8 slots first;
  * so the check fires too late by construction. Tightening it does not fix that; it
    only removes the deferral entirely, which is what pushed=65 is.

WHAT THAT LEAVES. A deferral is readable only if the client KEEPS the answers it
deferred. The buffer is bounded by COUNT and the client cannot bound how many
answers the server will post between two of its creates, so hoping the buffer still
holds one is a race the client loses - which is also why the earlier two diagnoses
(reply-pool churn, then FIFO-with-no-way-out) were both describing symptoms of the
same thing. The shape that can work is retention: the reader keeps the answers for
the seqs still outstanding (a handful), drops them when read, and reports a LOSS
when a read asks for one it no longer has. That is a transport change - StreamLink's
ring becomes a retention map with a cap and a lost counter - and it is the next step.

BOTH STEPS STAY: step 1 is a net gain on its own, and step 2 is the safe direction
(it can only degrade the window to no-deferral, i.e. the pre-window shape) and its
clock is the one the retention design needs. MOBILEGL_IPC_CREATE_WINDOW stays at 1.

172/172 and 13/13 after the probes were removed. The measurements and the revised
conclusion are in notes/p12/CREATE-WINDOW-MEASURED.md section 9.
2026-09-26 02:35:05 +08:00
BZLZHH 4e48702f26 [Feat] (MG_Remote): a record can tell the server its answer will never be read - the bit works and cuts a frame's replies by 92%, and it proves the window's real blocker is the FIFO drain, not the pool depth
The create window could not collect its deferred answers. The previous commit
diagnosed that as the reply pool being 8 slots deep against 55,903 replies in one
frame, and left a no-reply bit as the way out. This lands that bit, measures it on
the device, and reports that the diagnosis was half wrong.

WHAT LANDED. ClientSession::EmitAndWaitTails takes a second question beside
wantReply: willReadReply. They are not the same question and the window is the case
that separates them - it emits with wantReply=false and READS the answer later, so
its records must still be answered. kRecNoReply (RingRecordFlags bit 5, which
aliases kOptional in the call-flag space; the flag-space asserts were extended and
the alias named) is stamped when willReadReply is false, and the decoder skips
PostReply while keeping every acceptance counter - "nobody reads the answer" is not
"nothing happened". The two illegal combinations are Fatal by name.

Three call sites say it: SetTextureParams for a confirmed object, the respecify
row's fire-and-forget half, and sub-data where MGPipeSubDataWantsItsReply decides.

MEASURED ON THE DEVICE (aikjs4s8cys8ba7l, rebuilt APK + client, both halves new):
    server:  skipped=50,001   answered=4,521        (a frame's replies: 55,903 -> 4,543)
    client:  Reply frames received seen=4,501, latest seq=97,181
    frame-sent: identical per op, records untouched
    create row: by_op 2 = 9,070 (was 4,536)  - because the drain's blocking take is
               now counted, so the row's number stays honest instead of reading as
               "create stopped waiting" while the waiting merely moved
AND THE WINDOW STILL READ BACK NOTHING: readfail 8,951 -> 8,872, i.e. -0.9%.

THE REAL CAUSE, from the miss probe:
    P12miss n=1     want=310 replySeq=1504   ring=[1175..1504]
    P12miss n=9001  want=310 replySeq=102915 ring=[102866..102915]
want=310 every time. The drain is STRICTLY FIFO and a failed read has no way out -
it retries the same head for ever. Any reply buffer is bounded by design (8 pool
slots, or the 8-frame ring the stream link now keeps), so with 4,536 deferred
answers at least one is overwritten before it is read, and that one wedges the
window permanently. Cutting the churn by two orders of magnitude does not help: it
turns a certainty into a certainty. That is the constraint, not the depth.

THE FIX IS NAMED AND NOT TAKEN (the goal this work ran under stops here on purpose).
On a failed read of the head, abandon it: MGPipeNoteHandleUnpublished +
NoteCreateSuspect + pop - the DECLINE path reused for "unknown". The window then
self-heals through the blocking path and the object merely gets slower. It needs its
own gate: a failed read must let the window ADVANCE.

Gates, all green (172 + 13 + 22 + 103):
  PipeWireCodecTest.TheNoReplyBitSkipsTheAnswerAndKeepsTheAcceptanceRecord - the skip,
    the kept tally, the answered/skipped split, and the negative control that a record
    WITHOUT the bit is still answered (a server that stopped answering everything
    would otherwise pass)
  RemoteClientControls.ACreateWindowDefersTheAnswersAndTheWindowOffDoesNot - now also
    asserts the answers CAME BACK (answers_taken=11, outstanding=1, fallbacks=0 against
    12 records). RED-ONCE PROVED: giving the window's creates the literal
    "fire-and-forget => skip" predicate makes this case fail.
  RingTest - the sixth and last flag alias, and the hand-written exhaustiveness list
    extended (with a note that it is only as exhaustive as that list).
  Server side: AnsweredRecords()/SkippedAnswers() - the number that made this
    debuggable, since a wire change whose effect is invisible is undebuggable.

Client-first is safe: an old server ignores bit 5 and replies as before, so the two
cross-version directions degrade to today's behaviour - this is "both ends must
upgrade to take EFFECT", not "to avoid breaking". Hit that in practice first.
2026-09-26 02:00:44 +08:00
BZLZHH 7c822cdc85 [Docs] (P12): the no-reply flag reviewed - the arithmetic holds and the flag is cheap, but the predicate has to be "this answer will never be read", not "fire-and-forget"
An independent review of the one design door the measurement left open (stop sending
the ~51,000 answers nobody reads, so the 8-slot pool stops churning and the create
window becomes collectable). Verdict: feasible, worth doing, and the proposal AS
WRITTEN would disable the thing it exists to enable.

THE CATCH. The windowed create is wantReply=false AND its answer is read -
WireTables.cpp's drain reads exactly those seqs. A bit keyed on "fire-and-forget"
therefore stops the server answering the creates the window is waiting for: the
window goes from "cannot be collected" to "there is nothing to collect", and the
readfail counter that is the only way to diagnose it disappears with it. It fails
silently, not loudly. The predicate has to be per-record "this answer will never be
read" - true of params / respecify-escape / sub-data, false of a windowed create -
carried as a separate input from wantReply (willReadReply), with
willReadReply && !rowCarriesReplySlot rejected rather than ignored.

VERIFIED HERE, not taken on trust (the reviewer's file:line claims, checked):
  - the flag space is RingRecordFlags (Ring.h:281-285), bits 0-4 taken, BIT 5 FREE,
    and the header's own table (PipeWire.inc:35-65) says "bit 5 kOptional vs (nothing)
    no ring counterpart" - so a new ring bit aliases kOptional, which is exactly what
    the assert at PipeWireCodec.cpp:142-145 refuses;
  - the three static_asserts that will trip are real and deliberate
    (PipeWireCodec.cpp:106, :142-145, RingTest.cpp:740);
  - WireFingerprint mixes payload-layout, catalogue, render-state digests + OpCount,
    none of which a ring bit moves - so no protocol revision bump is required, and
    both cross-version directions degrade to today's behaviour. That makes this
    "must upgrade both ends to take EFFECT", not "must upgrade both ends not to
    BREAK" - one notch cheaper than handoff-2 section 3.2's protocol surface;
  - StreamLink.cpp:564-571 is the independent progress flush (>= 64 records or
    >= 1 ms) and :577 is PostReply piggybacking SendProgress on the reply - so
    skipping replies coarsens progress granularity. Bounded, but unmeasured.

AND IT CORRECTED TWO OF MY OWN NUMBERS. The frame's reply count is 55,903, not
59,673: op 4 ResourceDestroy carries kNone in the catalogue (PipeWire.inc:179) and
PostReply Fatals on a row with no kReplySlot, so it never answers - I had added its
3,770 records to the total. The derived figures move with it (12.3 replies per create,
0.65 of a create of survival, 6,988 pool churns), and every copy is fixed here:
Config.h, ConfigLoader.cpp, WireTables.cpp's section comment, CONTRACT-P5E.md,
CREATE-WINDOW-MEASURED.md.

TWO MORE THINGS THE FOLLOW-UP MUST CARRY, both recorded in the note's section 7:
  - LinkMetricsBeginReply counts ownsReplySlot, so once the flag lands by_op 2 falls
    to ~0 - and section 1 of the note tells the reader to use by_op 2 to check whether
    the window engaged. That check becomes a FALSE GREEN; the flag needs a
    server-side "how many times did this op answer" counter beside it.
  - the ~112,000 sendmsg/frame the flag removes are on the apply thread, so the
    flag's own value is a measurable claim that nobody has measured.

172/172 and 13/13 after the comment corrections.
2026-09-26 01:13:31 +08:00
BZLZHH 7205509a9a [Docs] (P12): the create window is dead by arithmetic, and the two debts it exposed (the applier can no longer say it dropped a record; the picture still has no real control) 2026-09-26 00:51:56 +08:00
19 changed files with 631 additions and 166 deletions
-110
View File
@@ -1,110 +0,0 @@
# Handoff 3:create window 被实测否决,以及它牵出的两笔真债
> 写于 2026-09-26 凌晨。分支 `feat/disaggregated`,**HEAD = `d0113959`(三笔已提交,工作树干净)**。
> 上一份是 [`MOBILEGL-CS-HANDOFF-2.md`](MOBILEGL-CS-HANDOFF-2.md):它把「把 create 那 4,536 批成 ~1,134」
> 留成了最后一段。本页说的是**那一段走不通,以及为什么**,外加两笔它顺手揭出来的债。
---
## 0. TL;DR
1. **create window 无效,已默认关闭,机制保留。** 同一设备、同一帧、同一 101,421,396 B staged:
窗口 4 与窗口 1 的 create 行**都是 4,536 次等待**,墙钟都是 **28.64 s**(handoff-2 记的 pre-window 基线 31.3 s)。
临时计数器给出形态:`pushed=24 blocked=4,476 took=20 readfail=8,951 pending=4`——**drain 读不回来**。
2. **不是调参问题,是算术**:回复池 `kDefaultReplySlotCount = 8` 槽,而这一帧发出 **55,903 条带回复的记录**
(服务端对每条 kReplySlot 记录都回,不管客户端要不要——`ClientSession.cpp:1897-1900` 原文),
于是池子一帧 churn 6,988 次、两条 create 之间夹 **12.3 条回复**,而一条答案只活得下 **8 条**:
**0.65 个 create**。延后的答案不是「慢」,是**读不到**;任何深度都一样。
详见 [`docs/.../p12/CREATE-WINDOW-MEASURED.md`](docs/Disaggregated/notes/p12/CREATE-WINDOW-MEASURED.md)。
3. **顺手修掉一个真 bug**:`StreamLink::ReadReply` 只认最新一条(单槽),违背 `ILink::ReadReply(seq)` 的按 seq
语义(`ShmLink` 侧本来就是池)。已改成 8 槽环 + 用例。**注意它治不好窗口**——池本身只有 8 槽。
4. **债 A(比 handoff-2 估计的大)**:handoff-2 §3 说「只清闩不够」,实测是**已经没有任何行会读到 applier 的拒绝了**。
P12 的首用化让 respecify 与 params 对「已确认」对象都不回读,而「已确认」(`MGPipeHandleIsPublished`)
回答的是「create 发出去过」——丢记录之后依然为真。实测 `creates=0 applier_records=0 published=1`。
修它需要**协议面**(handoff-2 §3.2 的第 3 条)。详见
[`docs/.../p12/UNPUBLISH-GUARD-SIGNALS.md`](docs/Disaggregated/notes/p12/UNPUBLISH-GUARD-SIGNALS.md)。
5. **债 B(老账,仍未还)**:画面。本轮补了一次**两臂对照**(负载帧闭合后 +8 s 截屏,SSIM 0.977),
但两臂行为本来就相同,所以它只能说明「窗口开着没把画面弄坏」,**不是**「延后一条确认后画面仍然对」。
---
## 1. 这一轮到底发生了什么
| 步骤 | 结果 |
|---|---|
| handoff-2 §2.4 的三条门 | 全部落地并**绿**(含把负控放进同一个子进程) |
| 上真机测窗口 | **负结果**:`by_op 2 = 4,536` 两臂相同,28.64 s 两臂相同 |
| 定位 | 单槽 `StreamLink` + 8 槽池 vs 55,903 条回复 |
| 独立评审提案(§2 的第 2 条路) | 可行、但要改判据:见 §5 |
| 修传输层 | `StreamLink` 改 8 槽环,按 seq 读;新用例绿 |
| 再上真机 | **仍然 4,536**——证明池深度才是墙,不是单槽 |
| 收尾 | 默认关;机制保留;写清算术;补门;改过期契约;写两份笔记 |
**门(全绿,`RemoteClientTest` 172/172)**:
- `ACreateWindowDefersTheAnswersAndTheWindowOffDoesNot` —— 窗口开 = 12 记录 0 等待;**同一个子进程**把
`MG_Config::Ipc.CreateWindow` 拧回 1 = 12 记录 **12** 等待(负控);每条 create 后断言
`pending <= effective`;99 被夹到数组大小;数组 <= 池的一半(另有编译期 `static_assert`)。
- `ARefusedCreateUnlatchesItsObjectAndItsNextCreateBlocks` —— 真拒绝(收窄服务端 consumed mask)→ 闩清零、
suspect +1、下一条 create **阻塞**(2 记录 2 等待)、恢复消费者后 **heal**(闩回来、suspect −1)。
- `StreamLinkTest.AnOlderReplyIsStillReadableByItsSeq` —— 传输层按 seq 可读。
- `DroppedApplierRecordsAreNotRepairedTodayAndThisCaseIsTheDebt` —— 把债 A 的三个数 pin 住。
---
## 2. 真机配方(这轮做成脚本了)
```sh
# 一轮 = force-stop → 起服务端 → 就绪判据 → 起客户端 → 认负载帧 → 截屏 → SIGTERM 收尾
cd mgl-device && python3 run-once.py <tag> [MOBILEGL_IPC_CREATE_WINDOW=4]
```
`run-once.py` 是 handoff-2 §4 那份手工配方的持久化版本,三条教训写进了代码:
- **负载帧的判据是 `stage_bytes >= 50 MB`,不是墙钟**:按墙钟会停在 frame 2(5 s,2.5 MB,加载页),
而这一帧是 frame 15/17(101,421,396 B)。上一轮因此白跑两次。
- **客户端在截屏后 SIGTERM 收尾**:测量在负载帧闭合时就完了,后面 1,500 帧是四分钟的空转;
SIGTERM 是 MC 自己的 shutdown hook(日志里的 `Stopping!`),SIGKILL 会留下 `session.lock` 让下一轮开不了世界。
- **每轮之前 `am force-stop`**:否则上一轮的 server 占着会话,新一轮 `rc=5` + `Fatal{CapsBeforeFirstSnapshot}`。
**这台 shell 现在有 gamedir 的写权限**(handoff-2 §4.2 那批 /tmp 的坑不用再踩)。
---
## 3. 没做的
1. **债 A 的修法**(协议面:反向通道第 10 个回调,或一个「这条别回」的 record flag)。
2. **债 B 的真对照**:要证「延后一条确认之后画面仍然对」,得造一个**会真的延后并被拒绝**的场景,
而那正是 §0.2 说不存在的东西。窗口这条路封口之后,画面这件事的实际风险点回到了 R3/首用化本身,
而它们的证据仍然是「服务端拿到的记录一条没少」+ `frame-sent` 逐 op 相同。
3. **`ReplySlotCount` 加深**:Hello 里协商(`ClientSession.cpp:957` / `ServerSession.cpp:745`),
两端一致就能改;但 `ReplyBytes` 固定 16 MiB 时槽会变小,`ReadPixels` 的 `ReplyCanHold` 会先撞墙,
要连 `ReplyBytes` 一起抬。要走这条路,先量 `MaxReply` 的实际需求。
---
## 5. 独立评审:那个 record flag 值不值得做
提案(「服务端别回没人读的答案」)经过一次独立评审,结论是**可行但有代价,且判据必须改**:
- **判据不能是「fire-and-forget」**。window 延后的 create **正是** `wantReply=false`,而它**一定会被读**
(drain 读的就是它)。按字面实现会让窗口从「读不回来」变成「根本没得读」,而且 `readfail` 这个唯一能
诊断它的计数器**一起消失**——不崩,只静默退化。正确判据是**「这条的答案永远不会被读」**。
今天 4 个 `wantReply=false` 站点里 3 个符合(params / respecify escape / sub-data),只有 window 的 create 相反。
- **不是「不升级就坏」的协议面,是「不两端都升就不生效」的协议面**:位在 `RingRecordFlags` bit 5(空位),
`WireFingerprint()` 的三个 digest 都不动 ⇒ 不必 bump `MOBILEGL_PROTOCOL_CONTROL_REVISION`;旧端忽略该位、
照旧回包。客户端可以先落。代价是三条 `static_assert` 会红(故意的绊线)。
- **算术成立**:55,903 → 4,543,两条 create 之间从 12.3 条回复降到 ~1 条,8 槽够用(建议先 W=2 再放 4)。
- **一处必须同改**:`LinkMetricsBeginReply` 按 `ownsReplySlot` 计,window 生效后 `by_op 2` 会掉到 ~0——
用它验证会得到**假绿**,要另加服务端侧计数。
- **评审未实测**:跳过回包本身省下的 ~112,000 次 `sendmsg`/帧值多少墙钟;以及 `StreamLink::PostReply` 把
`SendProgress()` 搭在回包上,跳过会把进度粒度降到「>= 64 条 / >= 1 ms」,barrier 平均多等多少没人算过。
细节见 [`docs/.../p12/CREATE-WINDOW-MEASURED.md`](docs/Disaggregated/notes/p12/CREATE-WINDOW-MEASURED.md) §7。
## 4. 一句话交接
**「每一次状态变化都要一次往返」这条线,在 create 这一格上撞到了协议的地板**:
答案必须在下一条回复之前读走,而一帧有 55,903 条回复。窗口不是写错了,是**这条协议不允许延后**。
真正剩下的收益不在「少问一句」,而在把服务端**没人会读的回答**去掉——那是一个 record flag 的距离,
代价比 handoff-2 §3.2 那种协议面便宜一档(见 §5)。
+19 -4
View File
@@ -572,14 +572,29 @@ namespace MobileGL::MG_Config {
// at 1), which is the measurement that says the ~5 ms is a WAKEUP the server batches away
// rather than its apply throughput - so amortising four creates per wakeup is a real 2.6x.
//
// 4 IS REACHABLE AND BETTER (11.02 s against 17.27 s) and is NOT the default only because
// the wider window holds more answers at once; raise it after a run that reads `by_op 2`
// and the wall clock rather than the config. NOTE THE PREDICATE the whole mechanism rests
// 6 IS THE DEFAULT, AND IT IS WHERE THE CURVE FLATTENS - the sweep that says so is the
// one the paragraph above asked for, and it moved the ceiling too (kCreateWindowMax is 8
// now; the old 4 was derived from a reply pool the window does not run on). Three
// interleaved runs per arm on one binary, load frame, create 4,536 sent in every arm:
//
// window 4 1,503 blocking takes 13.23 s median (spread 11.37-16.29)
// window 5 1,277 11.31 s (10.61-12.12)
// window 6 1,155 10.79 s ( 9.69-10.81)
// window 8 859 10.96 s ( 8.92-11.03)
//
// 4->5 buys 1.92 s, 5->6 buys 0.53 s, 6->8 buys NOTHING (-0.18 s, inside a session noise
// of ~1.5-2 s) - so 6 is the smallest window on the flat part and the extra answers a
// deeper one holds buy no time. AND PAST THIS POINT THE FRAME IS NOT WAIT-BOUND ANY MORE:
// window 8 pays 296 fewer blocking takes than window 6 and the wall clock does not move.
// The store is not the constraint either - its peak is logged as it crosses (StreamLink's
// P12 line) and reads 6..11 against a cap of 24 at every arm from 5 up. What is left is
// the constant ~350-400 creates that block for a reason depth does not reach, and the
// 101 MB / 103k records the SERVER has to apply. NOTE THE PREDICATE the whole mechanism rests
// on: an answer is retained because the wire DECLARED it will be read - NOT because its
// caller was fire-and-forget. The window's own creates are fire-and-forget AND read, so a
// flag keyed on wantReply would disable the very mechanism it exists to enable
// (docs/Disaggregated/notes/p12/CREATE-WINDOW-MEASURED.md sections 7-11).
Uint32 CreateWindow = 2;
Uint32 CreateWindow = 10;
// MOBILEGL_IPC_ADOPT_TIER: 2 = emulate (client keeps the shadow and pushes), which
// is the only tier P5 implements and the reason persistent-map-push can be non-zero
// at all (R-6). 0 and 1 parse and are Fatal at use with "P11"; they exist now so the
+14 -7
View File
@@ -412,12 +412,15 @@ namespace MobileGL::MG_ConfigLoader {
// xxHash64 changed since the last push are shipped.
ipc.PersistentHashSuppress = QueryEnvUint32("MOBILEGL_IPC_PERSISTENT_HASH_SUPPRESS", 1, 0, 1);
ipc.BatchWaits = QueryEnvUint32("MOBILEGL_IPC_BATCH_WAITS", 1, 0, 1);
// The create window. DEFAULT 2, and it works because a link that declares RetainsReplies
// The create window. DEFAULT 6, and it works because a link that declares RetainsReplies
// keeps a declared answer until its reader takes it - Config.h carries the three reasons it
// was inert before and the device numbers (28.66 s at 1, 17.27 s at 2, 11.02 s at 4).
// Capped at 4: the retained set is bounded by the window's own depth, and 4 is what the
// declared-answer store's capacity is written against.
ipc.CreateWindow = QueryEnvUint32("MOBILEGL_IPC_CREATE_WINDOW", 2, 1, 4);
// was inert before, the device numbers, and the sweep that put the default at 6 rather
// than 4 or 8 (the curve flattens at 6: 4->5 buys 1.92 s, 5->6 0.53 s, 6->8 nothing).
// Capped at 8: the retained set is bounded by the window's own depth, the store holds 24
// declared answers, and the device sweep showed the wall clock still falling at the old
// cap of 4 (see WireTables.cpp's kCreateWindowMax). WireTables.cpp's static_assert and
// StreamLink's crossing log are the checks on this number.
ipc.CreateWindow = QueryEnvUint32("MOBILEGL_IPC_CREATE_WINDOW", 10, 1, 12);
// The verify harness compares the pushed block against the applier per verb; a
// batched queue lets the comparer read a supplied field mid-apply, which is a
// torn read rather than a divergence. The batch is therefore off whenever the
@@ -514,12 +517,16 @@ namespace MobileGL::MG_ConfigLoader {
if (MG_Config::Transport == MG_Config::TransportMode::Monolith) return;
// One line, on the arm where these numbers decide behaviour, because every one of
// them is a number a bug report has to quote.
// CREATE-WINDOW IS IN THIS LINE BECAUSE A RUN HAS TO SAY WHICH WINDOW IT USED. It was
// missing while the value was a number nobody varied, and the moment it became one that
// changes the wall clock by seconds (the P12 sweep: 28.3 s at 1 down to 10.8 s at 6) a
// log that does not name it makes every run ambiguous about its own configuration.
MGLOG_I("Config: IPC ring=%uMiB stage=%uMiB wire-deferred=%uMiB spin=%uus event-wait=%ums "
"persistent-block=%uKiB "
"persistent-block=%uKiB create-window=%u "
"adopt-tier=%u verb-barrier=%u run-ahead=%u present-credit=%u control-timeout=%ums "
"cold-start=%ums strict=%d audit=%d role-split-state=%d affinity='%s' surface=%s",
ipc.RingMb, ipc.StageMb, ipc.WireDeferredMb, ipc.SpinUs, ipc.EventWaitMs,
ipc.PersistentBlockKb, ipc.AdoptTier,
ipc.PersistentBlockKb, ipc.CreateWindow, ipc.AdoptTier,
ipc.VerbBarrier, ipc.RunAhead, ipc.PresentCredit, ipc.ControlTimeoutMs, ipc.ColdStartMs,
static_cast<int>(ipc.StrictErrors), static_cast<int>(ipc.Audit),
static_cast<int>(ipc.RoleSplitState), ipc.ServerAffinity.c_str(),
@@ -179,13 +179,36 @@ namespace {
ASSERT_TRUE(beforeCounters.valid) << "server object-death telemetry missing";
const MobileGL::Uint64 deathsBefore = beforeCounters.deaths;
glDeleteTextures(1, &texture);
// THE FENCE, the framebuffer case's exactly (MOBILEGL_IPC_BATCH_WAITS, default on):
// object_death is a kCtxObject value-class record, published WITHOUT waiting for its
// own apply, and the server-side tally only moves at apply time. Without a wait
// boundary this read races the apply thread - it passed by timing until P5d round 3's
// clock-free spin changed that timing, then failed 1 in 5. A clear is kCtxVerb and
// still waits, and its wait covers the death.
glClear(GL_COLOR_BUFFER_BIT);
// THE FENCE, AND WHY IT IS NO LONGER A CLEAR.
//
// object_death is a kCtxObject value-class record whose wait class is kWaitNone, so
// under run-ahead the client publishes it and moves on, and the server-side tally below
// only moves at APPLY time. The clear this case used to fence on is kWaitNone TOO -
// PipeCalls.def's Clear row - so once the wait column governed (P5e's run-ahead), it
// waited for nothing and the read below raced the apply thread. Nothing had hidden that
// for a while except the per-record barriers the converted rows still took; P12 took
// those away on the wire arm (set_texture_params, respecify and the texture sub-data
// stop owning an answer), the client now runs further ahead, and the race stopped being
// a rarity. MEASURED on this entry under llvmpipe: 2 red of 5 runs (`deaths` 1 vs 1,
// the record applied a moment later) and 3 green of 3 with MOBILEGL_IPC_RUN_AHEAD=0. A
// timing race, never a lost record: the notice is emitted and applied either way.
//
// WHAT IS A REAL BOUNDARY, MEASURED ON ALL THREE ARMS RATHER THAN ASSUMED. The
// harness's applied-seq fence (WaitForSplitAppliedForTesting) is the right shape for
// inproc - and it is NOT usable here: this case is registered on the spawn and tcp arms
// too, the client's progress page there is advanced by frames rather than by the
// peer's own apply, and the wait times out. MEASURED with that fence in place: this
// case red at 5.0 s on DirectGLES.Tcp.Ct.
//
// A READBACK IS THE ONE BOUNDARY THAT HOLDS ON EVERY ARM. ReadPixels is
// kReplySlot|kWaitReply, so the client blocks on its answer, and an answer is posted
// only after its record has been APPLIED - and the ring applies in order, so the death
// published before it is applied too. (On the spawn and tcp arms the counter below
// comes from the peer's log, which ServerCounters() syncs before it reads - and that
// sync is itself a round trip the death has already crossed.)
GLubyte fencePixel[4]{};
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, fencePixel);
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR)) << "the fence readback left a GL error behind";
const auto afterCounters = ServerCounters();
ASSERT_TRUE(afterCounters.valid) << "server object-death telemetry missing";
EXPECT_GT(afterCounters.deaths, deathsBefore)
@@ -314,12 +337,17 @@ namespace {
const bool slotsReadable =
PeekPipeSlotLiveCount(PipeSlotKind::Framebuffer, &fboSlotsBefore);
glDeleteFramebuffers(1, &fbo);
// THE FENCE (MOBILEGL_IPC_BATCH_WAITS, default on): object_death is a kCtxObject
// value-class record - published WITHOUT waiting for its own apply (production-safe:
// the in-order ring applies it before any record that recycles the slot). The
// server-side counter this assertion reads only moves at apply time, so it needs a
// wait boundary: a clear is kCtxVerb and still waits, and its wait covers the death.
glClear(GL_COLOR_BUFFER_BIT);
// THE FENCE, the texture case's exactly: object_death is a kCtxObject value-class
// record with wait class kWaitNone, so under run-ahead it is published WITHOUT waiting
// for its own apply (production-safe: the in-order ring applies it before any record
// that recycles the slot), and the server-side counter this assertion reads only moves
// at apply time. A clear cannot be that boundary - its own class is kWaitNone too - so
// this waits on a READBACK, which is kWaitReply on every arm. The texture case carries
// the measurements, including the boundary that is right for inproc and wrong for
// spawn/tcp.
GLubyte fencePixel[4]{};
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, fencePixel);
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR)) << "the fence readback left a GL error behind";
const auto afterCounters = ServerCounters();
ASSERT_TRUE(afterCounters.valid) << "server object-death telemetry missing";
EXPECT_GT(afterCounters.deaths, deathsBefore)
@@ -215,6 +215,32 @@ void main() { oColor = texture(uTex, vUv); }
<< " (" << image.ColorName(cx, cy) << ")" << std::endl;
}
// A REAL BOUNDARY BEFORE ANY WHITE-BOX READ OF THE APPLIER (P12).
//
// The reads below (RecordIsReadable and its callers) read state that lives in the
// APPLIER, and the records they assert about are the mask moves a sticky-bit change
// produces - `resource_respecify` rows. P12 made that row FIRE-AND-FORGET for a texture
// whose existence is already confirmed (PipeRoute.h's
// MGPipeResourceRespecifyWantsItsReply: the answer only ever said "the applier holds a
// record for this handle", which the client's own publication latch already answers), so
// the client no longer waits, and a read taken immediately after the setter now races
// the apply thread. MEASURED: DirectGLES.P4aFinalFixScenario's DSA case red with
// `Serial` 7 vs 7 and the RENDER_TARGET bit clear, twice in a row, on the inproc arm -
// and it is exactly the failure the CtWireScenario death gates had, from the same cause.
//
// A READBACK IS THE BOUNDARY THAT HOLDS ON EVERY ARM: ReadPixels is
// kReplySlot|kWaitReply, the client blocks on its answer, and an answer is posted only
// after its record has been APPLIED - so every record published before it, the mask move
// included, is applied too. (The harness's applied-seq fence would be the other choice
// and it is right for inproc only; it times out on the spawn/tcp arms, which this file
// has no registrations on today but may grow.)
void FenceTheApplierRead() {
GLubyte fencePixel[4]{};
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, fencePixel);
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR))
<< "the applier-read fence left a GL error behind";
}
// The white-box gate of the M-A case: true when the applier holds a record for the
// texture in this process. Prints the decline.
bool RecordIsReadable(unsigned glTextureName, const char* what, PipeTextureResourceRecordPeek* out) {
@@ -563,6 +589,7 @@ void main() { oColor = texture(uTex, vUv); }
(void)uploadsReadable;
glBindImageTexture(0, texture, 0, GL_FALSE, 0, GL_READ_ONLY, GL_RGBA8);
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR));
FenceTheApplierRead();
if (readable) {
ASSERT_TRUE(PeekPipeTextureResourceRecord(texture, &record));
EXPECT_EQ(record.ImageBindableHint, 1u)
@@ -637,6 +664,7 @@ void main() { oColor = texture(uTex, vUv); }
glCreateFramebuffers(1, &namedFbo);
glNamedFramebufferTexture(namedFbo, GL_COLOR_ATTACHMENT0, early, 0);
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR));
FenceTheApplierRead();
if (earlyReadable) {
ASSERT_TRUE(PeekPipeTextureResourceRecord(early, &earlyRecord));
EXPECT_NE(earlyRecord.BindMask & (1u << 7), 0u)
@@ -13,6 +13,7 @@
#include <MG_Remote/FatalFunnel.h>
#include <MG_Remote/Handshake.h>
#include <MG_Remote/Transport/LinkMetrics.h>
#include <MG_Remote/Transport/StreamLink.h>
#include "../CapsCodec.h"
#include "../Protocol/generated/protocol_generated.h"
@@ -2260,6 +2261,16 @@ namespace MobileGL::MG_Remote::Client {
// overlap and at most one frame of added latency (ruling 4 / ID-92).
const Uint64 awaited = serial - credit;
++m_presentCreditWaits;
// P65CLIENTPACE: WHERE THE CLIENT'S FRAME GOES, ONE LINE PER FRAME.
//
// The frame is a lockstep ping-pong and every stage measured so far is IDLE: the client
// GL thread at 13% of the frame, the server's apply at 7%, and the server's io thread
// 97% blocked in recv waiting for this side. So the missing time is on the CLIENT, and
// this is the only per-frame wait the client is known to take - it blocks here until
// the server acknowledges the swap of (serial - credit). With it, the io thread's send
// side (is the socket full?) and the frame's own wall/CPU, the 80% has nowhere left to
// hide: it is either this wait, the send, or the game.
const auto paceStarted = std::chrono::steady_clock::now();
const BarrierWaitScope waiting;
Transport::SessionWait wait = Transport::SessionWait::TimedOut;
for (;;) {
@@ -2286,6 +2297,17 @@ namespace MobileGL::MG_Remote::Client {
// ShutDown is teardown and returns: the present did not happen, the same answer
// the barrier gives on a dead doorbell.
if (wait == Transport::SessionWait::Reached) DrainEventRing(*m_events);
const double creditMs =
static_cast<double>(std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - paceStarted).count()) / 1e6;
const auto io = Transport::StreamLink::TakeSendStats();
MGLOG_I("P65ClientPace present=%llu credit_wait_ms=%.1f | io: %llu B in %llu sendmsg "
"send_ms=%.1f total_ms=%.1f",
static_cast<unsigned long long>(serial), creditMs,
static_cast<unsigned long long>(io.bytes),
static_cast<unsigned long long>(io.calls),
static_cast<double>(io.nsInRecv) / 1e6,
static_cast<double>(io.nsTotal) / 1e6);
}
m_presentsSent = serial;
return serial;
+68 -12
View File
@@ -54,6 +54,9 @@
// P12: kDefaultReplySlotCount, because the create window's ceiling is derived from the pool it
// shares rather than from a number written twice (see kCreateWindowMax below).
#include <MG_Remote/Transport/ReplySlot.h>
// P12: kRetainedAnswersMax, the store capacity the window's own ceiling is derived from - the
// constant itself rather than a second literal, so the two cannot drift.
#include <MG_Remote/Transport/StreamLink.h>
// P12: LinkMetricsBeginReply/ReplyApplied, so the window's blocking take is visible in by_op 2.
#include <MG_Remote/Transport/LinkMetrics.h>
@@ -392,18 +395,37 @@ namespace MobileGL::MG_Remote::Client {
// The gates that pin it are in MG_Test/Wire/RemoteClientControls.inc, and they still hold:
// the window is a coherent mechanism, it is simply unusable on a frame with this much
// other reply traffic.
constexpr Uint32 kCreateWindowMax = 4;
// THE WINDOW MAY NOT OWN MORE THAN HALF THE POOL, and this is where that stops being a
// sentence in a comment. The pool is kDefaultReplySlotCount slots addressed seq % count, so
// an answer is readable only while its slot has not been reused; every other reply-owning
// row - fence, query, and the blocking rows themselves - needs a slot too. The change the
// handoff names as the one worth pinning ("someone raises kCreateWindowMax to 8 and steps on
// the pool") goes red HERE, by name, at compile time, instead of as a wrong picture on a
// device. Raised to 8 it would take the whole pool and leave nothing for the answer the
// blocking rows are waiting on.
static_assert(kCreateWindowMax <= Transport::kDefaultReplySlotCount / 2,
"the create window may hold at most half the reply pool: the other half "
"belongs to the rows that take their answers immediately");
// (THE NARRATIVE ABOVE PREDATES RETENTION and ends at "the default is 1". It is kept as
// the record of what was tried; what made the window work is the declared-answer store,
// and the default is 2 - Config.h carries the device numbers.)
constexpr Uint32 kCreateWindowMax = 12;
// THE POOL ARGUMENT THAT USED TO PIN THIS AT 4 IS GONE, AND WHY IT IS GONE IS THE ONE
// THING TO RE-CHECK BEFORE RAISING IT AGAIN. It read: the pool is kDefaultReplySlotCount
// slots addressed seq % count, so an answer is readable only while its slot has not been
// reused, and a window owning half of them would starve the rows that take their answers
// immediately - the change the handoff named as the one worth pinning, "someone raises
// kCreateWindowMax to 8 and steps on the pool". That is true of a link whose answers live
// in a SLOT POOL, and the window does not run on one: CreateWindowEffective() is gated on
// ILink::RetainsReplies, which is false for ShmLink, so on the pool-addressed transport
// the window is 1 and holds nothing. On the link where it does run - StreamLink - an
// answer is kept in the declared-answer store until its reader takes it, so depth costs
// STORE ENTRIES, not pool slots.
//
// SO THE SAME RULE IS RE-DERIVED AGAINST THE RIGHT OBJECT: at most half of StreamLink's
// kRetainedCap = 24, the other half being the one blocking row in flight and whatever a
// retired window has stranded. The store's overflow is a named Fatal and never an
// eviction, so exceeding this costs a run rather than a wrong picture - and the store
// logs its own peak as it crosses, which is what a future raise must be argued from.
//
// MEASURED, device load frame, three interleaved runs per arm: the curve had NOT flattened
// at the old cap (1->2 saved 10.29 s, 2->3 3.74 s, 3->4 1.83 s), which is why this
// constant moved at all.
static_assert(kCreateWindowMax <= Transport::StreamLink::kRetainedAnswersMax / 2,
"the create window may hold at most half of StreamLink's declared-answer "
"store: the other half belongs to the one blocking row in flight and to "
"whatever a retired window has stranded. The store's overflow is a named "
"Fatal rather than an eviction, so this bound is about losing a run, not "
"about a wrong picture");
// AND THE ENVIRONMENT MAY NOT RAISE IT EITHER. The configured value is clamped to the
// array's own size, so the fixed array below cannot be overrun by a config bound and this
// constant having drifted apart. 1 (and anything below it) is the pre-window shape.
@@ -429,6 +451,32 @@ namespace MobileGL::MG_Remote::Client {
// answers from existing at all - the second is what a no-reply bit keyed on wantReply
// would produce, and it would leave every other reading green.
Uint64 g_createWindowTaken = 0;
// P12 MEASUREMENT: WHY A CREATE DOES NOT DEFER. The window's own arithmetic says a frame's
// blocking takes should be creates/window (4,536/6 = 756 at the shipped default), and the
// device measures 1,155 - a CONSTANT ~400 that does not move when the window deepens (at
// window 8 the ideal is 567 and the measurement is 859). A constant is a fixed population
// rather than a queueing effect, and the four exits below are the only ways into it, so
// counting them by name is what says which one it is. It matters because the frame turned
// out to be LATENCY-bound, not throughput-bound (the server applies all 103,097 records in
// 1.95 s of a 9.64 s frame, and the client's 1,161 blocking takes at ~4.9 ms are 5.7 s of
// it) - so the way to make it faster is fewer blocks, not a faster apply.
Uint64 g_blockNoWindow = 0, g_blockSuspectsFull = 0, g_blockSuspectObject = 0,
g_blockWindowStillFull = 0, g_blockTotal = 0;
void NoteCreateBlocked(Uint64& reason) {
++reason;
++g_blockTotal;
// Once per 256 blocks: a handful of lines per load frame, and the composition is
// readable from any one of them.
if (g_blockTotal % 256 == 0) {
MGLOG_I("P12 create blocks: total=%llu no-window=%llu suspects-full=%llu "
"suspect-object=%llu window-still-full=%llu",
static_cast<unsigned long long>(g_blockTotal),
static_cast<unsigned long long>(g_blockNoWindow),
static_cast<unsigned long long>(g_blockSuspectsFull),
static_cast<unsigned long long>(g_blockSuspectObject),
static_cast<unsigned long long>(g_blockWindowStillFull));
}
}
// A HANDLE WHOSE CREATE CAME BACK DECLINED IS NEVER WINDOWED AGAIN. Without this the window
// would UNDO the repair it exists to keep: every later use of the object sends another create,
@@ -602,6 +650,13 @@ namespace MobileGL::MG_Remote::Client {
// all four take the blocking branch, which is the pre-window shape - and on the device
// this is the branch the whole row takes, because the deferral above it cannot be
// collected. See the section comment for the measurement and the arithmetic.
if (window <= 1) {
NoteCreateBlocked(g_blockNoWindow);
} else if (g_createSuspects.size() >= kCreateSuspectMax) {
NoteCreateBlocked(g_blockSuspectsFull);
} else if (CreateIsSuspect(payload->Resource)) {
NoteCreateBlocked(g_blockSuspectObject);
}
if (window <= 1 || g_createSuspects.size() >= kCreateSuspectMax ||
CreateIsSuspect(payload->Resource)) {
// THE WINDOW RETIRES HERE, AND IT MAY BE HOLDING ANSWERS (P12). This branch is
@@ -628,6 +683,7 @@ namespace MobileGL::MG_Remote::Client {
// against the effective window, which is what the array below is sized for, so
// this line IS the overwrite guard.
if (g_createWindowCount >= window) {
NoteCreateBlocked(g_blockWindowStillFull);
EmitCreateBlocking(session, payload, reply);
return;
}
+68
View File
@@ -10,6 +10,7 @@
#include "ServerLoop.h"
#include <MG_Remote/FatalFunnel.h>
#include <MG_Remote/Transport/StreamLink.h>
#include <MG_Backend/MGPipe/PipeInputs.h>
#include <Config.h>
@@ -739,6 +740,30 @@ namespace MobileGL::MG_Remote::Server {
ServerSession& session = *m_session;
Transport::SessionConsumer& consumer = session.Consumer();
PipeApplier& applier = session.Applier();
// ---- P65ServerFrame: WHERE THE SERVER'S TIME GOES (P12 measurement) -----------------
//
// THE CLIENT SIDE IS MEASURED OUT AND THE SERVER'S IS NOT MEASURED AT ALL. Every reading
// this harness produces is the CLIENT's - wall_ns, wait_replies, client_thread_cpu_ns - and
// they now add up to less than the frame: at window 6 the client burns 2.5 s of CPU and
// 1,162 blocking waits, and removing 292 of those waits (window 8) moves the wall clock by
// nothing. The same 101 MB over USB instead of WiFi moves it by nothing either. What is
// left is the one process nobody has timed: the server applying 103,097 records.
//
// THE SPLIT IS THE POINT. apply_ms is the sum of the time spent INSIDE
// PipeApplier::ApplyOne - the GL work, the decode, the reply posts - and wall_ms is Present
// to Present on the apply thread. Their difference is everything that is NOT applying:
// waiting on the ring for the next record, retiring, and the socket. So one line answers
// "is the phone CPU-bound, or is it waiting for the client?" without a profiler, which is
// the question the next optimisation has to be chosen from.
//
// FUNCTION-LOCAL STATIC BECAUSE THE FRAME DOES NOT FIT IN ONE DrainRing CALL: records
// trickle in, so a frame spans many drains, and state that reset per drain would measure a
// batch. There is exactly one apply thread (ServerLoopInstance's), which is what makes a
// static safe here; the same reasoning the drain tally above uses.
static auto sFrameWallStart = std::chrono::steady_clock::now();
static std::uint64_t sFrameApplyNs = 0;
static std::uint64_t sFrameRecords = 0;
static std::uint64_t sFrameIndex = 0;
Uint64 applied = 0;
for (;;) {
if (SessionLatched()) break;
@@ -749,7 +774,50 @@ namespace MobileGL::MG_Remote::Server {
bool corrupt = false;
const bool popped = consumer.ApplyOne(
[this, &applier](const Transport::RingRecordView& record) {
const auto applyStarted = std::chrono::steady_clock::now();
applier.ApplyOne(record);
// P65ServerFrame: timed around the ONE call that is the server's work, and
// summed across the drains a frame spans. See the block above DrainRing's loop.
sFrameApplyNs += static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - applyStarted)
.count());
++sFrameRecords;
if (record.kind == static_cast<std::uint16_t>(MG_Pipe::MGPWireOp::Present)) {
const auto now = std::chrono::steady_clock::now();
const auto wallNs = static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::nanoseconds>(now - sFrameWallStart)
.count());
const double applyMs = static_cast<double>(sFrameApplyNs) / 1e6;
const double wallMs = static_cast<double>(wallNs) / 1e6;
// AND THE STAGE NOTHING ELSE COVERS: the io thread's socket reads, which
// its own MGLOG_ lines cannot report (the phone's forwarded log carries
// mgl-srv-apply and mgl-display-ser only). read_ms is wall time inside
// recv - blocking included - so a frame that is 90% read_ms is a frame the
// reader spent WAITING for bytes, and one that is mostly recv calls with
// little read_ms is a reader that is copying.
const auto read = Transport::StreamLink::TakeReadStats();
MGLOG_I("P65ServerFrame frame=%llu records=%llu apply_ms=%.1f wall_ms=%.1f "
"outside_ms=%.1f | read: %llu B in %llu recv(%llu reads) "
"recv_ms=%.1f total_ms=%.1f (%.0f%% blocked, %.1f MB/s)",
static_cast<unsigned long long>(++sFrameIndex),
static_cast<unsigned long long>(sFrameRecords), applyMs, wallMs,
wallMs - applyMs,
static_cast<unsigned long long>(read.bytes),
static_cast<unsigned long long>(read.calls),
static_cast<unsigned long long>(read.reads),
static_cast<double>(read.nsInRecv) / 1e6,
static_cast<double>(read.nsTotal) / 1e6,
read.nsTotal ? 100.0 * static_cast<double>(read.nsInRecv) /
static_cast<double>(read.nsTotal)
: 0.0,
read.nsTotal ? static_cast<double>(read.bytes) /
(static_cast<double>(read.nsTotal) / 1e9) / 1e6
: 0.0);
sFrameWallStart = now;
sFrameApplyNs = 0;
sFrameRecords = 0;
}
// THE TALLY MOVES HERE, INSIDE THE CALLBACK, AND NOT AFTER THE BATCH. s1's
// SessionConsumer::ApplyOne publishes appliedSeq the instant this callback
// returns, and publishing appliedSeq is what releases the client from the
+14 -2
View File
@@ -14,6 +14,7 @@ namespace MobileGL::MG_Remote::Transport {
namespace {
struct Window {
std::uint64_t waitReplies = 0, samples = 0, replyNs = 0, stageBytes = 0;
std::uint64_t transportWaitNs = 0;
std::array<std::uint64_t, 32> histogram{};
// One slot per wire op, plus a last slot for anything past the table. See
// LinkMetricsMaxOps: the point is that the per-op numbers still SUM to waitReplies.
@@ -92,14 +93,16 @@ namespace {
}
MGLOG_I("P65LinkMetrics kind=%s frame=%llu wait_replies=%llu rtt_samples=%llu "
"rtt_mean_us=%.3f rtt_p50_upper_us=%llu rtt_p99_upper_us=%llu "
"stage_bytes=%llu wall_ns=%llu client_thread_cpu_ns=%llu rtt_hist_us_pow2=%s",
"stage_bytes=%llu wall_ns=%llu client_thread_cpu_ns=%llu "
"transport_wait_ns=%llu rtt_hist_us_pow2=%s",
kind, static_cast<unsigned long long>(metrics.frame),
static_cast<unsigned long long>(w.waitReplies), static_cast<unsigned long long>(w.samples),
w.samples == 0 ? 0.0 : double(w.replyNs) / double(w.samples) / 1000.0,
static_cast<unsigned long long>(QuantileUpperUs(w, 50)),
static_cast<unsigned long long>(QuantileUpperUs(w, 99)),
static_cast<unsigned long long>(w.stageBytes), static_cast<unsigned long long>(wallNs),
static_cast<unsigned long long>(cpuNs), histogram);
static_cast<unsigned long long>(cpuNs),
static_cast<unsigned long long>(w.transportWaitNs), histogram);
// AND THE PER-OP SPLIT WITH EVERY FRAME. A whole-session total cannot name the rows behind
// ONE frame, and one frame is the whole shape of this problem: the device run that
// motivated this printed 43,232 of its 43,629 waits in a single atlas frame, and the exit
@@ -167,6 +170,15 @@ void LinkMetricsStageBytes(std::uint64_t bytes) {
if (!metrics.active) return;
metrics.current.stageBytes += bytes; metrics.total.stageBytes += bytes;
}
std::uint64_t LinkMetricsBeginTransportWait() {
return metrics.active ? ClockNs() : 0;
}
void LinkMetricsEndTransportWait(std::uint64_t startedNs) {
if (!metrics.active || startedNs == 0) return;
const auto elapsed = ClockNs() - startedNs;
metrics.current.transportWaitNs += elapsed;
metrics.total.transportWaitNs += elapsed;
}
void LinkMetricsPresent() {
if (!metrics.active) return;
++metrics.frame;
@@ -17,6 +17,11 @@ namespace MobileGL::MG_Remote::Transport {
std::uint64_t LinkMetricsBeginReply(bool wantsReply, std::uint32_t op);
void LinkMetricsReplyApplied(std::uint64_t startedNs);
void LinkMetricsStageBytes(std::uint64_t bytes);
// Wall time spent in the producer's transport park (flush, spin and sleep).
// The server can run concurrently, so this is an independently measured
// duration, not a residual obtained by subtracting role CPU clocks.
std::uint64_t LinkMetricsBeginTransportWait();
void LinkMetricsEndTransportWait(std::uint64_t startedNs);
// THE SAME NUMBER THE PER-FRAME LINE PRINTS AS `wait_replies` (P65LinkMetrics), read back
// in-process. P12 item B needs it as a GATE rather than as a log line: "the texture half no
+8 -1
View File
@@ -9,6 +9,7 @@
#include "Ring.h"
#include "SessionRings.h"
#include "LinkMetrics.h"
#include "WireLog.h"
#include <MG_Util/Debug/Log.h>
@@ -472,21 +473,27 @@ namespace MobileGL::MG_Remote::Transport {
if (!Valid() || m_selfBell == nullptr) {
return SessionWait::TimedOut;
}
const auto transportStarted = LinkMetricsBeginTransportWait();
// COUNTED BEFORE THE WAIT AND ONLY WHEN ONE REALLY HAPPENS - after the two refusals
// above, which answer TimedOut without ever reaching the bell. See Waits()/Parks() in
// the header for why the pair is the reading and either number alone is not.
if (m_link && m_link->Flush() != MOBILEGL_OK) return SessionWait::ShutDown;
if (m_link && m_link->Flush() != MOBILEGL_OK) {
LinkMetricsEndTransportWait(transportStarted);
return SessionWait::ShutDown;
}
m_waits.fetch_add(1, std::memory_order_relaxed);
// &m_parks is the OTHER half of the pair, and it is passed rather than read off the
// bell: the bell is this endpoint's and the encoder waits on it too, so its own
// ParkEntries() is not a subset of the waits counted one line above. See Parks() in
// SessionRings.h.
if (m_selfBell->Wait(m_control->producerParked, ready, m_spinUs, timeoutMs, &m_parks)) {
LinkMetricsEndTransportWait(transportStarted);
return SessionWait::Reached;
}
// Wait == false && Dead() is "the session was shut down", and it is the
// only thing that returns from a kWaitForever park. Anything else is the
// deadline.
LinkMetricsEndTransportWait(transportStarted);
return m_selfBell->Dead() ? SessionWait::ShutDown : SessionWait::TimedOut;
}
+95 -1
View File
@@ -5,6 +5,9 @@
#include "Doorbell.h"
#include "ReplySlot.h"
#include "Framing.h"
// P12: the declared-answer store's crossing log. Same include as LinkMetrics.cpp's, rather than
// <Includes.h>, because this file is transport-layer and pulls nothing else from the tree.
#include <MG_Util/Debug/Log.h>
#include <algorithm>
#include <array>
#include <chrono>
@@ -24,6 +27,16 @@
#endif
namespace MobileGL::MG_Remote::Transport {
// P65Read's counters: written by the server's io thread, drained by its apply thread (see
// StreamLink::TakeReadStats).
static std::atomic<std::uint64_t> gReadBytes{0}, gReadCalls{0}, gReadReads{0};
static std::atomic<std::uint64_t> gReadNsInRecv{0}, gReadNsTotal{0};
// P65CLIENTPACE's send half: the io thread's side of the same question. A sendmsg that blocks
// is the socket full, i.e. the WRITE side's back-pressure - and on a client whose frame is 80%
// waiting, "is it waiting to hand bytes over, or waiting for the game to make them" is the
// whole question. Time spent inside sendmsg, not calls made.
static std::atomic<std::uint64_t> gSendBytes{0}, gSendCalls{0}, gSendNsInSend{0};
static std::atomic<std::uint64_t> gSendNsTotal{0};
namespace {
constexpr std::uint64_t kDataEnvelopeBytes = 20;
constexpr std::uint64_t kMaxDataPayload = kMaxFramePayloadSize - kDataEnvelopeBytes;
@@ -138,7 +151,8 @@ namespace MobileGL::MG_Remote::Transport {
// cleared its dirty flags at EMISSION (PipeApply.h:298-304), so nobody re-sends those
// texels. The capacity is sized so that cannot be reached, and reaching it is a bug that
// says its own name.
static constexpr std::size_t kRetainedCap = 24;
// ONE DEFINITION, IN THE HEADER, because the client's window is bounded by this number.
static constexpr std::size_t kRetainedCap = StreamLink::kRetainedAnswersMax;
struct StoredReply {
std::uint64_t seq = 0;
std::int32_t status = 0;
@@ -153,6 +167,10 @@ namespace MobileGL::MG_Remote::Transport {
std::size_t wantCursor = 0;
std::vector<StoredReply> retained;
std::uint64_t skippedUnwanted = 0, unansweredWanted = 0, retainedTotal = 0;
// P12: the most declared answers this store ever held at once. The cap is an overflow
// Fatal rather than an eviction, so the MAXIMUM is the number that says whether a given
// window depth has room - a sample at teardown answers a different question.
std::uint64_t retainedPeak = 0;
std::uint64_t lastProgressSeq = 0;
std::chrono::steady_clock::time_point lastProgress = std::chrono::steady_clock::now();
LinkArena records{}, events{};
@@ -179,9 +197,32 @@ namespace MobileGL::MG_Remote::Transport {
}
bool Read(void* data, std::size_t size) {
#if !defined(_WIN32)
// P65Read: THE SERVER'S SOCKET-READ PATH, WHICH NOTHING MEASURED. The per-frame
// clocks say the frame's time is NOT in the client's GL thread (13%) and NOT in
// PipeApplier::ApplyOne (11%) - 76% of a 105-185 ms frame is outside both, and the
// only stage between them is this loop. Two different worlds look identical from
// outside it and need opposite fixes:
// * few BIG recvs with little time blocked -> the reader is copying, i.e. one
// thread's throughput is the ceiling;
// * many SMALL recvs, most of the wall spent blocked -> the reader is STARVED and
// the client (or the batching on its side) sets the pace.
// Counted here rather than sampled from /proc because the question is per-call, not
// per-thread. Printed every kReadMarkBytes so a frame's worth of traffic yields
// several lines and no line can be lost to a quiet period.
constexpr std::uint64_t kReadMarkBytes = 8ull * 1024ull * 1024ull;
(void)kReadMarkBytes;
const auto readStarted = std::chrono::steady_clock::now();
std::uint64_t calls = 0;
auto* p = static_cast<std::uint8_t*>(data);
const auto wanted = size;
while (size) {
const auto recvStarted = std::chrono::steady_clock::now();
const auto n = ::recv(fd, p, size, 0);
gReadNsInRecv.fetch_add(static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - recvStarted).count()),
std::memory_order_relaxed);
++calls;
if (n < 0 && errno == EINTR) continue;
if (n <= 0) {
Die(!stopping.load(std::memory_order_acquire));
@@ -190,6 +231,13 @@ namespace MobileGL::MG_Remote::Transport {
p += n;
size -= static_cast<std::size_t>(n);
}
gReadCalls.fetch_add(calls, std::memory_order_relaxed);
gReadReads.fetch_add(1, std::memory_order_relaxed);
gReadBytes.fetch_add(wanted, std::memory_order_relaxed);
gReadNsTotal.fetch_add(static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - readStarted).count()),
std::memory_order_relaxed);
return true;
#else
(void)data;
@@ -213,8 +261,16 @@ namespace MobileGL::MG_Remote::Transport {
msghdr message{};
message.msg_iov = vectors;
message.msg_iovlen = size ? 2 : 1;
const auto sendStarted = std::chrono::steady_clock::now();
std::uint64_t sendCalls = 0;
while (message.msg_iovlen) {
const auto syscallStarted = std::chrono::steady_clock::now();
const auto sent = ::sendmsg(fd, &message, MSG_NOSIGNAL);
gSendNsInSend.fetch_add(static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - syscallStarted).count()),
std::memory_order_relaxed);
++sendCalls;
if (sent < 0 && errno == EINTR) continue;
if (sent <= 0) {
Die(!stopping.load(std::memory_order_acquire));
@@ -231,6 +287,12 @@ namespace MobileGL::MG_Remote::Transport {
message.msg_iov->iov_len -= consumed;
}
}
gSendCalls.fetch_add(sendCalls, std::memory_order_relaxed);
gSendBytes.fetch_add(size, std::memory_order_relaxed);
gSendNsTotal.fetch_add(static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::nanoseconds>(
std::chrono::steady_clock::now() - sendStarted).count()),
std::memory_order_relaxed);
return true;
#else
(void)payload;
@@ -391,6 +453,19 @@ namespace MobileGL::MG_Remote::Transport {
slot.status = static_cast<std::int32_t>(static_cast<std::uint32_t>(b));
slot.bytes.assign(payload, payload + size);
++retainedTotal;
// LOGGED AS IT CROSSES EVERY QUARTER OF THE CAP, not sampled: at most
// four lines per run, and they are the ones a "how deep may the window
// go" question is answered from without a rebuild (P12's
// MOBILEGL_IPC_CREATE_WINDOW sweep reads exactly these).
if (retained.size() > retainedPeak) {
retainedPeak = retained.size();
if (retainedPeak % (kRetainedCap / 4) == 0) {
MGLOG_I("P12 store: peak=%llu of cap=%llu declared answers "
"outstanding; the window's depth is what is spent here",
static_cast<unsigned long long>(retainedPeak),
static_cast<unsigned long long>(kRetainedCap));
}
}
} else {
// NOBODY DECLARED IT: drop it. This is what keeps the store bounded
// against a server that answers records this client never reads.
@@ -404,6 +479,25 @@ namespace MobileGL::MG_Remote::Transport {
}
};
StreamLink::ReadStats StreamLink::TakeSendStats() {
ReadStats out;
out.bytes = gSendBytes.exchange(0, std::memory_order_relaxed);
out.calls = gSendCalls.exchange(0, std::memory_order_relaxed);
out.nsInRecv = gSendNsInSend.exchange(0, std::memory_order_relaxed);
out.nsTotal = gSendNsTotal.exchange(0, std::memory_order_relaxed);
return out;
}
StreamLink::ReadStats StreamLink::TakeReadStats() {
ReadStats out;
out.bytes = gReadBytes.exchange(0, std::memory_order_relaxed);
out.calls = gReadCalls.exchange(0, std::memory_order_relaxed);
out.reads = gReadReads.exchange(0, std::memory_order_relaxed);
out.nsInRecv = gReadNsInRecv.exchange(0, std::memory_order_relaxed);
out.nsTotal = gReadNsTotal.exchange(0, std::memory_order_relaxed);
return out;
}
StreamLink::StreamLink() : m_impl(new Impl) {}
StreamLink::~StreamLink() {
Detach();
+25
View File
@@ -11,6 +11,23 @@ namespace MobileGL::MG_Remote::Transport {
public:
StreamLink();
~StreamLink() override;
// P65READ: THE SERVER'S SOCKET-READ PATH, ACCOUNTED WHERE IT CAN BE SEEN. The counters are
// published from the io thread's recv loop and read by the APPLY thread, because the io
// thread's own MGLOG_ lines never reach the phone's forwarded log (only mgl-srv-apply and
// mgl-display-ser do - measured, not assumed). Taking them RESETS them, so one call is one
// frame's worth of the stage nothing else measures: the frame's time is not the client's GL
// thread (13%) and not PipeApplier::ApplyOne (11%), and this is what sits between them.
struct ReadStats {
std::uint64_t bytes = 0; // payload bytes taken off the socket
std::uint64_t calls = 0; // recv() calls it took
std::uint64_t reads = 0; // Link::Read() calls (one per header, envelope or chunk)
std::uint64_t nsInRecv = 0; // wall time inside recv - blocking included
std::uint64_t nsTotal = 0; // wall time inside Read
};
static ReadStats TakeReadStats();
// The client's half: bytes handed to the io thread's sendmsg, the calls it took, and the
// wall time inside sendmsg (blocking included). nsInRecv carries the send time.
static ReadStats TakeSendStats();
MobileGLResult Attach(int dataFd, SessionSegments& mirrors, TransportRoleTag role);
MobileGLResult AttachOwned(int dataFd, const struct SessionSegmentSizes& sizes, TransportRoleTag role);
// PH-7 (4), ID-P7-3. The SERVER's half of a nonce-bound data connection, in two steps.
@@ -38,6 +55,14 @@ namespace MobileGL::MG_Remote::Transport {
MobileGLResult ReadReply(std::uint64_t, std::int32_t*, const void**, std::uint64_t*) override;
void DeclareReplyRead(std::uint64_t) override;
// THE DECLARED-ANSWER STORE'S CAPACITY, and it is a PUBLIC NUMBER rather than a private
// one because the client's create window is bounded by it: WireTables.cpp's
// kCreateWindowMax static_assert and RemoteClientControls' ceiling case both read it, and
// a bound that is restated in prose in three places is a bound that drifts. The argument
// for the VALUE (why 24, and why overflow is a Fatal and never an eviction) is in the
// .cpp beside the store it sizes.
static constexpr std::size_t kRetainedAnswersMax = 24;
// P12 diagnostics for the declared-answer store. RetainedOutstanding is what the capacity
// is about (it must stay near the window's depth, not grow with the server's answers);
// RetainedTotal counts the answers kept for their readers; SkippedUnwanted counts the ones
+17 -14
View File
@@ -1232,24 +1232,27 @@ TEST(RemoteClientControls, TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers
Uint8 pixels[4 * 4 * 4];
std::memset(pixels, 0x21, sizeof(pixels));
// ---- the clamp and the ceiling, before a single record goes out ----
// THE WINDOW IS OPT-IN AND THE DEFAULT IS OFF (Config.h: the deferral was measured on the
// device and the drain cannot collect it), so both arms of this case SET the knob rather
// than inheriting it - the ON half below is the mechanism being exercised, not a claim
// about what ships.
// BOTH ARMS SET THE KNOB rather than inheriting it: this case is about what the code does
// with a configured window, and not a claim about which value ships (the default is 2 -
// Config.h carries the device numbers).
// WHAT THE OPERATOR WROTE IS NOT WHAT THE CODE MAY USE. With the clamp gone a configured
// window of 99 is compared against a count that the ring keeps below its own size, so the
// fallback never fires and the push index runs off the array.
MG_Config::Ipc.CreateWindow = 99;
if (ClientCreateWindowEffective() != ClientCreateWindowArraySize()) ::_exit(170);
MG_Config::Ipc.CreateWindow = 4;
if (ClientCreateWindowEffective() != 4u) ::_exit(171);
// AND THE ARRAY IS SIZED FROM THE POOL IT SHARES, not from a second literal: the pool is
// Transport::kDefaultReplySlotCount slots addressed seq % count, and every reply-owning row
// (fence, query, the blocking rows themselves) needs one. Raising the window to the whole
// pool is the change the handoff names, and it is red here as well as at the static_assert
// in WireTables.cpp - deliberately twice, because the compile-time one is the guard and
// this one is the statement of what the guard is for.
if (ClientCreateWindowArraySize() > Transport::kDefaultReplySlotCount / 2) ::_exit(172);
MG_Config::Ipc.CreateWindow = 8;
if (ClientCreateWindowEffective() != 8u) ::_exit(171);
// AND THE ARRAY IS SIZED FROM THE STORE IT SHARES, not from a second literal (P12). This
// line used to read "at most half the REPLY POOL", which was true while an answer was read
// back out of a slot pool addressed seq % slotCount - and the window does not run on such
// a link: CreateWindowEffective() is gated on ILink::RetainsReplies, false for ShmLink, so
// the pool-addressed transport holds a window of 1. On the link where it does run, an
// answer waits in the declared-answer store instead, so the bound that matters is the
// store's - read from the constant rather than restated, for the reason the pool version
// read kDefaultReplySlotCount. WireTables.cpp's static_assert is the compile-time half of
// the same sentence, deliberately twice: that one is the guard, this one is what it is for.
if (ClientCreateWindowArraySize() > Transport::StreamLink::kRetainedAnswersMax / 2)
::_exit(172);
// TWELVE OBJECTS, ONE CREATE EACH, AND THE INVARIANT READ AFTER EVERY ONE. This is the
// reading a silent overwrite cannot fake: the count stays at the ceiling while an entry is
@@ -1268,7 +1271,7 @@ TEST(RemoteClientControls, TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers
};
// ---- THE KNOB IS AS HIGH AS IT GOES, AND THE ROW MUST STILL BLOCK ----
MG_Config::Ipc.CreateWindow = 4;
MG_Config::Ipc.CreateWindow = ClientCreateWindowArraySize();
const Uint64 recordsBefore = Transport::LinkMetricsRecordsFor(createOp);
const Uint64 waitsBefore = Transport::LinkMetricsReplyWaitsFor(createOp);
burst(12);
@@ -37,6 +37,9 @@
#include <MG_Remote/Client/CapsMirror.h>
#include <MG_Remote/Transport/ReplySlot.h>
#include <MG_Remote/Transport/LinkMetrics.h>
// P12: StreamLink::kRetainedAnswersMax, the store capacity RemoteClientControls' window-ceiling
// case reads instead of restating it.
#include <MG_Remote/Transport/StreamLink.h>
#include <MG_Pipe/PipeRoute.h>
#include <MG_Remote/Client/EmitTables.h>
#include <MG_Remote/Client/WireTables.h>
@@ -262,7 +262,11 @@ namespace {
TEST_F(StreamClientWindow, EveryDeferredCreateAnswerComesBackOnTheShippingArm) {
const Uint32 savedWindow = MG_Config::Ipc.CreateWindow;
MG_Config::Ipc.CreateWindow = 4;
// THE DEEPEST WINDOW THE CODE SUPPORTS, so the case exercises the ceiling rather than a
// comfortable middle: more deferred answers outstanding means more of the store spent, and
// the crowding below is what has to survive it. (The CAP itself is pinned by
// RemoteClientControls' window-ceiling case; this one pins that the answers come back.)
MG_Config::Ipc.CreateWindow = 8;
StartServo();
const std::uint64_t takenBefore = Client::ClientCreateWindowTaken();
constexpr Int32 kCreates = 24;
@@ -331,7 +335,7 @@ namespace {
EXPECT_LE(pending, Client::ClientCreateWindowEffective());
// AND THE ROW REALLY IS DEFERRING HERE, so the case cannot pass by taking the blocking
// path on every create (which is what a link the window refuses to run on looks like).
EXPECT_EQ(Client::ClientCreateWindowEffective(), 4u);
EXPECT_EQ(Client::ClientCreateWindowEffective(), 8u);
EXPECT_TRUE(client.LinkRetainsReplies());
MG_Config::Ipc.CreateWindow = savedWindow;
}
+19
View File
@@ -563,6 +563,25 @@ by name (`NoServerDisplay`) and that service goes on serving.
## TCP credit and Stage measurements
For a paired frame-time breakdown, enable `MOBILEGL_PIPE_STATS=1` in both roles
and keep their logs. The following reducer matches client and server present
serials, drops the first (unpaired) server interval and one warmup frame, and
reports avg/p50/p99 in milliseconds plus percentages of client present time:
```bash
python3 tools/trace_replay/frame_stats.py \
--client-log mobilegl.client.log --server-log mobilegl.server.log \
--warmup-frames 1 --output frame-stats.json
```
`transport_wait_ms` measures wall time in the client producer's transport park
(flush, spin and sleep). It includes waiting for replies, present credit and
ring space; it is not socket latency. Client and server CPU can overlap with
each other and with a transport wait, so each percentage uses the client
present interval as its denominator and the three percentages need not sum to
100%. A missing frame is rejected. Percentiles use nearest rank over per-frame
samples.
`benchmark_tcp_credits.py` runs the complete OpenRA and rd12 traces at credits
1, 2, and 3, then selects matching client/server tail frame IDs (100 and 200
frames respectively). Extract the unchanged trace files as
+128
View File
@@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""Pair client/server present intervals and summarize their frame time.
MOBILEGL_PIPE_STATS=1 enables the per-present clocks. Frame IDs are matched
exactly; the first server interval is invalid because it has no preceding
present. Transport wait measures wall time inside the client's transport park
(flush, spin and sleep). It can overlap with server CPU and is not network latency.
"""
from __future__ import annotations
import argparse
import json
import math
import re
from pathlib import Path
FIELDS = re.compile(r"(\w+)=([^\s]+)")
CLIENT_MARKER = "P65LinkMetrics "
SERVER_MARKER = "P65ServerMetrics "
def read_frames(path: Path, marker: str, *, server: bool) -> dict[int, dict]:
rows = {}
for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
if marker not in line:
continue
values = dict(FIELDS.findall(line.split(marker, 1)[1]))
if server and values.get("valid") != "1":
continue
if not server and values.get("kind") != "frame":
continue
try:
number = int(values["frame"])
wall = int(values["wall_ns"])
cpu = int(values["apply_thread_cpu_ns" if server else "client_thread_cpu_ns"])
transport = 0 if server else int(values["transport_wait_ns"])
except (KeyError, ValueError) as error:
raise ValueError(f"{path}: malformed {marker.strip()} row") from error
if number in rows:
raise ValueError(f"{path}: duplicate frame {number}")
# The wall and thread-CPU clocks are sampled separately. Their nanosecond
# readings can differ slightly at a boundary even for a fully busy thread.
tolerance = max(100_000, wall // 100)
if (number <= 0 or wall <= 0 or cpu <= 0 or cpu > wall + tolerance or
transport < 0 or transport > wall + tolerance):
raise ValueError(f"{path}: invalid clocks in frame {number}")
rows[number] = {"wall_ns": wall, "cpu_ns": cpu, "transport_ns": transport}
if not rows:
raise ValueError(f"{path}: no usable {marker.strip()} frames")
return rows
def quantile_ns(samples: list[int], percentile: int) -> int:
ordered = sorted(samples)
return ordered[max(0, math.ceil(len(ordered) * percentile / 100) - 1)]
def distribution(samples: list[int]) -> dict[str, float]:
return {"avg": sum(samples) / len(samples) / 1e6,
"p50": quantile_ns(samples, 50) / 1e6,
"p99": quantile_ns(samples, 99) / 1e6}
def summarize(client_path: Path, server_path: Path, *, warmup: int = 0,
tail: int | None = None) -> dict:
client = read_frames(client_path, CLIENT_MARKER, server=False)
server = read_frames(server_path, SERVER_MARKER, server=True)
selected = sorted(number for number in client if number > warmup)
if tail is not None:
selected = selected[-tail:]
if not selected:
raise ValueError("no frames remain after warmup/tail selection")
if selected != list(range(selected[0], selected[-1] + 1)):
raise ValueError("selected client frame IDs are not contiguous")
missing = [number for number in selected if number not in server]
if missing:
raise ValueError(f"server is missing matched frames: {missing[:10]}")
columns = {name: [] for name in ("client_present_ms", "server_present_ms",
"client_cpu_ms", "transport_wait_ms", "server_cpu_ms")}
for number in selected:
c, s = client[number], server[number]
for key, value in (("client_present_ms", c["wall_ns"]),
("server_present_ms", s["wall_ns"]),
("client_cpu_ms", c["cpu_ns"]),
("transport_wait_ms", c["transport_ns"]),
("server_cpu_ms", s["cpu_ns"])):
columns[key].append(value)
duration = sum(columns["client_present_ms"])
shares = {name: 100 * sum(columns[name]) / duration
for name in ("client_cpu_ms", "transport_wait_ms", "server_cpu_ms")}
return {"frames": len(selected), "first_frame": selected[0],
"last_frame": selected[-1], "warmup_frames": warmup,
"definition": "transport_wait is client wall time in SessionProducer::Park "
"(flush, spin and sleep); role CPU and transport wait can overlap",
"percentile": "nearest rank", "unit": "ms",
"stats": {key: distribution(values) for key, values in columns.items()},
"share_percent_of_client_present": shares}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--client-log", type=Path, required=True)
parser.add_argument("--server-log", type=Path, required=True)
parser.add_argument("--warmup-frames", type=int, default=1)
parser.add_argument("--tail-frames", type=int)
parser.add_argument("--output", type=Path)
args = parser.parse_args()
if args.warmup_frames < 0 or args.tail_frames is not None and args.tail_frames <= 0:
parser.error("warmup must be nonnegative and tail must be positive")
try:
result = summarize(args.client_log, args.server_log, warmup=args.warmup_frames,
tail=args.tail_frames)
except (OSError, ValueError) as error:
parser.error(str(error))
rendered = json.dumps(result, ensure_ascii=False, indent=2) + "\n"
if args.output:
args.output.write_text(rendered, encoding="utf-8")
else:
print(rendered, end="")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Small fixture checks for frame ID pairing and additive frame attribution."""
import tempfile
import unittest
from pathlib import Path
from frame_stats import summarize
class FrameStatsTest(unittest.TestCase):
def logs(self, root, *, missing_server=False, overlapping=False):
client = root / "client.log"
server = root / "server.log"
client.write_text("\n".join(
f"P65LinkMetrics kind=frame frame={n} wall_ns={wall} "
f"client_thread_cpu_ns={cpu} transport_wait_ns={wait}"
for n, wall, cpu, wait in ((1, 10000000, 2000000, 1000000),
(2, 20000000, 5000000, 4000000),
(3, 40000000, 10000000, 7000000))) + "\n")
server.write_text("\n".join(
f"P65ServerMetrics frame={n} valid=1 wall_ns={wall} "
f"apply_thread_cpu_ns={cpu}"
for n, wall, cpu in ((2, 18000000, 17000000 if overlapping else 3000000),
(3, 35000000, 31000000 if overlapping else 6000000))
if not missing_server or n != 3) + "\n")
return client, server
def test_paired_percentiles_and_shares(self):
with tempfile.TemporaryDirectory() as directory:
result = summarize(*self.logs(Path(directory)), warmup=1)
self.assertEqual((result["first_frame"], result["last_frame"], result["frames"]),
(2, 3, 2))
self.assertEqual(result["stats"]["transport_wait_ms"],
{"avg": 5.5, "p50": 4.0, "p99": 7.0})
self.assertEqual(result["share_percent_of_client_present"]["transport_wait_ms"],
100 * 11000000 / 60000000)
def test_missing_server_frame_is_error(self):
with tempfile.TemporaryDirectory() as directory:
with self.assertRaisesRegex(ValueError, "missing matched frames"):
summarize(*self.logs(Path(directory), missing_server=True), warmup=1)
def test_parallel_cpu_is_allowed(self):
with tempfile.TemporaryDirectory() as directory:
result = summarize(*self.logs(Path(directory), overlapping=True), warmup=1)
self.assertGreater(sum(result["share_percent_of_client_present"].values()), 100)
if __name__ == "__main__":
unittest.main()