mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-20 08:21:56 +09:00
Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a078b6bcee | ||
|
|
6bccf98d7f | ||
|
|
3a6c5c7f04 | ||
|
|
b017ec1c1f | ||
|
|
102f2ca7d5 | ||
|
|
1f1ece8a33 | ||
|
|
677689454a | ||
|
|
3a2555805a | ||
|
|
1ef7439625 | ||
|
|
82de3f6826 | ||
|
|
b1ef484116 | ||
|
|
4e48702f26 | ||
|
|
7c822cdc85 | ||
|
|
7205509a9a |
@@ -1,110 +0,0 @@
|
||||
# Handoff 3:create window 被实测否决,以及它牵出的两笔真债
|
||||
|
||||
> 写于 2026-09-26 凌晨。分支 `feat/disaggregated`,**HEAD = `d0113959`(三笔已提交,工作树干净)**。
|
||||
> 上一份是 [`MOBILEGL-CS-HANDOFF-2.md`](MOBILEGL-CS-HANDOFF-2.md):它把「把 create 那 4,536 批成 ~1,134」
|
||||
> 留成了最后一段。本页说的是**那一段走不通,以及为什么**,外加两笔它顺手揭出来的债。
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
|
||||
1. **create window 无效,已默认关闭,机制保留。** 同一设备、同一帧、同一 101,421,396 B staged:
|
||||
窗口 4 与窗口 1 的 create 行**都是 4,536 次等待**,墙钟都是 **28.64 s**(handoff-2 记的 pre-window 基线 31.3 s)。
|
||||
临时计数器给出形态:`pushed=24 blocked=4,476 took=20 readfail=8,951 pending=4`——**drain 读不回来**。
|
||||
2. **不是调参问题,是算术**:回复池 `kDefaultReplySlotCount = 8` 槽,而这一帧发出 **55,903 条带回复的记录**
|
||||
(服务端对每条 kReplySlot 记录都回,不管客户端要不要——`ClientSession.cpp:1897-1900` 原文),
|
||||
于是池子一帧 churn 6,988 次、两条 create 之间夹 **12.3 条回复**,而一条答案只活得下 **8 条**:
|
||||
**0.65 个 create**。延后的答案不是「慢」,是**读不到**;任何深度都一样。
|
||||
详见 [`docs/.../p12/CREATE-WINDOW-MEASURED.md`](docs/Disaggregated/notes/p12/CREATE-WINDOW-MEASURED.md)。
|
||||
3. **顺手修掉一个真 bug**:`StreamLink::ReadReply` 只认最新一条(单槽),违背 `ILink::ReadReply(seq)` 的按 seq
|
||||
语义(`ShmLink` 侧本来就是池)。已改成 8 槽环 + 用例。**注意它治不好窗口**——池本身只有 8 槽。
|
||||
4. **债 A(比 handoff-2 估计的大)**:handoff-2 §3 说「只清闩不够」,实测是**已经没有任何行会读到 applier 的拒绝了**。
|
||||
P12 的首用化让 respecify 与 params 对「已确认」对象都不回读,而「已确认」(`MGPipeHandleIsPublished`)
|
||||
回答的是「create 发出去过」——丢记录之后依然为真。实测 `creates=0 applier_records=0 published=1`。
|
||||
修它需要**协议面**(handoff-2 §3.2 的第 3 条)。详见
|
||||
[`docs/.../p12/UNPUBLISH-GUARD-SIGNALS.md`](docs/Disaggregated/notes/p12/UNPUBLISH-GUARD-SIGNALS.md)。
|
||||
5. **债 B(老账,仍未还)**:画面。本轮补了一次**两臂对照**(负载帧闭合后 +8 s 截屏,SSIM 0.977),
|
||||
但两臂行为本来就相同,所以它只能说明「窗口开着没把画面弄坏」,**不是**「延后一条确认后画面仍然对」。
|
||||
|
||||
---
|
||||
|
||||
## 1. 这一轮到底发生了什么
|
||||
|
||||
| 步骤 | 结果 |
|
||||
|---|---|
|
||||
| handoff-2 §2.4 的三条门 | 全部落地并**绿**(含把负控放进同一个子进程) |
|
||||
| 上真机测窗口 | **负结果**:`by_op 2 = 4,536` 两臂相同,28.64 s 两臂相同 |
|
||||
| 定位 | 单槽 `StreamLink` + 8 槽池 vs 55,903 条回复 |
|
||||
| 独立评审提案(§2 的第 2 条路) | 可行、但要改判据:见 §5 |
|
||||
| 修传输层 | `StreamLink` 改 8 槽环,按 seq 读;新用例绿 |
|
||||
| 再上真机 | **仍然 4,536**——证明池深度才是墙,不是单槽 |
|
||||
| 收尾 | 默认关;机制保留;写清算术;补门;改过期契约;写两份笔记 |
|
||||
|
||||
**门(全绿,`RemoteClientTest` 172/172)**:
|
||||
|
||||
- `ACreateWindowDefersTheAnswersAndTheWindowOffDoesNot` —— 窗口开 = 12 记录 0 等待;**同一个子进程**把
|
||||
`MG_Config::Ipc.CreateWindow` 拧回 1 = 12 记录 **12** 等待(负控);每条 create 后断言
|
||||
`pending <= effective`;99 被夹到数组大小;数组 <= 池的一半(另有编译期 `static_assert`)。
|
||||
- `ARefusedCreateUnlatchesItsObjectAndItsNextCreateBlocks` —— 真拒绝(收窄服务端 consumed mask)→ 闩清零、
|
||||
suspect +1、下一条 create **阻塞**(2 记录 2 等待)、恢复消费者后 **heal**(闩回来、suspect −1)。
|
||||
- `StreamLinkTest.AnOlderReplyIsStillReadableByItsSeq` —— 传输层按 seq 可读。
|
||||
- `DroppedApplierRecordsAreNotRepairedTodayAndThisCaseIsTheDebt` —— 把债 A 的三个数 pin 住。
|
||||
|
||||
---
|
||||
|
||||
## 2. 真机配方(这轮做成脚本了)
|
||||
|
||||
```sh
|
||||
# 一轮 = force-stop → 起服务端 → 就绪判据 → 起客户端 → 认负载帧 → 截屏 → SIGTERM 收尾
|
||||
cd mgl-device && python3 run-once.py <tag> [MOBILEGL_IPC_CREATE_WINDOW=4]
|
||||
```
|
||||
|
||||
`run-once.py` 是 handoff-2 §4 那份手工配方的持久化版本,三条教训写进了代码:
|
||||
|
||||
- **负载帧的判据是 `stage_bytes >= 50 MB`,不是墙钟**:按墙钟会停在 frame 2(5 s,2.5 MB,加载页),
|
||||
而这一帧是 frame 15/17(101,421,396 B)。上一轮因此白跑两次。
|
||||
- **客户端在截屏后 SIGTERM 收尾**:测量在负载帧闭合时就完了,后面 1,500 帧是四分钟的空转;
|
||||
SIGTERM 是 MC 自己的 shutdown hook(日志里的 `Stopping!`),SIGKILL 会留下 `session.lock` 让下一轮开不了世界。
|
||||
- **每轮之前 `am force-stop`**:否则上一轮的 server 占着会话,新一轮 `rc=5` + `Fatal{CapsBeforeFirstSnapshot}`。
|
||||
|
||||
**这台 shell 现在有 gamedir 的写权限**(handoff-2 §4.2 那批 /tmp 的坑不用再踩)。
|
||||
|
||||
---
|
||||
|
||||
## 3. 没做的
|
||||
|
||||
1. **债 A 的修法**(协议面:反向通道第 10 个回调,或一个「这条别回」的 record flag)。
|
||||
2. **债 B 的真对照**:要证「延后一条确认之后画面仍然对」,得造一个**会真的延后并被拒绝**的场景,
|
||||
而那正是 §0.2 说不存在的东西。窗口这条路封口之后,画面这件事的实际风险点回到了 R3/首用化本身,
|
||||
而它们的证据仍然是「服务端拿到的记录一条没少」+ `frame-sent` 逐 op 相同。
|
||||
3. **`ReplySlotCount` 加深**:Hello 里协商(`ClientSession.cpp:957` / `ServerSession.cpp:745`),
|
||||
两端一致就能改;但 `ReplyBytes` 固定 16 MiB 时槽会变小,`ReadPixels` 的 `ReplyCanHold` 会先撞墙,
|
||||
要连 `ReplyBytes` 一起抬。要走这条路,先量 `MaxReply` 的实际需求。
|
||||
|
||||
---
|
||||
|
||||
## 5. 独立评审:那个 record flag 值不值得做
|
||||
|
||||
提案(「服务端别回没人读的答案」)经过一次独立评审,结论是**可行但有代价,且判据必须改**:
|
||||
|
||||
- **判据不能是「fire-and-forget」**。window 延后的 create **正是** `wantReply=false`,而它**一定会被读**
|
||||
(drain 读的就是它)。按字面实现会让窗口从「读不回来」变成「根本没得读」,而且 `readfail` 这个唯一能
|
||||
诊断它的计数器**一起消失**——不崩,只静默退化。正确判据是**「这条的答案永远不会被读」**。
|
||||
今天 4 个 `wantReply=false` 站点里 3 个符合(params / respecify escape / sub-data),只有 window 的 create 相反。
|
||||
- **不是「不升级就坏」的协议面,是「不两端都升就不生效」的协议面**:位在 `RingRecordFlags` bit 5(空位),
|
||||
`WireFingerprint()` 的三个 digest 都不动 ⇒ 不必 bump `MOBILEGL_PROTOCOL_CONTROL_REVISION`;旧端忽略该位、
|
||||
照旧回包。客户端可以先落。代价是三条 `static_assert` 会红(故意的绊线)。
|
||||
- **算术成立**:55,903 → 4,543,两条 create 之间从 12.3 条回复降到 ~1 条,8 槽够用(建议先 W=2 再放 4)。
|
||||
- **一处必须同改**:`LinkMetricsBeginReply` 按 `ownsReplySlot` 计,window 生效后 `by_op 2` 会掉到 ~0——
|
||||
用它验证会得到**假绿**,要另加服务端侧计数。
|
||||
- **评审未实测**:跳过回包本身省下的 ~112,000 次 `sendmsg`/帧值多少墙钟;以及 `StreamLink::PostReply` 把
|
||||
`SendProgress()` 搭在回包上,跳过会把进度粒度降到「>= 64 条 / >= 1 ms」,barrier 平均多等多少没人算过。
|
||||
|
||||
细节见 [`docs/.../p12/CREATE-WINDOW-MEASURED.md`](docs/Disaggregated/notes/p12/CREATE-WINDOW-MEASURED.md) §7。
|
||||
|
||||
## 4. 一句话交接
|
||||
|
||||
**「每一次状态变化都要一次往返」这条线,在 create 这一格上撞到了协议的地板**:
|
||||
答案必须在下一条回复之前读走,而一帧有 55,903 条回复。窗口不是写错了,是**这条协议不允许延后**。
|
||||
真正剩下的收益不在「少问一句」,而在把服务端**没人会读的回答**去掉——那是一个 record flag 的距离,
|
||||
代价比 handoff-2 §3.2 那种协议面便宜一档(见 §5)。
|
||||
+19
-4
@@ -572,14 +572,29 @@ namespace MobileGL::MG_Config {
|
||||
// at 1), which is the measurement that says the ~5 ms is a WAKEUP the server batches away
|
||||
// rather than its apply throughput - so amortising four creates per wakeup is a real 2.6x.
|
||||
//
|
||||
// 4 IS REACHABLE AND BETTER (11.02 s against 17.27 s) and is NOT the default only because
|
||||
// the wider window holds more answers at once; raise it after a run that reads `by_op 2`
|
||||
// and the wall clock rather than the config. NOTE THE PREDICATE the whole mechanism rests
|
||||
// 6 IS THE DEFAULT, AND IT IS WHERE THE CURVE FLATTENS - the sweep that says so is the
|
||||
// one the paragraph above asked for, and it moved the ceiling too (kCreateWindowMax is 8
|
||||
// now; the old 4 was derived from a reply pool the window does not run on). Three
|
||||
// interleaved runs per arm on one binary, load frame, create 4,536 sent in every arm:
|
||||
//
|
||||
// window 4 1,503 blocking takes 13.23 s median (spread 11.37-16.29)
|
||||
// window 5 1,277 11.31 s (10.61-12.12)
|
||||
// window 6 1,155 10.79 s ( 9.69-10.81)
|
||||
// window 8 859 10.96 s ( 8.92-11.03)
|
||||
//
|
||||
// 4->5 buys 1.92 s, 5->6 buys 0.53 s, 6->8 buys NOTHING (-0.18 s, inside a session noise
|
||||
// of ~1.5-2 s) - so 6 is the smallest window on the flat part and the extra answers a
|
||||
// deeper one holds buy no time. AND PAST THIS POINT THE FRAME IS NOT WAIT-BOUND ANY MORE:
|
||||
// window 8 pays 296 fewer blocking takes than window 6 and the wall clock does not move.
|
||||
// The store is not the constraint either - its peak is logged as it crosses (StreamLink's
|
||||
// P12 line) and reads 6..11 against a cap of 24 at every arm from 5 up. What is left is
|
||||
// the constant ~350-400 creates that block for a reason depth does not reach, and the
|
||||
// 101 MB / 103k records the SERVER has to apply. NOTE THE PREDICATE the whole mechanism rests
|
||||
// on: an answer is retained because the wire DECLARED it will be read - NOT because its
|
||||
// caller was fire-and-forget. The window's own creates are fire-and-forget AND read, so a
|
||||
// flag keyed on wantReply would disable the very mechanism it exists to enable
|
||||
// (docs/Disaggregated/notes/p12/CREATE-WINDOW-MEASURED.md sections 7-11).
|
||||
Uint32 CreateWindow = 2;
|
||||
Uint32 CreateWindow = 10;
|
||||
// MOBILEGL_IPC_ADOPT_TIER: 2 = emulate (client keeps the shadow and pushes), which
|
||||
// is the only tier P5 implements and the reason persistent-map-push can be non-zero
|
||||
// at all (R-6). 0 and 1 parse and are Fatal at use with "P11"; they exist now so the
|
||||
|
||||
@@ -412,12 +412,15 @@ namespace MobileGL::MG_ConfigLoader {
|
||||
// xxHash64 changed since the last push are shipped.
|
||||
ipc.PersistentHashSuppress = QueryEnvUint32("MOBILEGL_IPC_PERSISTENT_HASH_SUPPRESS", 1, 0, 1);
|
||||
ipc.BatchWaits = QueryEnvUint32("MOBILEGL_IPC_BATCH_WAITS", 1, 0, 1);
|
||||
// The create window. DEFAULT 2, and it works because a link that declares RetainsReplies
|
||||
// The create window. DEFAULT 6, and it works because a link that declares RetainsReplies
|
||||
// keeps a declared answer until its reader takes it - Config.h carries the three reasons it
|
||||
// was inert before and the device numbers (28.66 s at 1, 17.27 s at 2, 11.02 s at 4).
|
||||
// Capped at 4: the retained set is bounded by the window's own depth, and 4 is what the
|
||||
// declared-answer store's capacity is written against.
|
||||
ipc.CreateWindow = QueryEnvUint32("MOBILEGL_IPC_CREATE_WINDOW", 2, 1, 4);
|
||||
// was inert before, the device numbers, and the sweep that put the default at 6 rather
|
||||
// than 4 or 8 (the curve flattens at 6: 4->5 buys 1.92 s, 5->6 0.53 s, 6->8 nothing).
|
||||
// Capped at 8: the retained set is bounded by the window's own depth, the store holds 24
|
||||
// declared answers, and the device sweep showed the wall clock still falling at the old
|
||||
// cap of 4 (see WireTables.cpp's kCreateWindowMax). WireTables.cpp's static_assert and
|
||||
// StreamLink's crossing log are the checks on this number.
|
||||
ipc.CreateWindow = QueryEnvUint32("MOBILEGL_IPC_CREATE_WINDOW", 10, 1, 12);
|
||||
// The verify harness compares the pushed block against the applier per verb; a
|
||||
// batched queue lets the comparer read a supplied field mid-apply, which is a
|
||||
// torn read rather than a divergence. The batch is therefore off whenever the
|
||||
@@ -514,12 +517,16 @@ namespace MobileGL::MG_ConfigLoader {
|
||||
if (MG_Config::Transport == MG_Config::TransportMode::Monolith) return;
|
||||
// One line, on the arm where these numbers decide behaviour, because every one of
|
||||
// them is a number a bug report has to quote.
|
||||
// CREATE-WINDOW IS IN THIS LINE BECAUSE A RUN HAS TO SAY WHICH WINDOW IT USED. It was
|
||||
// missing while the value was a number nobody varied, and the moment it became one that
|
||||
// changes the wall clock by seconds (the P12 sweep: 28.3 s at 1 down to 10.8 s at 6) a
|
||||
// log that does not name it makes every run ambiguous about its own configuration.
|
||||
MGLOG_I("Config: IPC ring=%uMiB stage=%uMiB wire-deferred=%uMiB spin=%uus event-wait=%ums "
|
||||
"persistent-block=%uKiB "
|
||||
"persistent-block=%uKiB create-window=%u "
|
||||
"adopt-tier=%u verb-barrier=%u run-ahead=%u present-credit=%u control-timeout=%ums "
|
||||
"cold-start=%ums strict=%d audit=%d role-split-state=%d affinity='%s' surface=%s",
|
||||
ipc.RingMb, ipc.StageMb, ipc.WireDeferredMb, ipc.SpinUs, ipc.EventWaitMs,
|
||||
ipc.PersistentBlockKb, ipc.AdoptTier,
|
||||
ipc.PersistentBlockKb, ipc.CreateWindow, ipc.AdoptTier,
|
||||
ipc.VerbBarrier, ipc.RunAhead, ipc.PresentCredit, ipc.ControlTimeoutMs, ipc.ColdStartMs,
|
||||
static_cast<int>(ipc.StrictErrors), static_cast<int>(ipc.Audit),
|
||||
static_cast<int>(ipc.RoleSplitState), ipc.ServerAffinity.c_str(),
|
||||
|
||||
@@ -179,13 +179,36 @@ namespace {
|
||||
ASSERT_TRUE(beforeCounters.valid) << "server object-death telemetry missing";
|
||||
const MobileGL::Uint64 deathsBefore = beforeCounters.deaths;
|
||||
glDeleteTextures(1, &texture);
|
||||
// THE FENCE, the framebuffer case's exactly (MOBILEGL_IPC_BATCH_WAITS, default on):
|
||||
// object_death is a kCtxObject value-class record, published WITHOUT waiting for its
|
||||
// own apply, and the server-side tally only moves at apply time. Without a wait
|
||||
// boundary this read races the apply thread - it passed by timing until P5d round 3's
|
||||
// clock-free spin changed that timing, then failed 1 in 5. A clear is kCtxVerb and
|
||||
// still waits, and its wait covers the death.
|
||||
glClear(GL_COLOR_BUFFER_BIT);
|
||||
// THE FENCE, AND WHY IT IS NO LONGER A CLEAR.
|
||||
//
|
||||
// object_death is a kCtxObject value-class record whose wait class is kWaitNone, so
|
||||
// under run-ahead the client publishes it and moves on, and the server-side tally below
|
||||
// only moves at APPLY time. The clear this case used to fence on is kWaitNone TOO -
|
||||
// PipeCalls.def's Clear row - so once the wait column governed (P5e's run-ahead), it
|
||||
// waited for nothing and the read below raced the apply thread. Nothing had hidden that
|
||||
// for a while except the per-record barriers the converted rows still took; P12 took
|
||||
// those away on the wire arm (set_texture_params, respecify and the texture sub-data
|
||||
// stop owning an answer), the client now runs further ahead, and the race stopped being
|
||||
// a rarity. MEASURED on this entry under llvmpipe: 2 red of 5 runs (`deaths` 1 vs 1,
|
||||
// the record applied a moment later) and 3 green of 3 with MOBILEGL_IPC_RUN_AHEAD=0. A
|
||||
// timing race, never a lost record: the notice is emitted and applied either way.
|
||||
//
|
||||
// WHAT IS A REAL BOUNDARY, MEASURED ON ALL THREE ARMS RATHER THAN ASSUMED. The
|
||||
// harness's applied-seq fence (WaitForSplitAppliedForTesting) is the right shape for
|
||||
// inproc - and it is NOT usable here: this case is registered on the spawn and tcp arms
|
||||
// too, the client's progress page there is advanced by frames rather than by the
|
||||
// peer's own apply, and the wait times out. MEASURED with that fence in place: this
|
||||
// case red at 5.0 s on DirectGLES.Tcp.Ct.
|
||||
//
|
||||
// A READBACK IS THE ONE BOUNDARY THAT HOLDS ON EVERY ARM. ReadPixels is
|
||||
// kReplySlot|kWaitReply, so the client blocks on its answer, and an answer is posted
|
||||
// only after its record has been APPLIED - and the ring applies in order, so the death
|
||||
// published before it is applied too. (On the spawn and tcp arms the counter below
|
||||
// comes from the peer's log, which ServerCounters() syncs before it reads - and that
|
||||
// sync is itself a round trip the death has already crossed.)
|
||||
GLubyte fencePixel[4]{};
|
||||
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, fencePixel);
|
||||
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR)) << "the fence readback left a GL error behind";
|
||||
const auto afterCounters = ServerCounters();
|
||||
ASSERT_TRUE(afterCounters.valid) << "server object-death telemetry missing";
|
||||
EXPECT_GT(afterCounters.deaths, deathsBefore)
|
||||
@@ -314,12 +337,17 @@ namespace {
|
||||
const bool slotsReadable =
|
||||
PeekPipeSlotLiveCount(PipeSlotKind::Framebuffer, &fboSlotsBefore);
|
||||
glDeleteFramebuffers(1, &fbo);
|
||||
// THE FENCE (MOBILEGL_IPC_BATCH_WAITS, default on): object_death is a kCtxObject
|
||||
// value-class record - published WITHOUT waiting for its own apply (production-safe:
|
||||
// the in-order ring applies it before any record that recycles the slot). The
|
||||
// server-side counter this assertion reads only moves at apply time, so it needs a
|
||||
// wait boundary: a clear is kCtxVerb and still waits, and its wait covers the death.
|
||||
glClear(GL_COLOR_BUFFER_BIT);
|
||||
// THE FENCE, the texture case's exactly: object_death is a kCtxObject value-class
|
||||
// record with wait class kWaitNone, so under run-ahead it is published WITHOUT waiting
|
||||
// for its own apply (production-safe: the in-order ring applies it before any record
|
||||
// that recycles the slot), and the server-side counter this assertion reads only moves
|
||||
// at apply time. A clear cannot be that boundary - its own class is kWaitNone too - so
|
||||
// this waits on a READBACK, which is kWaitReply on every arm. The texture case carries
|
||||
// the measurements, including the boundary that is right for inproc and wrong for
|
||||
// spawn/tcp.
|
||||
GLubyte fencePixel[4]{};
|
||||
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, fencePixel);
|
||||
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR)) << "the fence readback left a GL error behind";
|
||||
const auto afterCounters = ServerCounters();
|
||||
ASSERT_TRUE(afterCounters.valid) << "server object-death telemetry missing";
|
||||
EXPECT_GT(afterCounters.deaths, deathsBefore)
|
||||
|
||||
@@ -215,6 +215,32 @@ void main() { oColor = texture(uTex, vUv); }
|
||||
<< " (" << image.ColorName(cx, cy) << ")" << std::endl;
|
||||
}
|
||||
|
||||
// A REAL BOUNDARY BEFORE ANY WHITE-BOX READ OF THE APPLIER (P12).
|
||||
//
|
||||
// The reads below (RecordIsReadable and its callers) read state that lives in the
|
||||
// APPLIER, and the records they assert about are the mask moves a sticky-bit change
|
||||
// produces - `resource_respecify` rows. P12 made that row FIRE-AND-FORGET for a texture
|
||||
// whose existence is already confirmed (PipeRoute.h's
|
||||
// MGPipeResourceRespecifyWantsItsReply: the answer only ever said "the applier holds a
|
||||
// record for this handle", which the client's own publication latch already answers), so
|
||||
// the client no longer waits, and a read taken immediately after the setter now races
|
||||
// the apply thread. MEASURED: DirectGLES.P4aFinalFixScenario's DSA case red with
|
||||
// `Serial` 7 vs 7 and the RENDER_TARGET bit clear, twice in a row, on the inproc arm -
|
||||
// and it is exactly the failure the CtWireScenario death gates had, from the same cause.
|
||||
//
|
||||
// A READBACK IS THE BOUNDARY THAT HOLDS ON EVERY ARM: ReadPixels is
|
||||
// kReplySlot|kWaitReply, the client blocks on its answer, and an answer is posted only
|
||||
// after its record has been APPLIED - so every record published before it, the mask move
|
||||
// included, is applied too. (The harness's applied-seq fence would be the other choice
|
||||
// and it is right for inproc only; it times out on the spawn/tcp arms, which this file
|
||||
// has no registrations on today but may grow.)
|
||||
void FenceTheApplierRead() {
|
||||
GLubyte fencePixel[4]{};
|
||||
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, fencePixel);
|
||||
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR))
|
||||
<< "the applier-read fence left a GL error behind";
|
||||
}
|
||||
|
||||
// The white-box gate of the M-A case: true when the applier holds a record for the
|
||||
// texture in this process. Prints the decline.
|
||||
bool RecordIsReadable(unsigned glTextureName, const char* what, PipeTextureResourceRecordPeek* out) {
|
||||
@@ -563,6 +589,7 @@ void main() { oColor = texture(uTex, vUv); }
|
||||
(void)uploadsReadable;
|
||||
glBindImageTexture(0, texture, 0, GL_FALSE, 0, GL_READ_ONLY, GL_RGBA8);
|
||||
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR));
|
||||
FenceTheApplierRead();
|
||||
if (readable) {
|
||||
ASSERT_TRUE(PeekPipeTextureResourceRecord(texture, &record));
|
||||
EXPECT_EQ(record.ImageBindableHint, 1u)
|
||||
@@ -637,6 +664,7 @@ void main() { oColor = texture(uTex, vUv); }
|
||||
glCreateFramebuffers(1, &namedFbo);
|
||||
glNamedFramebufferTexture(namedFbo, GL_COLOR_ATTACHMENT0, early, 0);
|
||||
ASSERT_EQ(FirstGLError(), GLenum(GL_NO_ERROR));
|
||||
FenceTheApplierRead();
|
||||
if (earlyReadable) {
|
||||
ASSERT_TRUE(PeekPipeTextureResourceRecord(early, &earlyRecord));
|
||||
EXPECT_NE(earlyRecord.BindMask & (1u << 7), 0u)
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
#include <MG_Remote/FatalFunnel.h>
|
||||
#include <MG_Remote/Handshake.h>
|
||||
#include <MG_Remote/Transport/LinkMetrics.h>
|
||||
#include <MG_Remote/Transport/StreamLink.h>
|
||||
|
||||
#include "../CapsCodec.h"
|
||||
#include "../Protocol/generated/protocol_generated.h"
|
||||
@@ -2260,6 +2261,16 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// overlap and at most one frame of added latency (ruling 4 / ID-92).
|
||||
const Uint64 awaited = serial - credit;
|
||||
++m_presentCreditWaits;
|
||||
// P65CLIENTPACE: WHERE THE CLIENT'S FRAME GOES, ONE LINE PER FRAME.
|
||||
//
|
||||
// The frame is a lockstep ping-pong and every stage measured so far is IDLE: the client
|
||||
// GL thread at 13% of the frame, the server's apply at 7%, and the server's io thread
|
||||
// 97% blocked in recv waiting for this side. So the missing time is on the CLIENT, and
|
||||
// this is the only per-frame wait the client is known to take - it blocks here until
|
||||
// the server acknowledges the swap of (serial - credit). With it, the io thread's send
|
||||
// side (is the socket full?) and the frame's own wall/CPU, the 80% has nowhere left to
|
||||
// hide: it is either this wait, the send, or the game.
|
||||
const auto paceStarted = std::chrono::steady_clock::now();
|
||||
const BarrierWaitScope waiting;
|
||||
Transport::SessionWait wait = Transport::SessionWait::TimedOut;
|
||||
for (;;) {
|
||||
@@ -2286,6 +2297,17 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// ShutDown is teardown and returns: the present did not happen, the same answer
|
||||
// the barrier gives on a dead doorbell.
|
||||
if (wait == Transport::SessionWait::Reached) DrainEventRing(*m_events);
|
||||
const double creditMs =
|
||||
static_cast<double>(std::chrono::duration_cast<std::chrono::nanoseconds>(
|
||||
std::chrono::steady_clock::now() - paceStarted).count()) / 1e6;
|
||||
const auto io = Transport::StreamLink::TakeSendStats();
|
||||
MGLOG_I("P65ClientPace present=%llu credit_wait_ms=%.1f | io: %llu B in %llu sendmsg "
|
||||
"send_ms=%.1f total_ms=%.1f",
|
||||
static_cast<unsigned long long>(serial), creditMs,
|
||||
static_cast<unsigned long long>(io.bytes),
|
||||
static_cast<unsigned long long>(io.calls),
|
||||
static_cast<double>(io.nsInRecv) / 1e6,
|
||||
static_cast<double>(io.nsTotal) / 1e6);
|
||||
}
|
||||
m_presentsSent = serial;
|
||||
return serial;
|
||||
|
||||
@@ -54,6 +54,9 @@
|
||||
// P12: kDefaultReplySlotCount, because the create window's ceiling is derived from the pool it
|
||||
// shares rather than from a number written twice (see kCreateWindowMax below).
|
||||
#include <MG_Remote/Transport/ReplySlot.h>
|
||||
// P12: kRetainedAnswersMax, the store capacity the window's own ceiling is derived from - the
|
||||
// constant itself rather than a second literal, so the two cannot drift.
|
||||
#include <MG_Remote/Transport/StreamLink.h>
|
||||
// P12: LinkMetricsBeginReply/ReplyApplied, so the window's blocking take is visible in by_op 2.
|
||||
#include <MG_Remote/Transport/LinkMetrics.h>
|
||||
|
||||
@@ -392,18 +395,37 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// The gates that pin it are in MG_Test/Wire/RemoteClientControls.inc, and they still hold:
|
||||
// the window is a coherent mechanism, it is simply unusable on a frame with this much
|
||||
// other reply traffic.
|
||||
constexpr Uint32 kCreateWindowMax = 4;
|
||||
// THE WINDOW MAY NOT OWN MORE THAN HALF THE POOL, and this is where that stops being a
|
||||
// sentence in a comment. The pool is kDefaultReplySlotCount slots addressed seq % count, so
|
||||
// an answer is readable only while its slot has not been reused; every other reply-owning
|
||||
// row - fence, query, and the blocking rows themselves - needs a slot too. The change the
|
||||
// handoff names as the one worth pinning ("someone raises kCreateWindowMax to 8 and steps on
|
||||
// the pool") goes red HERE, by name, at compile time, instead of as a wrong picture on a
|
||||
// device. Raised to 8 it would take the whole pool and leave nothing for the answer the
|
||||
// blocking rows are waiting on.
|
||||
static_assert(kCreateWindowMax <= Transport::kDefaultReplySlotCount / 2,
|
||||
"the create window may hold at most half the reply pool: the other half "
|
||||
"belongs to the rows that take their answers immediately");
|
||||
// (THE NARRATIVE ABOVE PREDATES RETENTION and ends at "the default is 1". It is kept as
|
||||
// the record of what was tried; what made the window work is the declared-answer store,
|
||||
// and the default is 2 - Config.h carries the device numbers.)
|
||||
constexpr Uint32 kCreateWindowMax = 12;
|
||||
// THE POOL ARGUMENT THAT USED TO PIN THIS AT 4 IS GONE, AND WHY IT IS GONE IS THE ONE
|
||||
// THING TO RE-CHECK BEFORE RAISING IT AGAIN. It read: the pool is kDefaultReplySlotCount
|
||||
// slots addressed seq % count, so an answer is readable only while its slot has not been
|
||||
// reused, and a window owning half of them would starve the rows that take their answers
|
||||
// immediately - the change the handoff named as the one worth pinning, "someone raises
|
||||
// kCreateWindowMax to 8 and steps on the pool". That is true of a link whose answers live
|
||||
// in a SLOT POOL, and the window does not run on one: CreateWindowEffective() is gated on
|
||||
// ILink::RetainsReplies, which is false for ShmLink, so on the pool-addressed transport
|
||||
// the window is 1 and holds nothing. On the link where it does run - StreamLink - an
|
||||
// answer is kept in the declared-answer store until its reader takes it, so depth costs
|
||||
// STORE ENTRIES, not pool slots.
|
||||
//
|
||||
// SO THE SAME RULE IS RE-DERIVED AGAINST THE RIGHT OBJECT: at most half of StreamLink's
|
||||
// kRetainedCap = 24, the other half being the one blocking row in flight and whatever a
|
||||
// retired window has stranded. The store's overflow is a named Fatal and never an
|
||||
// eviction, so exceeding this costs a run rather than a wrong picture - and the store
|
||||
// logs its own peak as it crosses, which is what a future raise must be argued from.
|
||||
//
|
||||
// MEASURED, device load frame, three interleaved runs per arm: the curve had NOT flattened
|
||||
// at the old cap (1->2 saved 10.29 s, 2->3 3.74 s, 3->4 1.83 s), which is why this
|
||||
// constant moved at all.
|
||||
static_assert(kCreateWindowMax <= Transport::StreamLink::kRetainedAnswersMax / 2,
|
||||
"the create window may hold at most half of StreamLink's declared-answer "
|
||||
"store: the other half belongs to the one blocking row in flight and to "
|
||||
"whatever a retired window has stranded. The store's overflow is a named "
|
||||
"Fatal rather than an eviction, so this bound is about losing a run, not "
|
||||
"about a wrong picture");
|
||||
// AND THE ENVIRONMENT MAY NOT RAISE IT EITHER. The configured value is clamped to the
|
||||
// array's own size, so the fixed array below cannot be overrun by a config bound and this
|
||||
// constant having drifted apart. 1 (and anything below it) is the pre-window shape.
|
||||
@@ -429,6 +451,32 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// answers from existing at all - the second is what a no-reply bit keyed on wantReply
|
||||
// would produce, and it would leave every other reading green.
|
||||
Uint64 g_createWindowTaken = 0;
|
||||
// P12 MEASUREMENT: WHY A CREATE DOES NOT DEFER. The window's own arithmetic says a frame's
|
||||
// blocking takes should be creates/window (4,536/6 = 756 at the shipped default), and the
|
||||
// device measures 1,155 - a CONSTANT ~400 that does not move when the window deepens (at
|
||||
// window 8 the ideal is 567 and the measurement is 859). A constant is a fixed population
|
||||
// rather than a queueing effect, and the four exits below are the only ways into it, so
|
||||
// counting them by name is what says which one it is. It matters because the frame turned
|
||||
// out to be LATENCY-bound, not throughput-bound (the server applies all 103,097 records in
|
||||
// 1.95 s of a 9.64 s frame, and the client's 1,161 blocking takes at ~4.9 ms are 5.7 s of
|
||||
// it) - so the way to make it faster is fewer blocks, not a faster apply.
|
||||
Uint64 g_blockNoWindow = 0, g_blockSuspectsFull = 0, g_blockSuspectObject = 0,
|
||||
g_blockWindowStillFull = 0, g_blockTotal = 0;
|
||||
void NoteCreateBlocked(Uint64& reason) {
|
||||
++reason;
|
||||
++g_blockTotal;
|
||||
// Once per 256 blocks: a handful of lines per load frame, and the composition is
|
||||
// readable from any one of them.
|
||||
if (g_blockTotal % 256 == 0) {
|
||||
MGLOG_I("P12 create blocks: total=%llu no-window=%llu suspects-full=%llu "
|
||||
"suspect-object=%llu window-still-full=%llu",
|
||||
static_cast<unsigned long long>(g_blockTotal),
|
||||
static_cast<unsigned long long>(g_blockNoWindow),
|
||||
static_cast<unsigned long long>(g_blockSuspectsFull),
|
||||
static_cast<unsigned long long>(g_blockSuspectObject),
|
||||
static_cast<unsigned long long>(g_blockWindowStillFull));
|
||||
}
|
||||
}
|
||||
|
||||
// A HANDLE WHOSE CREATE CAME BACK DECLINED IS NEVER WINDOWED AGAIN. Without this the window
|
||||
// would UNDO the repair it exists to keep: every later use of the object sends another create,
|
||||
@@ -602,6 +650,13 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// all four take the blocking branch, which is the pre-window shape - and on the device
|
||||
// this is the branch the whole row takes, because the deferral above it cannot be
|
||||
// collected. See the section comment for the measurement and the arithmetic.
|
||||
if (window <= 1) {
|
||||
NoteCreateBlocked(g_blockNoWindow);
|
||||
} else if (g_createSuspects.size() >= kCreateSuspectMax) {
|
||||
NoteCreateBlocked(g_blockSuspectsFull);
|
||||
} else if (CreateIsSuspect(payload->Resource)) {
|
||||
NoteCreateBlocked(g_blockSuspectObject);
|
||||
}
|
||||
if (window <= 1 || g_createSuspects.size() >= kCreateSuspectMax ||
|
||||
CreateIsSuspect(payload->Resource)) {
|
||||
// THE WINDOW RETIRES HERE, AND IT MAY BE HOLDING ANSWERS (P12). This branch is
|
||||
@@ -628,6 +683,7 @@ namespace MobileGL::MG_Remote::Client {
|
||||
// against the effective window, which is what the array below is sized for, so
|
||||
// this line IS the overwrite guard.
|
||||
if (g_createWindowCount >= window) {
|
||||
NoteCreateBlocked(g_blockWindowStillFull);
|
||||
EmitCreateBlocking(session, payload, reply);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
|
||||
#include "ServerLoop.h"
|
||||
#include <MG_Remote/FatalFunnel.h>
|
||||
#include <MG_Remote/Transport/StreamLink.h>
|
||||
#include <MG_Backend/MGPipe/PipeInputs.h>
|
||||
|
||||
#include <Config.h>
|
||||
@@ -739,6 +740,30 @@ namespace MobileGL::MG_Remote::Server {
|
||||
ServerSession& session = *m_session;
|
||||
Transport::SessionConsumer& consumer = session.Consumer();
|
||||
PipeApplier& applier = session.Applier();
|
||||
// ---- P65ServerFrame: WHERE THE SERVER'S TIME GOES (P12 measurement) -----------------
|
||||
//
|
||||
// THE CLIENT SIDE IS MEASURED OUT AND THE SERVER'S IS NOT MEASURED AT ALL. Every reading
|
||||
// this harness produces is the CLIENT's - wall_ns, wait_replies, client_thread_cpu_ns - and
|
||||
// they now add up to less than the frame: at window 6 the client burns 2.5 s of CPU and
|
||||
// 1,162 blocking waits, and removing 292 of those waits (window 8) moves the wall clock by
|
||||
// nothing. The same 101 MB over USB instead of WiFi moves it by nothing either. What is
|
||||
// left is the one process nobody has timed: the server applying 103,097 records.
|
||||
//
|
||||
// THE SPLIT IS THE POINT. apply_ms is the sum of the time spent INSIDE
|
||||
// PipeApplier::ApplyOne - the GL work, the decode, the reply posts - and wall_ms is Present
|
||||
// to Present on the apply thread. Their difference is everything that is NOT applying:
|
||||
// waiting on the ring for the next record, retiring, and the socket. So one line answers
|
||||
// "is the phone CPU-bound, or is it waiting for the client?" without a profiler, which is
|
||||
// the question the next optimisation has to be chosen from.
|
||||
//
|
||||
// FUNCTION-LOCAL STATIC BECAUSE THE FRAME DOES NOT FIT IN ONE DrainRing CALL: records
|
||||
// trickle in, so a frame spans many drains, and state that reset per drain would measure a
|
||||
// batch. There is exactly one apply thread (ServerLoopInstance's), which is what makes a
|
||||
// static safe here; the same reasoning the drain tally above uses.
|
||||
static auto sFrameWallStart = std::chrono::steady_clock::now();
|
||||
static std::uint64_t sFrameApplyNs = 0;
|
||||
static std::uint64_t sFrameRecords = 0;
|
||||
static std::uint64_t sFrameIndex = 0;
|
||||
Uint64 applied = 0;
|
||||
for (;;) {
|
||||
if (SessionLatched()) break;
|
||||
@@ -749,7 +774,50 @@ namespace MobileGL::MG_Remote::Server {
|
||||
bool corrupt = false;
|
||||
const bool popped = consumer.ApplyOne(
|
||||
[this, &applier](const Transport::RingRecordView& record) {
|
||||
const auto applyStarted = std::chrono::steady_clock::now();
|
||||
applier.ApplyOne(record);
|
||||
// P65ServerFrame: timed around the ONE call that is the server's work, and
|
||||
// summed across the drains a frame spans. See the block above DrainRing's loop.
|
||||
sFrameApplyNs += static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::nanoseconds>(
|
||||
std::chrono::steady_clock::now() - applyStarted)
|
||||
.count());
|
||||
++sFrameRecords;
|
||||
if (record.kind == static_cast<std::uint16_t>(MG_Pipe::MGPWireOp::Present)) {
|
||||
const auto now = std::chrono::steady_clock::now();
|
||||
const auto wallNs = static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::nanoseconds>(now - sFrameWallStart)
|
||||
.count());
|
||||
const double applyMs = static_cast<double>(sFrameApplyNs) / 1e6;
|
||||
const double wallMs = static_cast<double>(wallNs) / 1e6;
|
||||
// AND THE STAGE NOTHING ELSE COVERS: the io thread's socket reads, which
|
||||
// its own MGLOG_ lines cannot report (the phone's forwarded log carries
|
||||
// mgl-srv-apply and mgl-display-ser only). read_ms is wall time inside
|
||||
// recv - blocking included - so a frame that is 90% read_ms is a frame the
|
||||
// reader spent WAITING for bytes, and one that is mostly recv calls with
|
||||
// little read_ms is a reader that is copying.
|
||||
const auto read = Transport::StreamLink::TakeReadStats();
|
||||
MGLOG_I("P65ServerFrame frame=%llu records=%llu apply_ms=%.1f wall_ms=%.1f "
|
||||
"outside_ms=%.1f | read: %llu B in %llu recv(%llu reads) "
|
||||
"recv_ms=%.1f total_ms=%.1f (%.0f%% blocked, %.1f MB/s)",
|
||||
static_cast<unsigned long long>(++sFrameIndex),
|
||||
static_cast<unsigned long long>(sFrameRecords), applyMs, wallMs,
|
||||
wallMs - applyMs,
|
||||
static_cast<unsigned long long>(read.bytes),
|
||||
static_cast<unsigned long long>(read.calls),
|
||||
static_cast<unsigned long long>(read.reads),
|
||||
static_cast<double>(read.nsInRecv) / 1e6,
|
||||
static_cast<double>(read.nsTotal) / 1e6,
|
||||
read.nsTotal ? 100.0 * static_cast<double>(read.nsInRecv) /
|
||||
static_cast<double>(read.nsTotal)
|
||||
: 0.0,
|
||||
read.nsTotal ? static_cast<double>(read.bytes) /
|
||||
(static_cast<double>(read.nsTotal) / 1e9) / 1e6
|
||||
: 0.0);
|
||||
sFrameWallStart = now;
|
||||
sFrameApplyNs = 0;
|
||||
sFrameRecords = 0;
|
||||
}
|
||||
// THE TALLY MOVES HERE, INSIDE THE CALLBACK, AND NOT AFTER THE BATCH. s1's
|
||||
// SessionConsumer::ApplyOne publishes appliedSeq the instant this callback
|
||||
// returns, and publishing appliedSeq is what releases the client from the
|
||||
|
||||
@@ -14,6 +14,7 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
namespace {
|
||||
struct Window {
|
||||
std::uint64_t waitReplies = 0, samples = 0, replyNs = 0, stageBytes = 0;
|
||||
std::uint64_t transportWaitNs = 0;
|
||||
std::array<std::uint64_t, 32> histogram{};
|
||||
// One slot per wire op, plus a last slot for anything past the table. See
|
||||
// LinkMetricsMaxOps: the point is that the per-op numbers still SUM to waitReplies.
|
||||
@@ -92,14 +93,16 @@ namespace {
|
||||
}
|
||||
MGLOG_I("P65LinkMetrics kind=%s frame=%llu wait_replies=%llu rtt_samples=%llu "
|
||||
"rtt_mean_us=%.3f rtt_p50_upper_us=%llu rtt_p99_upper_us=%llu "
|
||||
"stage_bytes=%llu wall_ns=%llu client_thread_cpu_ns=%llu rtt_hist_us_pow2=%s",
|
||||
"stage_bytes=%llu wall_ns=%llu client_thread_cpu_ns=%llu "
|
||||
"transport_wait_ns=%llu rtt_hist_us_pow2=%s",
|
||||
kind, static_cast<unsigned long long>(metrics.frame),
|
||||
static_cast<unsigned long long>(w.waitReplies), static_cast<unsigned long long>(w.samples),
|
||||
w.samples == 0 ? 0.0 : double(w.replyNs) / double(w.samples) / 1000.0,
|
||||
static_cast<unsigned long long>(QuantileUpperUs(w, 50)),
|
||||
static_cast<unsigned long long>(QuantileUpperUs(w, 99)),
|
||||
static_cast<unsigned long long>(w.stageBytes), static_cast<unsigned long long>(wallNs),
|
||||
static_cast<unsigned long long>(cpuNs), histogram);
|
||||
static_cast<unsigned long long>(cpuNs),
|
||||
static_cast<unsigned long long>(w.transportWaitNs), histogram);
|
||||
// AND THE PER-OP SPLIT WITH EVERY FRAME. A whole-session total cannot name the rows behind
|
||||
// ONE frame, and one frame is the whole shape of this problem: the device run that
|
||||
// motivated this printed 43,232 of its 43,629 waits in a single atlas frame, and the exit
|
||||
@@ -167,6 +170,15 @@ void LinkMetricsStageBytes(std::uint64_t bytes) {
|
||||
if (!metrics.active) return;
|
||||
metrics.current.stageBytes += bytes; metrics.total.stageBytes += bytes;
|
||||
}
|
||||
std::uint64_t LinkMetricsBeginTransportWait() {
|
||||
return metrics.active ? ClockNs() : 0;
|
||||
}
|
||||
void LinkMetricsEndTransportWait(std::uint64_t startedNs) {
|
||||
if (!metrics.active || startedNs == 0) return;
|
||||
const auto elapsed = ClockNs() - startedNs;
|
||||
metrics.current.transportWaitNs += elapsed;
|
||||
metrics.total.transportWaitNs += elapsed;
|
||||
}
|
||||
void LinkMetricsPresent() {
|
||||
if (!metrics.active) return;
|
||||
++metrics.frame;
|
||||
|
||||
@@ -17,6 +17,11 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
std::uint64_t LinkMetricsBeginReply(bool wantsReply, std::uint32_t op);
|
||||
void LinkMetricsReplyApplied(std::uint64_t startedNs);
|
||||
void LinkMetricsStageBytes(std::uint64_t bytes);
|
||||
// Wall time spent in the producer's transport park (flush, spin and sleep).
|
||||
// The server can run concurrently, so this is an independently measured
|
||||
// duration, not a residual obtained by subtracting role CPU clocks.
|
||||
std::uint64_t LinkMetricsBeginTransportWait();
|
||||
void LinkMetricsEndTransportWait(std::uint64_t startedNs);
|
||||
|
||||
// THE SAME NUMBER THE PER-FRAME LINE PRINTS AS `wait_replies` (P65LinkMetrics), read back
|
||||
// in-process. P12 item B needs it as a GATE rather than as a log line: "the texture half no
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
#include "Ring.h"
|
||||
|
||||
#include "SessionRings.h"
|
||||
#include "LinkMetrics.h"
|
||||
#include "WireLog.h"
|
||||
|
||||
#include <MG_Util/Debug/Log.h>
|
||||
@@ -472,21 +473,27 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
if (!Valid() || m_selfBell == nullptr) {
|
||||
return SessionWait::TimedOut;
|
||||
}
|
||||
const auto transportStarted = LinkMetricsBeginTransportWait();
|
||||
// COUNTED BEFORE THE WAIT AND ONLY WHEN ONE REALLY HAPPENS - after the two refusals
|
||||
// above, which answer TimedOut without ever reaching the bell. See Waits()/Parks() in
|
||||
// the header for why the pair is the reading and either number alone is not.
|
||||
if (m_link && m_link->Flush() != MOBILEGL_OK) return SessionWait::ShutDown;
|
||||
if (m_link && m_link->Flush() != MOBILEGL_OK) {
|
||||
LinkMetricsEndTransportWait(transportStarted);
|
||||
return SessionWait::ShutDown;
|
||||
}
|
||||
m_waits.fetch_add(1, std::memory_order_relaxed);
|
||||
// &m_parks is the OTHER half of the pair, and it is passed rather than read off the
|
||||
// bell: the bell is this endpoint's and the encoder waits on it too, so its own
|
||||
// ParkEntries() is not a subset of the waits counted one line above. See Parks() in
|
||||
// SessionRings.h.
|
||||
if (m_selfBell->Wait(m_control->producerParked, ready, m_spinUs, timeoutMs, &m_parks)) {
|
||||
LinkMetricsEndTransportWait(transportStarted);
|
||||
return SessionWait::Reached;
|
||||
}
|
||||
// Wait == false && Dead() is "the session was shut down", and it is the
|
||||
// only thing that returns from a kWaitForever park. Anything else is the
|
||||
// deadline.
|
||||
LinkMetricsEndTransportWait(transportStarted);
|
||||
return m_selfBell->Dead() ? SessionWait::ShutDown : SessionWait::TimedOut;
|
||||
}
|
||||
|
||||
|
||||
@@ -5,6 +5,9 @@
|
||||
#include "Doorbell.h"
|
||||
#include "ReplySlot.h"
|
||||
#include "Framing.h"
|
||||
// P12: the declared-answer store's crossing log. Same include as LinkMetrics.cpp's, rather than
|
||||
// <Includes.h>, because this file is transport-layer and pulls nothing else from the tree.
|
||||
#include <MG_Util/Debug/Log.h>
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <chrono>
|
||||
@@ -24,6 +27,16 @@
|
||||
#endif
|
||||
|
||||
namespace MobileGL::MG_Remote::Transport {
|
||||
// P65Read's counters: written by the server's io thread, drained by its apply thread (see
|
||||
// StreamLink::TakeReadStats).
|
||||
static std::atomic<std::uint64_t> gReadBytes{0}, gReadCalls{0}, gReadReads{0};
|
||||
static std::atomic<std::uint64_t> gReadNsInRecv{0}, gReadNsTotal{0};
|
||||
// P65CLIENTPACE's send half: the io thread's side of the same question. A sendmsg that blocks
|
||||
// is the socket full, i.e. the WRITE side's back-pressure - and on a client whose frame is 80%
|
||||
// waiting, "is it waiting to hand bytes over, or waiting for the game to make them" is the
|
||||
// whole question. Time spent inside sendmsg, not calls made.
|
||||
static std::atomic<std::uint64_t> gSendBytes{0}, gSendCalls{0}, gSendNsInSend{0};
|
||||
static std::atomic<std::uint64_t> gSendNsTotal{0};
|
||||
namespace {
|
||||
constexpr std::uint64_t kDataEnvelopeBytes = 20;
|
||||
constexpr std::uint64_t kMaxDataPayload = kMaxFramePayloadSize - kDataEnvelopeBytes;
|
||||
@@ -138,7 +151,8 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
// cleared its dirty flags at EMISSION (PipeApply.h:298-304), so nobody re-sends those
|
||||
// texels. The capacity is sized so that cannot be reached, and reaching it is a bug that
|
||||
// says its own name.
|
||||
static constexpr std::size_t kRetainedCap = 24;
|
||||
// ONE DEFINITION, IN THE HEADER, because the client's window is bounded by this number.
|
||||
static constexpr std::size_t kRetainedCap = StreamLink::kRetainedAnswersMax;
|
||||
struct StoredReply {
|
||||
std::uint64_t seq = 0;
|
||||
std::int32_t status = 0;
|
||||
@@ -153,6 +167,10 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
std::size_t wantCursor = 0;
|
||||
std::vector<StoredReply> retained;
|
||||
std::uint64_t skippedUnwanted = 0, unansweredWanted = 0, retainedTotal = 0;
|
||||
// P12: the most declared answers this store ever held at once. The cap is an overflow
|
||||
// Fatal rather than an eviction, so the MAXIMUM is the number that says whether a given
|
||||
// window depth has room - a sample at teardown answers a different question.
|
||||
std::uint64_t retainedPeak = 0;
|
||||
std::uint64_t lastProgressSeq = 0;
|
||||
std::chrono::steady_clock::time_point lastProgress = std::chrono::steady_clock::now();
|
||||
LinkArena records{}, events{};
|
||||
@@ -179,9 +197,32 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
}
|
||||
bool Read(void* data, std::size_t size) {
|
||||
#if !defined(_WIN32)
|
||||
// P65Read: THE SERVER'S SOCKET-READ PATH, WHICH NOTHING MEASURED. The per-frame
|
||||
// clocks say the frame's time is NOT in the client's GL thread (13%) and NOT in
|
||||
// PipeApplier::ApplyOne (11%) - 76% of a 105-185 ms frame is outside both, and the
|
||||
// only stage between them is this loop. Two different worlds look identical from
|
||||
// outside it and need opposite fixes:
|
||||
// * few BIG recvs with little time blocked -> the reader is copying, i.e. one
|
||||
// thread's throughput is the ceiling;
|
||||
// * many SMALL recvs, most of the wall spent blocked -> the reader is STARVED and
|
||||
// the client (or the batching on its side) sets the pace.
|
||||
// Counted here rather than sampled from /proc because the question is per-call, not
|
||||
// per-thread. Printed every kReadMarkBytes so a frame's worth of traffic yields
|
||||
// several lines and no line can be lost to a quiet period.
|
||||
constexpr std::uint64_t kReadMarkBytes = 8ull * 1024ull * 1024ull;
|
||||
(void)kReadMarkBytes;
|
||||
const auto readStarted = std::chrono::steady_clock::now();
|
||||
std::uint64_t calls = 0;
|
||||
auto* p = static_cast<std::uint8_t*>(data);
|
||||
const auto wanted = size;
|
||||
while (size) {
|
||||
const auto recvStarted = std::chrono::steady_clock::now();
|
||||
const auto n = ::recv(fd, p, size, 0);
|
||||
gReadNsInRecv.fetch_add(static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::nanoseconds>(
|
||||
std::chrono::steady_clock::now() - recvStarted).count()),
|
||||
std::memory_order_relaxed);
|
||||
++calls;
|
||||
if (n < 0 && errno == EINTR) continue;
|
||||
if (n <= 0) {
|
||||
Die(!stopping.load(std::memory_order_acquire));
|
||||
@@ -190,6 +231,13 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
p += n;
|
||||
size -= static_cast<std::size_t>(n);
|
||||
}
|
||||
gReadCalls.fetch_add(calls, std::memory_order_relaxed);
|
||||
gReadReads.fetch_add(1, std::memory_order_relaxed);
|
||||
gReadBytes.fetch_add(wanted, std::memory_order_relaxed);
|
||||
gReadNsTotal.fetch_add(static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::nanoseconds>(
|
||||
std::chrono::steady_clock::now() - readStarted).count()),
|
||||
std::memory_order_relaxed);
|
||||
return true;
|
||||
#else
|
||||
(void)data;
|
||||
@@ -213,8 +261,16 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
msghdr message{};
|
||||
message.msg_iov = vectors;
|
||||
message.msg_iovlen = size ? 2 : 1;
|
||||
const auto sendStarted = std::chrono::steady_clock::now();
|
||||
std::uint64_t sendCalls = 0;
|
||||
while (message.msg_iovlen) {
|
||||
const auto syscallStarted = std::chrono::steady_clock::now();
|
||||
const auto sent = ::sendmsg(fd, &message, MSG_NOSIGNAL);
|
||||
gSendNsInSend.fetch_add(static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::nanoseconds>(
|
||||
std::chrono::steady_clock::now() - syscallStarted).count()),
|
||||
std::memory_order_relaxed);
|
||||
++sendCalls;
|
||||
if (sent < 0 && errno == EINTR) continue;
|
||||
if (sent <= 0) {
|
||||
Die(!stopping.load(std::memory_order_acquire));
|
||||
@@ -231,6 +287,12 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
message.msg_iov->iov_len -= consumed;
|
||||
}
|
||||
}
|
||||
gSendCalls.fetch_add(sendCalls, std::memory_order_relaxed);
|
||||
gSendBytes.fetch_add(size, std::memory_order_relaxed);
|
||||
gSendNsTotal.fetch_add(static_cast<std::uint64_t>(
|
||||
std::chrono::duration_cast<std::chrono::nanoseconds>(
|
||||
std::chrono::steady_clock::now() - sendStarted).count()),
|
||||
std::memory_order_relaxed);
|
||||
return true;
|
||||
#else
|
||||
(void)payload;
|
||||
@@ -391,6 +453,19 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
slot.status = static_cast<std::int32_t>(static_cast<std::uint32_t>(b));
|
||||
slot.bytes.assign(payload, payload + size);
|
||||
++retainedTotal;
|
||||
// LOGGED AS IT CROSSES EVERY QUARTER OF THE CAP, not sampled: at most
|
||||
// four lines per run, and they are the ones a "how deep may the window
|
||||
// go" question is answered from without a rebuild (P12's
|
||||
// MOBILEGL_IPC_CREATE_WINDOW sweep reads exactly these).
|
||||
if (retained.size() > retainedPeak) {
|
||||
retainedPeak = retained.size();
|
||||
if (retainedPeak % (kRetainedCap / 4) == 0) {
|
||||
MGLOG_I("P12 store: peak=%llu of cap=%llu declared answers "
|
||||
"outstanding; the window's depth is what is spent here",
|
||||
static_cast<unsigned long long>(retainedPeak),
|
||||
static_cast<unsigned long long>(kRetainedCap));
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// NOBODY DECLARED IT: drop it. This is what keeps the store bounded
|
||||
// against a server that answers records this client never reads.
|
||||
@@ -404,6 +479,25 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
}
|
||||
};
|
||||
|
||||
StreamLink::ReadStats StreamLink::TakeSendStats() {
|
||||
ReadStats out;
|
||||
out.bytes = gSendBytes.exchange(0, std::memory_order_relaxed);
|
||||
out.calls = gSendCalls.exchange(0, std::memory_order_relaxed);
|
||||
out.nsInRecv = gSendNsInSend.exchange(0, std::memory_order_relaxed);
|
||||
out.nsTotal = gSendNsTotal.exchange(0, std::memory_order_relaxed);
|
||||
return out;
|
||||
}
|
||||
|
||||
StreamLink::ReadStats StreamLink::TakeReadStats() {
|
||||
ReadStats out;
|
||||
out.bytes = gReadBytes.exchange(0, std::memory_order_relaxed);
|
||||
out.calls = gReadCalls.exchange(0, std::memory_order_relaxed);
|
||||
out.reads = gReadReads.exchange(0, std::memory_order_relaxed);
|
||||
out.nsInRecv = gReadNsInRecv.exchange(0, std::memory_order_relaxed);
|
||||
out.nsTotal = gReadNsTotal.exchange(0, std::memory_order_relaxed);
|
||||
return out;
|
||||
}
|
||||
|
||||
StreamLink::StreamLink() : m_impl(new Impl) {}
|
||||
StreamLink::~StreamLink() {
|
||||
Detach();
|
||||
|
||||
@@ -11,6 +11,23 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
public:
|
||||
StreamLink();
|
||||
~StreamLink() override;
|
||||
// P65READ: THE SERVER'S SOCKET-READ PATH, ACCOUNTED WHERE IT CAN BE SEEN. The counters are
|
||||
// published from the io thread's recv loop and read by the APPLY thread, because the io
|
||||
// thread's own MGLOG_ lines never reach the phone's forwarded log (only mgl-srv-apply and
|
||||
// mgl-display-ser do - measured, not assumed). Taking them RESETS them, so one call is one
|
||||
// frame's worth of the stage nothing else measures: the frame's time is not the client's GL
|
||||
// thread (13%) and not PipeApplier::ApplyOne (11%), and this is what sits between them.
|
||||
struct ReadStats {
|
||||
std::uint64_t bytes = 0; // payload bytes taken off the socket
|
||||
std::uint64_t calls = 0; // recv() calls it took
|
||||
std::uint64_t reads = 0; // Link::Read() calls (one per header, envelope or chunk)
|
||||
std::uint64_t nsInRecv = 0; // wall time inside recv - blocking included
|
||||
std::uint64_t nsTotal = 0; // wall time inside Read
|
||||
};
|
||||
static ReadStats TakeReadStats();
|
||||
// The client's half: bytes handed to the io thread's sendmsg, the calls it took, and the
|
||||
// wall time inside sendmsg (blocking included). nsInRecv carries the send time.
|
||||
static ReadStats TakeSendStats();
|
||||
MobileGLResult Attach(int dataFd, SessionSegments& mirrors, TransportRoleTag role);
|
||||
MobileGLResult AttachOwned(int dataFd, const struct SessionSegmentSizes& sizes, TransportRoleTag role);
|
||||
// PH-7 (4), ID-P7-3. The SERVER's half of a nonce-bound data connection, in two steps.
|
||||
@@ -38,6 +55,14 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
MobileGLResult ReadReply(std::uint64_t, std::int32_t*, const void**, std::uint64_t*) override;
|
||||
void DeclareReplyRead(std::uint64_t) override;
|
||||
|
||||
// THE DECLARED-ANSWER STORE'S CAPACITY, and it is a PUBLIC NUMBER rather than a private
|
||||
// one because the client's create window is bounded by it: WireTables.cpp's
|
||||
// kCreateWindowMax static_assert and RemoteClientControls' ceiling case both read it, and
|
||||
// a bound that is restated in prose in three places is a bound that drifts. The argument
|
||||
// for the VALUE (why 24, and why overflow is a Fatal and never an eviction) is in the
|
||||
// .cpp beside the store it sizes.
|
||||
static constexpr std::size_t kRetainedAnswersMax = 24;
|
||||
|
||||
// P12 diagnostics for the declared-answer store. RetainedOutstanding is what the capacity
|
||||
// is about (it must stay near the window's depth, not grow with the server's answers);
|
||||
// RetainedTotal counts the answers kept for their readers; SkippedUnwanted counts the ones
|
||||
|
||||
@@ -1232,24 +1232,27 @@ TEST(RemoteClientControls, TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers
|
||||
Uint8 pixels[4 * 4 * 4];
|
||||
std::memset(pixels, 0x21, sizeof(pixels));
|
||||
// ---- the clamp and the ceiling, before a single record goes out ----
|
||||
// THE WINDOW IS OPT-IN AND THE DEFAULT IS OFF (Config.h: the deferral was measured on the
|
||||
// device and the drain cannot collect it), so both arms of this case SET the knob rather
|
||||
// than inheriting it - the ON half below is the mechanism being exercised, not a claim
|
||||
// about what ships.
|
||||
// BOTH ARMS SET THE KNOB rather than inheriting it: this case is about what the code does
|
||||
// with a configured window, and not a claim about which value ships (the default is 2 -
|
||||
// Config.h carries the device numbers).
|
||||
// WHAT THE OPERATOR WROTE IS NOT WHAT THE CODE MAY USE. With the clamp gone a configured
|
||||
// window of 99 is compared against a count that the ring keeps below its own size, so the
|
||||
// fallback never fires and the push index runs off the array.
|
||||
MG_Config::Ipc.CreateWindow = 99;
|
||||
if (ClientCreateWindowEffective() != ClientCreateWindowArraySize()) ::_exit(170);
|
||||
MG_Config::Ipc.CreateWindow = 4;
|
||||
if (ClientCreateWindowEffective() != 4u) ::_exit(171);
|
||||
// AND THE ARRAY IS SIZED FROM THE POOL IT SHARES, not from a second literal: the pool is
|
||||
// Transport::kDefaultReplySlotCount slots addressed seq % count, and every reply-owning row
|
||||
// (fence, query, the blocking rows themselves) needs one. Raising the window to the whole
|
||||
// pool is the change the handoff names, and it is red here as well as at the static_assert
|
||||
// in WireTables.cpp - deliberately twice, because the compile-time one is the guard and
|
||||
// this one is the statement of what the guard is for.
|
||||
if (ClientCreateWindowArraySize() > Transport::kDefaultReplySlotCount / 2) ::_exit(172);
|
||||
MG_Config::Ipc.CreateWindow = 8;
|
||||
if (ClientCreateWindowEffective() != 8u) ::_exit(171);
|
||||
// AND THE ARRAY IS SIZED FROM THE STORE IT SHARES, not from a second literal (P12). This
|
||||
// line used to read "at most half the REPLY POOL", which was true while an answer was read
|
||||
// back out of a slot pool addressed seq % slotCount - and the window does not run on such
|
||||
// a link: CreateWindowEffective() is gated on ILink::RetainsReplies, false for ShmLink, so
|
||||
// the pool-addressed transport holds a window of 1. On the link where it does run, an
|
||||
// answer waits in the declared-answer store instead, so the bound that matters is the
|
||||
// store's - read from the constant rather than restated, for the reason the pool version
|
||||
// read kDefaultReplySlotCount. WireTables.cpp's static_assert is the compile-time half of
|
||||
// the same sentence, deliberately twice: that one is the guard, this one is what it is for.
|
||||
if (ClientCreateWindowArraySize() > Transport::StreamLink::kRetainedAnswersMax / 2)
|
||||
::_exit(172);
|
||||
|
||||
// TWELVE OBJECTS, ONE CREATE EACH, AND THE INVARIANT READ AFTER EVERY ONE. This is the
|
||||
// reading a silent overwrite cannot fake: the count stays at the ceiling while an entry is
|
||||
@@ -1268,7 +1271,7 @@ TEST(RemoteClientControls, TheWindowRefusesToDeferOnALinkThatCannotRetainAnswers
|
||||
};
|
||||
|
||||
// ---- THE KNOB IS AS HIGH AS IT GOES, AND THE ROW MUST STILL BLOCK ----
|
||||
MG_Config::Ipc.CreateWindow = 4;
|
||||
MG_Config::Ipc.CreateWindow = ClientCreateWindowArraySize();
|
||||
const Uint64 recordsBefore = Transport::LinkMetricsRecordsFor(createOp);
|
||||
const Uint64 waitsBefore = Transport::LinkMetricsReplyWaitsFor(createOp);
|
||||
burst(12);
|
||||
|
||||
@@ -37,6 +37,9 @@
|
||||
#include <MG_Remote/Client/CapsMirror.h>
|
||||
#include <MG_Remote/Transport/ReplySlot.h>
|
||||
#include <MG_Remote/Transport/LinkMetrics.h>
|
||||
// P12: StreamLink::kRetainedAnswersMax, the store capacity RemoteClientControls' window-ceiling
|
||||
// case reads instead of restating it.
|
||||
#include <MG_Remote/Transport/StreamLink.h>
|
||||
#include <MG_Pipe/PipeRoute.h>
|
||||
#include <MG_Remote/Client/EmitTables.h>
|
||||
#include <MG_Remote/Client/WireTables.h>
|
||||
|
||||
@@ -262,7 +262,11 @@ namespace {
|
||||
|
||||
TEST_F(StreamClientWindow, EveryDeferredCreateAnswerComesBackOnTheShippingArm) {
|
||||
const Uint32 savedWindow = MG_Config::Ipc.CreateWindow;
|
||||
MG_Config::Ipc.CreateWindow = 4;
|
||||
// THE DEEPEST WINDOW THE CODE SUPPORTS, so the case exercises the ceiling rather than a
|
||||
// comfortable middle: more deferred answers outstanding means more of the store spent, and
|
||||
// the crowding below is what has to survive it. (The CAP itself is pinned by
|
||||
// RemoteClientControls' window-ceiling case; this one pins that the answers come back.)
|
||||
MG_Config::Ipc.CreateWindow = 8;
|
||||
StartServo();
|
||||
const std::uint64_t takenBefore = Client::ClientCreateWindowTaken();
|
||||
constexpr Int32 kCreates = 24;
|
||||
@@ -331,7 +335,7 @@ namespace {
|
||||
EXPECT_LE(pending, Client::ClientCreateWindowEffective());
|
||||
// AND THE ROW REALLY IS DEFERRING HERE, so the case cannot pass by taking the blocking
|
||||
// path on every create (which is what a link the window refuses to run on looks like).
|
||||
EXPECT_EQ(Client::ClientCreateWindowEffective(), 4u);
|
||||
EXPECT_EQ(Client::ClientCreateWindowEffective(), 8u);
|
||||
EXPECT_TRUE(client.LinkRetainsReplies());
|
||||
MG_Config::Ipc.CreateWindow = savedWindow;
|
||||
}
|
||||
|
||||
@@ -563,6 +563,25 @@ by name (`NoServerDisplay`) and that service goes on serving.
|
||||
|
||||
## TCP credit and Stage measurements
|
||||
|
||||
For a paired frame-time breakdown, enable `MOBILEGL_PIPE_STATS=1` in both roles
|
||||
and keep their logs. The following reducer matches client and server present
|
||||
serials, drops the first (unpaired) server interval and one warmup frame, and
|
||||
reports avg/p50/p99 in milliseconds plus percentages of client present time:
|
||||
|
||||
```bash
|
||||
python3 tools/trace_replay/frame_stats.py \
|
||||
--client-log mobilegl.client.log --server-log mobilegl.server.log \
|
||||
--warmup-frames 1 --output frame-stats.json
|
||||
```
|
||||
|
||||
`transport_wait_ms` measures wall time in the client producer's transport park
|
||||
(flush, spin and sleep). It includes waiting for replies, present credit and
|
||||
ring space; it is not socket latency. Client and server CPU can overlap with
|
||||
each other and with a transport wait, so each percentage uses the client
|
||||
present interval as its denominator and the three percentages need not sum to
|
||||
100%. A missing frame is rejected. Percentiles use nearest rank over per-frame
|
||||
samples.
|
||||
|
||||
`benchmark_tcp_credits.py` runs the complete OpenRA and rd12 traces at credits
|
||||
1, 2, and 3, then selects matching client/server tail frame IDs (100 and 200
|
||||
frames respectively). Extract the unchanged trace files as
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Pair client/server present intervals and summarize their frame time.
|
||||
|
||||
MOBILEGL_PIPE_STATS=1 enables the per-present clocks. Frame IDs are matched
|
||||
exactly; the first server interval is invalid because it has no preceding
|
||||
present. Transport wait measures wall time inside the client's transport park
|
||||
(flush, spin and sleep). It can overlap with server CPU and is not network latency.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
FIELDS = re.compile(r"(\w+)=([^\s]+)")
|
||||
CLIENT_MARKER = "P65LinkMetrics "
|
||||
SERVER_MARKER = "P65ServerMetrics "
|
||||
|
||||
|
||||
def read_frames(path: Path, marker: str, *, server: bool) -> dict[int, dict]:
|
||||
rows = {}
|
||||
for line in path.read_text(encoding="utf-8", errors="replace").splitlines():
|
||||
if marker not in line:
|
||||
continue
|
||||
values = dict(FIELDS.findall(line.split(marker, 1)[1]))
|
||||
if server and values.get("valid") != "1":
|
||||
continue
|
||||
if not server and values.get("kind") != "frame":
|
||||
continue
|
||||
try:
|
||||
number = int(values["frame"])
|
||||
wall = int(values["wall_ns"])
|
||||
cpu = int(values["apply_thread_cpu_ns" if server else "client_thread_cpu_ns"])
|
||||
transport = 0 if server else int(values["transport_wait_ns"])
|
||||
except (KeyError, ValueError) as error:
|
||||
raise ValueError(f"{path}: malformed {marker.strip()} row") from error
|
||||
if number in rows:
|
||||
raise ValueError(f"{path}: duplicate frame {number}")
|
||||
# The wall and thread-CPU clocks are sampled separately. Their nanosecond
|
||||
# readings can differ slightly at a boundary even for a fully busy thread.
|
||||
tolerance = max(100_000, wall // 100)
|
||||
if (number <= 0 or wall <= 0 or cpu <= 0 or cpu > wall + tolerance or
|
||||
transport < 0 or transport > wall + tolerance):
|
||||
raise ValueError(f"{path}: invalid clocks in frame {number}")
|
||||
rows[number] = {"wall_ns": wall, "cpu_ns": cpu, "transport_ns": transport}
|
||||
if not rows:
|
||||
raise ValueError(f"{path}: no usable {marker.strip()} frames")
|
||||
return rows
|
||||
|
||||
|
||||
def quantile_ns(samples: list[int], percentile: int) -> int:
|
||||
ordered = sorted(samples)
|
||||
return ordered[max(0, math.ceil(len(ordered) * percentile / 100) - 1)]
|
||||
|
||||
|
||||
def distribution(samples: list[int]) -> dict[str, float]:
|
||||
return {"avg": sum(samples) / len(samples) / 1e6,
|
||||
"p50": quantile_ns(samples, 50) / 1e6,
|
||||
"p99": quantile_ns(samples, 99) / 1e6}
|
||||
|
||||
|
||||
def summarize(client_path: Path, server_path: Path, *, warmup: int = 0,
|
||||
tail: int | None = None) -> dict:
|
||||
client = read_frames(client_path, CLIENT_MARKER, server=False)
|
||||
server = read_frames(server_path, SERVER_MARKER, server=True)
|
||||
selected = sorted(number for number in client if number > warmup)
|
||||
if tail is not None:
|
||||
selected = selected[-tail:]
|
||||
if not selected:
|
||||
raise ValueError("no frames remain after warmup/tail selection")
|
||||
if selected != list(range(selected[0], selected[-1] + 1)):
|
||||
raise ValueError("selected client frame IDs are not contiguous")
|
||||
missing = [number for number in selected if number not in server]
|
||||
if missing:
|
||||
raise ValueError(f"server is missing matched frames: {missing[:10]}")
|
||||
|
||||
columns = {name: [] for name in ("client_present_ms", "server_present_ms",
|
||||
"client_cpu_ms", "transport_wait_ms", "server_cpu_ms")}
|
||||
for number in selected:
|
||||
c, s = client[number], server[number]
|
||||
for key, value in (("client_present_ms", c["wall_ns"]),
|
||||
("server_present_ms", s["wall_ns"]),
|
||||
("client_cpu_ms", c["cpu_ns"]),
|
||||
("transport_wait_ms", c["transport_ns"]),
|
||||
("server_cpu_ms", s["cpu_ns"])):
|
||||
columns[key].append(value)
|
||||
|
||||
duration = sum(columns["client_present_ms"])
|
||||
shares = {name: 100 * sum(columns[name]) / duration
|
||||
for name in ("client_cpu_ms", "transport_wait_ms", "server_cpu_ms")}
|
||||
return {"frames": len(selected), "first_frame": selected[0],
|
||||
"last_frame": selected[-1], "warmup_frames": warmup,
|
||||
"definition": "transport_wait is client wall time in SessionProducer::Park "
|
||||
"(flush, spin and sleep); role CPU and transport wait can overlap",
|
||||
"percentile": "nearest rank", "unit": "ms",
|
||||
"stats": {key: distribution(values) for key, values in columns.items()},
|
||||
"share_percent_of_client_present": shares}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--client-log", type=Path, required=True)
|
||||
parser.add_argument("--server-log", type=Path, required=True)
|
||||
parser.add_argument("--warmup-frames", type=int, default=1)
|
||||
parser.add_argument("--tail-frames", type=int)
|
||||
parser.add_argument("--output", type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.warmup_frames < 0 or args.tail_frames is not None and args.tail_frames <= 0:
|
||||
parser.error("warmup must be nonnegative and tail must be positive")
|
||||
try:
|
||||
result = summarize(args.client_log, args.server_log, warmup=args.warmup_frames,
|
||||
tail=args.tail_frames)
|
||||
except (OSError, ValueError) as error:
|
||||
parser.error(str(error))
|
||||
rendered = json.dumps(result, ensure_ascii=False, indent=2) + "\n"
|
||||
if args.output:
|
||||
args.output.write_text(rendered, encoding="utf-8")
|
||||
else:
|
||||
print(rendered, end="")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Small fixture checks for frame ID pairing and additive frame attribution."""
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from frame_stats import summarize
|
||||
|
||||
|
||||
class FrameStatsTest(unittest.TestCase):
|
||||
def logs(self, root, *, missing_server=False, overlapping=False):
|
||||
client = root / "client.log"
|
||||
server = root / "server.log"
|
||||
client.write_text("\n".join(
|
||||
f"P65LinkMetrics kind=frame frame={n} wall_ns={wall} "
|
||||
f"client_thread_cpu_ns={cpu} transport_wait_ns={wait}"
|
||||
for n, wall, cpu, wait in ((1, 10000000, 2000000, 1000000),
|
||||
(2, 20000000, 5000000, 4000000),
|
||||
(3, 40000000, 10000000, 7000000))) + "\n")
|
||||
server.write_text("\n".join(
|
||||
f"P65ServerMetrics frame={n} valid=1 wall_ns={wall} "
|
||||
f"apply_thread_cpu_ns={cpu}"
|
||||
for n, wall, cpu in ((2, 18000000, 17000000 if overlapping else 3000000),
|
||||
(3, 35000000, 31000000 if overlapping else 6000000))
|
||||
if not missing_server or n != 3) + "\n")
|
||||
return client, server
|
||||
|
||||
def test_paired_percentiles_and_shares(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
result = summarize(*self.logs(Path(directory)), warmup=1)
|
||||
self.assertEqual((result["first_frame"], result["last_frame"], result["frames"]),
|
||||
(2, 3, 2))
|
||||
self.assertEqual(result["stats"]["transport_wait_ms"],
|
||||
{"avg": 5.5, "p50": 4.0, "p99": 7.0})
|
||||
self.assertEqual(result["share_percent_of_client_present"]["transport_wait_ms"],
|
||||
100 * 11000000 / 60000000)
|
||||
|
||||
def test_missing_server_frame_is_error(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
with self.assertRaisesRegex(ValueError, "missing matched frames"):
|
||||
summarize(*self.logs(Path(directory), missing_server=True), warmup=1)
|
||||
|
||||
def test_parallel_cpu_is_allowed(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
result = summarize(*self.logs(Path(directory), overlapping=True), warmup=1)
|
||||
self.assertGreater(sum(result["share_percent_of_client_present"].values()), 100)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user