From 68b55ea37114323b1afab9af2d0c8285ca6bb314 Mon Sep 17 00:00:00 2001 From: Swung0x48 Date: Mon, 21 Sep 2026 02:54:47 -0400 Subject: [PATCH 1/2] [MG_Remote] (Disaggregated): the inproc barrier watchdog is a deadlock detector, so its budget leaves room for a leg lavapipe/llvmpipe stall the apply thread for tens of seconds inside texture-handle registration (vkCreateImageView -> lvp_CreateImageView -> llvmpipe_register_texture, a JIT/futex path): one measured stall is 29.3-39.7 s and a whole inproc retrace run accumulates ~44.6 s of it. At a 30 s bound those runs red intermittently on Fatal{BarrierTimeout, "ResourceRespecify"} while the same stall under monolith is merely slow - monolith has no cap on that path. Since the monolith shares the stall, the bound, not the workload, was the defect. kBarrierTimeoutMs becomes 120000, >3x the largest observed stall, which still leaves the watchdog job: a record that is never retired still terminates the run and the Fatal still names the seq. The comment now says deadlock detector rather than performance gate and carries the measurement. The fake-clock case that pins the budget moves with it: FenceWaitBudgetHonorsGl- TimeoutAndFiniteTransportChunks expected 30000/30001/90000 and now expects 120000/120001/180000, so the assertion still pins the value instead of drifting. Texts that wrote the old number down are updated: ARCHITECTURE 17.1, CONTRACT-P5E 2.4 (which now names kBarrierTimeoutMs rather than the literal), and the two bring-up comments in PipeRoute.h / WireTables.cpp that dated a self-wait to "thirty seconds". --- MobileGL/MG_Pipe/PipeRoute.h | 2 +- MobileGL/MG_Remote/CONTRACT-P5E.md | 2 +- MobileGL/MG_Remote/Client/ClientSession.cpp | 21 +++++++++++++------ MobileGL/MG_Remote/Client/WireTables.cpp | 2 +- MobileGL/MG_Remote/Transport/Doorbell.h | 2 +- .../MG_Test/Wire/RemoteClientControls.inc | 13 +++++++----- docs/Disaggregated/ARCHITECTURE.md | 2 +- 7 files changed, 28 insertions(+), 16 deletions(-) diff --git a/MobileGL/MG_Pipe/PipeRoute.h b/MobileGL/MG_Pipe/PipeRoute.h index 1eceb59e..42d186e8 100644 --- a/MobileGL/MG_Pipe/PipeRoute.h +++ b/MobileGL/MG_Pipe/PipeRoute.h @@ -195,7 +195,7 @@ namespace MobileGL::MG_Pipe { // reaches the very same `MG_Impl/Pipe` emitters the client does, and a wire emitter there // publishes a record and then waits for the apply thread to apply it. That thread IS the // apply thread, so it waits for itself: `Fatal{BarrierTimeout, "ResourceRespecify"}`, - // logged by `mgl-srv-apply`, thirty seconds after bring-up starts. + // logged by `mgl-srv-apply` one barrier budget after bring-up starts. // // So the client emitters ask "am I the server role right now?" and, if so, run the // monolith adapter - which is exactly what `PipeWireCodec` already does on the decode side diff --git a/MobileGL/MG_Remote/CONTRACT-P5E.md b/MobileGL/MG_Remote/CONTRACT-P5E.md index 406f7f24..cfb8f4a9 100644 --- a/MobileGL/MG_Remote/CONTRACT-P5E.md +++ b/MobileGL/MG_Remote/CONTRACT-P5E.md @@ -164,7 +164,7 @@ keeps its lockstep meaning (nothing but reply rows waits); under run-ahead it re ### 2.4 Present pacing `EmitPresent` (`EmitTables.cpp:977-1053`): `BeforeReadOnlyVerb(); if (m_presentsSent >= PresentCredit) -WaitForPresentAck(m_presentsSent + 1 - PresentCredit, 30 s)` — `ShutDown` returns, `TimedOut` is +WaitForPresentAck(m_presentsSent + 1 - PresentCredit, kBarrierTimeoutMs)` — `ShutDown` returns, `TimedOut` is `Fatal{PresentCreditTimeout}`; then `DrainEventRing`, encode, publish, `PumpControlPlane`. Credit 1 = the client publishes frame N+1 while the server applies and swaps frame N: one frame of overlap, at most one frame of added latency. Budget at ~850 draws + ~0.36 MB pmap/frame: SEG_CMD ~70 KB + diff --git a/MobileGL/MG_Remote/Client/ClientSession.cpp b/MobileGL/MG_Remote/Client/ClientSession.cpp index b029c854..243d24be 100644 --- a/MobileGL/MG_Remote/Client/ClientSession.cpp +++ b/MobileGL/MG_Remote/Client/ClientSession.cpp @@ -129,11 +129,20 @@ namespace MobileGL::MG_Remote::Client { BarrierWaitScope& operator=(const BarrierWaitScope&) = delete; }; - // BOUNDED, AND THE BOUND IS GENEROUS RATHER THAN TIGHT. The barrier is a correctness - // device, not a watchdog: a slow readback on a software rasterizer is a legitimate - // second-scale wait, while a lost record never completes at all. 30 s separates the two - // without turning a loaded CI machine into a red lane, and the Fatal names the seq. - constexpr Uint32 kBarrierTimeoutMs = 30000; + // A DEADLOCK DETECTOR, NOT A PERFORMANCE GATE. The barrier exists so that a record the + // applier never retires is reported instead of hanging: a lost record never completes at + // all, while a slow apply completes late. The bound only has to sit far above the slowest + // legitimate apply. + // + // lavapipe/llvmpipe have legitimate apply-thread stalls that reach the tens of seconds: + // texture-handle registration (vkCreateImageView -> lvp_CreateImageView -> + // llvmpipe_register_texture) blocks the apply thread on a JIT/futex path, measured at + // 29.3-39.7 s in a single stall, ~44.6 s cumulatively over one inproc retrace run. A 30 s + // bound therefore red-lanes those runs intermittently while the same stall on a monolith + // run is simply slow - the monolith has no equivalent cap on this path. 120 s is >3x the + // largest observed stall and keeps the watchdog's job: a record that is truly lost still + // terminates the run, and the Fatal names the seq. + constexpr Uint32 kBarrierTimeoutMs = 120000; Uint64 AppliedWaitBudgetMs(MG_Pipe::MGPWireOp op, const void* payload) { if (op != MG_Pipe::MGPWireOp::FenceWait) return kBarrierTimeoutMs; @@ -711,7 +720,7 @@ namespace MobileGL::MG_Remote::Client { // would go to a server this client has not been told consumes that family; // - after ServerLoop::Start (step 8), because EmitAndWait BLOCKS on appliedSeq and // with no apply thread nothing advances it: the first resource_create would spend - // 30 seconds in the barrier and then Fatal{BarrierTimeout}; + // the whole kBarrierTimeoutMs in the barrier and then Fatal{BarrierTimeout}; // - on THIS thread, the one that called MG_Backend::Init(), because it is the GL // thread and table 3 makes gPipeInputs its to touch while the barrier holds. // The publication is safe without a fence because the apply thread never reads these diff --git a/MobileGL/MG_Remote/Client/WireTables.cpp b/MobileGL/MG_Remote/Client/WireTables.cpp index 452378d9..f94c7f8f 100644 --- a/MobileGL/MG_Remote/Client/WireTables.cpp +++ b/MobileGL/MG_Remote/Client/WireTables.cpp @@ -61,7 +61,7 @@ namespace MobileGL::MG_Remote::Client { // apply thread running the server's own backend - the EGL bring-up, InitCapabilities, // the applier - reaches these very emitters. A record published there would be waited // for by the thread that is supposed to apply it: `Fatal{BarrierTimeout, - // "ResourceRespecify"}` from `mgl-srv-apply`, thirty seconds into bring-up, which is + // "ResourceRespecify"}` from `mgl-srv-apply` one barrier budget into bring-up, which is // exactly how this was found. // // THE ANSWER IS NOT "SUPPRESS THE RECORD" - it is "run the server's own code", because diff --git a/MobileGL/MG_Remote/Transport/Doorbell.h b/MobileGL/MG_Remote/Transport/Doorbell.h index 11992c07..1f31552b 100644 --- a/MobileGL/MG_Remote/Transport/Doorbell.h +++ b/MobileGL/MG_Remote/Transport/Doorbell.h @@ -249,7 +249,7 @@ namespace MobileGL::MG_Remote::Transport { // caller asked for is no longer charged against its timeout, so the // total is bounded by `timeoutMs + the spin budget` rather than by // `timeoutMs`. spinUs is 50 by default and the timeouts that reach - // here are milliseconds (the verb barrier's 30 s, the drain's bound), + // here are milliseconds (the verb barrier's budget, the drain's bound), // so the drift is under a part in a thousand - and erring LONG is the // safe direction for a deadline whose expiry is reported as a timeout. const auto deadline = timeoutMs == kWaitForever diff --git a/MobileGL/MG_Test/Wire/RemoteClientControls.inc b/MobileGL/MG_Test/Wire/RemoteClientControls.inc index ad6eade0..9dd747bb 100644 --- a/MobileGL/MG_Test/Wire/RemoteClientControls.inc +++ b/MobileGL/MG_Test/Wire/RemoteClientControls.inc @@ -218,13 +218,16 @@ TEST(RemoteClientControls, UnknownReadPixelsReplyStatusRefusesTightAndBounce) { } TEST(RemoteClientControls, FenceWaitBudgetHonorsGlTimeoutAndFiniteTransportChunks) { + // 120000 below is ClientSession.cpp's kBarrierTimeoutMs, the transport grace every CPU bound + // wait adds on top of the GL timeout. It was 30 s; lavapipe's texture-handle registration + // (vkCreateImageView -> llvmpipe_register_texture) exceeds that in a single apply stall. struct WaitCase { Uint64 timeoutNs; MGPWireOp op; Uint32 expectedParkMs; }; const WaitCase cases[] = { - {0, MGPWireOp::FenceWait, 30000}, - {1, MGPWireOp::FenceWait, 30001}, - {60000000000ull, MGPWireOp::FenceWait, 90000}, + {0, MGPWireOp::FenceWait, 120000}, + {1, MGPWireOp::FenceWait, 120001}, + {60000000000ull, MGPWireOp::FenceWait, 180000}, {~Uint64(0), MGPWireOp::FenceWait, Transport::kWaitForever - 1}, - {~Uint64(0), MGPWireOp::FenceWaitServer, 30000}, + {~Uint64(0), MGPWireOp::FenceWaitServer, 120000}, }; for (const auto test : cases) { const auto child = RunInChild([test] { @@ -245,7 +248,7 @@ TEST(RemoteClientControls, FenceWaitBudgetHonorsGlTimeoutAndFiniteTransportChunk producer.*PeerMember(ProducerBellTag{}) = bell.original; const auto observed = bell.firstMs.load(); // Doorbell converts the remaining deadline to whole milliseconds. Allow - // scheduler delay, but neither a 30s clamp nor an unbounded sentinel. + // scheduler delay, but neither the barrier clamp nor an unbounded sentinel. if (!bell.entered.load() || observed > test.expectedParkMs || observed < test.expectedParkMs - 100 || observed == Transport::kWaitForever) ::_exit(93); if (test.op == MGPWireOp::FenceWait && diff --git a/docs/Disaggregated/ARCHITECTURE.md b/docs/Disaggregated/ARCHITECTURE.md index 2f8e4ac7..1ab732f1 100644 --- a/docs/Disaggregated/ARCHITECTURE.md +++ b/docs/Disaggregated/ARCHITECTURE.md @@ -539,7 +539,7 @@ P5 的 `inproc` 是真第二线程，但还是 **lockstep**：每条 class-B ver 同一地址空间不得成为旁路（R-2）：encoder 把 `MGHostSpan::Ptr` 恒写成 `nullptr`，内容 blob 必须带真实 `SEG_STAGE` offset / 非零 size；decoder 对四种形状分别 `Fatal{ProtocolCorruption}`。`MapPersistent` 在 split 恒 decline；`MOBILEGL_IPC_AUDIT=1` 在 retire 后把 staging 填 `0xDD`，让跨 applier 返回持针的实现下一次读取时可见地失败（R-11：任何 widened read 先过 `RequireStagedCoverageForPendingRanges`，否则 `Fatal{StageSnapshotTooNarrow}`）。apply 角色不得进入 client 的 persistent-map producer（r1）。 -reply mailbox 以**记录序号作为 slot id**（R-3）：slot header 是 `{Seq, Status, Size}`，acceptance 的 Bool 答案与 `MapPersistent` decline 都在既有 verb wait 内读取；`Status=ERROR` 一律 `Fatal{ReplyError}`，未知 status 是 `Fatal{ReplyStatusInvalid}`；读到 reply 之前先由 `appliedSeq` 证明该记录已离开 applier（R-5）。P9 才把这套同步 mailbox 推广成异步池。等待预算：普通 verb / 容量等待 30 s（`Fatal{BarrierTimeout}`）；`ClientWaitSync` 的 applied/reply 预算 = ceil(timeout ns / 1e6) + 30 000 ms，有限 chunk 避开溢出。 +reply mailbox 以**记录序号作为 slot id**（R-3）：slot header 是 `{Seq, Status, Size}`，acceptance 的 Bool 答案与 `MapPersistent` decline 都在既有 verb wait 内读取；`Status=ERROR` 一律 `Fatal{ReplyError}`，未知 status 是 `Fatal{ReplyStatusInvalid}`；读到 reply 之前先由 `appliedSeq` 证明该记录已离开 applier（R-5）。P9 才把这套同步 mailbox 推广成异步池。等待预算：普通 verb / 容量等待 120 s（`Fatal{BarrierTimeout}`，`ClientSession.cpp` 的 `kBarrierTimeoutMs`）；`ClientWaitSync` 的 applied/reply 预算 = ceil(timeout ns / 1e6) + 120 000 ms，有限 chunk 避开溢出。这条看门狗只判**死锁**（记录永不退役），不是性能门：lavapipe/llvmpipe 在 texture-handle 注册（`vkCreateImageView` → `llvmpipe_register_texture`）上可合法阻塞 apply 线程 29.3–39.7 s，故预算从原 30 s 提到 120 s——monolith 走同一路径本就没有这个上限。 ### 17.2 71 槽的三类、caps 与 tight readback From 0689fc96370f9b935aff82a915066ca1938e6f4b Mon Sep 17 00:00:00 2001 From: Swung0x48 Date: Mon, 21 Sep 2026 03:15:21 -0400 Subject: [PATCH 2/2] [Docs] (Disaggregated): the P5b review row keeps its historical 30 s and points at the current 120 s MEASUREMENTS 7.3 records what 82683d4a did, so the 30 s in that row stays as written; a note under the table now says which value is live and where the reason for the change is argued, so a reader looking up the barrier budget by grep does not come away with the superseded number. --- docs/Disaggregated/MEASUREMENTS.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Disaggregated/MEASUREMENTS.md b/docs/Disaggregated/MEASUREMENTS.md index 6cc5f2e1..c693318c 100644 --- a/docs/Disaggregated/MEASUREMENTS.md +++ b/docs/Disaggregated/MEASUREMENTS.md @@ -403,11 +403,13 @@ G1 27,814 符号 0/0/0/0、`.text` 10,806,611 → 10,806,611；G5 两族 + pin | 项 | 问题 | 处置 | |---|---|---| | Major 1 | user-index span 在段内但 Size 可能短于 Count × IndexSize | `a021e3cc`：encoder / decoder / sink 共用 shape/extent gate（单 range、宽度 1/2/4、`Uint64(Count) × IndexSize ≤ Size`），三端短 span 均拒绝、段末 exact-fit 通过；定向 9/9 | -| Major 2 | `ClientWaitSync` 保留 64-bit timeout 但 applied/reply 等待一律 30 s | `82683d4a`：预算 = ceil(timeout ns / 1e6) + 30,000 ms，有限 chunk；普通容量等待 / `FenceWaitServer` 仍 30 s，shutdown 可唤醒 | +| Major 2 | `ClientWaitSync` 保留 64-bit timeout 但 applied/reply 等待一律 30 s | `82683d4a`：预算 = ceil(timeout ns / 1e6) + 30,000 ms，有限 chunk；普通容量等待 / `FenceWaitServer` 仍 30 s，shutdown 可唤醒。**该 30 s 常量后来在 CI 修复轮改成 120 s（`68b55ea3`），原因见本表下注** | | Minor 3 | ReadPixels exact-size reply 的未知 status 穿过 production helper | `82683d4a`：tight 与 bounce 路径拒绝 status=3，`Fatal{ReplyStatusInvalid}` | 定向 `RemoteClientTest` 7/7（0 ns、1 ns、60 s、UINT64_MAX 预算，`FenceWaitServer`、shutdown、未知状态）。合并后 quickgate 里 split 单元车道仅有的 2 个失败（`PipeWireCodecTest.UserIndexSpan*`）是测试日志捕获缺陷（库按进程截断日志，fork 子进程增量读读空），`7cb29d46` 修复后 3/3。本阶段不再做第二轮全门或审查；原始全门源头仍为 `348d22a4`，三项修复只以定向证据补齐。 +> 上表 Major 2 的 30 s 是 `82683d4a` 当时的值，保留为历史记录。**当前值是 120 s**（`68b55ea3`，`ClientSession.cpp` 的 `kBarrierTimeoutMs`），因为 lavapipe 在 texture-handle 注册（`vkCreateImageView` → `llvmpipe_register_texture`）上可合法阻塞 apply 线程 29.3–39.7 s，30 s 的看门狗会把合法慢路径间歇判红；详因见 `ARCHITECTURE.md` §17.1。`RemoteClientControls.FenceWaitBudgetHonorsGlTimeoutAndFiniteTransportChunks` 的期望值（30000/30001/90000）已同步改为 120000/120001/180000。 + ### 7.4 Redmi 出口与四臂 A/B（`82683d4a`，APK `p5bcodex2`） Redmi `2f7cbe2e`，证据根 `MobileGL/.trace-work/p5b-redmi/p5bcodex2/2f7cbe2e/`；全部组合显式 `MOBILEGL_IPC_STAGE_MB=256`。**钉频口径变更**：2026-09-11 的厂商 GPU 上限（1050 MHz）已消失，本次 deterministic pin 为 **1100 MHz**；与 1050 MHz 时代的活动不可比钟频，只有同场四臂配对可比。