[MG_Remote] (Disaggregated): the inproc barrier watchdog is a deadlock detector, so its budget leaves room for a leg

lavapipe/llvmpipe stall the apply thread for tens of seconds inside texture-handle
registration (vkCreateImageView -> lvp_CreateImageView -> llvmpipe_register_texture,
a JIT/futex path): one measured stall is 29.3-39.7 s and a whole inproc retrace run
accumulates ~44.6 s of it. At a 30 s bound those runs red intermittently on
Fatal{BarrierTimeout, "ResourceRespecify"} while the same stall under monolith is
merely slow - monolith has no cap on that path. Since the monolith shares the stall,
the bound, not the workload, was the defect.

kBarrierTimeoutMs becomes 120000, >3x the largest observed stall, which still leaves
the watchdog job: a record that is never retired still terminates the run and the
Fatal still names the seq. The comment now says deadlock detector rather than
performance gate and carries the measurement.

The fake-clock case that pins the budget moves with it: FenceWaitBudgetHonorsGl-
TimeoutAndFiniteTransportChunks expected 30000/30001/90000 and now expects
120000/120001/180000, so the assertion still pins the value instead of drifting.
Texts that wrote the old number down are updated: ARCHITECTURE 17.1, CONTRACT-P5E
2.4 (which now names kBarrierTimeoutMs rather than the literal), and the two
bring-up comments in PipeRoute.h / WireTables.cpp that dated a self-wait to
"thirty seconds".
This commit is contained in:
2026-09-21 02:54:47 -04:00
parent d56c0e8346
commit 68b55ea371
7 changed files with 28 additions and 16 deletions
+1 -1
View File
@@ -195,7 +195,7 @@ namespace MobileGL::MG_Pipe {
// reaches the very same `MG_Impl/Pipe` emitters the client does, and a wire emitter there
// publishes a record and then waits for the apply thread to apply it. That thread IS the
// apply thread, so it waits for itself: `Fatal{BarrierTimeout, "ResourceRespecify"}`,
// logged by `mgl-srv-apply`, thirty seconds after bring-up starts.
// logged by `mgl-srv-apply` one barrier budget after bring-up starts.
//
// So the client emitters ask "am I the server role right now?" and, if so, run the
// monolith adapter - which is exactly what `PipeWireCodec` already does on the decode side
+1 -1
View File
@@ -164,7 +164,7 @@ keeps its lockstep meaning (nothing but reply rows waits); under run-ahead it re
### 2.4 Present pacing
`EmitPresent` (`EmitTables.cpp:977-1053`): `BeforeReadOnlyVerb(); if (m_presentsSent >= PresentCredit)
WaitForPresentAck(m_presentsSent + 1 - PresentCredit, 30 s)` — `ShutDown` returns, `TimedOut` is
WaitForPresentAck(m_presentsSent + 1 - PresentCredit, kBarrierTimeoutMs)` — `ShutDown` returns, `TimedOut` is
`Fatal{PresentCreditTimeout}`; then `DrainEventRing`, encode, publish, `PumpControlPlane`. Credit 1
= the client publishes frame N+1 while the server applies and swaps frame N: one frame of overlap,
at most one frame of added latency. Budget at ~850 draws + ~0.36 MB pmap/frame: SEG_CMD ~70 KB +
+15 -6
View File
@@ -129,11 +129,20 @@ namespace MobileGL::MG_Remote::Client {
BarrierWaitScope& operator=(const BarrierWaitScope&) = delete;
};
// BOUNDED, AND THE BOUND IS GENEROUS RATHER THAN TIGHT. The barrier is a correctness
// device, not a watchdog: a slow readback on a software rasterizer is a legitimate
// second-scale wait, while a lost record never completes at all. 30 s separates the two
// without turning a loaded CI machine into a red lane, and the Fatal names the seq.
constexpr Uint32 kBarrierTimeoutMs = 30000;
// A DEADLOCK DETECTOR, NOT A PERFORMANCE GATE. The barrier exists so that a record the
// applier never retires is reported instead of hanging: a lost record never completes at
// all, while a slow apply completes late. The bound only has to sit far above the slowest
// legitimate apply.
//
// lavapipe/llvmpipe have legitimate apply-thread stalls that reach the tens of seconds:
// texture-handle registration (vkCreateImageView -> lvp_CreateImageView ->
// llvmpipe_register_texture) blocks the apply thread on a JIT/futex path, measured at
// 29.3-39.7 s in a single stall, ~44.6 s cumulatively over one inproc retrace run. A 30 s
// bound therefore red-lanes those runs intermittently while the same stall on a monolith
// run is simply slow - the monolith has no equivalent cap on this path. 120 s is >3x the
// largest observed stall and keeps the watchdog's job: a record that is truly lost still
// terminates the run, and the Fatal names the seq.
constexpr Uint32 kBarrierTimeoutMs = 120000;
Uint64 AppliedWaitBudgetMs(MG_Pipe::MGPWireOp op, const void* payload) {
if (op != MG_Pipe::MGPWireOp::FenceWait) return kBarrierTimeoutMs;
@@ -711,7 +720,7 @@ namespace MobileGL::MG_Remote::Client {
// would go to a server this client has not been told consumes that family;
// - after ServerLoop::Start (step 8), because EmitAndWait BLOCKS on appliedSeq and
// with no apply thread nothing advances it: the first resource_create would spend
// 30 seconds in the barrier and then Fatal{BarrierTimeout};
// the whole kBarrierTimeoutMs in the barrier and then Fatal{BarrierTimeout};
// - on THIS thread, the one that called MG_Backend::Init(), because it is the GL
// thread and table 3 makes gPipeInputs its to touch while the barrier holds.
// The publication is safe without a fence because the apply thread never reads these
+1 -1
View File
@@ -61,7 +61,7 @@ namespace MobileGL::MG_Remote::Client {
// apply thread running the server's own backend - the EGL bring-up, InitCapabilities,
// the applier - reaches these very emitters. A record published there would be waited
// for by the thread that is supposed to apply it: `Fatal{BarrierTimeout,
// "ResourceRespecify"}` from `mgl-srv-apply`, thirty seconds into bring-up, which is
// "ResourceRespecify"}` from `mgl-srv-apply` one barrier budget into bring-up, which is
// exactly how this was found.
//
// THE ANSWER IS NOT "SUPPRESS THE RECORD" - it is "run the server's own code", because
+1 -1
View File
@@ -249,7 +249,7 @@ namespace MobileGL::MG_Remote::Transport {
// caller asked for is no longer charged against its timeout, so the
// total is bounded by `timeoutMs + the spin budget` rather than by
// `timeoutMs`. spinUs is 50 by default and the timeouts that reach
// here are milliseconds (the verb barrier's 30 s, the drain's bound),
// here are milliseconds (the verb barrier's budget, the drain's bound),
// so the drift is under a part in a thousand - and erring LONG is the
// safe direction for a deadline whose expiry is reported as a timeout.
const auto deadline = timeoutMs == kWaitForever
@@ -218,13 +218,16 @@ TEST(RemoteClientControls, UnknownReadPixelsReplyStatusRefusesTightAndBounce) {
}
TEST(RemoteClientControls, FenceWaitBudgetHonorsGlTimeoutAndFiniteTransportChunks) {
// 120000 below is ClientSession.cpp's kBarrierTimeoutMs, the transport grace every CPU bound
// wait adds on top of the GL timeout. It was 30 s; lavapipe's texture-handle registration
// (vkCreateImageView -> llvmpipe_register_texture) exceeds that in a single apply stall.
struct WaitCase { Uint64 timeoutNs; MGPWireOp op; Uint32 expectedParkMs; };
const WaitCase cases[] = {
{0, MGPWireOp::FenceWait, 30000},
{1, MGPWireOp::FenceWait, 30001},
{60000000000ull, MGPWireOp::FenceWait, 90000},
{0, MGPWireOp::FenceWait, 120000},
{1, MGPWireOp::FenceWait, 120001},
{60000000000ull, MGPWireOp::FenceWait, 180000},
{~Uint64(0), MGPWireOp::FenceWait, Transport::kWaitForever - 1},
{~Uint64(0), MGPWireOp::FenceWaitServer, 30000},
{~Uint64(0), MGPWireOp::FenceWaitServer, 120000},
};
for (const auto test : cases) {
const auto child = RunInChild([test] {
@@ -245,7 +248,7 @@ TEST(RemoteClientControls, FenceWaitBudgetHonorsGlTimeoutAndFiniteTransportChunk
producer.*PeerMember(ProducerBellTag{}) = bell.original;
const auto observed = bell.firstMs.load();
// Doorbell converts the remaining deadline to whole milliseconds. Allow
// scheduler delay, but neither a 30s clamp nor an unbounded sentinel.
// scheduler delay, but neither the barrier clamp nor an unbounded sentinel.
if (!bell.entered.load() || observed > test.expectedParkMs ||
observed < test.expectedParkMs - 100 || observed == Transport::kWaitForever) ::_exit(93);
if (test.op == MGPWireOp::FenceWait &&
+1 -1
View File
@@ -539,7 +539,7 @@ P5 的 `inproc` 是真第二线程,但还是 **lockstep**:每条 class-B ver
同一地址空间不得成为旁路(R-2):encoder 把 `MGHostSpan::Ptr` 恒写成 `nullptr`,内容 blob 必须带真实 `SEG_STAGE` offset / 非零 size;decoder 对四种形状分别 `Fatal{ProtocolCorruption}`。`MapPersistent` 在 split 恒 decline;`MOBILEGL_IPC_AUDIT=1` 在 retire 后把 staging 填 `0xDD`,让跨 applier 返回持针的实现下一次读取时可见地失败(R-11:任何 widened read 先过 `RequireStagedCoverageForPendingRanges`,否则 `Fatal{StageSnapshotTooNarrow}`)。apply 角色不得进入 client 的 persistent-map producer(r1)。
reply mailbox 以**记录序号作为 slot id**(R-3):slot header 是 `{Seq, Status, Size}`,acceptance 的 Bool 答案与 `MapPersistent` decline 都在既有 verb wait 内读取;`Status=ERROR` 一律 `Fatal{ReplyError}`,未知 status 是 `Fatal{ReplyStatusInvalid}`;读到 reply 之前先由 `appliedSeq` 证明该记录已离开 applier(R-5)。P9 才把这套同步 mailbox 推广成异步池。等待预算:普通 verb / 容量等待 30 s(`Fatal{BarrierTimeout}`);`ClientWaitSync` 的 applied/reply 预算 = ceil(timeout ns / 1e6) + 30 000 ms,有限 chunk 避开溢出。
reply mailbox 以**记录序号作为 slot id**(R-3):slot header 是 `{Seq, Status, Size}`,acceptance 的 Bool 答案与 `MapPersistent` decline 都在既有 verb wait 内读取;`Status=ERROR` 一律 `Fatal{ReplyError}`,未知 status 是 `Fatal{ReplyStatusInvalid}`;读到 reply 之前先由 `appliedSeq` 证明该记录已离开 applier(R-5)。P9 才把这套同步 mailbox 推广成异步池。等待预算:普通 verb / 容量等待 120 s(`Fatal{BarrierTimeout}`,`ClientSession.cpp` 的 `kBarrierTimeoutMs`);`ClientWaitSync` 的 applied/reply 预算 = ceil(timeout ns / 1e6) + 120 000 ms,有限 chunk 避开溢出。这条看门狗只判**死锁**(记录永不退役),不是性能门:lavapipe/llvmpipe 在 texture-handle 注册(`vkCreateImageView` → `llvmpipe_register_texture`)上可合法阻塞 apply 线程 29.3–39.7 s,故预算从原 30 s 提到 120 s——monolith 走同一路径本就没有这个上限。
### 17.2 71 槽的三类、caps 与 tight readback