mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-18 15:33:22 +09:00
Two things land together: the plan and contract draft P6 needs before it starts, and the
working notes that until now existed on exactly one machine.
1. P6's plan and contract draft (neither started; not one line of code)
docs/Disaggregated/P6-SPAWN-PLAN.md the package plan
docs/Disaggregated/P6-CONTRACT-DRAFT.md the contract draft (moves to
MG_Remote/CONTRACT-P6.md when c6 lands)
The three findings that mattered, all written into the documents:
a. P6 is SMALLER than its roadmap cell suggests. The transport primitives are already in
the tree and already tested: SocketDoorbell including hangup detection and the two
lost-wakeup fences (Doorbell.cpp:211-360, four cases in FdPassingTest), SCM_RIGHTS fd
passing, ShmSegment::Adopt, and the SurfaceOp / WindowKind schema already defined in
protocol.fbs. inproc uses a real second shm mapping rather than an alias, and
SessionRings.h:22-28 says that was done so today is not the first time it runs. P6 is
assembly plus a control plane plus a process, not "write a socket transport".
b. The evidence for "P6 is only a transport swap" has EXPIRED. It was bought by P5c, whose
audit found 59 direct memory accesses in a tree that already looked clean. P5e changed
the tree again. So the first package, a6, is a read-only audit that writes no code, and
every contract row that depends on it is marked PENDING a6 rather than decided in
advance.
c. P6 does not get a game running on the phone. ANativeWindow* is meaningless in the child,
and protocol.fbs:186-196's own comment says Android transfers the window out of band -
that band is P12. P6 lands pbuffer / surfaceless / offscreen, which is what HeadlessGL,
the split-equivalent lane and trace replay need, and which is exactly what P6's own exit
gate measures. A real window reaching the server is a named refusal. This gets its own
section because "P6 is done" would otherwise read as "the game runs now".
The contract adds rule G: a role may not name a process-local handle. It extends P5c's
rule E (a role may not name the other role's memory) from memory to identity - EGLDisplay /
EGLSurface / EGLContext / ANativeWindow* / fd number / pid / mapped address either cross as
a token this contract defines or they do not cross, and a descriptor crosses only through
SCM_RIGHTS, where the kernel renumbers it.
What P5e left for P6 gets its own section. The latch semantics are already designed
(ARCHITECTURE.md:504) and are copied unchanged. What is new is that after run-ahead the
client no longer waits while the peer process can DIE: the client holds records that were
published and will never be applied, and Android's freezer suspends the child without
closing anything. So slow and dead must be distinguishable WITHOUT a timeout threshold.
That is P6's only new design; everything else is carriage. It shares a predicate with E1's
outstanding re-specification (ID-122), and the two should be designed together.
Two gates carried over from P5e's lessons: G2 requires integration-spawn's test-name set to
match integration-split's name for name, because a differing set means the lane is quietly
running less; and the arm-proof gate requires every entry to record the child pid and
transport=spawn in its OWN private log - ID-124 established that this kind of false green
really happens, and a spawn lane that actually ran monolith would pass every other gate.
S2's shape must be settled before it is written, and the document says why: E1 is the
cautionary case.
2. The working notes move into the repository (docs/Disaggregated/notes/, 415 .md files)
ROADMAP / CURRENT_STAGE_PROGRESS / MEASUREMENTS cite ruling numbers, audit line numbers and
the provenance of gate figures from ~/w7/notes/, a directory that exists on one machine and
is under no version control. A record cited hundreds of times that lives only locally is not
a record, so it moves in whole, with its directory structure intact.
Only .md moves. Scripts, run logs, census and gate result JSON, patches, APKs and nm dumps
stay where they are: they are evidence DATA rather than documents - large, regenerated per
run, and mostly carrying absolute paths and device serials that mean something only on the
machine that produced them. The mapping and this split are written in
docs/Disaggregated/notes/README.md.
Two exceptions are recorded there too: P6's BRIEF has been promoted to a first-class
document and keeps no copy under notes/, so the two cannot drift apart; and recovered/ holds
material salvaged from failed workflow runs, which may duplicate the like-named phase
directories and is kept because a few of those were the only complete version at the time.
CI's citation lint takes docs/Disaggregated/*.md only (non-recursive), and notes/ is
deliberately outside it: those notes' file:line citations were written against their own
code heads, and resolving them against today's HEAD would fail wholesale - which is not an
error, it is their correct state as a historical record. To check one, use the revision it
declares (--rev).
3. Inconsistencies fixed in passing
- README.md's status line still said P5e was in progress while the stage table already said
closed.
- CURRENT_STAGE_PROGRESS's next-step item 1 still listed the exit gate as pending, though
(a) and (b) were run in the previous commit; rewritten to closed plus the one item that
actually remains (E1).
- ROADMAP's P6 row still had the pre-phase plan in three of its cells; it now carries the
package order, the scope boundary and the two new gates.
Checked with check_doc_citations.py: 125 citations across docs/Disaggregated/*.md, 0 problems.
450 lines
38 KiB
Markdown
450 lines
38 KiB
Markdown
# v1-v3 — server role round 3: every review-v2 item closed with an executed gate, the census corrected with its cause, and one v1 defect the census exposed and this round fixes
|
||
|
||
Branch `p5/v1`, round-3 range **`c9d84e33..b13f2b02`**: `ca92308b` = `git merge feat/disaggregated` at `c3e736a3`
|
||
(one conflict, `MG_Test/Wire/CMakeLists.txt`, s1's `SessionHandshakeTest` block beside v1's - both kept; the
|
||
merge touched no v1-owned path), then six one-line commits `a838c787`, `49fecc16`, `13043e3f`, `1646049a`,
|
||
`74060e8b` (ID-67, which arrived mid-round) and `b13f2b02` (what the first full gate and red-check run on
|
||
`74060e8b` caught: two pull symbols resized by the ID-67 hunk's `const Bool created` temp - restored byte for
|
||
byte in the `#else` arm - one red-check perturbation red for the wrong reason after N-9's helper thread, and
|
||
the two C6 null-check gates that stayed green on RGB8). `feat/disaggregated` moved again while this ran (c1f
|
||
`fbeea876`, docs `edcdf771`); this branch was not re-merged onto them - the brief's step 0 named `c3e736a3`,
|
||
and the joint scratch measures against `p5/c1@5682f429`, one c1 fix behind c1f.
|
||
Identity `Swung0x48 <swung0x48@outlook.com>`; no trailer. Links toggled per PACKAGE-PREAMBLE around every
|
||
git and build. Files: `MG_Remote/Server/{ServerLoop.h,ServerLoop.cpp,PipeApplier.h}`, `MG_Backend/Init.cpp`,
|
||
`MG_Backend/DirectGLES/{Managers.cpp,BackendObject_DirectGLES.cpp}` (the last under ID-52/ID-59's grant, every
|
||
hunk `#if MOBILEGL_BUILD_DISAGGREGATED`), `MG_Test/Wire/ServerLoopTest.cpp`. Nothing of c1's was committed to
|
||
`p5/v1`; the joint scratch is `p5/v1-joint3` (`~/w7/p5-v1-joint`, = this head + `p5/c1@5682f429` + the two seam
|
||
hunks committed on the scratch only).
|
||
|
||
Temp files kept because they are cited: `~/w7/p5-v1-r3-*.{sh,py,log,txt}`, `~/w7/p5-v1-redcheck.py` (the
|
||
runner, now 36 perturbations), `~/w7/p5-v1-gate.sh` (Part 2c added; the r2 copy is `p5-v1-gate.sh.r2bak`).
|
||
|
||
---
|
||
|
||
## 0. What changed in one paragraph
|
||
|
||
The review's verdict was "the code is right, nothing can notice it going away". This round makes every one
|
||
of those things noticeable, and in doing so found that one of them was NOT right. A new fixture in
|
||
`ServerLoopTest` brings the SERVER's real DirectGLES backend up on the apply thread through v1's twelve
|
||
forwarders alone (mesa surfaceless / llvmpipe; no client object, no emit table), counts native `eglMakeCurrent`
|
||
at the EGL function table the backend dispatches through, and drives the buffer twins through the real op
|
||
table on that context. On it: C7 is now ONE native bind per process (was 2) and ZERO native releases,
|
||
measured at the driver on the joint lane too; N-3's tuple is forgotten on every context-ending event; the nine
|
||
N-5 deletions each go red on their own string; item 10's M-2 control exists and is executed; and the census's
|
||
"36-entry drop" turns out to be c1 round 2's `BlobMissing` abort (reversed by c1 round 3) PLUS six entries that
|
||
v1's own round-2 M-3 refusal aborted - the ordinary `glBufferData(NULL)` + partial `glBufferSubData` streaming
|
||
idiom - which this round fixes by keying the two whole-store refusals on the descriptor's own
|
||
`HasDefinedContent`, with a positive gate for the idiom and the death gates kept for the missing-record shape.
|
||
|
||
---
|
||
|
||
## 1. The EGL fixture (the thing every N-5 gate stands on)
|
||
|
||
`ServerLoopTest.cpp`, `EglServerFixture` (round-3 section). `CreateBackend(DirectGLES)` on the app thread (step
|
||
1 of `InitSplitRoles`), then `ServerInitializeEGLDisplay(0x1)`, `ServerCreateEGLPbufferSurface(0x1, 64, 64)`,
|
||
`ServerMakeEGLCurrent(0x1, 0x1, 0x1, 0x1)` as blocking control requests on `mgl-srv-apply` - the client's
|
||
virtual handles are literally `0x1` on this host, which is the N-3 finding's whole point. The native count is
|
||
read where the driver reads it: a counting trampoline in front of `g_EGLFuncs.eglMakeCurrent`, installed with
|
||
`SetEGLFuncsTable`, forwarding to the driver's entry point; no production counter is trusted for that number.
|
||
The frontend `GLContext` is installed for the case's life (`SplitBufferTest`'s shape) because the backend's
|
||
draw-shaped paths walk it. No usable EGL is a `GTEST_SKIP` naming the step, or a `FAIL` naming it when
|
||
`MOBILEGL_ITEST_REQUIRE_GPU` is set (the harness's own rule; N-1's lesson) - the gate, the red-check runner and
|
||
the joint run all set it, and every EGL case RAN (0 skipped) in every run below.
|
||
|
||
Death gates fork after the bring-up (the harness's pre-flight shape): the child's only thread is a copy of the
|
||
test thread and holds no context, and every refusal under test fires BEFORE the first GL call that matters, so
|
||
the child reaches it. The two positive M-3 gates bring the server up INSIDE the child and exit 0/7/8, so a
|
||
Fatal on the apply thread ends the child, not the case.
|
||
|
||
---
|
||
|
||
## 2. Item by item (each with its red-once line; every death names its diagnostic)
|
||
|
||
Red-once lines are the runner's names (`~/w7/p5-v1-redcheck.py`), all executed on the final head:
|
||
`C9 META-CONTROL PASS; 36 of 36 perturbations RED for their own reason; `P5_V1_REDCHECK_ALL_WENT_RED``.
|
||
|
||
### Item 1 — M-3 (was NOT CLOSED as a gate): both whole-store readers, gated, and one of them fixed
|
||
|
||
* `StagedShadowProductionTest.ASparseShadowForcedThroughAPoolReuseUploadIsFatalByName`: A =
|
||
`glBufferData(64, data)` drawn and deleted on the apply thread (its id retires to the pool; two presents with a
|
||
finish move the completion watermark past its retire serial); B = a data-carrying respecify whose 64-byte
|
||
subdata arrived short (16 bytes). B's first ensure takes A's id from the pool and dies
|
||
`Fatal{StageSnapshotTooNarrow, "pool_reuse_whole_store"}` (`EXPECT_EXIT` `KilledBySignal(SIGABRT)` + the
|
||
exact string). **Red once:** `m3-poolreuse-require-deleted` → `RED, rc=8, `not the pool-reuse reader``.
|
||
* `...ASparseShadowForcedThroughAStorageRespecifyIsFatalByName`: `glBufferData(64, data)` drawn, then
|
||
`glBufferData(64, data2)` again off the context thread (the store goes PENDING exactly as an emitted record
|
||
leaves it), then only 16 of the 64 bytes → the next ensure dies
|
||
`Fatal{StageSnapshotTooNarrow, "respecify_whole_store"}`. **Red once:** `m3-respecify-require-deleted` →
|
||
`RED, rc=8, `not the respecify reader``.
|
||
* **The defect (§6 found it).** Round 2's refusal did not look at WHY the shadow was sparse. For a store the
|
||
application ORPHANED (`glBufferData(size, NULL)`, `glBufferStorage(NULL)`) every byte it has not staged since
|
||
is undefined by its own declaration, and the ordinary streaming idiom - orphan, partial `glBufferSubData`,
|
||
draw - reaches the respecify reader with the frontend object's `HasDefinedContent()` flipped to true by the
|
||
subdata and a shadow that covers only the subdata. Round 2 refused it. On the joint that aborted six
|
||
entries by name (`DirectGLES.{,ResourceSubsystemOn.}LargeArenaAdoptionScenario.{RespecifiedVertexArenaKeepsVaoBindings,SubDataAfterAnInFlightDrawReachesTheNextDraw}`,
|
||
`DirectGLES.MapPersistentRoundtrips.LargeArenaAdoptionScenario.AnAdoptionCostsExactlyOneMapPersistentRoundtrip`,
|
||
`DirectGLES.ResourceSubsystemControl.On.ResourceSubsystemControlScenario.ClearingTheP3aBitsStopsTheEmissionsAndNotThePixels`),
|
||
every one probed with a private log: `Fatal{StageSnapshotTooNarrow, "respecify_whole_store"}`
|
||
(`~/w7/p5-v1-r3-probe-stillA-*.log`, `mpr-large-arena-adoption-DirectGLES.log`,
|
||
`resource-subsystem-on-DirectGLES.log`). They passed on the round-1 joint; on the round-2 joint they aborted
|
||
under c1's `BlobMissing` first, so neither review could see whose abort was underneath.
|
||
**The rule now** (`Managers.cpp`, `ResourceContentIsDeclared` beside `ResourceWidthOf`): the whole-store
|
||
coverage is required iff the applier's DESCRIPTOR says the application SUPPLIED the content
|
||
(`Desc.HasDefinedContent != 0`, `glBufferData(size, data)`), because under split that content arrives as
|
||
`resource_subdata` behind the respecify (table 1 row 19) and a gap at the draw is a missing record; for an
|
||
orphaned store the zero-fill past the coverage is exactly what the monolith arm uploads from `MappedData()`
|
||
and is legal. The pending-range clamps (`ensure_flush_pending`, `readback_flush_pending`) and
|
||
`UploadRangeFrom`'s own `upload_range` check are unchanged: a queued range nothing staged is always a missing
|
||
record. Positive gates: `TheStreamingIdiomOrphanThenPartialSubDataIsUploadedNotRefused` (respecify reader)
|
||
and `TheStreamingIdiomThroughAPoolReuseIsUploadedNotRefused` (pool reader) - the idiom end to end in a forked
|
||
child, the driver's store read back through a READ map: 16 bytes of pattern, 48 of zero. **Red once:**
|
||
`m3-refusal-unconditional` (both readers refuse whatever the descriptor says) →
|
||
`RED, rc=8, `refused by name``. On the joint after the fix all six entries pass
|
||
(`~/w7/p5-v1-r3-joint-final2.sh`, each `100% tests passed` standalone, and the census moves by exactly those
|
||
six, §6).
|
||
**This needs the integrator's eye** (§9): it narrows a refusal the v2 review accepted as correct.
|
||
* **ID-41:** `FlushPendingRangesFrom` untouched; both G5 scripts on the final head
|
||
(`~/w7/p5-v1-r3-g5-and-probe6.sh`): `172b022273db01b16e772d15b269ffcd797fe38c767f7354d83ce113a66040d0
|
||
FlushPendingRangesFrom`, "the 11 ... functions are byte-identical", "the 17 ... regions are byte-identical
|
||
between ff2994d9 and HEAD".
|
||
|
||
### Item 2 — the seven PARTIALs
|
||
|
||
| review item | closed by | red-once line |
|
||
|---|---|---|
|
||
| 2, M-1's coverage half | `AQueuedRangeOutsideTheStagedCoverageIsFatalAtTheDrainByName`: `glBufferData(64, NULL)` + `glBufferSubData(0, 16)` + draw (legal), then a `resource_flush_range` over [16, 48) with no bytes (item 12's row) queues a range nothing staged; the next ensure dies `Fatal{StageSnapshotTooNarrow, "ensure_flush_pending"}` | `require-pending-coverage-neutered` → `RED, rc=8, `not the pending-coverage clamp`` (the child then dies one tier later, in `UploadRangeFrom`'s `"upload_range"`, which the assertion refuses - so the clamp's own site is what is gated, not "some Fatal") |
|
||
| 5, m-5 | `ATwinSurvivingContextLossDropsItsFreedShadowBase`: drawn twin → `ServerReleaseEGLResources` (DropAll; the twin table survives, verified) → recreate → re-ensure → `hostBytes == nullptr`; without the null the re-arming respecify reads the freed vector (the ASan shape, asserted as the dangling pointer) | `m5-production-null-deleted` → `RED, rc=8, `still names the freed server shadow`` |
|
||
| 6, codex 11 | `TheResolvedAffinityMaskIsTheKernelsEffectiveSetNotTheRequest`: the request is one cpu this thread MAY use (taken from `sched_getaffinity` itself) plus cpu 63, absent on any <64-cpu box; `sched_setaffinity` accepts the mask minus the absent cpu, so the effective set differs from the request without a cpuset; resolved must be the effective one and the log line names it. The false `:373` red-once claim is replaced | `affinity-reports-requested` → `RED, rc=8, `reports the REQUESTED set`` |
|
||
| 7, M-6 | `ShutdownSplitRolesAfterAnEarlyStartFailureDropsTheServersPrivateBackend`: step 1's backend, then a REAL early `ClientSession::Start` failure (`TransportMode::Spawn` is refused by name before Accept), then `ShutdownSplitRoles()` → `Backend() == nullptr` and the next `CreateBackend` succeeds | `m6-shutdown-stop-deleted` → `RED, rc=8, `left the server's private backend alive`` |
|
||
| 9, C6 | `TheSevenFormatCapabilityReadsAnswerFromTheServersBackendNotTheGlobal`: server cache empty, global = a second object whose cache says RGB8 is caveat-only and RGBA8 probed at 6 samples → the caveat decision (the `HasCachedFormatCapability` pair) and `ClampSamplesToBackendSupport` (its pair) answer from the server; then global = NULL, server present → `GenerateTextureFormatInfo` keeps native RGB8_SNORM and `BackendTextureFormatAddsAlpha` answers false (the two null-check reads, distinguishable only in the step-1-to-step-4 window; RGB8_SNORM because the fallback's three-channel widening applies to it and not to RGB8 - the first cut used RGB8 and both arms agreed, which the runner caught). Six of the seven reads by answer; the seventh - `UsesWidenedPacked16NormStorage`'s gate before a memoised driver probe - is a null check whose two arms give the same answer on the reference rasterizer and has no answer-level control here (said, not claimed) | one site at a time: `c6-caveat-pair-reverted` → `RED, rc=8, `caveat decision followed the global``; `c6-clampsamples-reverted` → `RED, rc=8, `sample clamp answered from the global``; `c6-generateformatinfo-nullcheck-reverted` → `RED, rc=8, `format info followed the global``; `c6-addsalpha-nullcheck-reverted` → `RED, rc=8, `adds-alpha answer followed the global`` |
|
||
| 10, C7 / ID-54 | §3 below | three ways, §3 |
|
||
| 14, the census correction | §6 below | - |
|
||
|
||
### Item 3 — N-3, the tuple outlives the context
|
||
|
||
`ServerLoop::ForgetCurrentTuple()` / `ForgetCurrentTupleIfItNames(surface)` (apply-thread only, like the tuple),
|
||
called from `ServerReleaseEGLResources` (after `DestroyEGLContext`), `ServerReleaseEGLSurface` for a surface
|
||
the tuple names, both `ServerCreateEGL*Surface` on success (BackendObject_DirectGLES destroys the context to
|
||
create a different surface), `ApplyThreadMain`'s backend destruction and `Stop()`'s `!joinable` arm.
|
||
Forgetting is always safe: a forgotten-but-live tuple costs one forwarded bind the backend object dedups
|
||
natively (§3); a remembered-but-dead one is a silent no-context-current. Control
|
||
`ServerLoopEglTest.ADestroyedContextForgetsTheTupleSoTheSameHandleValuesBindAgain`: bind, swap (works),
|
||
`ServerReleaseEGLResources`, re-create under the same `0x1` handle values, bind again → `NativeBindCount()`
|
||
1 → 2, the swap after the re-create works (it fails "no current context attached" when the make-current was
|
||
treated as a repeat), native binds 2 (one per context lifetime). **Red once:** `n3-tuple-never-forgotten`
|
||
(`ForgetCurrentTuple` a no-op, which also no-ops `...IfItNames`) → `RED, rc=8, `classified as a RepeatNoOp``.
|
||
|
||
### Item 4 — N-5, the nine deletions
|
||
|
||
| deletion (review v2 N-5 row) | went red on |
|
||
|---|---|
|
||
| M-1's coverage half (`Managers.cpp` loop body → `continue`) | `RED, rc=8, `not the pending-coverage clamp`` |
|
||
| M-3 pool-reuse | `RED, rc=8, `not the pool-reuse reader`` |
|
||
| M-3 respecify | `RED, rc=8, `not the respecify reader`` |
|
||
| m-5's null | `RED, rc=8, `still names the freed server shadow`` |
|
||
| M-6's call (`Init.cpp` `ServerLoopInstance().Stop()`) | `RED, rc=8, `left the server's private backend alive`` |
|
||
| codex 11's read-back → `return requested` | `RED, rc=8, `reports the REQUESTED set`` |
|
||
| C7's classification (`ApplyMakeCurrent` → always `NativeBind`) | `RED, rc=8, `forwarded as a native bind`` |
|
||
| C6's routing, one site at a time (four sites) | the four lines under item 2, row 9 |
|
||
| ID-49's tight size (`:265` post at `DstSize`; `:231` scratch from `DstSize`; `:216` `tight = DstSize`) | `RED, rc=8, `posted at the client's DstSize``; `RED, rc=8, `scratch was sized from the client``; `RED, rc=8, `posted at the client's DstSize`` |
|
||
|
||
ID-49's gate is `ServerLoopEglTest.AReadPixelsReplyIsTheTightExtentWhateverDstSizeTheClientSent`: the client's
|
||
half of the verb is run first (`MGPipeValidateForVerb(ReadPixels)`, the same residual fill `glReadPixels`
|
||
performs before it reaches a backend - without it the apply thread walks a null texture-unit base, which is
|
||
how the first version of the case died), then a `read_pixels` record with `DstSize = 80` for a 4x3 RGBA8 read
|
||
whose tight extent is 48; the applier posts 48 (`ReadbackBytes`), grew its scratch to 48
|
||
(`ReadbackScratchBytes`, the new diagnostic), and the client's own read of the slot sees seq/OK/48.
|
||
|
||
### Item 5 — N-2, re-triaged on the right lanes (and the joint's attribution confirmed)
|
||
|
||
The round-2 probes ran `DirectGLES.PrimitivesGeneratedNoXfbScenario.*` and
|
||
`DirectGLES.HandleRecycleScenario.*`, which are SKIPPED on that backend; `[ PASSED ] 0 tests` is not evidence
|
||
and "flaky under `-j`" is withdrawn. This round ran each of the eight failing entries on its own lane twice
|
||
under inproc and once under monolith (`~/w7/p5-v1-r3-census-probes.sh`, `p5-v1-r3-probe-n2-*.log`), one
|
||
test each, rc=1 every inproc run, rc=0 every monolith run - the same result as the joint package's 24 extra
|
||
runs (`~/w7/p5-joint-probe-{12..15,24..39}.log`, `joint-v1.md` §4). Deterministic split-path wrong answers, no
|
||
Fatal in any private log. The joint's attribution, confirmed with a probed line each:
|
||
|
||
* **`HandleRecycleScenario.AFramebufferAtARecycledAddressDoesNotInheritItsPredecessorsTwin`** (5 entries,
|
||
both backends, every arm): `HandleRecycleScenario.cpp:1369: replacementIsNotGreen Which is: 16 ... (first
|
||
texel rgba=0,0,0,0)`. The scenario reads both attachments through `glGetTexImage` (`:1318-1322`); under
|
||
split `GetTexImage` (`GL_Texture.cpp:6797-6809`) takes the CLIENT-shadow fallback because the client object's
|
||
table has no `GL.GetTexImage`, and the client's texture shadow never saw the server's clear - so every texel
|
||
reads 0,0,0,0 and 16 "offenders" is the whole 4x4 image. A P4b/P7 texture-readback debt, not a recycled-handle
|
||
defect; not v1's.
|
||
* **`PrimitivesGeneratedNoXfbScenario`** (3 entries, DirectVulkan): `:295 generated Which is: 0 ... 2u` and
|
||
`:541 ... Which is: 0 ... 1u`. `GL_Query.cpp:262-265` reads the result through
|
||
`gBackendFunctionsTable.GL.GetQueryResult64` only when the query has a backend handle and the slot is
|
||
non-null; under split the client object's table carries no query slots, so the frontend's default 0 is the
|
||
answer. P7 query accounting, outside BRIEF §4's six verbs; not v1's.
|
||
|
||
### Item 6 — N-4, the census, corrected, with the cause of every movement
|
||
|
||
All counts by the reviewer's `census.sh` token rule (`***Exception`, not the "Subprocess aborted" grep that
|
||
double-counts), per-test transitions by `~/w7/p5-v1-r3-censusdiff.py`:
|
||
|
||
| joint | head | passed | skipped | aborted | failed | segfault | sum |
|
||
|---|---|---:|---:|---:|---:|---:|---:|
|
||
| round 1 (`p5/v1-joint`) | v1 r1 + c1 r1 | 428 | 185 | 506 | 28 | 2 | 1149 |
|
||
| round 2 (`p5/v1-joint2`, the reviewer's) | v1 r2 + c1 r2 | 392 | 183 | 551 | 23 | 0 | 1149 |
|
||
| round 3, pre-fix (`p5/v1-joint3` `e939174c`) | v1 r2(+merge) + c1 r3 | 426 | 185 | 511 | 27 | 0 | 1149 |
|
||
| round 3, final (`p5/v1-joint3` `ca97a4a9`) | v1 r3 + c1 r3 | **432** | 185 | 505 | 27 | 0 | 1149 |
|
||
|
||
Round 2's "575" was `passed + skipped`; the reviewer's 392 is right. **The 36-entry drop (428 → 392) is 40
|
||
passed→aborted, 2 segfault→passed (ID-49), 1 aborted→passed and 1 failed→passed (item 12).** The 40 are:
|
||
* **34 c1 round-2**: `Fatal{BlobMissing, "SetDynamicState"}` - c1's round-2 `MGP_WIRE_BLOB` wrapper made a
|
||
version-only `set_dynamic_state` (zero bytes, `ChunkMask == 0`) a blob-shape abort (c1-v3.md §1 item (b));
|
||
the verifier's pre-round-3 probe of one of them reads verbatim `Fatal{BlobMissing, "SetDynamicState"} - the
|
||
row's decoder requires a declared blob and the call site handed over 0 bytes`
|
||
(`~/w7/p5-verify3-probe-DirectGLES_HandleRecycle_Handles_...log`). c1 round 3 fixed it: on `joint3` all 34
|
||
pass again standalone with an empty library log (`p5-v1-r3-census-probes.sh`, `r2r3-A_P-*`), 4 more come
|
||
back as their honest wrong answers (2 HandleRecycle, 2 LayeredCubeMapArrayDepth - the 23 → 27 failed) and
|
||
2 as skips. So r2 → r3 has **zero** passed→aborted transitions.
|
||
* **6 v1 round-2**: the M-3 refusal of item 1, `Fatal{StageSnapshotTooNarrow, "respecify_whole_store"}`, every
|
||
one probed. They were hidden under `BlobMissing` on the round-2 joint and surfaced on `joint3`, which is the
|
||
only reason the pre-fix round-3 census reads 426 and not 432.
|
||
|
||
**The brief's hypothesis** - that c1's B1 routed the deletes and moved entries from "passed by accident" to
|
||
class-C aborts - is refuted by measurement: no entry went passed→aborted between c1 round 2 and c1 round 3,
|
||
and the class-C population (`Fatal{UnmigratedVerb}`, e.g. `DrawElements`, `BeginTransformFeedback` per the
|
||
joint package's samples) did not grow. The "passed by accident" direction the brief expected would be correct
|
||
if it had happened; it did not.
|
||
|
||
**The final census is the pre-fix one plus exactly the six** (`~/w7/p5-v1-r3-r3final2-transitions.txt`: 426
|
||
passed→passed, 185 skipped→skipped, 505 aborted→aborted, 27 failed→failed, 6 aborted→passed - the two
|
||
`LargeArenaAdoptionScenario` families, `MapPersistentRoundtrips` and `ResourceSubsystemControl.On` - and nothing
|
||
else). Against c1's own round-3 baseline on its head (c1-v3.md §2: 420/182/517/27/3) this round is
|
||
**+12 passed, −12 aborted, −3 segfault, failed unchanged**; against the joint package's measurement of the
|
||
round-2 head (joint-v1.md §4: 426/185/511/27/0) it is +6/−6.
|
||
|
||
**One transient, recorded not explained.** The FIRST census run on the round-3 head (`~/w7/p5-v1-r3-jointfinal-inproc.log`,
|
||
run immediately after the C7 instrumentation's two library rebuilds) read 377/186/511/75/0: 48 entries
|
||
passed→failed (`AdvertisedLimits` 15, `SpirvShaderBinary` 8, `DoublePrecision` 4, `AsyncCompile` 6,
|
||
`CopyImageLevelRange` 2, `HandleRecycle` 6, `ProgramPipeline` 2, one each of five others). All 48 pass
|
||
standalone (four probed with private logs, no Fatal line), pass twice more as a set at the census's own `-j 6`
|
||
(`~/w7/p5-v1-r3-rerun48{,b}.log`, 48/48 both), and pass in the final census above, which ran the same
|
||
binary. I cannot name the cause; I can say it is not deterministic, not v1's code (the same code passes the
|
||
same entries three times since), and that it was spread across the whole run rather than clustered.
|
||
|
||
### Item 7 — N-1, what the pre-flight does under the knob, and where the Fatal belongs
|
||
|
||
Measured on `joint3` (`~/w7/p5-v1-r3-n1probe.sh`, one Split entry, private log):
|
||
* `MOBILEGL_IPC_VERB_BARRIER=0`, `REQUIRE_GPU` unset: `***Skipped`, ctest `100% tests passed`. The private log,
|
||
line 78, immediately after `LogBackendInfo()`'s extension list (i.e. the first routed verb after
|
||
`MG_Backend::Init` returned, still inside the pre-flight child's `eglMakeCurrent`):
|
||
`[Linux MobileGLIntegra/FATAL]: MGPipe: Fatal{BarrierViolation, "SetTextureParams"} - the apply thread is
|
||
inside the applier while the GL thread is emitting`.
|
||
* `MOBILEGL_IPC_VERB_BARRIER=0 MOBILEGL_ITEST_REQUIRE_GPU=1`: `***Failed`, but for "the EGL bring-up ABORTS on
|
||
this platform: a forked pre-flight child died on signal 6" - red for the platform's reason, not the knob's.
|
||
|
||
**Statement.** The pre-flight child runs literally the same bring-up the parent will (HeadlessGL.cpp's whole
|
||
argument for forking), and the bring-up already emits routed verbs (`SetTextureParams` from the first texture
|
||
state after Init, under c1's R-17). With the barrier off the client does not wait on `appliedSeq`, the next
|
||
emit finds the applier inside, and `EmitAndWait` refuses by name. That is the barrier being load-bearing at the
|
||
first place it can be; the Fatal is raised where it belongs. It cannot be moved "inside the entry" without a
|
||
pre-flight that does not emit, and a non-emitting pre-flight is not a pre-flight of this bring-up - so there is
|
||
no v1 arming to change (the server emits nothing; the emitting side is c1's, the fork is the harness's).
|
||
The fix is the control's, as ID-62 rules: a selection whose entries SKIPPED is a control FAILURE by name - and
|
||
the joint package's gate already reads E1 red through j0's script with `Fatal{BarrierViolation}` in all 14
|
||
selected private files (joint-v1.md §3). One harness improvement worth j0's ear: when the pre-flight child dies
|
||
on SIGABRT, read its private log and report the `Fatal{...}` line as the skip/fail reason, so a knob-caused
|
||
death is never reported as an unusable platform.
|
||
|
||
### Item 8 — the four minors, and the prose
|
||
|
||
* **N-6**: `NativeBindCount()` / `ClientReleaseCount()` now have two readers (the C7 and N-3 controls), and
|
||
their header comment says what they count - forwarded binds - and that the driver's count is a different
|
||
number read at the EGL table.
|
||
* **N-7**: the m-5 null is `MG_Config::Transport != Monolith &&` guarded like the other two split hunks in
|
||
`Managers.cpp`; "under monolith nothing changes" is now true by construction.
|
||
* **N-8**: `Init.cpp` remembers the resource op table step 2 saw (`g_resourceOpsAtStep2`) and, after the
|
||
re-run of `AssertConsumerMaskIsHonest`, refuses a different pointer as
|
||
`Fatal{ConsumerMaskLie, "resource ops table replaced"}` - the swap the old comment claimed the re-run caught
|
||
and could not (a client-registered table keeps "is a table registered" true). The comment says exactly what
|
||
each check can and cannot see. No executable control on `p5/v1`: `Init()` needs the client object c1 ships;
|
||
c1's `BackendObject_Remote::Initialize` registers no table (read, `p5/c1@5682f429`), so the joint stays
|
||
green through it (verified: §8's joint numbers run through `Init()`).
|
||
* **N-9**: `drained-tally-removed` expects the tally case's own new message (`DrainedRecords) did not move`);
|
||
`park-predicate-loses-control` expects `did not un-park the apply thread within` - the park case now posts
|
||
from a helper thread with a 5 s deadline and says so in its own words (Stop() frees the helper: the
|
||
predicate keeps `m_stopRequested`, and the exit block answers the still-posted request NOT_INITIALIZED).
|
||
* **Item 10's prose**: `ServerLoop.h`'s header block and the twelve-calls block, `ServerLoop.cpp`'s
|
||
`ServerMakeEGLCurrent` comment, `ServerLoopTest.cpp`'s C7 pure-decision comment and the `:373` affinity
|
||
comment now state what is measured (§3) and where each half of the dedup lives; the "one-off startup cost"
|
||
sentence is gone and replaced by the measured one-per-context-lifetime. The gate's "201 split-only" line now
|
||
counts v1's four suites separately from the rest.
|
||
* Claims-not-supported #8 (the R-12 republish decision on a repeat): `ApplyMakeCurrent`'s `RepeatNoOp` arm
|
||
says it in place - the decision for an identical repeat is "nothing to republish" because nothing that
|
||
moves the caps ran, and c1's `InitCapabilities` asks the server through `ServerInitCapabilities` either way.
|
||
ID-67 has since ruled exactly that (§3), and c1f's control that expected a republish on a repeat is being
|
||
corrected on `p5/c1g`.
|
||
|
||
### Item 10 (coordinator) — M-2's control, shipped and executed
|
||
|
||
`StagedShadowProductionTest.TheEnsurePathUploadsTheServerShadowNotTheClientObjectsBytes`: a real frontend
|
||
`BufferObject` whose shadow is pattern A (`Respecify(64, A)`), a server shadow staged with pattern B through
|
||
the real op table, and the production ensure given BOTH; the driver's store, read back through a READ map on
|
||
the apply thread, holds B, and the frontend object still holds A. That is the executable form of "corrupt only
|
||
the staged bytes and the draw shows it". **Red once:** `m2-mappeddata-fallback-restored` (the transport arm
|
||
of `liveHostBase()` deleted, `MappedData()` read again) → `RED, rc=8, `uploaded the CLIENT object's bytes``.
|
||
|
||
---
|
||
|
||
## 3. C7 / ID-54 — bind once, measured at the driver
|
||
|
||
Two layers, because one is not enough: `DirectGLES::MakeCurrent()` is reached from the surface's own creation
|
||
(`InitPbufferSurface`) AND from the client's first `eglMakeCurrent`, and the forwarder cannot see the first.
|
||
(1) `ApplyMakeCurrent` classifies against the tuple it last bound (unchanged); (2)
|
||
`BackendObject_DirectGLES::MakeEGLCurrent` (v1's edit under ID-52/ID-59's grant, `#if
|
||
MOBILEGL_BUILD_DISAGGREGATED`, transport-guarded) skips the native call when the requested draw surface is the
|
||
active one, `IsBackendContextCurrentOnThisThread()` - EGL ground truth - says this thread holds the context,
|
||
AND the virtual tuple is the one that bind was for (`NativeBindCanBeSkippedFor`: the surface's creation bind is
|
||
"fresh" and adopted by the first tuple; after that only the exact recorded tuple skips; every context-ending
|
||
event resets it). The base class's bookkeeping (InitCapabilities, the current-thread record `SwapEGLBuffers`
|
||
needs) still runs.
|
||
|
||
**ID-67 (mid-round).** The first round-3 cut keyed the skip on the surface alone, which would have let a
|
||
DIFFERENT virtual context onto the same surface skip the native call - and with it the seven invalidations
|
||
that describe the frontend context - so it was corrected in `74060e8b` before it was gated: a different tuple is
|
||
a real native bind AND a caps republish (R-12 arm (a), so the client adopts without a pump or a Present); an
|
||
identical tuple is neither (the mirror generation does not move, nothing accumulates). The forwarder's half
|
||
was already so (`RepeatNoOp` → `boundNatively = false` → no republish; a forwarded bind → republish); the
|
||
republish is now tallied (`ServerLoop::MakeCurrentRepublishCount`). Control
|
||
`ServerLoopEglTest.ADifferentTupleBindsNativelyAndRepublishesAnIdenticalRepeatDoesNeither`, on the fixture
|
||
with the session given its backend: creation = 1 native bind / 0 republishes; the first tuple = +0 / +1
|
||
(adopts the creation's bind, publishes the real caps); the same tuple again = +0 / +0; a second context
|
||
`0x2` onto the same surface = +1 / +1; `0x2` again = +0 / +0; back to `0x1` = +1 / +1. **Red once, three ways:**
|
||
`id67-different-tuple-native-skipped` (the backend's skip answers "same" for any tuple) →
|
||
`RED, rc=8, `did not bind natively``; `id67-repeat-republishes` →
|
||
`RED, rc=8, `identical repeat republished``; `id67-republish-first-bind-only` → `RED, rc=8, `did not republish the caps snapshot; the client would adopt nothing``.
|
||
The reduced-path lane has one tuple per process, so the joint's "1 native bind per process" (below) is
|
||
unchanged by this.
|
||
|
||
* **Unit, at the EGL table** (`ServerLoopEglTest.MakeCurrentBindsNativelyOncePerContextAndNeverForwardsAClientRelease`):
|
||
surface creation = 1 native bind; the first make-current, an identical repeat, a client release and a bind
|
||
after the release add **0** native binds and **0** native releases; `NativeBindCount()` 1, `ClientReleaseCount()`
|
||
1; the apply thread still owns the context by ground truth and the app thread does not. **Red once, three
|
||
ways:** `c7-applymakecurrent-always-binds` → `RED, rc=8, `forwarded as a native bind``;
|
||
`c7-backend-native-bind-unconditional` → `RED, rc=8, `second native eglMakeCurrent``;
|
||
`c7-client-release-forwarded` → `RED, rc=8, `reached the driver``.
|
||
* **Joint, the reviewer's own instrumentation re-homed** (`~/w7/p5-v1-r3-c7-instr.py`, applied, measured,
|
||
restored, the scratch clean after): on all three reduced-path families the native bind count per process is
|
||
**1** (parent and pre-flight child alike; was 2 and 2), native releases **0**, `C7_ACTION` shows one
|
||
`NativeBind` (held=0) and one `ClientRelease` (held=1) per process, no `RepeatNoOp` needed:
|
||
|
||
```
|
||
===== clear native binds, per pid: 1 C7_NATIVE_BIND pid=443959; 1 C7_NATIVE_BIND pid=443960; releases: (none)
|
||
===== triangle native binds, per pid: 1 C7_NATIVE_BIND pid=445225; 1 C7_NATIVE_BIND pid=445227; releases: (none)
|
||
===== pmap native binds, per pid: 1 C7_NATIVE_BIND pid=446511; 1 C7_NATIVE_BIND pid=446512; releases: (none)
|
||
```
|
||
|
||
So DirectGLES.cpp's six cache invalidations run once per context lifetime, which is now a measured sentence.
|
||
The release half (1 → 0) was already real in round 2; the bind half (2 → 1) is new.
|
||
|
||
---
|
||
|
||
## 4. The numbers (final head `b13f2b02`)
|
||
|
||
**Red-check** (`~/w7/p5-v1-r3-redcheck.log`): `C9 META-CONTROL PASS; 36 of 36 perturbations RED for their own reason; `P5_V1_REDCHECK_ALL_WENT_RED``.
|
||
|
||
**Gate** (`~/w7/p5-v1-gate.sh`, `~/w7/p5-v1-r3-gaterun.log`, head `b13f2b02`, all four flavours rebuilt inside it):
|
||
|
||
| | pull | push | verify | split |
|
||
|---|---|---|---|---|
|
||
| build | rc=0 | rc=0 | rc=0 | rc=0 |
|
||
| `ctest -L unit` | **1814/1814** | **1814/1814** | **1814/1814** | **2018, 1 known red** (`PipeWireCodecTest.NoReplyIsWrittenForARowTheCatalogueGivesNoReplySlot` = c1's ID-31; 2052/2052 on the joint, which has c1's fix) |
|
||
|
||
* **G1** (pull vs `p5-before-libMobileGL.so`): `.text 10806611 -> 10806611 (+0)`, `27814 -> 27814: 0 added, 0
|
||
removed, 0 resized, 0 renamed`. (The gate on `74060e8b` read **2 resized** - the two `Create*Surface`
|
||
functions, +2 bytes each, from the ID-67 hunk's `const Bool created` temp in the pull arm; `b13f2b02` put the
|
||
original statements back in the `#else` arm and G1 is clean again.) `nm` pull `MG_Remote` = 0, split = 443,
|
||
split `ServerLoop` = 31; include-closure 0 problems; the three generators up to date.
|
||
* **Part 2b**, v1's four suites x100 under `-j 8` with `REQUIRE_GPU=1`: `100% tests passed, 0 tests failed
|
||
out of 33`. **Part 2c**, the suites with `REQUIRE_GPU=1`: **33/33, 0 skipped** - every EGL-backed case ran.
|
||
* **Part 3**, push `integration-gpu` under monolith: **1128/1128**; G2 pull-vs-push name diff 0; G14 pull and
|
||
push 0 removed / 40 added (wave 1's), split-only 225 = v1's four suites (33) + wave 1 + s1 + c1 (192).
|
||
* **Part 3b**, split `integration-gpu` under monolith: **1149/1149**.
|
||
* The whole gate ran in four minutes after the red-check (ccache); both are re-runnable as
|
||
`~/w7/p5-v1-r3-chain.sh`.
|
||
|
||
**Joint** (`p5/v1-joint3` `0b5539c8` = `e939174c` (`ca92308b` + `p5/c1@5682f429`) + the two seam hunks committed
|
||
on the scratch + `p5/v1@b13f2b02`; `~/w7/p5-v1-r3-jointfinal{2,3,4}-*.log`; the same lanes were run at each of
|
||
the four round-3 heads, `13043e3f` / `1646049a` / `74060e8b` / `b13f2b02`, with the same results except the six
|
||
entries and the census, which moved exactly as item 1 says between the first two; the census was taken at
|
||
`1646049a` and `74060e8b` and is entry-for-entry identical at both, and `b13f2b02` changes no split
|
||
behaviour - a pull-arm restoration and two test edits - so it was not retaken):
|
||
|
||
| lane | result |
|
||
|---|---|
|
||
| `ctest -L integration-split` (inproc) | **21/21** (19 pass + the 2 `TheMapLandsInTheArmItsLaneDeclares` design skips) |
|
||
| `MOBILEGL_IPC_AUDIT=1 ctest -L integration-split` | **21/21** |
|
||
| `MOBILEGL_IPC_STRICT_ERRORS=1` (recorded, R-7 debt; measured at `13043e3f`) | 19 of 21 red, as before |
|
||
| v1's four suites on the joint, `MOBILEGL_ITEST_REQUIRE_GPU=1` | **33/33**, 0 skipped |
|
||
| split `ctest -L unit` on the joint | **2052/2052** |
|
||
| the six formerly-aborting `LargeArenaAdoption` / `ResourceSubsystemControl` entries | 6/6 pass |
|
||
| inproc `integration-gpu` census | **432 / 185 / 505 / 27 / 0** (= 1149), entry-for-entry identical at `1646049a` and `74060e8b` |
|
||
| C7 native binds per process, reduced-path lane (the reviewer's instrumentation, at `13043e3f`) | **1 / 1 / 1** (was 2), releases **0** |
|
||
|
||
---
|
||
|
||
## 5. Claims this report withdraws or corrects from v1-v2.md
|
||
|
||
1. "575 passed" → 392 (the reviewer's number); the round-2 census fell, and §6 says why and whose.
|
||
2. "PrimitivesGeneratedNoXfbScenario / HandleRecycleScenario pass standalone, flake under -j" → withdrawn; both
|
||
deterministic under inproc on their own lanes, green under monolith, attributed in item 5.
|
||
3. "M-3: the production wiring is exercised end-to-end on the joint" → it was not gated, and it was wrong for the
|
||
orphan idiom; both are fixed above, and the 23 → "none is v1's" sentence needed the abort class too: six
|
||
aborts WERE v1's.
|
||
4. "M-6: red once (unit StopWithoutAStartedThreadStillDropsThePrivateBackend)" and "m-5: red once
|
||
(HasShadowIsTrue...)" → replaced by the gates in item 2, both executed.
|
||
5. `ServerLoopTest.cpp:373` "red once by making ApplyAffinity return requested" → replaced; the explicit-mask case
|
||
does not gate the read-back and says so; the new case does.
|
||
6. "the instrumented native-bind COUNT is a joint-lane control" → it is now a unit control at the EGL table AND
|
||
a measured joint number (§3); the round-2 sentence that the six invalidations were a one-off was untrue then
|
||
and is true now.
|
||
7. "the R-12 republish decision" on a repeat → stated precisely in code (item 8).
|
||
|
||
---
|
||
|
||
## 6. What I could not verify, and why
|
||
|
||
* The seventh C6 read (`UsesWidenedPacked16NormStorage`'s backend gate) by answer: on the reference
|
||
rasterizer the probe behind it answers false, the same as the no-backend arm.
|
||
* N-8's Fatal by a red: `Init()` is not unit-drivable on `p5/v1`; verified only by the joint running through
|
||
it green and by reading c1's object.
|
||
* The cause of the 48-entry transient in §6 (one census run; the same 48 pass standalone, twice as a set at
|
||
`-j 6`, and in the final census).
|
||
* codex 11 under a real cgroup cpuset; the absent-cpu control is the cpuset-free equivalent, not a cgroup.
|
||
* ASan builds: none configured; m-5's control asserts the dangling pointer, not the read through it.
|
||
|
||
---
|
||
|
||
## 7. Debts and hand-offs
|
||
|
||
* The 27 ordinary inproc failures on the joint are P4b/P7 texture-readback (14 Layered, 3 packed-depth
|
||
GetTexImage, 5 HandleRecycle), P7 query accounting (3 PrimGen), the swizzle inspection harness (1) and the
|
||
P4a FBO/RBO lifetime seam (1) - the joint package's table, confirmed here for the two families ID-62 asked
|
||
about; none is v1's.
|
||
* The unit lane on CI: the EGL-backed cases skip where libEGL/mesa is unusable unless
|
||
`MOBILEGL_ITEST_REQUIRE_GPU` is set; the split unit lane's runner installs `libegl-mesa0` (test.yml) and the
|
||
variable should be set there so these cases cannot skip silently (t1/j0).
|
||
* After context loss under split the client's bytes are gone with the server shadow (m-5 nulls the base, the
|
||
respecify uploads NULL); re-staging after context loss is c1's/P6's, noted while writing the m-5 gate.
|
||
* The pre-flight's death-reason plumbing (item 7's last sentence), j0's.
|
||
|
||
---
|
||
|
||
## 8. The three things the integrator most needs to know
|
||
|
||
1. **M-3 changed shape, and it needed to.** The round-2 refusal aborted the ordinary streaming idiom - six joint
|
||
entries by name, hidden under c1's `BlobMissing` until c1 round 3 - and OpenRA / P6 use that idiom on every
|
||
frame. The rule is now the descriptor's own `HasDefinedContent`; the death gates keep the missing-record
|
||
shape red and two positive gates keep the idiom green. It narrows a refusal the v2 review accepted, so it
|
||
should be ratified, not merely merged.
|
||
2. **The census drop was c1 round 2's `BlobMissing` (34) plus v1's M-3 (6), and c1 round 3 + this round
|
||
reverse both;** the brief's B1 hypothesis is refuted by the per-test transitions (432 passed on the final
|
||
joint, +12 over c1's own round-3 baseline, failed unchanged at 27, none of them v1's).
|
||
3. **C7 is one native bind per process at the driver, N-3 is closed, and every N-5 deletion is red on its own
|
||
string** - all on a unit fixture that runs the server's real backend without c1, so the gates travel with
|
||
`p5/v1` and do not wait for the joint.
|