swung0x48/MobileGL
1
0
Fork 0
mirror of https://github.com/MobileGL-Dev/MobileGL synced 2026-11-07 18:13:23 +09:00
Code Issues Packages Projects Releases Wiki Activity

[tools] (P7): window-2 reducers enforce the contract's repeats, arms and case set, and gate Pass-to-NotSupported losses

Browse Source
This commit is contained in:
swung0x48
2026-09-23 17:04:14 -04:00
parent 02db0e4d9d
commit 8f3d179ced
4 changed files with 858 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Disaggregated/notes/p7/device-window-2/dry-run/pre-14e1c8b9-cts/verdict-codex-ruling.txt
+70
View File
@@ -0,0 +1,70 @@
# 60-reduce.py (gate rate Pass/(Pass+Fail+L), ruling: integrator 2026-09-23 after codex closeout review; p7/cf-reducer) on
# ~/w7/logs/devprep/w2/pre-14e1c8b9 - CTS AFTER dry run of pipe head 14e1c8b9's arm64 lib; that run had no gate 3 (NOT-RUN below
# is expected). The contract-literal Pass/(Pass+Fail) reading: verdict-contract.txt; the pre-ruling one: verdict.txt.
== GATE 3 ==
GATE3 NOT-RUN (no gate3/cases.txt)
== CTS AFTER (inproc x DirectVulkan) vs BASE (monolith x DirectVulkan, p7w1) ==
gate (CONTRACT-P7 7.3 + ruling: integrator 2026-09-23 after codex closeout review):
rate = Pass/(Pass+Fail+L), L = the $BASE-Pass cases AFTER turned NotSupported or another non-Pass,
non-Fail status (a lost supported case is NON-PASS); other NS, warnings W and crashes X are not in the
denominator. Per block: delta >= -0.5 pp, 0 NEW crash, and every L case adjudicated in
cts/adjudication.tsv (else FAIL); all five blocks required. '(literal)' = the contract's literal
Pass/(Pass+Fail) delta and 'info' the pre-ruling Pass/(results-NS): printed for comparison, never gating.
block cases BASE P/F/NS/W/X rate AFTER P/F/NS/W/X L rate delta pp (literal) crash regr fixed verdict
shader-image 125 63/6/56/0/0 91.304% 65/3/56/0/1 0 95.588% +4.284 (+4.284) 1 1 3 FAIL
info literal Pass/(Pass+Fail): BASE 91.304% AFTER 95.588% delta +4.284 pp (not gating)
info pre-ruling Pass/(results-NS): BASE 91.304% AFTER 94.203% delta +2.899 pp (not gating)
! 1 NEW crash(es)
! note: AFTER is 4.284 pp ABOVE BASE (not a regression; review the fixed list)
NEW CRASH KHR-GL46.shader_image_load_store.incomplete_textures
Pass->not KHR-GL46.shader_image_load_store.advanced-sync-vertexArray
not->Pass KHR-GL46.shader_image_load_store.advanced-sync-drawIndirect
not->Pass KHR-GL46.shader_image_load_store.advanced-sync-textureUpdate
not->Pass KHR-GL46.shader_image_load_store.non-layered_binding
ssbo 124 124/0/0/0/0 100.000% 124/0/0/0/0 0 100.000% +0.000 (+0.000) 0 0 0 PASS
info literal Pass/(Pass+Fail): BASE 100.000% AFTER 100.000% delta +0.000 pp (not gating)
info pre-ruling Pass/(results-NS): BASE 100.000% AFTER 100.000% delta +0.000 pp (not gating)
dsa 371 368/2/0/1/0 99.459% 366/3/0/1/1 1 98.919% -0.541 (-0.272) 1 2 0 FAIL
info literal Pass/(Pass+Fail): BASE 99.459% AFTER 99.187% delta -0.272 pp (not gating)
info pre-ruling Pass/(results-NS): BASE 99.191% AFTER 98.652% delta -0.539 pp (not gating)
! 1 NEW crash(es)
! delta -0.541 pp < -0.5 pp (Pass/(Pass+Fail+L))
! 1 $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status with no cts/adjudication.tsv line (ruling: integrator 2026-09-23 after codex closeout review)
NEW CRASH KHR-GL46.direct_state_access.renderbuffers_storage
LOST KHR-GL46.direct_state_access.renderbuffers_storage (Pass -> Crash) UNADJUDICATED
Pass->not KHR-GL46.direct_state_access.renderbuffers_storage_multisample
texture 1054 815/18/221/0/0 97.839% 823/10/221/0/0 0 98.800% +0.960 (+0.960) 0 1 9 PASS
info literal Pass/(Pass+Fail): BASE 97.839% AFTER 98.800% delta +0.960 pp (not gating)
info pre-ruling Pass/(results-NS): BASE 97.839% AFTER 98.800% delta +0.960 pp (not gating)
! note: AFTER is 0.960 pp ABOVE BASE (not a regression; review the fixed list)
Pass->not KHR-GL46.texture_border_clamp.Texture2DArrayCompressed
not->Pass KHR-GL46.texture_barrier.disjoint-texels
not->Pass KHR-GL46.texture_barrier.overlapping-texels
not->Pass KHR-GL46.texture_barrier.same-texel-rw
not->Pass KHR-GL46.texture_barrier.same-texel-rw-multipass
not->Pass KHR-GL46.texture_size_promotion.functional
not->Pass KHR-GL46.texture_swizzle.functional_format_idx_19_target_idx_7
not->Pass KHR-GL46.texture_swizzle.functional_format_idx_19_target_idx_8
not->Pass KHR-GL46.texture_swizzle.functional_format_idx_8_target_idx_7
not->Pass KHR-GL46.texture_swizzle.functional_format_idx_8_target_idx_8
packed-pixels 4732 4732/0/0/0/0 100.000% 4732/0/0/0/0 0 100.000% +0.000 (+0.000) 0 0 0 PASS
info literal Pass/(Pass+Fail): BASE 100.000% AFTER 100.000% delta +0.000 pp (not gating)
info pre-ruling Pass/(results-NS): BASE 100.000% AFTER 100.000% delta +0.000 pp (not gating)
ubo unrun (not in this glcts): the device glcts has no GTF module (ID-P7-16); recorded, not gated (GTF-GL46.gtf31.GL3Tests.uniform_buffer_object*)
CTS FAIL
== TIMING ==
preflight 5s 2026-09-23T09:59:01 -> 2026-09-23T09:59:06 reboot=0 pin=1
cts-shader-image 9s 2026-09-23T09:59:10 -> 2026-09-23T09:59:19 limit=0
cts-ssbo 3s 2026-09-23T09:59:20 -> 2026-09-23T09:59:23 limit=0
cts-dsa 9s 2026-09-23T09:59:24 -> 2026-09-23T09:59:33 limit=0
cts-texture 108s 2026-09-23T09:59:34 -> 2026-09-23T10:01:22 limit=0
cts-packed-pixels 220s 2026-09-23T10:01:23 -> 2026-09-23T10:05:03 limit=0
cts 358s 2026-09-23T09:59:06 -> 2026-09-23T10:05:04 blocks=[shader-image ssbo dsa texture packed-pixels] limit=0
restore 5s 2026-09-23T10:05:04 -> 2026-09-23T10:05:09
restore 6s 2026-09-23T10:08:05 -> 2026-09-23T10:08:11
restore 11s 2026-09-23T10:08:36 -> 2026-09-23T10:08:47
WINDOW2 GATE3 NOT-RUN | CTS FAIL -> exit 1
reduce-rc=1
docs/Disaggregated/notes/p7/device-window-2/tools/60-reduce-selftest.py
Executable → Regular
+402 -111
View File
@@ -4,8 +4,10 @@
Self-test of 60-reduce.py: builds canned window output directories in a temp dir and runs the
reducer on each exactly as window.sh does (a subprocess, <stamp> --logroot --tools), then checks
the printed verdict, verdict.json and the exit status. Gate 3 is canned at the gate's own
denominator (36 cases x monolith/inproc x3/spawn x3/ra0, one boot_id, reboot-clean) unless a
scenario says otherwise. The scenarios:
denominator - the canonical 36, derived here as 30-gate3.sh derives gate3/cases.txt (the tree's
trace_cases.json CI split DirectVulkan cases minus CONTRACT-P7 7.1's three exclusions) - with
monolith / inproc x3 / spawn x3 / ra0, one boot_id, reboot-clean, unless a scenario says
otherwise. The scenarios:
control gate 3 and all five CTS blocks equal to $BASE -> GATE3 PASS, CTS PASS, exit 0
missing-block ssbo never finished, dsa's report is empty, texture's has no results; and a
@@ -15,36 +17,58 @@ scenario says otherwise. The scenarios:
-> GATE3 INVALID-SESSION
monolith-arm the monolith repeat's log carries Config: IPC / has no log; a split repeat's
transport-proof did not pass -> that case FAIL
cross-arm spawn's pictures differ from inproc's and monolith's, each arm self-identical
-> WARN line, GATE3 still PASS
one inproc repeat differs from its siblings -> FAIL (within-arm identity)
repeats the contract's 3, not the runner's record: a --repeat 1 run -> GATE3 FAIL-REPEATS;
the same archive under an arms.txt that claims repeat=3 -> FAIL-REPEATS; one
finished spawn pair of 2 repeats -> FAIL-REPEATS; the same pair unfinished
(no .done) -> INCOMPLETE (a resume re-runs it); each exit != 0
arms all four arms on every case: a run started without inproc-ra0 (or spawn) ->
GATE3 FAIL-ARMS; ra0 named in arms.txt but not reached / one case without its
ra0 -> INCOMPLETE; a ra0 repeat whose log says run-ahead=1 -> that case FAIL
case-set cases.txt must EQUAL the canonical 36: 35 + create-indirect, 35 + a name outside
the manifest, 36 + an exclusion, a case listed twice -> GATE3 FAIL-CASESET
cross-arm cross-arm identity is not a per-case gate, but every mismatch needs an
adjudication: spawn's pictures differ on every case -> WARN + px counts,
GATE3 PASS-NEEDS-ADJUDICATION, exit 1; every case in gate3/adjudication.tsv ->
GATE3 PASS, exit 0; one case left out (a line without a reason) -> still
NEEDS-ADJUDICATION naming it; one case differing, adjudicated (+ a stale line) ->
PASS; one inproc repeat differs from its siblings -> FAIL (within-arm identity)
unfinished a pair without .done; a repeat without an actual PNG -> GATE3 INCOMPLETE
rate-gate the contract rate alone decides (no crash anywhere): k $BASE-Pass dsa cases
rate-gate the gate rate alone decides (no crash anywhere): k $BASE-Pass dsa cases
turn Fail. k=1: 367/370, -0.270 pp -> PASS, exit 0; k=2: -0.541 pp -> FAIL on
the rate; k=4: 364/6/0/1/0, -1.081 pp -> FAIL, CTS FAIL
cts-lost a $BASE-Pass case AFTER turns NotSupported / another non-Pass, non-Fail status
(L): dsa Pass->NS -> NON-PASS in the rate (367/370, -0.270 pp; the literal
Pass/(Pass+Fail) -0.001 pp printed as info), listed, dsa FAIL, CTS FAIL; with a
cts/adjudication.tsv line -> dsa PASS, exit 0 (still counted in the rate); a line
without a reason adjudicates nothing; ssbo Pass->NS adjudicated -> still FAIL on
the rate (-0.806 pp); codex's example, 123 of 124 ssbo Passes -> NS: literal
100% but gate rate 0.806% -> FAIL; Pass->CompatibilityWarning -> L; Pass->Crash
adjudicated -> still FAIL (a NEW crash is never adjudicated)
vacuous no verdict from nothing: the caselist a run names is gone (the dry run's AFTER
data, two new crashes) -> every block MISSING, CTS INCOMPLETE - and with the
caselist copy the window keeps, the real reading (CTS FAIL); no caselist at all /
an empty one -> MISSING; every AFTER result NotSupported -> every block FAIL
not-a-gate a pass that is not the gate's is never PASS / exit 0: 2 of 36 cases -> GATE3
PASS-SUBSET; not reboot-clean -> GATE3 PASS-NOT-REBOOT-CLEAN; a --limit CTS
(gate rate 0, every $BASE Pass an L case)
not-a-gate a pass that is not the gate's is never PASS / exit 0: 2 of the canonical 36 ->
GATE3 PASS-SUBSET; not reboot-clean -> GATE3 PASS-NOT-REBOOT-CLEAN; a --limit CTS
block -> CTS PASS-SUBSET; each exit != 0
warnings information lines: a case $BASE lacks (AFTER Fail) stays out of both rates
(texture delta 0) with a WARN; a Pass that became NotSupported -> WARN on the NS
count, block still PASS
(texture delta 0) with a WARN
check-block 60-reduce.py --check-block (50-cts-after.sh's test before a block's .done):
complete -> 0; unrun.txt non-empty / a case without result / caselist gone /
empty report -> 3
dry-run the pre-14e1c8b9 CTS dry run rebuilt from $BASE + its recorded differences
(60-reduce-selftest-pre14e1c8b9.json): contract rate Pass/(Pass+Fail) gives
dsa -0.272 pp (the pre-ruling Pass/(results-NS) -0.539 pp is printed as info
only, with its rates 99.191% -> 98.652%), shader-image +4.284, texture +0.960,
ssbo/packed-pixels 0; dsa and shader-image FAIL on one NEW crash each, not on
the rate -> CTS FAIL
(60-reduce-selftest-pre14e1c8b9.json): gate rate Pass/(Pass+Fail+L) gives dsa
-0.541 pp (renderbuffers_storage Pass->Crash is an L case; the literal
Pass/(Pass+Fail) -0.272 pp and the pre-ruling Pass/(results-NS) -0.539 pp, rates
99.191% -> 98.652%, are printed as info only), shader-image +4.284, texture
+0.960, ssbo/packed-pixels 0; dsa FAIL on its NEW crash, the rate and the
unadjudicated loss, shader-image on one NEW crash -> CTS FAIL
When --dry-run-dir (default ~/w7/logs/devprep/w2/pre-14e1c8b9) exists, the reducer also runs on
that real output (verdict JSON to the temp dir) and must give the same CTS numbers.
TREE (compare_actuals.py + device-window-1/CTS-base + tools/cts/caselists) defaults to the
MobileGL tree this file sits in, else $W2_TOOLS_OVERRIDE, else $W2_PIPE / ~/w7/pipe (only read).
TREE (compare_actuals.py + trace_cases.json + device-window-1/CTS-base + tools/cts/caselists)
defaults to the MobileGL tree this file sits in, else $W2_TOOLS_OVERRIDE, else $W2_PIPE /
~/w7/pipe (only read).
--reducer runs the checks against another 60-reduce.py (red-once of an older reducer).
Exit 0 when every check holds.
"""
@@ -66,13 +90,19 @@ CTS_BASE_REL = "docs/Disaggregated/notes/p7/device-window-1/CTS-base"
BLOCKS = ("shader-image", "ssbo", "dsa", "texture", "packed-pixels")
BOOT = "11111111-2222-3333-4444-555555555555"
OTHER_BOOT = "99999999-8888-7777-6666-555555555555"
# The gate's own denominator (CONTRACT-P7 7.2): 36 cases; the first two are the ones scenarios poke.
CASES = ("caseA-in-world", "caseB-main-menu") + tuple("case%02d-filler" % i for i in range(3, 37))
# CONTRACT-P7 7.1: the three cases red on monolith, out of the 39 CI split DirectVulkan cases.
EXCLUDED = ("minecraft-1.21.1-neoforge-create-indirect-in-world", "minecraft-1.21.11-main-menu",
"minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world")
# Filled by main() from the tree's trace_cases.json: the canonical 36 (manifest order) and the two
# non-OpenRA cases the scenarios poke.
CASES, A, B = (), None, None
OPENRA = "OpenRA"
REPEAT = 3
GONE = "/nonexistent/p7w7-tree-deadbeef/tools/cts/caselists/p7-%s-gl46.txt"
ARMS = (("monolith", 1), ("inproc", REPEAT), ("spawn", REPEAT), ("inproc-ra0", 1))
DSA_STORAGE = "KHR-GL46.direct_state_access.renderbuffers_storage"
SI_INCOMPLETE = "KHR-GL46.shader_image_load_store.incomplete_textures"
CODEX_RULING = "ruling: integrator 2026-09-23 after codex closeout review"
IPC = ("Config: IPC ring=8MiB stage=32MiB wire-deferred=64MiB spin=50us persistent-block=64KiB adopt-tier=2 "
"verb-barrier=1 run-ahead=%d present-credit=1 strict=1 audit=0 role-split-state=1 affinity='auto'")
TAG = "[12:00:00] [Android MobileGLTraceRe/INFO]: "
@@ -107,11 +137,21 @@ def find_tree(arg):
candidates = [arg] + [str(p) for p in HERE.parents] + [
os.environ.get("W2_TOOLS_OVERRIDE"), os.environ.get("W2_PIPE"), str(Path.home() / "w7/pipe")]
for c in candidates:
if c and (Path(c) / "tools/trace_replay/compare_actuals.py").is_file() and (Path(c) / CTS_BASE_REL).is_dir():
if (c and (Path(c) / "tools/trace_replay/compare_actuals.py").is_file()
and (Path(c) / "tools/trace_replay/trace_cases.json").is_file() and (Path(c) / CTS_BASE_REL).is_dir()):
return Path(c)
return None
def canonical_cases():
"""CONTRACT-P7 7.1's denominator, derived here independently of the reducer, the way 30-gate3.sh
writes gate3/cases.txt: -> (the 39 CI split DirectVulkan cases, those minus EXCLUDED)."""
from trace_cases import ci_backends, ci_trace_cases, load_trace_cases, split_trace_cases
split_dv = [c["name"] for c in split_trace_cases(ci_trace_cases(load_trace_cases()))
if "DirectVulkan" in ci_backends(c)]
return split_dv, tuple(c for c in split_dv if c not in EXCLUDED)
# ------------------------------------------------------------------------------------------------
# canned window directories
# ------------------------------------------------------------------------------------------------
@@ -122,26 +162,34 @@ class Canned:
def png(self, value):
return self.ca._png_bytes(8, 8, self.ca._solid(8, 8, value))
def gate3(self, out, picture=None, boot=BOOT, cases=CASES, reboot_clean=True):
def gate3(self, out, picture=None, boot=BOOT, cases=None, reboot_clean=True, arms=ARMS, arms_txt=None,
excluded=EXCLUDED):
"""30-gate3.sh's output. picture: {arm or (arm, case): grey value} (default 128 everywhere);
arms: ((arm, repeats), ...) actually run; arms_txt: gate3/arms.txt's text (default: what
30-gate3.sh writes for `arms`)."""
picture = picture or {}
cases = CASES if cases is None else cases
g = out / "gate3"
write(g / "cases.txt", "\n".join(cases) + "\n")
write(g / "excluded.txt", "x-red-on-monolith-case\n")
write(g / "arms.txt", "arms=monolith inproc spawn inproc-ra0\nrepeat=%d\n" % REPEAT)
write(g / "excluded.txt", "\n".join(excluded) + "\n")
repeat = max([n for a, n in arms if a in ("inproc", "spawn")] or [REPEAT])
write(g / "arms.txt", arms_txt if arms_txt is not None
else "arms=%s\nrepeat=%d\n" % (" ".join(a for a, _ in arms), repeat))
write(out / "session/boot-id.txt", boot + "\n")
write(out / "session/reboot-clean.txt",
("reboot-clean: 00000000-0000-0000-0000-000000000000 -> %s\n" % boot) if reboot_clean
else ("NOT reboot-clean (no --reboot): boot_id %s, up 5000.00s\n" % boot))
for arm, n in ARMS:
for arm, n in arms:
for case in cases:
name = case + "-DirectVulkan"
openra = case == OPENRA
for i in range(1, n + 1):
rep = g / "archive" / arm / name / ("repeat-%02d" % i)
write(rep / "result.json", json.dumps({
"passed": True, "statusCode": 0, "backend": "DirectVulkan", "ssim": 0.998,
"ssimThreshold": 0.99, "mismatchPixels": 17, "matchedGoldenPath": "/in/golden.png",
"cropX": 0, "cropY": 0, "cropWidth": 0, "cropHeight": 0}))
write(rep / (name + "-actual.png"), self.png(picture.get(arm, 128)))
"passed": True, "statusCode": 0, "backend": "DirectVulkan", "ssim": 1.0 if openra else 0.998,
"ssimThreshold": 0.99, "mismatchPixels": 0 if openra else 17,
"matchedGoldenPath": "/in/golden.png", "cropX": 0, "cropY": 0, "cropWidth": 0, "cropHeight": 0}))
write(rep / (name + "-actual.png"), self.png(picture.get((arm, case), picture.get(arm, 128))))
write(rep / "logcat.txt", "canned logcat %s %s repeat %d\n" % (arm, case, i))
write(rep / "mobilegl.log", LOGS[arm])
if arm in ("inproc", "spawn"):
@@ -158,15 +206,19 @@ class Canned:
def caselist(self, b):
return self.tree / "tools/cts/caselists" / ("p7-%s-gl46.txt" % b)
def cts(self, out, differences=None, drop=(), empty=(), no_results=(), copy=True, named=None, lists=None):
def cts(self, out, differences=None, drop=(), empty=(), no_results=(), copy=True, named=None, lists=None,
adjudication=None):
"""50-cts-after.sh's output for the five blocks: AFTER = $BASE + `differences` per block.
copy: keep runs/<b>/caselist.txt as 50-cts-after.sh does; named: b -> what caselist.path
says (None = the tools tree's list, '' = no caselist.path); lists: b -> the cases the run
was asked to run (a --limit subset, or a list with cases $BASE lacks)."""
was asked to run (a --limit subset, or a list with cases $BASE lacks); adjudication: the
text of cts/adjudication.tsv."""
differences, lists = differences or {}, lists or {}
skip = set(read_list(self.tree / "tools/cts/caselists/p7-skip.txt"))
c = out / "cts"
c.mkdir(parents=True, exist_ok=True)
if adjudication is not None:
write(c / "adjudication.tsv", adjudication)
rebuilt = {}
for b in BLOCKS:
if b in drop:
@@ -255,6 +307,19 @@ def case(verdict, name):
return {}
def overall(verdict):
return verdict.get("gate3", {}).get("overall")
def g3(verdict, key, default=None):
"""verdict.json's gate3[key], or `default` (an older reducer, --reducer, may lack the key)."""
return verdict.get("gate3", {}).get(key, default)
def contract(verdict, key):
return (g3(verdict, "contract") or {}).get(key)
def r3(v):
return None if v is None else round(v, 3)
@@ -267,17 +332,23 @@ def scenario_control(k, cn, tree):
cn.window("control")
rc, text, v = run_reducer(cn.root, "control", tree)
k.expect("exit 0", rc == 0, "rc=%d\n%s" % (rc, text))
k.expect("GATE3 PASS 36/36", v.get("gate3", {}).get("overall") == "PASS" and "GATE3 PASS: 36/36 cases PASS\n" in text,
text)
k.expect("GATE3 PASS 36/36", overall(v) == "PASS" and "GATE3 PASS: 36/36 cases PASS\n" in text, text)
k.expect("the contract constants held (repeats, four arms, canonical case set)",
g3(v, "contract") == {"repeats": [], "arms": [], "cases": []}
and "case set == the canonical 36 (39 CI split DirectVulkan - 3 exclusions): held" in text,
g3(v, "contract"))
k.expect("CTS PASS, all five blocks", v.get("cts", {}).get("overall") == "PASS"
and [b["block"] for b in v["cts"]["blocks"]] == list(BLOCKS), text)
k.expect("every block delta 0.000 pp", all(r3(block(v, b).get("delta_pp")) == 0.0 for b in BLOCKS))
k.expect("every block delta 0.000 pp, no L case", all(r3(block(v, b).get("delta_pp")) == 0.0
and block(v, b).get("lost") == [] for b in BLOCKS))
k.expect("UBO recorded 'unrun (not in this glcts)'", v.get("cts", {}).get("ubo", {}).get("verdict") == "UNRUN"
and "unrun (not in this glcts)" in text)
s = v.get("gate3", {}).get("session", {})
records = 2 * len(CASES) * (len(ARMS) + sum(n for _, n in ARMS)) # (.done + repeats) x (boot_id, before)
k.expect("session valid: one boot_id over %d records" % records,
s.get("valid") is True and s.get("boot_ids") == [BOOT] and s.get("records") == records, s)
k.expect("RUN_AHEAD=0 control on 36/36 cases, proven", "RUN_AHEAD=0 control (required on every case, picture "
"recorded, not gating): 36/36 case(s); proofs: run-ahead=0" in text, text)
k.expect("verdict line", "WINDOW2 GATE3 PASS | CTS PASS -> exit 0" in text, text[-400:])
@@ -295,7 +366,7 @@ def scenario_missing_block(k, cn, tree):
and "has no results" in " ".join(block(v, "texture").get("reasons", [])), block(v, "texture"))
k.expect("the two complete blocks still PASS", block(v, "shader-image").get("verdict") == "PASS"
and block(v, "packed-pixels").get("verdict") == "PASS")
k.expect("GATE3 unaffected (PASS)", v.get("gate3", {}).get("overall") == "PASS")
k.expect("GATE3 unaffected (PASS)", overall(v) == "PASS")
cn.window("no-cts", cts=False)
rc, text, v = run_reducer(cn.root, "no-cts", tree)
k.expect("no cts/ at all -> CTS INCOMPLETE, exit != 0", rc != 0 and v.get("cts", {}).get("overall") == "INCOMPLETE"
@@ -305,84 +376,220 @@ def scenario_missing_block(k, cn, tree):
def scenario_boot_id(k, cn, tree):
print("[boot-id-change] every gate-3 record must carry the session's one boot_id")
out = cn.window("boot-done")
done = out / "gate3/state/spawn" / (CASES[1] + ".done")
done = out / "gate3/state/spawn" / (B + ".done")
done.write_text(done.read_text().replace(BOOT, OTHER_BOOT))
rc, text, v = run_reducer(cn.root, "boot-done", tree)
s = v.get("gate3", {}).get("session", {})
k.expect("spawn .done on another boot -> GATE3 INVALID-SESSION, exit != 0",
rc != 0 and v.get("gate3", {}).get("overall") == "INVALID-SESSION" and "GATE3 INVALID-SESSION" in text, text)
rc != 0 and overall(v) == "INVALID-SESSION" and "GATE3 INVALID-SESSION" in text, text)
k.expect("the other boot_id is named", any(OTHER_BOOT in p for p in s.get("problems", [])), s)
k.expect("...although every case's pictures PASS", all(e["verdict"] == "PASS" for e in v["gate3"]["cases"]))
k.expect("...although every case's pictures PASS", all(e["verdict"] == "PASS" for e in g3(v, "cases", [])))
out = cn.window("boot-repeat")
rep = out / "gate3/archive/inproc" / (CASES[0] + "-DirectVulkan") / "repeat-02" / "boot_id.txt"
rep = out / "gate3/archive/inproc" / (A + "-DirectVulkan") / "repeat-02" / "boot_id.txt"
rep.write_text("boot_id=%s boot_id_before=%s\n" % (OTHER_BOOT, BOOT))
rc, text, v = run_reducer(cn.root, "boot-repeat", tree)
k.expect("one repeat's boot_id.txt on another boot -> INVALID-SESSION",
rc != 0 and v.get("gate3", {}).get("overall") == "INVALID-SESSION", text)
rc != 0 and overall(v) == "INVALID-SESSION", text)
out = cn.window("boot-old-format")
done = out / "gate3/state/monolith" / (CASES[0] + ".done")
done = out / "gate3/state/monolith" / (A + ".done")
done.write_text("rc=0 seconds=8 attempts=1 passed_lines=1 finished=2026-09-23T09:40:14-04:00\n")
rc, text, v = run_reducer(cn.root, "boot-old-format", tree)
k.expect(".done without boot_id (pre-ruling format) -> INVALID-SESSION",
rc != 0 and v.get("gate3", {}).get("overall") == "INVALID-SESSION"
rc != 0 and overall(v) == "INVALID-SESSION"
and any("carry no boot_id" in p for p in v["gate3"]["session"]["problems"]), text)
out = cn.window("boot-no-session")
(out / "session/boot-id.txt").unlink()
rc, text, v = run_reducer(cn.root, "boot-no-session", tree)
k.expect("no session/boot-id.txt -> INVALID-SESSION",
rc != 0 and v.get("gate3", {}).get("overall") == "INVALID-SESSION", text)
k.expect("no session/boot-id.txt -> INVALID-SESSION", rc != 0 and overall(v) == "INVALID-SESSION", text)
def scenario_monolith_arm(k, cn, tree):
print("[monolith-arm] the monolith arm must be the monolith; split arms must prove theirs")
out = cn.window("mono-ipc")
log = out / "gate3/archive/monolith" / (CASES[0] + "-DirectVulkan") / "repeat-01" / "mobilegl.log"
log = out / "gate3/archive/monolith" / (A + "-DirectVulkan") / "repeat-01" / "mobilegl.log"
log.write_text(LOGS["monolith"] + TAG + IPC % 1 + "\n")
rc, text, v = run_reducer(cn.root, "mono-ipc", tree)
e = case(v, CASES[0])
e = case(v, A)
k.expect("monolith log with Config: IPC -> case FAIL, GATE3 FAIL",
e.get("verdict") == "FAIL" and v["gate3"]["overall"] == "FAIL" and rc != 0
e.get("verdict") == "FAIL" and overall(v) == "FAIL" and rc != 0
and any("NOT proven monolith" in r for r in e.get("reasons", [])), e)
k.expect("the other case still PASS", case(v, CASES[1]).get("verdict") == "PASS")
k.expect("the other case still PASS", case(v, B).get("verdict") == "PASS")
k.expect("arm identity line counts 35/36 monolith", "monolith 35/36 repeat(s)" in text, text)
out = cn.window("mono-nolog")
(out / "gate3/archive/monolith" / (CASES[1] + "-DirectVulkan") / "repeat-01" / "mobilegl.log").unlink()
(out / "gate3/archive/monolith" / (B + "-DirectVulkan") / "repeat-01" / "mobilegl.log").unlink()
rc, text, v = run_reducer(cn.root, "mono-nolog", tree)
e = case(v, CASES[1])
e = case(v, B)
k.expect("monolith repeat without mobilegl.log -> FAIL (identity unproven)",
e.get("verdict") == "FAIL" and any("unproven" in r for r in e.get("reasons", [])), e)
out = cn.window("split-proof")
p = out / "gate3/archive/spawn" / (CASES[0] + "-DirectVulkan") / "repeat-02" / "transport-proof.json"
p = out / "gate3/archive/spawn" / (A + "-DirectVulkan") / "repeat-02" / "transport-proof.json"
p.write_text(json.dumps({"required_transport": "spawn", "passed": False}))
rc, text, v = run_reducer(cn.root, "split-proof", tree)
e = case(v, CASES[0])
e = case(v, A)
k.expect("spawn transport-proof not passed -> FAIL",
e.get("verdict") == "FAIL" and any("spawn rep2 arm proof" in r for r in e.get("reasons", [])), e)
def scenario_repeats(k, cn, tree):
print("[repeats] inproc and spawn need CONTRACT-P7 7.2's 3 repeats, whatever gate3/arms.txt says")
one = (("monolith", 1), ("inproc", 1), ("spawn", 1), ("inproc-ra0", 1))
cn.window("rep1", gate3_args={"arms": one})
rc, text, v = run_reducer(cn.root, "rep1", tree)
k.expect("a --repeat 1 run (arms.txt repeat=1), every picture passing -> GATE3 FAIL-REPEATS, exit != 0",
overall(v) == "FAIL-REPEATS" and "\nGATE3 FAIL-REPEATS: " in text and rc != 0
and "WINDOW2 GATE3 FAIL-REPEATS | CTS PASS -> exit 1" in text, text[-600:])
k.expect("...naming arms.txt's repeat=1 and the 72 short finished pairs",
any("repeat=1 < CONTRACT-P7 7.2's 3" in p for p in (contract(v, "repeats") or []))
and any(p.startswith("72 finished inproc/spawn pair(s)") for p in contract(v, "repeats")),
g3(v, "contract"))
k.expect("...each case FAIL on '1 repeat(s) in a finished pair'",
all(e["verdict"] == "FAIL" and any("inproc: 1 repeat(s) in a finished pair" in r for r in e["reasons"])
for e in g3(v, "cases", [])), case(v, A))
cn.window("rep1-forged", gate3_args={"arms": one, "arms_txt": "arms=monolith inproc spawn inproc-ra0\nrepeat=3\n"})
rc, text, v = run_reducer(cn.root, "rep1-forged", tree)
k.expect("the same one-repeat archive under an arms.txt that says repeat=3 -> still GATE3 FAIL-REPEATS",
overall(v) == "FAIL-REPEATS" and rc != 0 and g3(v, "repeat") == 3
and any(p.startswith("72 finished inproc/spawn pair(s)") for p in contract(v, "repeats")),
(overall(v), g3(v, "contract")))
out = cn.window("rep-short")
shutil.rmtree(out / "gate3/archive/spawn" / (A + "-DirectVulkan") / "repeat-03")
rc, text, v = run_reducer(cn.root, "rep-short", tree)
k.expect("one finished spawn pair with 2 repeats -> GATE3 FAIL-REPEATS naming spawn/%s (2)" % A,
overall(v) == "FAIL-REPEATS" and rc != 0
and any(("spawn/%s (2)" % A) in p for p in contract(v, "repeats")), g3(v, "contract"))
out = cn.window("rep-unfinished")
shutil.rmtree(out / "gate3/archive/spawn" / (A + "-DirectVulkan") / "repeat-03")
(out / "gate3/state/spawn" / (A + ".done")).unlink()
rc, text, v = run_reducer(cn.root, "rep-unfinished", tree)
k.expect("...the same pair without its .done (a resume re-runs it) -> INCOMPLETE, not FAIL-REPEATS",
overall(v) == "INCOMPLETE" and case(v, A).get("verdict") == "INCOMPLETE" and rc != 0
and contract(v, "repeats") == [], (overall(v), case(v, A).get("reasons")))
def scenario_arms(k, cn, tree):
print("[arms] monolith, inproc, spawn and inproc-ra0 on every case")
three = ARMS[:3]
cn.window("no-ra0", gate3_args={"arms": three})
rc, text, v = run_reducer(cn.root, "no-ra0", tree)
k.expect("a run started with --arms monolith,inproc,spawn -> GATE3 FAIL-ARMS, exit != 0",
overall(v) == "FAIL-ARMS" and "\nGATE3 FAIL-ARMS: " in text and rc != 0
and any("arm inproc-ra0 has no archive" in p for p in contract(v, "arms")), text[-600:])
k.expect("...each case INCOMPLETE on 'inproc-ra0: not run'",
all(any("inproc-ra0: not run" in r for r in e["reasons"]) for e in g3(v, "cases", [])), case(v, A))
cn.window("no-spawn", gate3_args={"arms": (ARMS[0], ARMS[1], ARMS[3])})
rc, text, v = run_reducer(cn.root, "no-spawn", tree)
k.expect("a run started without spawn -> GATE3 FAIL-ARMS", overall(v) == "FAIL-ARMS" and rc != 0
and any("arm spawn has no archive" in p for p in contract(v, "arms")), overall(v))
cn.window("ra0-planned", gate3_args={"arms": three, "arms_txt": "arms=monolith inproc spawn inproc-ra0\nrepeat=3\n"})
rc, text, v = run_reducer(cn.root, "ra0-planned", tree)
k.expect("inproc-ra0 named in arms.txt but not reached yet -> GATE3 INCOMPLETE (not PASS), exit != 0",
overall(v) == "INCOMPLETE" and rc != 0 and contract(v, "arms") == []
and all(e["verdict"] == "INCOMPLETE" for e in g3(v, "cases", [])), overall(v))
out = cn.window("ra0-one-case")
shutil.rmtree(out / "gate3/archive/inproc-ra0" / (B + "-DirectVulkan"))
(out / "gate3/state/inproc-ra0" / (B + ".done")).unlink()
rc, text, v = run_reducer(cn.root, "ra0-one-case", tree)
k.expect("one case without its inproc-ra0 -> that case INCOMPLETE, GATE3 INCOMPLETE",
case(v, B).get("verdict") == "INCOMPLETE" and overall(v) == "INCOMPLETE" and rc != 0
and case(v, A).get("verdict") == "PASS", (case(v, B).get("reasons"), overall(v)))
out = cn.window("ra0-not-proven")
(out / "gate3/archive/inproc-ra0" / (A + "-DirectVulkan") / "repeat-01" / "mobilegl.log").write_text(LOGS["inproc"])
rc, text, v = run_reducer(cn.root, "ra0-not-proven", tree)
k.expect("an inproc-ra0 repeat whose log says run-ahead=1 -> that case FAIL (not the control arm)",
case(v, A).get("verdict") == "FAIL" and overall(v) == "FAIL" and rc != 0
and any("not proven the RUN_AHEAD=0 arm" in r for r in case(v, A).get("reasons", [])), case(v, A))
def scenario_case_set(k, cn, tree):
print("[case-set] gate3/cases.txt must EQUAL the canonical denominator, not merely count 36")
swapped = tuple(c for c in CASES if c != B) + (EXCLUDED[0],)
cn.window("set-swap", gate3_args={"cases": swapped})
rc, text, v = run_reducer(cn.root, "set-swap", tree)
k.expect("36 cases = 35 canonical + create-indirect, all passing -> GATE3 FAIL-CASESET, exit != 0",
len(swapped) == 36 and overall(v) == "FAIL-CASESET" and "\nGATE3 FAIL-CASESET: 36/36 cases PASS" in text
and rc != 0, text[-600:])
k.expect("...naming the case outside and the canonical case missing",
any(EXCLUDED[0] in p and ("missing: " + B) in p for p in (contract(v, "cases") or [])),
g3(v, "contract"))
bogus = tuple(c for c in CASES if c != B) + ("minecraft-9.9-not-a-trace-case",)
cn.window("set-bogus", gate3_args={"cases": bogus})
rc, text, v = run_reducer(cn.root, "set-bogus", tree)
k.expect("36 cases = 35 canonical + a name outside the manifest -> FAIL-CASESET",
overall(v) == "FAIL-CASESET" and rc != 0, overall(v))
cn.window("set-37", gate3_args={"cases": CASES + (EXCLUDED[2],)})
rc, text, v = run_reducer(cn.root, "set-37", tree)
k.expect("the canonical 36 + photon-v1.3b (37) -> FAIL-CASESET", overall(v) == "FAIL-CASESET" and rc != 0, overall(v))
out = cn.window("set-dup")
write(out / "gate3/cases.txt", "\n".join(CASES + (B,)) + "\n")
rc, text, v = run_reducer(cn.root, "set-dup", tree)
k.expect("the canonical 36 with one listed twice -> FAIL-CASESET ('more than once')",
overall(v) == "FAIL-CASESET" and rc != 0
and any("more than once" in p and B in p for p in contract(v, "cases")), overall(v))
def scenario_cross_arm(k, cn, tree):
print("[cross-arm] cross-arm identity is reported (WARN), within-arm identity is the gate")
out = cn.root / "cross"
out.mkdir()
cn.gate3(out, picture={"spawn": 129})
cn.cts(out)
print("[cross-arm] cross-arm identity is not a per-case gate, but every mismatch needs an adjudication")
cn.window("cross", gate3_args={"picture": {"spawn": 129}})
rc, text, v = run_reducer(cn.root, "cross", tree)
k.expect("spawn != inproc != monolith pictures -> GATE3 still PASS, exit 0",
rc == 0 and v.get("gate3", {}).get("overall") == "PASS", text)
k.expect("spawn != inproc == monolith on every case -> GATE3 PASS-NEEDS-ADJUDICATION, exit 1",
rc == 1 and overall(v) == "PASS-NEEDS-ADJUDICATION" and "\nGATE3 PASS-NEEDS-ADJUDICATION: 36/36 cases PASS" in text
and "WINDOW2 GATE3 PASS-NEEDS-ADJUDICATION | CTS PASS -> exit 1" in text, text[-600:])
k.expect("...every case still PASS (not a per-case gate)", all(e["verdict"] == "PASS" for e in g3(v, "cases", [])))
k.expect("a WARN cross-arm line per case", all(("WARN cross-arm %s:" % c) in text for c in CASES), text)
k.expect("cross_arm.identical false in the JSON", all(case(v, c).get("cross_arm", {}).get("identical") is False for c in CASES))
k.expect("the ruling is named", "ruling: integrator 2026-09-23" in text)
k.expect("cross_arm.identical false and 36 unadjudicated in the JSON",
all(case(v, c).get("cross_arm", {}).get("identical") is False for c in CASES)
and g3(v, "cross_arm_unadjudicated") == list(CASES))
x = case(v, B).get("cross_arm", {})
k.expect("the list carries the archived px!=golden and the direct px counts (8x8: 64 between 128 and 129)",
x.get("px_vs_golden") == {"monolith": 17, "inproc": [17, 17, 17], "spawn": [17, 17, 17]}
and x.get("px_direct") == {"inproc-monolith": 0, "spawn-monolith": 64, "inproc-spawn": 64}
and "px!=golden (archived mismatchPixels) monolith 17, inproc 17..17, spawn 17..17; px between the arms' "
"first repeats: inproc-monolith 0, spawn-monolith 64, inproc-spawn 64" in text, x)
k.expect("both rulings are named", "ruling: integrator 2026-09-23;" in text and CODEX_RULING in text)
out = cn.window("cross-adj", gate3_args={"picture": {"spawn": 129}})
write(out / "gate3/adjudication.tsv", "# case\treason\n" + "".join(
"%s\tspawn +1 grey level everywhere, diffed by hand: driver rounding (canned)\n" % c for c in CASES))
rc, text, v = run_reducer(cn.root, "cross-adj", tree)
k.expect("...every mismatching case adjudicated in gate3/adjudication.tsv -> GATE3 PASS, exit 0",
rc == 0 and overall(v) == "PASS" and "GATE3 PASS: 36/36 cases PASS\n" in text
and text.count("adjudicated: spawn +1 grey level") == 36, text[-600:])
out = cn.window("cross-adj-partial", gate3_args={"picture": {"spawn": 129}})
write(out / "gate3/adjudication.tsv", "".join("%s\tcanned reason\n" % c for c in CASES if c != B) + B + "\n")
rc, text, v = run_reducer(cn.root, "cross-adj-partial", tree)
k.expect("...35 adjudicated + %s on a line without a reason -> still PASS-NEEDS-ADJUDICATION naming it" % B,
rc == 1 and overall(v) == "PASS-NEEDS-ADJUDICATION" and g3(v, "cross_arm_unadjudicated") == [B]
and "adjudication.tsv:36: %s has no reason" % B in text, (overall(v), g3(v, "cross_arm_unadjudicated")))
out = cn.window("cross-one", gate3_args={"picture": {("spawn", A): 129}})
rc, text, v = run_reducer(cn.root, "cross-one", tree)
k.expect("only %s's spawn differs -> PASS-NEEDS-ADJUDICATION listing that one case" % A,
rc == 1 and overall(v) == "PASS-NEEDS-ADJUDICATION" and g3(v, "cross_arm_mismatched") == [A]
and "UNADJUDICATED: no gate3/adjudication.tsv line" in text, (overall(v), g3(v, "cross_arm_mismatched")))
write(out / "gate3/adjudication.tsv", "%s\tone LSB on spawn, canned\n%s\tstale line\n" % (A, B))
rc, text, v = run_reducer(cn.root, "cross-one", tree)
k.expect("...adjudicated (plus a stale line for a case whose arms agree) -> GATE3 PASS, exit 0, stale noted",
rc == 0 and overall(v) == "PASS" and (g3(v, "adjudication") or {}).get("stale") == [B]
and "names case(s) whose arms agree (nothing to adjudicate): " + B in text, text[-600:])
out = cn.window("within")
rep = out / "gate3/archive/inproc" / (CASES[1] + "-DirectVulkan") / "repeat-03"
(rep / (CASES[1] + "-DirectVulkan-actual.png")).write_bytes(cn.png(127))
rep = out / "gate3/archive/inproc" / (B + "-DirectVulkan") / "repeat-03"
(rep / (B + "-DirectVulkan-actual.png")).write_bytes(cn.png(127))
rc, text, v = run_reducer(cn.root, "within", tree)
e = case(v, CASES[1])
e = case(v, B)
k.expect("one inproc repeat differs from its siblings -> FAIL",
e.get("verdict") == "FAIL" and any("inproc repeats NOT bit-identical" in r for r in e.get("reasons", [])), e)
@@ -390,16 +597,16 @@ def scenario_cross_arm(k, cn, tree):
def scenario_unfinished(k, cn, tree):
print("[unfinished] a pair without .done or with a repeat lacking its picture is INCOMPLETE")
out = cn.window("nodone")
(out / "gate3/state/inproc" / (CASES[1] + ".done")).unlink()
(out / "gate3/state/inproc" / (B + ".done")).unlink()
rc, text, v = run_reducer(cn.root, "nodone", tree)
e = case(v, CASES[1])
e = case(v, B)
k.expect("pair without .done -> case INCOMPLETE, GATE3 INCOMPLETE",
e.get("verdict") == "INCOMPLETE" and v["gate3"]["overall"] == "INCOMPLETE" and rc != 0
e.get("verdict") == "INCOMPLETE" and overall(v) == "INCOMPLETE" and rc != 0
and any("no state .done" in r for r in e.get("reasons", [])), e)
out = cn.window("nopng")
(out / "gate3/archive/spawn" / (CASES[0] + "-DirectVulkan") / "repeat-03" / (CASES[0] + "-DirectVulkan-actual.png")).unlink()
(out / "gate3/archive/spawn" / (A + "-DirectVulkan") / "repeat-03" / (A + "-DirectVulkan-actual.png")).unlink()
rc, text, v = run_reducer(cn.root, "nopng", tree)
e = case(v, CASES[0])
e = case(v, A)
k.expect("repeat without actual PNG -> INCOMPLETE", e.get("verdict") == "INCOMPLETE"
and any("no *-actual.png" in r for r in e.get("reasons", [])), e)
@@ -417,7 +624,7 @@ def dryrun_differences():
def scenario_rate_gate(k, cn, tree):
print("[rate-gate] the contract rate alone decides a block (no crash anywhere)")
print("[rate-gate] the gate rate alone decides a block (no crash anywhere)")
flips = pass_cases(cn, "dsa")
cn.window("rate-1", differences={"dsa": {x: "Fail" for x in flips[:1]}})
rc, text, v = run_reducer(cn.root, "rate-1", tree)
@@ -428,9 +635,9 @@ def scenario_rate_gate(k, cn, tree):
cn.window("rate-2", gate3=False, differences={"dsa": {x: "Fail" for x in flips[:2]}})
rc, text, v = run_reducer(cn.root, "rate-2", tree)
d = block(v, "dsa")
k.expect("2 dsa Pass->Fail: -0.541 pp < -0.5 pp -> dsa FAIL on the rate alone (0 new crash)",
k.expect("2 dsa Pass->Fail: -0.541 pp < -0.5 pp -> dsa FAIL on the rate alone (0 new crash, 0 L)",
r3(d.get("delta_pp")) == -0.541 and d.get("verdict") == "FAIL" and d.get("new_crash") == []
and d.get("reasons") == ["delta -0.541 pp < -0.5 pp (Pass/(Pass+Fail))"],
and d.get("lost") == [] and d.get("reasons") == ["delta -0.541 pp < -0.5 pp (Pass/(Pass+Fail+L))"],
(d.get("delta_pp"), d.get("verdict"), d.get("reasons")))
cn.window("rate-4", gate3=False, differences={"dsa": {x: "Fail" for x in flips[:4]}})
rc, text, v = run_reducer(cn.root, "rate-4", tree)
@@ -442,6 +649,74 @@ def scenario_rate_gate(k, cn, tree):
(d.get("after"), d.get("delta_pp"), d.get("verdict"), rc))
def scenario_cts_lost(k, cn, tree):
print("[cts-lost] a $BASE Pass that AFTER turns NotSupported / another non-Pass, non-Fail status is NON-PASS")
dsa, ssbo = pass_cases(cn, "dsa"), pass_cases(cn, "ssbo")
ns = dsa[0]
cn.window("lost-ns", differences={"dsa": {ns: "NotSupported"}})
rc, text, v = run_reducer(cn.root, "lost-ns", tree)
d = block(v, "dsa")
k.expect("dsa Pass->NS: gate rate 367/(367+2+1) = -0.270 pp, the literal Pass/(Pass+Fail) -0.001 pp as info",
r3(d.get("delta_pp")) == -0.27 and r3(d.get("delta_pp_literal_info")) == -0.001
and d.get("after", {}).get("lost") == 1 and r3(100 * d["after"]["rate"]) == 99.189
and "info literal Pass/(Pass+Fail): BASE 99.459% AFTER 99.458% delta -0.001 pp (not gating)" in text,
(d.get("delta_pp"), d.get("delta_pp_literal_info"), d.get("after")))
k.expect("...listed (LOST ... Pass -> NotSupported, UNADJUDICATED), dsa FAIL, CTS FAIL, exit != 0",
d.get("lost") == [{"case": ns, "after": "NotSupported", "adjudication": None}] and d.get("verdict") == "FAIL"
and ("LOST %s (Pass -> NotSupported) UNADJUDICATED" % ns) in text
and any("lost to NotSupported / another non-Pass, non-Fail status with no cts/adjudication.tsv line" in r
for r in d.get("reasons", [])) and v["cts"]["overall"] == "FAIL" and rc != 0
and "WARN dsa: NotSupported $BASE 0 -> AFTER 1" in text, (d.get("lost"), d.get("reasons"), rc))
reason = "CTS NotSupported: extension probe differs by design (canned)"
cn.window("lost-ns-adj", differences={"dsa": {ns: "NotSupported"}}, adjudication="%s\t%s\n" % (ns, reason))
rc, text, v = run_reducer(cn.root, "lost-ns-adj", tree)
d = block(v, "dsa")
k.expect("...with a cts/adjudication.tsv line: dsa PASS, CTS PASS, window exit 0, still NON-PASS in the rate",
d.get("verdict") == "PASS" and v["cts"]["overall"] == "PASS" and rc == 0 and r3(d.get("delta_pp")) == -0.27
and d.get("lost") == [{"case": ns, "after": "NotSupported", "adjudication": reason}]
and ("adjudicated: " + reason) in text, (d.get("verdict"), d.get("reasons"), rc))
cn.window("lost-ns-noreason", gate3=False, differences={"dsa": {ns: "NotSupported"}}, adjudication=ns + "\n")
rc, text, v = run_reducer(cn.root, "lost-ns-noreason", tree)
d = block(v, "dsa")
k.expect("...an adjudication line without a reason adjudicates nothing -> dsa FAIL, WARN on the line",
d.get("verdict") == "FAIL" and d.get("lost_unadjudicated") == [ns]
and ("cts/adjudication.tsv:1: %s has no reason" % ns) in text, (d.get("verdict"), text[-300:]))
cn.window("lost-ssbo-adj", gate3=False, differences={"ssbo": {ssbo[0]: "NotSupported"}},
adjudication="%s\t%s\n" % (ssbo[0], reason))
rc, text, v = run_reducer(cn.root, "lost-ssbo-adj", tree)
s = block(v, "ssbo")
k.expect("ssbo Pass->NS adjudicated: 123/(123+0+1) = -0.806 pp -> still FAIL on the rate (adjudication != rate)",
s.get("verdict") == "FAIL" and r3(s.get("delta_pp")) == -0.806 and s.get("lost_unadjudicated") == []
and "delta -0.806 pp < -0.5 pp (Pass/(Pass+Fail+L))" in s.get("reasons", []), (s.get("delta_pp"), s.get("reasons")))
cn.window("lost-codex", gate3=False, differences={"ssbo": {x: "NotSupported" for x in ssbo[1:]}})
rc, text, v = run_reducer(cn.root, "lost-codex", tree)
s = block(v, "ssbo")
k.expect("codex's example: 123 of 124 ssbo Passes -> NS, one still passes: literal 100% (+0.000 pp) but gate "
"rate 1/124 -> FAIL, all 123 listed",
r3(s.get("delta_pp_literal_info")) == 0.0 and r3(100 * s.get("after", {}).get("literal_rate", 0)) == 100.0
and r3(100 * s.get("after", {}).get("rate", 1)) == 0.806 and s.get("verdict") == "FAIL"
and len(s.get("lost", [])) == 123 and text.count("(Pass -> NotSupported) UNADJUDICATED") == 123,
(s.get("delta_pp_literal_info"), s.get("after"), s.get("verdict")))
cn.window("lost-warn", gate3=False, differences={"dsa": {ns: "CompatibilityWarning"}})
rc, text, v = run_reducer(cn.root, "lost-warn", tree)
d = block(v, "dsa")
k.expect("dsa Pass->CompatibilityWarning (any non-Pass, non-Fail status) -> an L case, dsa FAIL",
d.get("lost") == [{"case": ns, "after": "CompatibilityWarning", "adjudication": None}]
and d.get("verdict") == "FAIL" and r3(d.get("delta_pp")) == -0.27, (d.get("lost"), d.get("verdict")))
cn.window("lost-crash-adj", gate3=False, differences={"dsa": {ns: "Crash"}}, adjudication="%s\t%s\n" % (ns, reason))
rc, text, v = run_reducer(cn.root, "lost-crash-adj", tree)
d = block(v, "dsa")
k.expect("dsa Pass->Crash adjudicated -> the L case is adjudicated, but the NEW crash still makes dsa FAIL",
d.get("verdict") == "FAIL" and d.get("new_crash") == [ns] and d.get("lost_unadjudicated") == []
and d.get("reasons", [None])[0] == "1 NEW crash(es)", (d.get("verdict"), d.get("reasons")))
def scenario_vacuous(k, cn, tree):
print("[vacuous] a block with nothing to compare is never PASS")
_, diffs = dryrun_differences()
@@ -455,8 +730,8 @@ def scenario_vacuous(k, cn, tree):
cn.window("gone-copy", gate3=False, differences=diffs, named=lambda b: GONE % b)
rc, text, v = run_reducer(cn.root, "gone-copy", tree)
d = block(v, "dsa")
k.expect("...with the window's caselist.txt copy: the real reading (dsa -0.272 pp, 2 new crashes, CTS FAIL)",
r3(d.get("delta_pp")) == -0.272 and d.get("caselist", "").endswith("runs/dsa/caselist.txt")
k.expect("...with the window's caselist.txt copy: the real reading (dsa -0.541 pp, 2 new crashes, CTS FAIL)",
r3(d.get("delta_pp")) == -0.541 and d.get("caselist", "").endswith("runs/dsa/caselist.txt")
and v.get("cts", {}).get("overall") == "FAIL", (d.get("delta_pp"), d.get("caselist"), v.get("cts", {}).get("overall")))
cn.window("no-caselist", gate3=False, copy=False, named=lambda b: "")
rc, text, v = run_reducer(cn.root, "no-caselist", tree)
@@ -472,9 +747,11 @@ def scenario_vacuous(k, cn, tree):
ns = {b: {x: "NotSupported" for x in cn.base(b)} for b in BLOCKS}
cn.window("all-ns", differences=ns)
rc, text, v = run_reducer(cn.root, "all-ns", tree)
k.expect("every AFTER result NotSupported -> every block FAIL ('AFTER has no Pass/Fail'), CTS FAIL, exit != 0",
k.expect("every AFTER result NotSupported -> every block FAIL ('AFTER has no Pass/Fail'), gate rate 0, every "
"$BASE Pass an L case, CTS FAIL, exit != 0",
rc != 0 and v.get("cts", {}).get("overall") == "FAIL" and "CTS PASS" not in text
and all(block(v, b).get("verdict") == "FAIL" and block(v, b).get("delta_pp") is None
and all(block(v, b).get("verdict") == "FAIL" and block(v, b)["after"]["rate"] == 0.0
and block(v, b)["after"]["lost"] == block(v, b)["base"]["pass"]
and any("AFTER has no Pass/Fail" in r for r in block(v, b).get("reasons", [])) for b in BLOCKS), text)
k.expect("...and a WARN on each block's NotSupported count", all(("WARN %s: NotSupported $BASE" % b) in text for b in BLOCKS), text)
@@ -483,39 +760,34 @@ def scenario_not_a_gate(k, cn, tree):
print("[not-a-gate] a pass that is not the gate's own reading never exits 0")
cn.window("subset-g3", gate3_args={"cases": CASES[:2]})
rc, text, v = run_reducer(cn.root, "subset-g3", tree)
k.expect("2 of 36 gate-3 cases, all passing -> GATE3 PASS-SUBSET, exit != 0",
v.get("gate3", {}).get("overall") == "PASS-SUBSET" and "GATE3 PASS-SUBSET: 2/2 cases PASS" in text
k.expect("2 of the canonical 36, all passing -> GATE3 PASS-SUBSET, exit != 0",
overall(v) == "PASS-SUBSET" and "GATE3 PASS-SUBSET: 2/2 cases PASS" in text
and "2 of the canonical 36 case(s)" in text
and "WINDOW2 GATE3 PASS-SUBSET | CTS PASS -> exit 1" in text and rc != 0, text[-500:])
cn.window("not-clean", gate3_args={"reboot_clean": False})
rc, text, v = run_reducer(cn.root, "not-clean", tree)
k.expect("36/36 in a session that is not reboot-clean -> GATE3 PASS-NOT-REBOOT-CLEAN, exit != 0",
v.get("gate3", {}).get("overall") == "PASS-NOT-REBOOT-CLEAN" and rc != 0
and "session not reboot-clean" in text, text[-500:])
overall(v) == "PASS-NOT-REBOOT-CLEAN" and rc != 0 and "session not reboot-clean" in text, text[-500:])
first20 = read_list(cn.caselist("dsa"))[:20]
cn.window("subset-cts", lists={"dsa": first20})
rc, text, v = run_reducer(cn.root, "subset-cts", tree)
k.expect("a --limit 20 dsa block, all passing -> CTS PASS-SUBSET (dsa marked), GATE3 PASS, exit != 0",
v.get("cts", {}).get("overall") == "PASS-SUBSET" and block(v, "dsa").get("subset") is True
and block(v, "dsa").get("cases") == 20 and v.get("gate3", {}).get("overall") == "PASS" and rc != 0
and block(v, "dsa").get("cases") == 20 and overall(v) == "PASS" and rc != 0
and "SUBSET run: dsa" in text, text[-500:])
def scenario_warnings(k, cn, tree):
print("[warnings] like-for-like rates and the NotSupported count are reported, not gated")
print("[warnings] like-for-like rates are reported, not gated")
extra = "KHR-GL46.texture_selftest.not_in_base"
tex = read_list(cn.caselist("texture")) + [extra]
ssbo_ns = pass_cases(cn, "ssbo")[0]
cn.window("warn", gate3=False, lists={"texture": tex},
differences={"texture": {extra: "Fail"}, "ssbo": {ssbo_ns: "NotSupported"}})
cn.window("warn", gate3=False, lists={"texture": tex}, differences={"texture": {extra: "Fail"}})
rc, text, v = run_reducer(cn.root, "warn", tree)
t, s = block(v, "texture"), block(v, "ssbo")
t = block(v, "texture")
k.expect("a case $BASE lacks (AFTER Fail) stays out of both rates: texture delta 0.000, PASS, compared = cases - 1",
r3(t.get("delta_pp")) == 0.0 and t.get("verdict") == "PASS" and t.get("compared") == t.get("cases", 0) - 1
and t.get("missing_in_base") == [extra], (t.get("delta_pp"), t.get("verdict"), t.get("compared"), t.get("cases")))
k.expect("...with a WARN line naming it", "WARN texture: 1 case(s) have no $BASE result (e.g. %s)" % extra in text, text)
k.expect("an ssbo Pass that became NotSupported: rate unchanged (PASS), WARN on the NS count, in Pass->not",
s.get("verdict") == "PASS" and r3(s.get("delta_pp")) == 0.0 and ssbo_ns in s.get("regressed", [])
and "WARN ssbo: NotSupported $BASE 0 -> AFTER 1" in text, (s.get("verdict"), s.get("delta_pp"), text[-400:]))
def scenario_check_block(k, cn, tree):
@@ -543,33 +815,43 @@ def scenario_check_block(k, cn, tree):
def check_dryrun_numbers(k, v, text, label):
dsa, si, tex = block(v, "dsa"), block(v, "shader-image"), block(v, "texture")
k.expect("%s: dsa contract delta -0.272 pp" % label, r3(dsa.get("delta_pp")) == -0.272, dsa.get("delta_pp"))
k.expect("%s: dsa BASE 368/(368+2) = 99.459%%, AFTER 366/(366+3) = 99.187%%" % label,
r3(100 * dsa["base"]["rate"]) == 99.459 and r3(100 * dsa["after"]["rate"]) == 99.187
if dsa.get("base") else False, dsa)
k.expect("%s: dsa pre-ruling delta -0.539 pp kept as info" % label,
r3(dsa.get("delta_pp_results_info")) == -0.539 and "(-0.539)" in text, dsa.get("delta_pp_results_info"))
k.expect("%s: dsa pre-ruling rates printed (info line 99.191%% -> 98.652%%)" % label,
"info pre-ruling Pass/(results-NS): BASE 99.191% AFTER 98.652% delta -0.539 pp (not gating)" in text, text)
k.expect("%s: dsa FAIL on the new crash only, not on the rate" % label,
dsa.get("verdict") == "FAIL" and dsa.get("reasons") == ["1 NEW crash(es)"]
and dsa.get("new_crash") == [DSA_STORAGE], dsa.get("reasons"))
k.expect("%s: dsa counts P/F/NS/W/X 366/3/0/1/1" % label,
[dsa.get("after", {}).get(x) for x in ("pass", "fail", "ns", "warn", "crash")] == [366, 3, 0, 1, 1])
k.expect("%s: shader-image +4.284 pp, FAIL on new crash incomplete_textures" % label,
r3(si.get("delta_pp")) == 4.284 and si.get("verdict") == "FAIL" and si.get("new_crash") == [SI_INCOMPLETE],
(si.get("delta_pp"), si.get("verdict"), si.get("new_crash")))
k.expect("%s: dsa gate delta -0.541 pp (renderbuffers_storage Pass->Crash is an L case)" % label,
r3(dsa.get("delta_pp")) == -0.541, dsa.get("delta_pp"))
k.expect("%s: dsa BASE 368/(368+2) = 99.459%%, AFTER 366/(366+3+1) = 98.919%%, literal 366/(366+3) = 99.187%%" % label,
r3(100 * dsa["base"]["rate"]) == 99.459 and r3(100 * dsa["after"]["rate"]) == 98.919
and r3(100 * dsa["after"]["literal_rate"]) == 99.187 if dsa.get("base") else False, dsa)
k.expect("%s: dsa literal Pass/(Pass+Fail) -0.272 pp kept as info" % label,
r3(dsa.get("delta_pp_literal_info")) == -0.272 and "(-0.272)" in text
and "info literal Pass/(Pass+Fail): BASE 99.459% AFTER 99.187% delta -0.272 pp (not gating)" in text,
dsa.get("delta_pp_literal_info"))
k.expect("%s: dsa pre-ruling delta -0.539 pp kept as info (info line 99.191%% -> 98.652%%)" % label,
r3(dsa.get("delta_pp_results_info")) == -0.539
and "info pre-ruling Pass/(results-NS): BASE 99.191% AFTER 98.652% delta -0.539 pp (not gating)" in text,
dsa.get("delta_pp_results_info"))
k.expect("%s: dsa FAIL on the new crash, the rate and the unadjudicated loss" % label,
dsa.get("verdict") == "FAIL" and dsa.get("reasons") == [
"1 NEW crash(es)", "delta -0.541 pp < -0.5 pp (Pass/(Pass+Fail+L))",
"1 $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status with no "
"cts/adjudication.tsv line (%s)" % CODEX_RULING]
and dsa.get("new_crash") == [DSA_STORAGE]
and dsa.get("lost") == [{"case": DSA_STORAGE, "after": "Crash", "adjudication": None}], dsa.get("reasons"))
k.expect("%s: dsa counts P/F/NS/W/X 366/3/0/1/1, L 1" % label,
[dsa.get("after", {}).get(x) for x in ("pass", "fail", "ns", "warn", "crash", "lost")] == [366, 3, 0, 1, 1, 1])
k.expect("%s: shader-image +4.284 pp (L 0), FAIL on new crash incomplete_textures" % label,
r3(si.get("delta_pp")) == 4.284 and si.get("verdict") == "FAIL" and si.get("new_crash") == [SI_INCOMPLETE]
and si.get("lost") == [], (si.get("delta_pp"), si.get("verdict"), si.get("new_crash")))
k.expect("%s: texture +0.960 pp PASS" % label, r3(tex.get("delta_pp")) == 0.96 and tex.get("verdict") == "PASS",
(tex.get("delta_pp"), tex.get("verdict")))
k.expect("%s: ssbo, packed-pixels 0.000 pp PASS" % label,
all(r3(block(v, b).get("delta_pp")) == 0.0 and block(v, b).get("verdict") == "PASS"
for b in ("ssbo", "packed-pixels")))
k.expect("%s: CTS FAIL" % label, v.get("cts", {}).get("overall") == "FAIL" and "\nCTS FAIL" in text)
k.expect("%s: printed '-0.272'" % label, "-0.272" in text)
k.expect("%s: printed '-0.541' and the LOST line" % label, "-0.541" in text
and ("LOST %s (Pass -> Crash) UNADJUDICATED" % DSA_STORAGE) in text)
def scenario_dryrun(k, cn, tree, dry_dir):
print("[dry-run] contract formula on the pre-14e1c8b9 dry-run data")
print("[dry-run] the gate rate on the pre-14e1c8b9 dry-run data")
fixture, diffs = dryrun_differences()
out = cn.root / "dryrun"
out.mkdir()
@@ -587,8 +869,7 @@ def scenario_dryrun(k, cn, tree, dry_dir):
if line.strip():
print(" | " + line)
check_dryrun_numbers(k, v, text, "canned")
k.expect("canned: no gate3/ -> GATE3 NOT-RUN, exit != 0",
rc != 0 and v.get("gate3", {}).get("overall") == "NOT-RUN")
k.expect("canned: no gate3/ -> GATE3 NOT-RUN, exit != 0", rc != 0 and overall(v) == "NOT-RUN")
if dry_dir and (Path(dry_dir) / "cts").is_dir():
dry = Path(dry_dir)
rc2, text2, v2 = run_reducer(dry.parent, dry.name, tree, json_out=cn.root / "real-dry-run-verdict.json")
@@ -598,7 +879,7 @@ def scenario_dryrun(k, cn, tree, dry_dir):
def main():
global REDUCER
global REDUCER, CASES, A, B
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--tools", help="MobileGL tree (default: the tree holding this file, else $W2_TOOLS_OVERRIDE)")
ap.add_argument("--dry-run-dir", default=str(Path.home() / "w7/logs/devprep/w2/pre-14e1c8b9"))
@@ -608,12 +889,18 @@ def main():
REDUCER = Path(args.reducer).resolve()
tree = find_tree(args.tools)
if tree is None:
print("no MobileGL tree with tools/trace_replay/compare_actuals.py and %s: pass --tools" % CTS_BASE_REL,
file=sys.stderr)
print("no MobileGL tree with tools/trace_replay/{compare_actuals.py,trace_cases.json} and %s: pass --tools"
% CTS_BASE_REL, file=sys.stderr)
return 2
sys.path.insert(0, str(tree / "tools" / "trace_replay"))
sys.dont_write_bytecode = True # the tree is only read (the working copy falls back to ~/w7/pipe)
import compare_actuals as ca
split_dv, CASES = canonical_cases()
if len(split_dv) != 39 or len(CASES) != 36 or not all(x in split_dv for x in EXCLUDED) or OPENRA not in CASES:
print("%s's trace_cases.json no longer gives CONTRACT-P7 7.1's 39 - 3 = 36 (got %d, %d)"
% (tree, len(split_dv), len(CASES)), file=sys.stderr)
return 2
A, B = [c for c in CASES if c != OPENRA][:2]
root = Path(tempfile.mkdtemp(prefix="w2-reduce-selftest-"))
print("60-reduce self-test: reducer %s, tree %s, canned windows in %s" % (REDUCER, tree, root))
k = Checks()
@@ -623,9 +910,13 @@ def main():
scenario_missing_block(k, cn, tree)
scenario_boot_id(k, cn, tree)
scenario_monolith_arm(k, cn, tree)
scenario_repeats(k, cn, tree)
scenario_arms(k, cn, tree)
scenario_case_set(k, cn, tree)
scenario_cross_arm(k, cn, tree)
scenario_unfinished(k, cn, tree)
scenario_rate_gate(k, cn, tree)
scenario_cts_lost(k, cn, tree)
scenario_vacuous(k, cn, tree)
scenario_not_a_gate(k, cn, tree)
scenario_warnings(k, cn, tree)
docs/Disaggregated/notes/p7/device-window-2/tools/60-reduce.py
Executable → Regular
+332 -73
View File
@@ -4,7 +4,22 @@
Device window 2 verdicts, read from the window's own output directory (<logroot>/<stamp>).
GATE 3 (CONTRACT-P7 7.2), per case of gate3/cases.txt:
GATE 3 (CONTRACT-P7 7.1/7.2), per case of gate3/cases.txt:
* THE CONTRACT'S CONSTANTS, NOT THE RUNNER'S OWN RECORDS (ruling: integrator 2026-09-23 after
codex closeout review). gate3/arms.txt says what 30-gate3.sh was asked to run, not what the
gate needs, so the reducer holds the run to CONTRACT-P7 7.1/7.2 itself:
- REPEATS: inproc and spawn need >= 3 archived repeats on every case. arms.txt repeat < 3,
or a finished pair (state .done) with fewer than 3 archived repeats, is GATE3 FAIL-REPEATS
whatever arms.txt says;
- ARMS: monolith, inproc, spawn AND inproc-ra0 on every case. An arm with no archive that
arms.txt's arms= does not name (the run was started without it) is GATE3 FAIL-ARMS; a named
arm not reached yet, or a case an arm lacks, makes that case INCOMPLETE;
- CASE SET: gate3/cases.txt must EQUAL the canonical denominator, computed the way
30-gate3.sh computes it - the tools tree's trace_cases.json CI split cases whose
ci_backends carry DirectVulkan (39) - minus CONTRACT-P7 7.1's three exclusions, which are
this file's constants (CONTRACT_EXCLUDED), not gate3/excluded.txt. A case outside it, a
case listed twice, or a manifest that no longer gives 39 / 36 is GATE3 FAIL-CASESET; a
proper subset (a --cases run) is PASS-SUBSET at best - never merely "count 36";
* ONE SESSION: every gate3/state/<arm>/<case>.done and every archived repeat's boot_id.txt, of
every arm, carries one boot_id, equal to session/boot-id.txt. A second boot_id, a record that
carries none, or no session/boot-id.txt makes the whole gate GATE3 INVALID-SESSION, whatever
@@ -17,11 +32,18 @@ GATE 3 (CONTRACT-P7 7.2), per case of gate3/cases.txt:
within-arm identity is the gate), and |ssim - monolith ssim| <= 0.0005 on every repeat; a
repeat whose logcat is byte-identical to the previous repeat's is flagged STALE;
* OpenRA: 1.000000 and mismatch 0 on every repeat of both split arms AND on monolith;
* cross-arm picture identity (inproc vs spawn vs monolith) is REPORTED per case and a mismatch
prints a WARN line - information, not a gate (ruling: integrator 2026-09-23);
* inproc-ra0 (RUN_AHEAD=0): recorded (ssim, sha, run-ahead=0 proof), never gating;
* a reading that passes but is not the gate's - a denominator other than 36, or a session that
is not reboot-clean - is GATE3 PASS-SUBSET / PASS-NOT-REBOOT-CLEAN, never PASS.
* cross-arm picture identity (inproc vs spawn vs monolith) is not a per-case gate (ruling:
integrator 2026-09-23, ID-P7-59): a mismatch prints a WARN line and leaves the case's verdict
alone. But it is never waved through silently (ruling: integrator 2026-09-23 after codex
closeout review): every mismatching case must be listed in gate3/adjudication.tsv
('<case><TAB><reason>'), else an otherwise PASS gate is GATE3 PASS-NEEDS-ADJUDICATION (exit 1).
Each mismatch is listed with the archived mismatchPixels (vs golden) of every arm and the
direct pixel counts between the arms' first repeats;
* inproc-ra0 (RUN_AHEAD=0): required on every case (one archived repeat with its .done) and its
role log must prove run-ahead=0 (else that case FAIL: it is not the control arm); its picture
and ssim are recorded, never gating;
* a reading that passes but is not the gate's - a proper subset of the canonical 36, or a
session that is not reboot-clean - is GATE3 PASS-SUBSET / PASS-NOT-REBOOT-CLEAN, never PASS.
BSL STATS: per arm, maps-line / VmRSS / VmHWM peaks and the wbuf[] gauges (information).
CTS AFTER (CONTRACT-P7 7.3), per block vs device-window-1/CTS-base/report-<block>.json, over the
cases the AFTER run was asked to run (so a --limit subset is compared like for like):
@@ -31,21 +53,32 @@ CTS AFTER (CONTRACT-P7 7.3), per block vs device-window-1/CTS-base/report-<block
else the file caselist.path names); a missing one makes the block MISSING and CTS INCOMPLETE -
a block of 0 cases is never a reading. The UBO block (GTF-GL46 uniform_buffer_object) is
recorded as 'unrun (not in this glcts)': the device glcts carries no GTF module (ID-P7-16);
* rate = Pass / (Pass + Fail), the contract formula: NotSupported, the warnings and the crashes
are NOT in the denominator; gate: delta >= -0.5 pp on this rate. Both rates are taken over the
* rate = Pass / (Pass + Fail + L), L = the cases BASE passed that AFTER turned NotSupported or
any other non-Pass, non-Fail status (a warning, a crash): a lost supported case is NON-PASS, it
does not leave the denominator (ruling: integrator 2026-09-23 after codex closeout review -
under the contract's literal Pass / (Pass + Fail), 123 of 124 BASE-Pass cases turning
NotSupported would still read 100%). NotSupported, warnings and crashes that were not a BASE
Pass stay out of both rates. gate: delta >= -0.5 pp on this rate. Both rates are taken over the
cases that have a BASE and an AFTER result (a case BASE lacks prints a WARN, stays out of both
rates and still counts for new crashes). AFTER without a single Pass/Fail where BASE has some
is FAIL (the rate cannot be formed: every BASE Pass is gone); BASE without one is INCOMPLETE;
rates and still counts for new crashes). AFTER without a single Pass/Fail/L where BASE has some
is FAIL (the rate cannot be formed); BASE without one is INCOMPLETE;
* every L case is listed (Pass -> its AFTER status), and a block with one that
cts/adjudication.tsv ('<case><TAB><reason>') does not list is FAIL. An adjudication waives
only that rule: the case still counts as NON-PASS in the rate;
* crashes (Crash/Timeout/InternalError/ResourceError/DeviceHang/Incomplete, plus hung.txt) are
counted in their own column and are non-Pass; one that BASE does not already have is a NEW
crash, a hard red of its own (a Fail that became a Crash is a new crash);
crash, a hard red of its own that no adjudication waives (a Fail that became a Crash is a new
crash; a Pass that became a Crash is a new crash AND an L case);
* a block whose cases are not the tools tree's full p7-<block>-gl46.txt (a --limit run) makes an
all-PASS CTS reading 'PASS-SUBSET', never PASS;
* information only, never gating: the pre-ruling rate Pass / (results - NotSupported) of BASE
and AFTER and its delta, and a WARN when the NotSupported count moved.
Exit status: 0 only when GATE 3 is PASS (36/36, reboot-clean, one session) and CTS is PASS (five
full blocks); 1 otherwise (a section that did not run, a PASS-SUBSET / PASS-NOT-REBOOT-CLEAN
reading, INCOMPLETE / INVALID-SESSION, or FAIL); 2 on bad input.
* information only, never gating: the contract's literal Pass / (Pass + Fail) of BASE and AFTER
and its delta, the pre-ruling Pass / (results - NotSupported) and its delta, and a WARN when
the NotSupported count moved.
Exit status: 0 only when GATE 3 is PASS (the canonical 36, >= 3 repeats, four arms, reboot-clean,
one session, every cross-arm mismatch adjudicated) and CTS is PASS (five full blocks); 1 otherwise
(a section that did not run, a PASS-SUBSET / PASS-NOT-REBOOT-CLEAN / PASS-NEEDS-ADJUDICATION
reading, INCOMPLETE / INVALID-SESSION, FAIL-REPEATS / FAIL-ARMS / FAIL-CASESET, or FAIL); 2 on
bad input.
--check-block RUN_DIR REPORT_JSON: the completeness test 50-cts-after.sh applies before it writes
a block's .done (the same code the verdict uses): exit 0 when the report has a result for every
non-skipped case of the run's caselist and unrun.txt is empty, 3 when not (the reason is printed).
@@ -60,7 +93,16 @@ from collections import Counter
from pathlib import Path
TOL = 0.0005
# CONTRACT-P7 7.1/7.2's constants: the gate is held to these, never to what gate3/arms.txt,
# gate3/cases.txt or gate3/excluded.txt record about the run (ruling: integrator 2026-09-23 after
# codex closeout review).
GATE3_DENOMINATOR = 36
CI_SPLIT_DIRECTVULKAN = 39
CONTRACT_EXCLUDED = ("minecraft-1.21.1-neoforge-create-indirect-in-world", # OQ-17
"minecraft-1.21.11-main-menu", # UBO offset alignment
"minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world") # VK_ERROR_UNKNOWN
CONTRACT_REPEATS = 3
REQUIRED_ARMS = ("monolith", "inproc", "spawn", "inproc-ra0")
CTS_TOL_PP = 0.5
HARD = ("Crash", "Timeout", "InternalError", "ResourceError", "DeviceHang", "Incomplete")
SPLIT_ARMS = ("inproc", "spawn")
@@ -76,6 +118,7 @@ KV = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=(\S*)")
# "Config: IPC ring=..." are printed only when a transport resolves; the monolith prints neither.
NOT_MONOLITH = re.compile(r"Config: IPC|MOBILEGL_TRANSPORT=")
RULING = "ruling: integrator 2026-09-23"
CODEX_RULING = "ruling: integrator 2026-09-23 after codex closeout review"
def sha(path):
@@ -112,6 +155,27 @@ def first_line(path):
return lines[0].strip() if lines else ""
def read_adjudication(path):
"""gate3/adjudication.tsv, cts/adjudication.tsv: '<case><TAB><reason>' per line ('#' comments
and blank lines skipped; a case name holds no whitespace) -> ({case: reason}, problems). A line
without a reason adjudicates nothing and is reported."""
found, problems = {}, []
if not path.is_file():
return found, problems
for number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
text = line.strip()
if not text or text.startswith("#"):
continue
parts = text.split(None, 1)
reason = parts[1].strip() if len(parts) > 1 else ""
if not reason:
problems.append("%s/%s:%d: %s has no reason: it adjudicates nothing"
% (path.parent.name, path.name, number, parts[0]))
continue
found.setdefault(parts[0], reason)
return found, problems
# ------------------------------------------------------------------------------------------------
# gate 3
# ------------------------------------------------------------------------------------------------
@@ -171,19 +235,90 @@ def monolith_identity(row):
return None
def canonical_gate3_cases():
"""CONTRACT-P7 7.1's denominator, computed the way 30-gate3.sh computes gate3/cases.txt: the
tools tree's trace_cases.json CI split cases whose ci_backends carry DirectVulkan, minus the
contract's three exclusions (CONTRACT_EXCLUDED, never the run's own gate3/excluded.txt).
-> (cases in manifest order, or None; problems). A problem means the manifest no longer gives
the contract's 39 / 36, so no run can be scored against it."""
try:
from trace_cases import ci_backends, ci_trace_cases, load_trace_cases, split_trace_cases
split_dv = [c["name"] for c in split_trace_cases(ci_trace_cases(load_trace_cases()))
if "DirectVulkan" in ci_backends(c)]
except Exception as error: # an unreadable manifest is a finding of the reading, not a crash
return None, ["the tools tree's trace_cases.json cannot be read (%s: %s)" % (type(error).__name__, error)]
problems = []
if len(split_dv) != CI_SPLIT_DIRECTVULKAN:
problems.append("trace_cases.json gives %d CI split DirectVulkan case(s), CONTRACT-P7 7.1 names %d"
% (len(split_dv), CI_SPLIT_DIRECTVULKAN))
absent = [x for x in CONTRACT_EXCLUDED if x not in split_dv]
if absent:
problems.append("CONTRACT-P7 7.1 exclusion(s) not among them: " + ", ".join(absent))
canonical = [c for c in split_dv if c not in CONTRACT_EXCLUDED]
if len(set(canonical)) != GATE3_DENOMINATOR:
problems.append("the canonical denominator is %d case(s), CONTRACT-P7 7.1 says %d"
% (len(set(canonical)), GATE3_DENOMINATOR))
return canonical, problems
def px_between(ca, left, right):
"""Exact RGB pixel-inequality count between two archived pictures (compare_actuals'
mismatch_pixels, whole frame), or why it cannot be counted; None when a picture is missing."""
if not left or not right:
return None
try:
a, b = ca.read_png_rgba(left), ca.read_png_rgba(right)
return ca.mismatch_pixels(a, b, *ca.resolve_crop(a, b))
except Exception as error: # ImageError (a size mismatch), OSError, zlib: reported, not fatal
return "n/a (%s)" % error
def gate3(out, ca):
g = out / "gate3"
cases_file = g / "cases.txt"
if not cases_file.is_file():
return None
cases = [c for c in cases_file.read_text().split() if c]
listed = [c for c in cases_file.read_text().split() if c]
cases = list(dict.fromkeys(listed))
excluded = (g / "excluded.txt").read_text().split() if (g / "excluded.txt").is_file() else []
meta = read_kv(g / "arms.txt")
repeat = int(meta.get("repeat", "3"))
arms_present = [a for a in ("monolith", "inproc", "spawn", "inproc-ra0") if (g / "archive" / a).is_dir()]
try:
repeat = int(meta.get("repeat", str(CONTRACT_REPEATS)))
except ValueError:
repeat = 0
planned = meta.get("arms", "").split()
arms_present = [a for a in REQUIRED_ARMS if (g / "archive" / a).is_dir()]
summaries = {a: ca.summarize(g / "archive" / a) for a in arms_present}
session = gate3_session(out, g)
identity = {"monolith_repeats": 0, "monolith_proven": 0, "split_repeats": 0, "split_proofs_passed": 0}
adjudicated, adjudication_problems = read_adjudication(g / "adjudication.tsv")
# ---- the contract's constants (ruling: integrator 2026-09-23 after codex closeout review) ----
contract = {"repeats": [], "arms": [], "cases": []}
if repeat < CONTRACT_REPEATS:
contract["repeats"].append("gate3/arms.txt repeat=%s < CONTRACT-P7 7.2's %d: 30-gate3.sh froze that count "
"with the case list, a resume keeps it" % (meta.get("repeat", "?"), CONTRACT_REPEATS))
for arm in REQUIRED_ARMS:
if arm not in arms_present and arm not in planned:
contract["arms"].append("arm %s has no archive and gate3/arms.txt's arms=[%s] does not name it: the run "
"was started without it; CONTRACT-P7 7.2 needs %s on every case"
% (arm, " ".join(planned), " + ".join(REQUIRED_ARMS)))
canonical, canonical_problems = canonical_gate3_cases()
missing = []
if canonical is None or canonical_problems:
contract["cases"].append("no canonical denominator to hold gate3/cases.txt to: " + "; ".join(canonical_problems))
else:
outside = [c for c in cases if c not in set(canonical)]
missing = [c for c in canonical if c not in set(cases)]
if outside:
contract["cases"].append("%d case(s) outside CONTRACT-P7 7.1's denominator (%d CI split DirectVulkan - "
"the %d exclusions): %s%s" % (
len(outside), CI_SPLIT_DIRECTVULKAN, len(CONTRACT_EXCLUDED), ", ".join(outside),
"; canonical case(s) missing: " + ", ".join(missing) if missing else ""))
twice = sorted(c for c, n in Counter(listed).items() if n > 1)
if twice:
contract["cases"].append("case(s) listed more than once in gate3/cases.txt: " + ", ".join(twice))
short = []
def rows(arm, case):
s = summaries.get(arm)
@@ -205,6 +340,9 @@ def gate3(out, ca):
p = Path(row["dir"]) / "logcat.txt"
return sha(p) if p.is_file() else None
def picture(row):
return str(Path(row["dir"]) / row["actual"]) if row and row.get("actual") else None
def ra0_proof(row):
p = Path(row["dir"]) / "mobilegl.log"
if not p.is_file():
@@ -214,18 +352,21 @@ def gate3(out, ca):
return "run-ahead=0"
return "NOT-PROVEN(" + ";".join(lines)[:120] + ")"
def not_run(arm):
return "%s: not run%s" % (arm, "" if arm in planned else " (and gate3/arms.txt does not name the arm)")
results = []
for case in cases:
reasons, incomplete = [], []
entry = {"case": case, "arms": {}}
mono = rows("monolith", case)
mono_ssim = mono_sha = None
mono_row = mono_ssim = mono_sha = None
if "monolith" in summaries and not finished("monolith", case):
incomplete.append("monolith: no state .done (pair unfinished; 30-gate3.sh retries it)")
if len(mono) != 1 or mono[0].get("error"):
incomplete.append("monolith: %s" % (mono[0].get("error") if mono else "not run"))
incomplete.append("monolith: %s" % mono[0].get("error") if mono else not_run("monolith"))
else:
m = mono[0]
m = mono_row = mono[0]
mono_ssim, mono_sha = m.get("ssim_vs_golden"), m.get("actual_sha256")
not_mono = monolith_identity(m)
identity["monolith_repeats"] += 1
@@ -238,18 +379,23 @@ def gate3(out, ca):
reasons.append("monolith control did not pass (ssim %s)" % mono_ssim)
if case == OPENRA and not (m.get("mismatch_pixels_vs_golden") == 0 and (mono_ssim or 0) >= 0.9999995):
reasons.append("OpenRA monolith not 1.000000/0 (%s/%s)" % (mono_ssim, m.get("mismatch_pixels_vs_golden")))
split_rows = {}
for arm in SPLIT_ARMS:
rs = rows(arm, case)
rs = split_rows[arm] = rows(arm, case)
a = {"repeats": len(rs), "ssim": [r.get("ssim_vs_golden") for r in rs],
"sha": sorted({r.get("actual_sha256") for r in rs if r.get("actual_sha256")}), "proofs_passed": 0}
entry["arms"][arm] = a
if arm not in summaries:
incomplete.append(arm + ": not run")
incomplete.append(not_run(arm))
continue
if not finished(arm, case):
incomplete.append("%s: no state .done (pair unfinished; 30-gate3.sh retries it)" % arm)
if len(rs) != repeat:
incomplete.append("%s: %d/%d repeats" % (arm, len(rs), repeat))
if len(rs) < CONTRACT_REPEATS and finished(arm, case):
# The pair is final (30-gate3.sh wrote its .done): a resume will not add a repeat.
reasons.append("%s: %d repeat(s) in a finished pair < CONTRACT-P7 7.2's %d" % (arm, len(rs), CONTRACT_REPEATS))
short.append("%s/%s (%d)" % (arm, case, len(rs)))
for r in rs:
if r.get("error"):
incomplete.append("%s rep%d: %s" % (arm, r["repeat"], r["error"]))
@@ -276,54 +422,109 @@ def gate3(out, ca):
for i in range(1, len(lc)):
if lc[i] and lc[i] == lc[i - 1]:
reasons.append("%s rep%d STALE (logcat identical to rep%d)" % (arm, rs[i]["repeat"], rs[i - 1]["repeat"]))
# The RUN_AHEAD=0 control arm: required on every case (CONTRACT-P7 7.2 "RUN_AHEAD=0 对照臂一遍
# 记录"), and it must be that arm; its picture is recorded, never gating.
ra0 = rows("inproc-ra0", case)
if ra0:
if "inproc-ra0" in summaries and not finished("inproc-ra0", case):
incomplete.append("inproc-ra0: no state .done (pair unfinished; 30-gate3.sh retries it)")
if not ra0:
incomplete.append(not_run("inproc-ra0"))
else:
r = ra0[0]
entry["arms"]["inproc-ra0"] = {"ssim": r.get("ssim_vs_golden"), "sha": r.get("actual_sha256"),
"passed": r.get("passed"), "proof": ra0_proof(r), "error": r.get("error")}
# Cross-arm identity: reported, never gating (ruling d).
if r.get("error"):
incomplete.append("inproc-ra0 rep%d: %s" % (r["repeat"], r["error"]))
elif entry["arms"]["inproc-ra0"]["proof"] != "run-ahead=0":
reasons.append("inproc-ra0 is not proven the RUN_AHEAD=0 arm (%s)" % entry["arms"]["inproc-ra0"]["proof"])
# Cross-arm identity: not a per-case gate (ruling d), but every mismatch needs an adjudication.
ip = set(entry["arms"].get("inproc", {}).get("sha", []))
sp = set(entry["arms"].get("spawn", {}).get("sha", []))
entry["inproc_eq_spawn"] = bool(ip) and ip == sp
entry["split_eq_monolith"] = bool(ip) and ip == sp == {mono_sha}
entry["cross_arm"] = {"monolith": mono_sha, "inproc": sorted(ip), "spawn": sorted(sp),
"identical": (entry["split_eq_monolith"] if (ip and sp and mono_sha) else None)}
x = entry["cross_arm"] = {"monolith": mono_sha, "inproc": sorted(ip), "spawn": sorted(sp),
"identical": (entry["split_eq_monolith"] if (ip and sp and mono_sha) else None)}
if x["identical"] is False:
first = {arm: picture(split_rows[arm][0]) if split_rows[arm] else None for arm in SPLIT_ARMS}
x["px_vs_golden"] = {"monolith": mono_row.get("mismatch_pixels_vs_golden") if mono_row else None,
"inproc": [r.get("mismatch_pixels_vs_golden") for r in split_rows["inproc"]],
"spawn": [r.get("mismatch_pixels_vs_golden") for r in split_rows["spawn"]]}
x["px_direct"] = {"inproc-monolith": px_between(ca, first["inproc"], picture(mono_row)),
"spawn-monolith": px_between(ca, first["spawn"], picture(mono_row)),
"inproc-spawn": px_between(ca, first["inproc"], first["spawn"])}
x["adjudication"] = adjudicated.get(case)
entry["verdict"] = "INCOMPLETE" if incomplete else ("FAIL" if reasons else "PASS")
entry["reasons"] = incomplete + reasons
results.append(entry)
if short:
contract["repeats"].append("%d finished inproc/spawn pair(s) archived fewer than CONTRACT-P7 7.2's %d repeats "
"(arms.txt repeat=%s): %s%s" % (len(short), CONTRACT_REPEATS, meta.get("repeat", "?"),
", ".join(short[:4]), ", ..." if len(short) > 4 else ""))
mismatched = [e["case"] for e in results if e["cross_arm"]["identical"] is False]
unadjudicated = [c for c in mismatched if c not in adjudicated]
stale = sorted(c for c in adjudicated if c not in set(mismatched))
verdicts = [e["verdict"] for e in results]
# A reading that is not the gate's own (CONTRACT-P7 7.2: the 36-case denominator, one
# reboot-clean session) may pass, but never as PASS: exit 0 is reserved for the gate verdict.
# A reading that is not the gate's own (CONTRACT-P7 7.2: the canonical 36, one reboot-clean
# session, every cross-arm mismatch adjudicated) may pass, but never as PASS: exit 0 is
# reserved for the gate verdict.
not_a_gate = []
if len(cases) != GATE3_DENOMINATOR:
not_a_gate.append("denominator is %d, not the gate's %d: a subset run is NOT a gate verdict"
% (len(cases), GATE3_DENOMINATOR))
if missing and not contract["cases"]:
not_a_gate.append("%d of the canonical %d case(s) (missing %d, e.g. %s): a subset run is NOT a gate verdict"
% (len(cases), GATE3_DENOMINATOR, len(missing), missing[0]))
if not session["reboot_clean"].startswith("reboot-clean:"):
not_a_gate.append("session not reboot-clean: NOT a gate verdict")
if unadjudicated:
not_a_gate.append("%d cross-arm mismatch(es) without a gate3/adjudication.tsv line (%s): %s%s" % (
len(unadjudicated), CODEX_RULING, ", ".join(unadjudicated[:3]), ", ..." if len(unadjudicated) > 3 else ""))
if not session["valid"]:
overall = "INVALID-SESSION"
elif contract["repeats"]:
overall = "FAIL-REPEATS"
elif contract["arms"]:
overall = "FAIL-ARMS"
elif contract["cases"]:
overall = "FAIL-CASESET"
elif verdicts and all(v == "PASS" for v in verdicts):
overall = "PASS"
if len(cases) != GATE3_DENOMINATOR:
if missing:
overall = "PASS-SUBSET"
elif not_a_gate:
elif not session["reboot_clean"].startswith("reboot-clean:"):
overall = "PASS-NOT-REBOOT-CLEAN"
elif unadjudicated:
overall = "PASS-NEEDS-ADJUDICATION"
elif "FAIL" in verdicts:
overall = "FAIL"
else:
overall = "INCOMPLETE"
return {"cases": results, "excluded": excluded, "repeat": repeat, "denominator": len(cases),
"overall": overall, "not_a_gate": not_a_gate, "arms_present": arms_present,
"overall": overall, "not_a_gate": not_a_gate, "arms_present": arms_present, "arms_planned": planned,
"contract": contract, "canonical": canonical, "canonical_problems": canonical_problems,
"cross_arm_mismatched": mismatched, "cross_arm_unadjudicated": unadjudicated,
"adjudication": {"file": str(g / "adjudication.tsv"), "cases": adjudicated,
"problems": adjudication_problems, "stale": stale},
"session": session, "identity": identity}
def print_gate3(g3):
print("== GATE 3 (DirectVulkan x pbuffer; split repeat=%d; tol |ssim-monolith| <= %.4f) ==" % (g3["repeat"], TOL))
print("== GATE 3 (DirectVulkan x pbuffer; split repeat=%d, contract >= %d; tol |ssim-monolith| <= %.4f) =="
% (g3["repeat"], CONTRACT_REPEATS, TOL))
s = g3["session"]
print("-- session: boot_id %s; %d record(s) stamped; %s" % (s["session_boot_id"] or "?", s["records"],
s["reboot_clean"] or "no session/reboot-clean.txt"))
for problem in s["problems"]:
print(" ! INVALID-SESSION: " + problem)
c = g3["contract"]
held = not any(c.values())
print("-- contract (CONTRACT-P7 7.1/7.2; %s): inproc/spawn >= %d repeats per case, arms %s on every case, "
"case set == the canonical %d (%d CI split DirectVulkan - %d exclusions): %s"
% (CODEX_RULING, CONTRACT_REPEATS, "+".join(REQUIRED_ARMS), GATE3_DENOMINATOR, CI_SPLIT_DIRECTVULKAN,
len(CONTRACT_EXCLUDED), "held" if held else "VIOLATED"))
for key, label in (("repeats", "FAIL-REPEATS"), ("arms", "FAIL-ARMS"), ("cases", "FAIL-CASESET")):
for problem in c[key]:
print(" ! %s: %s" % (label, problem))
if sorted(g3["excluded"]) != sorted(CONTRACT_EXCLUDED):
print("WARN gate3/excluded.txt [%s] is not CONTRACT-P7 7.1's [%s] (the case set above is held to the contract's)"
% (" ".join(g3["excluded"]), " ".join(CONTRACT_EXCLUDED)))
w = max([len(e["case"]) for e in g3["cases"]] + [4])
print("%-*s %-10s %-11s %-24s %-24s %-11s %s" % (w, "case", "verdict", "monolith", "inproc(min..max) #sha", "spawn(min..max) #sha", "ra0", "i==s==m"))
for e in g3["cases"]:
@@ -344,22 +545,42 @@ def print_gate3(g3):
for reason in e["reasons"]:
print("%-*s ! %s" % (w, "", reason))
ra0s = [(e["case"], e["arms"]["inproc-ra0"]) for e in g3["cases"] if "inproc-ra0" in e["arms"]]
if ra0s:
print("-- RUN_AHEAD=0 control (recorded, not gating): %d case(s); proofs: %s" % (
len(ra0s), ", ".join(sorted({r["proof"] for _, r in ra0s}))))
print("-- RUN_AHEAD=0 control (required on every case, picture recorded, not gating): %d/%d case(s); proofs: %s" % (
len(ra0s), len(g3["cases"]), ", ".join(sorted({r["proof"] for _, r in ra0s})) or "-"))
i = g3["identity"]
print("-- arm identity: monolith %d/%d repeat(s) with no Config: IPC / MOBILEGL_TRANSPORT line; "
"inproc+spawn transport-proof passed %d/%d" % (i["monolith_proven"], i["monolith_repeats"],
i["split_proofs_passed"], i["split_repeats"]))
judged = [e for e in g3["cases"] if e["cross_arm"]["identical"] is not None]
print("-- cross-arm picture identity (reported, NOT a gate; %s): %d/%d case(s) inproc==spawn==monolith" % (
RULING, sum(e["cross_arm"]["identical"] for e in judged), len(judged)))
adj = g3["adjudication"]
print("-- cross-arm picture identity (NOT a per-case gate, %s; every mismatch needs a gate3/adjudication.tsv "
"line, %s): %d/%d case(s) inproc==spawn==monolith; %d mismatch(es), %d unadjudicated" % (
RULING, CODEX_RULING, sum(e["cross_arm"]["identical"] for e in judged), len(judged),
len(g3["cross_arm_mismatched"]), len(g3["cross_arm_unadjudicated"])))
def px(v):
if isinstance(v, list):
nums = [n for n in v if isinstance(n, int)]
return "%d..%d" % (min(nums), max(nums)) if nums else "-"
return "-" if v is None else str(v)
for e in judged:
if not e["cross_arm"]["identical"]:
x = e["cross_arm"]
g, d = x["px_vs_golden"], x["px_direct"]
print("WARN cross-arm %s: monolith %s | inproc %s | spawn %s" % (
e["case"], (x["monolith"] or "-")[:12], ",".join(v[:12] for v in x["inproc"]) or "-",
",".join(v[:12] for v in x["spawn"]) or "-"))
print(" px!=golden (archived mismatchPixels) monolith %s, inproc %s, spawn %s; px between the arms' "
"first repeats: inproc-monolith %s, spawn-monolith %s, inproc-spawn %s" % (
px(g["monolith"]), px(g["inproc"]), px(g["spawn"]), px(d["inproc-monolith"]),
px(d["spawn-monolith"]), px(d["inproc-spawn"])))
print(" " + ("adjudicated: " + x["adjudication"] if x.get("adjudication")
else "UNADJUDICATED: no gate3/adjudication.tsv line"))
for problem in adj["problems"]:
print("WARN " + problem)
if adj["stale"]:
print(" note: gate3/adjudication.tsv names case(s) whose arms agree (nothing to adjudicate): "
+ ", ".join(adj["stale"]))
print("-- excluded (red on monolith, CONTRACT-P7 7.1): " + (", ".join(g3["excluded"]) or "none"))
n = len(g3["cases"])
npass = sum(e["verdict"] == "PASS" for e in g3["cases"])
@@ -407,16 +628,20 @@ def read_list(path):
return [l.strip() for l in path.read_text(encoding="utf-8").splitlines() if l.strip() and not l.startswith("#")]
def rate(statuses):
"""CONTRACT-P7 7.3: rate = Pass / (Pass + Fail). NotSupported, the warnings (Compatibility /
Quality) and the crashes stay out of the denominator; crashes get their own count. The
pre-ruling Pass / (results - NotSupported) is kept as `results_rate`, information only."""
def rate(statuses, lost=0):
"""The gate's block rate = Pass / (Pass + Fail + lost), `lost` = the $BASE-Pass cases this side
turned NotSupported or another non-Pass, non-Fail status (a warning, a crash): a lost supported
case is NON-PASS, it does not leave the denominator (ruling: integrator 2026-09-23 after codex
closeout review). Other NotSupported / warnings / crashes stay out of the denominator; crashes
get their own count. Information only: CONTRACT-P7 7.3's literal Pass / (Pass + Fail) as
`literal_rate` and the pre-ruling Pass / (results - NotSupported) as `results_rate`."""
c = Counter(statuses)
p, f, ns = c["Pass"], c["Fail"], c["NotSupported"]
x = sum(c[s] for s in HARD)
n = len(statuses)
return {"pass": p, "fail": f, "ns": ns, "crash": x, "warn": n - p - f - ns - x, "results": n,
"rate": (p / (p + f)) if (p + f) else None,
return {"pass": p, "fail": f, "ns": ns, "crash": x, "warn": n - p - f - ns - x, "results": n, "lost": lost,
"rate": (p / (p + f + lost)) if (p + f + lost) else None,
"literal_rate": (p / (p + f)) if (p + f) else None,
"results_rate": (p / (n - ns)) if (n - ns) else None}
@@ -477,7 +702,8 @@ def block_scope(run, after):
return universe, src, [], incomplete
def cts_block(c, b, base_dir, canon_dir):
def cts_block(c, b, base_dir, canon_dir, adjudicated=None):
adjudicated = adjudicated or {}
run = c / "runs" / b
if not c.is_dir():
return {"block": b, "verdict": "MISSING", "reasons": ["no cts/ directory: 50-cts-after.sh did not run"]}
@@ -496,48 +722,63 @@ def cts_block(c, b, base_dir, canon_dir):
hung = read_list(run / "hung.txt")
# Like for like: both rates over the cases that have a BASE and an AFTER result.
both = [x for x in universe if x in after and x in base]
ra, rb = rate([after[x] for x in both]), rate([base[x] for x in both])
delta = None if ra["rate"] is None or rb["rate"] is None else (ra["rate"] - rb["rate"]) * 100.0
delta_results = (None if ra["results_rate"] is None or rb["results_rate"] is None
else (ra["results_rate"] - rb["results_rate"]) * 100.0)
# A $BASE Pass that AFTER turned NotSupported / a warning / a crash: NON-PASS in the gate rate,
# listed, and FAIL unless adjudicated (ruling: integrator 2026-09-23 after codex closeout review).
lost = sorted(x for x in both if base[x] == "Pass" and after[x] not in ("Pass", "Fail"))
ra, rb = rate([after[x] for x in both], lost=len(lost)), rate([base[x] for x in both])
def pp(key):
return None if ra[key] is None or rb[key] is None else (ra[key] - rb[key]) * 100.0
delta, delta_literal, delta_results = pp("rate"), pp("literal_rate"), pp("results_rate")
new_crash = sorted(x for x in universe if after.get(x) in HARD and base.get(x) not in HARD)
new_crash += sorted(x for x in hung if x not in new_crash)
regress = sorted(x for x in universe if base.get(x) == "Pass" and after.get(x) not in (None, "Pass"))
fixed = sorted(x for x in universe if base.get(x) not in (None, "Pass") and after.get(x) == "Pass")
missing_base = sorted(x for x in universe if x not in base)
unadjudicated = [x for x in lost if x not in adjudicated]
fail, notes, warnings = [], [], []
if new_crash:
fail.append("%d NEW crash(es)" % len(new_crash))
if rb["rate"] is None:
incomplete.append("$BASE has no Pass/Fail over this block's %d case(s): no contract rate to compare against"
% len(both))
elif ra["rate"] is None:
fail.append("AFTER has no Pass/Fail result (NS %d, W %d, X %d of %d): the contract rate cannot be formed "
"where $BASE's is %.3f%% - every $BASE Pass is lost" % (
ra["ns"], ra["warn"], ra["crash"], ra["results"], 100 * rb["rate"]))
elif ra["literal_rate"] is None:
fail.append("AFTER has no Pass/Fail result (NS %d, W %d, X %d of %d): the literal Pass/(Pass+Fail) cannot be "
"formed where $BASE's is %.3f%%; %d $BASE Pass(es) lost" % (
ra["ns"], ra["warn"], ra["crash"], ra["results"], 100 * rb["rate"], len(lost)))
if delta is not None and delta < -CTS_TOL_PP:
fail.append("delta %.3f pp < -%.1f pp (Pass/(Pass+Fail))" % (delta, CTS_TOL_PP))
fail.append("delta %.3f pp < -%.1f pp (Pass/(Pass+Fail+L))" % (delta, CTS_TOL_PP))
if delta is not None and delta > CTS_TOL_PP:
notes.append("note: AFTER is %.3f pp ABOVE BASE (not a regression; review the fixed list)" % delta)
if unadjudicated:
fail.append("%d $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status with no "
"cts/adjudication.tsv line (%s)" % (len(unadjudicated), CODEX_RULING))
elif lost:
notes.append("note: %d $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status, each "
"adjudicated in cts/adjudication.tsv (still NON-PASS in the rate)" % len(lost))
if missing_base:
warnings.append("%d case(s) have no $BASE result (e.g. %s): left out of both rates; a crash among them "
"still counts as NEW" % (len(missing_base), missing_base[0]))
if ra["ns"] != rb["ns"]:
warnings.append("NotSupported $BASE %d -> AFTER %d (not in the contract rate; a Pass that became NS is in "
"the Pass->not list)" % (rb["ns"], ra["ns"]))
warnings.append("NotSupported $BASE %d -> AFTER %d (a $BASE Pass among them is an L case: NON-PASS in the "
"rate, listed below)" % (rb["ns"], ra["ns"]))
canonical = canon_dir / ("p7-%s-gl46.txt" % b)
subset = set(read_list(src)) != set(read_list(canonical))
verdict = "FAIL" if fail else ("INCOMPLETE" if incomplete else "PASS")
return {"block": b, "cases": len(universe), "compared": len(both), "caselist": str(src),
"after": ra, "base": rb, "delta_pp": delta, "delta_pp_results_info": delta_results,
"after": ra, "base": rb, "delta_pp": delta, "delta_pp_literal_info": delta_literal,
"delta_pp_results_info": delta_results,
"new_crash": new_crash, "regressed": regress, "fixed": fixed, "missing_in_base": missing_base,
"lost": [{"case": x, "after": after[x], "adjudication": adjudicated.get(x)} for x in lost],
"lost_unadjudicated": unadjudicated,
"unrun": len(unrun), "hung": hung, "verdict": verdict, "reasons": incomplete + fail + notes,
"warnings": warnings, "subset": subset}
def cts(out, base_dir, canon_dir):
c = out / "cts"
blocks = [cts_block(c, b, base_dir, canon_dir) for b in CTS_BLOCKS]
adjudicated, adjudication_problems = read_adjudication(c / "adjudication.tsv")
blocks = [cts_block(c, b, base_dir, canon_dir, adjudicated) for b in CTS_BLOCKS]
extra = sorted(p.stem[len("report-"):] for p in c.glob("report-*.json")
if p.stem[len("report-"):] not in CTS_BLOCKS) if c.is_dir() else []
verdicts = [b["verdict"] for b in blocks]
@@ -548,17 +789,23 @@ def cts(out, base_dir, canon_dir):
overall = "PASS-SUBSET" if any(b.get("subset") for b in blocks) else "PASS"
else:
overall = "INCOMPLETE"
lost = {x["case"] for b in blocks for x in b.get("lost", [])}
return {"blocks": blocks, "overall": overall, "ubo": dict(UBO_BLOCK), "ignored_reports": extra,
"incomplete_blocks": [b["block"] for b in blocks if b["verdict"] not in ("PASS", "FAIL")]}
"incomplete_blocks": [b["block"] for b in blocks if b["verdict"] not in ("PASS", "FAIL")],
"adjudication": {"file": str(c / "adjudication.tsv"), "cases": adjudicated,
"problems": adjudication_problems, "stale": sorted(x for x in adjudicated if x not in lost)}}
def print_cts(c):
print("== CTS AFTER (inproc x DirectVulkan) vs BASE (monolith x DirectVulkan, p7w1) ==")
print("gate (CONTRACT-P7 7.3): rate = Pass/(Pass+Fail) [NS, warnings W and crashes X are not in the denominator];")
print(" per block delta >= -%.1f pp and 0 NEW crash; all five blocks required. 'info' = the pre-ruling" % CTS_TOL_PP)
print(" Pass/(results-NS) delta (its rates on each block's 'info' line), for comparison only, never gating.")
print("%-14s %6s %-20s %8s %-20s %8s %9s %9s %5s %4s %5s %s" % (
"block", "cases", "BASE P/F/NS/W/X", "rate", "AFTER P/F/NS/W/X", "rate", "delta pp", "(info)",
print("gate (CONTRACT-P7 7.3 + %s):" % CODEX_RULING)
print(" rate = Pass/(Pass+Fail+L), L = the $BASE-Pass cases AFTER turned NotSupported or another non-Pass,")
print(" non-Fail status (a lost supported case is NON-PASS); other NS, warnings W and crashes X are not in the")
print(" denominator. Per block: delta >= -%.1f pp, 0 NEW crash, and every L case adjudicated in" % CTS_TOL_PP)
print(" cts/adjudication.tsv (else FAIL); all five blocks required. '(literal)' = the contract's literal")
print(" Pass/(Pass+Fail) delta and 'info' the pre-ruling Pass/(results-NS): printed for comparison, never gating.")
print("%-14s %6s %-20s %8s %-20s %4s %8s %9s %9s %5s %4s %5s %s" % (
"block", "cases", "BASE P/F/NS/W/X", "rate", "AFTER P/F/NS/W/X", "L", "rate", "delta pp", "(literal)",
"crash", "regr", "fixed", "verdict"))
def counts(r):
@@ -573,11 +820,13 @@ def print_cts(c):
if "after" not in b:
print("%-14s %s: %s" % (b["block"], b["verdict"], "; ".join(b["reasons"])))
continue
print("%-14s %6d %-20s %8s %-20s %8s %9s %9s %5d %4d %5d %s%s" % (
print("%-14s %6d %-20s %8s %-20s %4d %8s %9s %9s %5d %4d %5d %s%s" % (
b["block"], b["cases"], counts(b["base"]), pct(b["base"]["rate"]), counts(b["after"]),
pct(b["after"]["rate"]), pp(b["delta_pp"]), "(%s)" % pp(b["delta_pp_results_info"]),
b["after"]["lost"], pct(b["after"]["rate"]), pp(b["delta_pp"]), "(%s)" % pp(b["delta_pp_literal_info"]),
len(b["new_crash"]), len(b["regressed"]), len(b["fixed"]), b["verdict"],
" (SUBSET)" if b["subset"] else ""))
print(" info literal Pass/(Pass+Fail): BASE %s AFTER %s delta %s pp (not gating)" % (
pct(b["base"]["literal_rate"]), pct(b["after"]["literal_rate"]), pp(b["delta_pp_literal_info"])))
print(" info pre-ruling Pass/(results-NS): BASE %s AFTER %s delta %s pp (not gating)" % (
pct(b["base"]["results_rate"]), pct(b["after"]["results_rate"]), pp(b["delta_pp_results_info"])))
for reason in b["reasons"]:
@@ -586,16 +835,26 @@ def print_cts(c):
print("WARN %s: %s" % (b["block"], warning))
for x in b["new_crash"]:
print(" NEW CRASH " + x)
for x in b["regressed"][:40]:
for x in b["lost"]:
print(" LOST %s (Pass -> %s) %s" % (x["case"], x["after"], "adjudicated: " + x["adjudication"]
if x["adjudication"] else "UNADJUDICATED"))
lost = {x["case"] for x in b["lost"]}
others = [x for x in b["regressed"] if x not in lost]
for x in others[:40]:
print(" Pass->not " + x)
if len(b["regressed"]) > 40:
print(" ... %d more regressions in the JSON" % (len(b["regressed"]) - 40))
if len(others) > 40:
print(" ... %d more regressions in the JSON" % (len(others) - 40))
for x in b["fixed"][:20]:
print(" not->Pass " + x)
u = c["ubo"]
print("%-14s %s (%s)" % (u["block"], u["reasons"][0], u["cases"]))
if c["ignored_reports"]:
print(" note: report(s) outside the five gate blocks ignored: " + ", ".join(c["ignored_reports"]))
for problem in c["adjudication"]["problems"]:
print("WARN " + problem)
if c["adjudication"]["stale"]:
print(" note: cts/adjudication.tsv names case(s) that are no $BASE-Pass loss (nothing to adjudicate): "
+ ", ".join(c["adjudication"]["stale"]))
tail = ""
if c["incomplete_blocks"]:
tail += " (INCOMPLETE: %s)" % ", ".join("%s %s" % (b["block"], b["verdict"]) for b in c["blocks"]
docs/Disaggregated/notes/p7/device-window-2/tools/RUNBOOK.md
+54 -21
View File
@@ -69,7 +69,7 @@ boot_id，与它不同立即停；每对的 `.done` 与每遍归档的 `repeat-N
| 4 | `30-gate3.sh <stamp>` | 36 例（`--matrix` 的 DirectVulkan 集减 `create-indirect` / `1.21.11-main-menu` / `photon-v1.3b`，冻结在 `gate3/cases.txt`）× 四臂；iterationrp 带 `apk.yml:460-462` 三个 Magma 旋钮；每遍 result.json / actual PNG / 双角色日志 / transport-proof / logcat 归档；每臂前后 pin check；每对前后核 boot_id（§1「会话」），`.done` 只在每遍都有 result.json + actual PNG 时写，否则退出 3 留待续跑 | 55–65 min |
| 5 | `40-bsl-stats.sh <stamp>` | bsl-esc-menu × {inproc, spawn} 各 1 遍，`MOBILEGL_PIPE_STATS=1 PERIOD=1`；先 force-stop 再起设备端 root 采样器（0.25 s：maps 行数、VmRSS、VmHWM；排除 argv 带 `tcp://` 的 supervisor server）；`wbuf[]` 从归档日志取 | ~1 min |
| 6 | `50-cts-after.sh <stamp>` | APK 的 arm64 `libMobileGL.so` 推到 `mgcts` 并在设备上核 sha256；AFTER 环境 = `$BASE` 的 flag + `MOBILEGL_TRANSPORT=inproc` + `ROLE_SPLIT_STATE=1 STRICT_ERRORS=1 RUN_AHEAD=1`；臂证明（mgprobe PASS 且 logcat 有 inproc 解析句与 `Config: IPC … strict=1 role-split-state=1 run-ahead=1`、0 Fatal；再用一个 glcts 用例证一次，其 StatusCode 记进 `arm-proof.txt`，不是 Pass / Fail 时打 WARN）；五块依次跑（顺序同 `$BASE`），每块 `qpa_report --json`（块的 `.done` 只在 `run_cts` 退出 0 且 `60-reduce.py --check-block` 判完整时写，否则退出 3 留待续跑），最后 `cts_multi_report` 出与 `$BASE` 同形的 JSON；UBO（GTF）块不跑：本 glcts 无 GTF 模块（ID-P7-16） | 6–7 min（pipe 头 lib inproc 实测 358 s；`$BASE` monolith 372 s） |
| 7 | `60-reduce.py <stamp>` | 写 `verdict.txt` / `verdict.json`（见 §3）；每遍都跑。自检：`python3 60-reduce-selftest.py`（主机，不占设备，~4 s） | <10 s |
| 7 | `60-reduce.py <stamp>` | 写 `verdict.txt` / `verdict.json`（见 §3）；每遍都跑。自检：`python3 60-reduce-selftest.py`（主机，不占设备，~7 s） | ~2 min（门 3 归档每遍一次纯 Python 的 SSIM-vs-first：`p7w7-6afa077f` 的 monolith+inproc+spawn 归档只读实测 108 s） |
| 8 | `90-restore.sh <stamp>` | Home；`am force-stop` 包（回放后缓存的 app 进程与 spawn 回放留下的 `libMobileGLServer.so @mgl-…` 子进程会一直占着几百 MB）；**把 `$BASE` 库推回 `/data/local/tmp/mgcts`**（设备 sha256 ≠ `$BASE` = `IDENTITY.txt` 的 `7f58aa0b…` 时才推，推后在设备上核 sha，结果行进 `restored.txt`）；supervisor 若开窗时在跑则以 `tcp_device_server.py start --allow-idle`（同 listen / token / Doze 状态文件）在**新 APK**上重启并核实在听；解钉频（MIUI 启动 boost 会让 `check` 短暂读成 DRIFT，重读至多 20 s）；stay-on 复原；每遍都跑（幂等） | ~15 s |
合计：构建 ~6 min + 设备 ~65–80 min ≈ **1.3–1.5 h**。外推依据见 §5。
@@ -78,19 +78,36 @@ boot_id，与它不同立即停；每对的 `.done` 与每遍归档的 `repeat-N
- **门 3**，先看 `-- session` 行：`session/boot-id.txt` 与四臂的每条记录（`gate3/state/<arm>/<case>.done` 的 `boot_id` / `boot_id_before`、
每遍 `repeat-NN/boot_id.txt`）必须是**同一个** boot_id；有第二个值、有记录不带 boot_id（含本规则之前的旧 `.done`）、或缺 `session/boot-id.txt`，
整门判 **`GATE3 INVALID-SESSION`**，与图无关。全部通过但不是门本身的读数时不记 `PASS`：分母不是 36 记 **`PASS-SUBSET`**，
会话不是 reboot-clean（`session/reboot-clean.txt`）记 **`PASS-NOT-REBOOT-CLEAN`**，末行注明「不是判据」，退出码非 0。
整门判 **`GATE3 INVALID-SESSION`**，与图无关。
- **门 3 按合同常量判，不按跑器自己的记录判（ruling: integrator 2026-09-23 after codex closeout review）**：`gate3/arms.txt` 只说明
`30-gate3.sh` 被要求跑什么，不说明门要什么；reducer 自己按 CONTRACT-P7 §7.1 / §7.2 核三件事（`-- contract` 行，违反的逐条 `! FAIL-…`）：
① **遍数**：inproc 与 spawn 每例归档 ≥ 3 遍。`arms.txt` 的 `repeat=` < 3，或已完成（有 `.done`）的对归档不足 3 遍 → **`GATE3 FAIL-REPEATS`**，
不管 `arms.txt` 写的是几（`--repeat 1` 跑出来的永远不是门）；没有 `.done` 的对不足 3 遍仍只是该例 INCOMPLETE（续跑整对重来）。
② **四臂**：monolith / inproc / spawn / inproc-ra0 每例都要有。某臂没有归档、且 `arms.txt` 的 `arms=` 也没列它（开跑时就没带）→ **`GATE3 FAIL-ARMS`**；
列了但还没跑到、或某例缺该臂 → 该例 INCOMPLETE。ra0 那一遍的 `mobilegl.log` 必须证明 `run-ahead=0`，否则该例 FAIL（那不是对照臂）；ra0 的图与 ssim 仍只记录、不判。
③ **用例集**：`gate3/cases.txt` 必须**等于**规范分母，不是「数到 36」——按 `30-gate3.sh` 的算法从工具树 `trace_cases.json` 取 CI split 且
`ci_backends` 含 DirectVulkan 的 39 例，减去 §7.1 的三个排除（reducer 的常量 `CONTRACT_EXCLUDED`，不读本次的 `gate3/excluded.txt`，两者不同只打 WARN）。
有集外用例、有重复行、或清单已给不出 39 / 36 → **`GATE3 FAIL-CASESET`**；是规范集的真子集（`--cases` 冒烟）→ 至多 `PASS-SUBSET`。
优先级：`INVALID-SESSION` > `FAIL-REPEATS` > `FAIL-ARMS` > `FAIL-CASESET` > 逐例 `FAIL` > `INCOMPLETE`。
- 全部通过但不是门本身的读数时不记 `PASS`：用例是规范 36 例的真子集记 **`PASS-SUBSET`**，
会话不是 reboot-clean（`session/reboot-clean.txt`）记 **`PASS-NOT-REBOOT-CLEAN`**，跨臂不同而没有裁定记 **`PASS-NEEDS-ADJUDICATION`**（见下），
末行注明「不是判据」，退出码非 0。
逐例 `PASS / FAIL / INCOMPLETE` + 原因：monolith 本身通过且**确是 monolith**（该遍 `mobilegl.log`（及 client / server 日志）没有
`Config: IPC` 也没有 `MOBILEGL_TRANSPORT=` 行；没有日志即「未证」）；inproc 与 spawn 各 3 遍齐全、该对有 `.done`、每遍臂证明
（`transport-proof.json` passed）通过、每遍 ssim ≥ 阈值、**同一臂内三遍 actual PNG SHA-256 相同**、每遍 `|ssim − monolith| ≤ 0.0005`；
OpenRA 在 monolith / inproc / spawn 每遍都 `1.000000` 且 mismatch 0；一遍的 logcat 与上一遍逐字节相同记 `STALE`。`-- arm identity`
行汇总 monolith 身份与 split 臂 transport-proof 的通过数。`RUN_AHEAD=0` 臂只记录
（ssim、SHA、`Config: IPC` 全是 `run-ahead=0` 的证明）；它的 runner rc=1 是 `--require-inproc` 要求 `run-ahead=1` 的已知副作用（E1）。
行汇总 monolith 身份与 split 臂 transport-proof 的通过数。`RUN_AHEAD=0` 臂每例必须有（见上 ②），且 `Config: IPC` 全是 `run-ahead=0`；
它的 ssim、SHA 只记录、不判；它的 runner rc=1 是 `--require-inproc` 要求 `run-ahead=1` 的已知副作用（E1）。
末行 `GATE3 PASS: 36/36` 才是门 3。
- **跨臂同图不是门（ruling: integrator 2026-09-23）**：§7.2 的「三遍逐位相同」是**臂内**判据——inproc 自己三遍、spawn 自己三遍各自逐位相同。
inproc / spawn / monolith 三臂之间 actual PNG 是否相同只**报告**：逐例列 `i==s==m`（`yes` / `i==s` / `no`），`-- cross-arm` 行给同图例数，
不同的例各打一行 `WARN cross-arm <case>: monolith <sha> | inproc <sha> | spawn <sha>`；WARN 不改变该例与门 3 的判定
（跨臂的门是每遍 `|ssim − monolith| ≤ 0.0005`）。
不同的例各打一行 `WARN cross-arm <case>: monolith <sha> | inproc <sha> | spawn <sha>`；WARN 不改变该例的判定
（跨臂的逐例门是每遍 `|ssim − monolith| ≤ 0.0005`）。
**但不静默放行（ruling: integrator 2026-09-23 after codex closeout review）**：每个跨臂不同的例都要在 `gate3/adjudication.tsv`
（每行 `<case><TAB><理由>`，`#` 为注释；没有理由的行不算，打 WARN）里有一行，否则本来 PASS 的门记 **`GATE3 PASS-NEEDS-ADJUDICATION`**（退出 1）。
WARN 行下给出该例各臂归档的 `mismatchPixels`（对 golden），以及三臂首遍之间直接数出的像素差（`inproc-monolith` / `spawn-monolith` / `inproc-spawn`，
compare_actuals 的 `mismatch_pixels`），再注 `adjudicated: <理由>` 或 `UNADJUDICATED`；文件里列了但各臂同图的例记 note（陈旧行）。
裁定只针对跨臂同图，不豁免任何逐例判据；裁定文件属于该 stamp 的那批图，重跑某对后要重新看图再写。
- **BSL**：每臂每进程（app / server）的 maps 行数峰值、VmRSS / VmHWM 峰值、`wbuf[wbufs wlivepk wdefpk wdefsync]`。
- **CTS**：五块（shader-image / ssbo / dsa / texture / packed-pixels）**缺一不可**：每块要有 `cts/runs/<block>/.done`、非空且有结果的
`cts/report-<block>.json`，以及读得到、非空的 caselist（先用输出目录里的副本 `runs/<block>/caselist.txt`，没有才用 `caselist.path`
@@ -98,24 +115,37 @@ boot_id，与它不同立即停；每对的 `.done` 与每遍归档的 `repeat-N
0 例的块永远不是读数。UBO 块（`GTF-GL46.gtf31.GL3Tests.uniform_buffer_object*`，
`p7-ubo-gl46.txt` 90 例）显式记 `unrun (not in this glcts)`（设备 glcts 无 GTF 模块，ID-P7-16），不入判据。
每块 BASE 与 AFTER 的 `P/F/NS/W/X`（W = CompatibilityWarning / QualityWarning 等，X = Crash / Timeout / InternalError / ResourceError / DeviceHang / Incomplete），
**rate = Pass / (Pass + Fail)**（§7.3 原文：NS 不入分母；warning 与 crash 也不在分母里），delta pp；块判据 delta ≥ −0.5 pp、
**新增 crash = 0 单独硬红**（AFTER 为 X 而 BASE 不是——Fail→Crash 也算——另加 hung.txt；X 也是非 Pass，进 Pass→非 Pass 名单）、`unrun.txt` 为空。
**rate = Pass / (Pass + Fail + L)**，L = `$BASE` 为 Pass、AFTER 变成 NotSupported 或其他任何非 Pass 非 Fail 状态（warning、crash）的用例：
丢掉的受支持用例按**非 Pass** 计，不从分母里消失（**ruling: integrator 2026-09-23 after codex closeout review**——按 §7.3 字面的
Pass / (Pass + Fail)，ssbo 124 个 `$BASE` Pass 里 123 个变 NS、1 个仍 Pass，读数仍是 100%）；其余 NS、warning、crash 不在分母里。
表里 `L` 列是其个数；§7.3 字面的 Pass / (Pass + Fail) 照印在 `(literal)` 列与每块的 `info literal` 行，只供对照、**不判**。
L 例逐条列 `LOST <case> (Pass -> <AFTER 状态>)`，块里有任何一条没在 `cts/adjudication.tsv`（每行 `<case><TAB><理由>`；没有理由的行不算，打 WARN）
裁定 → 该块 **FAIL**；裁定只免这一条，该例仍按非 Pass 进 rate（所以 ssbo 1 例 Pass→NS 即使裁定，−0.806 pp 仍红）。块判据：delta ≥ −0.5 pp、
未裁定的 L 例 = 0、**新增 crash = 0 单独硬红且不可裁定**（AFTER 为 X 而 BASE 不是——Fail→Crash 也算——另加 hung.txt；Pass→Crash 既是新 crash 也是 L 例）、`unrun.txt` 为空。
两边的 rate 只在**两边都有结果的用例**上算（同口径）；`$BASE` 没有的用例打 `WARN <block>: N case(s) have no $BASE result`，不进两边的 rate，
其中的 crash 仍算新增。AFTER 一个 Pass / Fail 都没有而 `$BASE` 有（例如全变 NotSupported：rate 无从算起，`$BASE` 的 Pass 全丢）→ 该块 **FAIL**；
其中的 crash 仍算新增。AFTER 一个 Pass / Fail 都没有而 `$BASE` 有（例如全变 NotSupported：`$BASE` 的 Pass 全成 L，rate 为 0）→ 该块 **FAIL**；
`$BASE` 自己没有 Pass / Fail → 该块 INCOMPLETE——算不出 delta 的块永远不是 PASS。NotSupported 数与 `$BASE` 不同时打
`WARN <block>: NotSupported …`（只作信息：合同口径 NS 不入分母；变成 NS 的 Pass 在 Pass→非 Pass 名单里）。
`(info)` 列与每块的 `info` 行是旧口径 Pass / (结果数 − NS)：BASE、AFTER 两个 rate 与 delta，只供对照，**不判**。另列 Pass→非 Pass 与非 Pass→Pass 的逐例名单；
`WARN <block>: NotSupported …`（信息；其中原为 Pass 的是 L 例，见上）。
每块第二个 `info` 行是旧口径 Pass / (结果数 − NS)：BASE、AFTER 两个 rate 与 delta，只供对照，**不判**。另列 Pass→Fail（`Pass->not`）与非 Pass→Pass 的逐例名单；
AFTER 高出 0.5 pp 以上只标注不判红。某块用的不是工具树里完整的 `p7-<block>-gl46.txt`（`--limit` 子集）时，全部通过也只记 **`CTS PASS-SUBSET`**。
- 末行 `WINDOW2 GATE3 <v> | CTS <v> -> exit <n>`：门 3 为 `PASS`（36/36、reboot-clean、同一会话）且 CTS 为 `PASS`（五块完整）才是 0；
`PASS-SUBSET` / `PASS-NOT-REBOOT-CLEAN` 不是判据，退出 1；门 3 没跑（无 `gate3/cases.txt`）记 `NOT-RUN`，同样非 0。
- 末行 `WINDOW2 GATE3 <v> | CTS <v> -> exit <n>`：门 3 为 `PASS`（规范 36 例、inproc / spawn 各 ≥ 3 遍、四臂齐、reboot-clean、同一会话、
跨臂不同全部裁定）且 CTS 为 `PASS`（五块完整、无未裁定的 L 例）才是 0；`PASS-SUBSET` / `PASS-NOT-REBOOT-CLEAN` / `PASS-NEEDS-ADJUDICATION`
不是判据，`FAIL-REPEATS` / `FAIL-ARMS` / `FAIL-CASESET` 是合同常量不符，均退出 1；门 3 没跑（无 `gate3/cases.txt`）记 `NOT-RUN`，同样非 0。
`reduce` 的退出码只作记录：红的门也要走完 restore。
- 自检（主机，~4 s）：`python3 60-reduce-selftest.py`（门 3 按 36 例造）——造目录验：缺块 → `CTS INCOMPLETE`；boot_id 变 / 旧 `.done` / 缺 `boot-id.txt` →
`GATE3 INVALID-SESSION`；monolith 日志带 `Config: IPC` → FAIL；跨臂不同 → WARN 仍 PASS；臂内不同 → FAIL；**只由 rate 定**的块
（dsa 1 / 2 / 4 例 Pass→Fail：−0.270 pp PASS、−0.541 pp FAIL、364/6/0/1/0 −1.081 pp FAIL，无 crash）；caselist 没了 / 为空 / 全 NotSupported
→ MISSING / FAIL 而非 PASS；2 例门 3、非 reboot-clean、`--limit` CTS → `PASS-SUBSET` / `PASS-NOT-REBOOT-CLEAN` 且退出非 0；
`$BASE` 缺的用例与 NS 变化 → WARN；`--check-block` 的 0 / 3；dry run 数据上的合同口径
（`60-reduce-selftest-pre14e1c8b9.json` = pre-14e1c8b9 五块相对 `$BASE` 的差异）dsa −0.272 pp；`~/w7/logs/devprep/w2/pre-14e1c8b9` 在时再对原始输出验一遍。
`--reducer <py>` 可对别的 reducer 跑同一套检查（red-once）。
- 自检（主机，~7 s）：`python3 60-reduce-selftest.py`（门 3 按规范 36 例造：用例名由自检自己按 `30-gate3.sh` 的算法从 `trace_cases.json` 算）——造目录验：
缺块 → `CTS INCOMPLETE`；boot_id 变 / 旧 `.done` / 缺 `boot-id.txt` → `GATE3 INVALID-SESSION`；monolith 日志带 `Config: IPC` → FAIL；
**合同常量**：`--repeat 1`、同一归档配上谎报 `repeat=3` 的 `arms.txt`、已完成的对只有 2 遍 → `FAIL-REPEATS`（同一对没有 `.done` → INCOMPLETE）；
开跑不带 ra0 / 不带 spawn → `FAIL-ARMS`，ra0 已列未跑 / 缺一例 → INCOMPLETE，ra0 日志 `run-ahead=1` → 该例 FAIL；
35 例 + create-indirect / 35 例 + 集外名 / 36 例 + 排除例 / 重复行 → `FAIL-CASESET`；
**跨臂**：spawn 每例不同 → WARN + 像素差、`PASS-NEEDS-ADJUDICATION`，全部写进 `gate3/adjudication.tsv` → PASS，缺一例（无理由行）→ 仍 NEEDS-ADJUDICATION，
只一例不同且已裁定（另有陈旧行）→ PASS；臂内不同 → FAIL；**只由 rate 定**的块（dsa 1 / 2 / 4 例 Pass→Fail：−0.270 pp PASS、−0.541 pp FAIL、
364/6/0/1/0 −1.081 pp FAIL，无 crash）；**L 例**：dsa Pass→NS 未裁定 → 367/370、−0.270 pp（字面 −0.001 pp 只作信息）、列出、dsa FAIL，裁定后 PASS
（仍进 rate），无理由的裁定行不算；ssbo Pass→NS 裁定后仍 −0.806 pp FAIL；codex 的例子 ssbo 123/124 变 NS：字面 100% 而门 rate 0.806% → FAIL；
Pass→CompatibilityWarning 也是 L；Pass→Crash 裁定后仍因新 crash FAIL；caselist 没了 / 为空 / 全 NotSupported → MISSING / FAIL 而非 PASS；
2 例门 3、非 reboot-clean、`--limit` CTS → `PASS-SUBSET` / `PASS-NOT-REBOOT-CLEAN` 且退出非 0；`$BASE` 缺的用例 → WARN；`--check-block` 的 0 / 3；
dry run 数据（`60-reduce-selftest-pre14e1c8b9.json` = pre-14e1c8b9 五块相对 `$BASE` 的差异）上 dsa 门 rate −0.541 pp（字面 −0.272 pp、
旧口径 −0.539 pp 只作信息）；`~/w7/logs/devprep/w2/pre-14e1c8b9` 在时再对原始输出验一遍。
`--reducer <py>` 可对别的 reducer 跑同一套检查（red-once：`6afa077f` 的 reducer 113 项里红 54 项）。
## 4. 入库清单 → `docs/Disaggregated/notes/p7/device-window-2/`
@@ -155,6 +185,9 @@ bsl-esc-menu 15–30 s）与本次 dry run 的 7–8 s/例一致，36 例一遍
- `…renderbuffers_storage_multisample`：monolith Pass，inproc Fail。两例合计 dsa 按合同口径 Pass/(Pass+Fail) 为 **−0.272 pp**
（368/370 → 366/369，在 0.5 pp 内；旧口径 Pass/(结果数 − NS) 为 −0.539 pp）——dsa 红在 1 个新 crash，不在 rate。
新 reducer 在该数据上的读数：`docs/Disaggregated/notes/p7/device-window-2/dry-run/pre-14e1c8b9-cts/verdict-contract.txt`。
**按 codex 收官复核后的裁定（ruling: integrator 2026-09-23 after codex closeout review）**，`renderbuffers_storage` 的 Pass→Crash 是 L 例：
门 rate 366/(366+3+1) = 98.919%，**−0.541 pp**，dsa 同时红在新 crash、rate 与未裁定的 L 例（字面 −0.272 pp 只作信息）；
读数 `…/dry-run/pre-14e1c8b9-cts/verdict-codex-ruling.txt`。
- `KHR-GL46.shader_image_load_store.incomplete_textures`：monolith Fail（`$BASE` 亦非 Pass），inproc **SIGABRT** `Fatal{UnmigratedVerb, "Magma:image-view-window"}`
（`MG_Backend/DirectVulkan/Renderer/UniformManager.cpp:151`）——Fail 变 Crash，按门 5 是新 crash。
- `…shader_image_load_store.advanced-sync-vertexArray`：`$BASE` Pass，pipe 头 monolith 与 inproc 都 Fail——不是分离缺陷，是 p7w1 之后两臂共有的漂移。