#!/usr/bin/env python3 """60-reduce.py [--logroot DIR] [--tools TREE] [--json OUT] 60-reduce.py --check-block RUN_DIR REPORT_JSON Device window 2 verdicts, read from the window's own output directory (/). GATE 3 (CONTRACT-P7 7.1/7.2), per case of gate3/cases.txt: * THE CONTRACT'S CONSTANTS, NOT THE RUNNER'S OWN RECORDS (ruling: integrator 2026-09-23 after codex closeout review). gate3/arms.txt says what 30-gate3.sh was asked to run, not what the gate needs, so the reducer holds the run to CONTRACT-P7 7.1/7.2 itself: - REPEATS: inproc and spawn need >= 3 archived repeats on every case. arms.txt repeat < 3, or a finished pair (state .done) with fewer than 3 archived repeats, is GATE3 FAIL-REPEATS whatever arms.txt says; - ARMS: monolith, inproc, spawn AND inproc-ra0 on every case. An arm with no archive that arms.txt's arms= does not name (the run was started without it) is GATE3 FAIL-ARMS; a named arm not reached yet, or a case an arm lacks, makes that case INCOMPLETE; - CASE SET: gate3/cases.txt must EQUAL the canonical denominator, computed the way 30-gate3.sh computes it - the tools tree's trace_cases.json CI split cases whose ci_backends carry DirectVulkan (39) - minus CONTRACT-P7 7.1's three exclusions, which are this file's constants (CONTRACT_EXCLUDED), not gate3/excluded.txt. A case outside it, a case listed twice, or a manifest that no longer gives 39 / 36 is GATE3 FAIL-CASESET; a proper subset (a --cases run) is PASS-SUBSET at best - never merely "count 36"; * ONE SESSION: every gate3/state//.done and every archived repeat's boot_id.txt, of every arm, carries one boot_id, equal to session/boot-id.txt. A second boot_id, a record that carries none, or no session/boot-id.txt makes the whole gate GATE3 INVALID-SESSION, whatever the pictures say (the gate is a same-session reading); * monolith x1 is the same-session control: it must itself pass, and its role logs must show no 'Config: IPC' / MOBILEGL_TRANSPORT line (the arm really was the monolith); * inproc and spawn: the pair finished (state .done), every expected repeat present and error-free, each repeat's arm proof (transport-proof.json passed), the device's own pass verdict (ssim >= case threshold), ALL repeats OF THE ARM bit-identical (actual PNG SHA-256; within-arm identity is the gate), and |ssim - monolith ssim| <= 0.0005 on every repeat; a repeat whose logcat is byte-identical to the previous repeat's is flagged STALE; * OpenRA: 1.000000 and mismatch 0 on every repeat of both split arms AND on monolith; * cross-arm picture identity (inproc vs spawn vs monolith) is not a per-case gate (ruling: integrator 2026-09-23, ID-P7-59): a mismatch prints a WARN line and leaves the case's verdict alone. But it is never waved through silently (ruling: integrator 2026-09-23 after codex closeout review): every mismatching case must be listed in gate3/adjudication.tsv (''), else an otherwise PASS gate is GATE3 PASS-NEEDS-ADJUDICATION (exit 1). Each mismatch is listed with the archived mismatchPixels (vs golden) of every arm and the direct pixel counts between the arms' first repeats; * inproc-ra0 (RUN_AHEAD=0): required on every case (one archived repeat with its .done) and its role log must prove run-ahead=0 (else that case FAIL: it is not the control arm); its picture and ssim are recorded, never gating; * a reading that passes but is not the gate's - a proper subset of the canonical 36, or a session that is not reboot-clean - is GATE3 PASS-SUBSET / PASS-NOT-REBOOT-CLEAN, never PASS. BSL STATS: per arm, maps-line / VmRSS / VmHWM peaks and the wbuf[] gauges (information). CTS AFTER (CONTRACT-P7 7.3), per block vs device-window-1/CTS-base/report-.json, over the cases the AFTER run was asked to run (so a --limit subset is compared like for like): * all five blocks (shader-image ssbo dsa texture packed-pixels) are REQUIRED, each with a finished run (cts/runs//.done), a non-empty cts/report-.json with results, and a readable, non-empty caselist (runs//caselist.txt, the copy 50-cts-after.sh keeps; else the file caselist.path names); a missing one makes the block MISSING and CTS INCOMPLETE - a block of 0 cases is never a reading. The UBO block (GTF-GL46 uniform_buffer_object) is recorded as 'unrun (not in this glcts)': the device glcts carries no GTF module (ID-P7-16); * rate = Pass / (Pass + Fail + L), L = the cases BASE passed that AFTER turned NotSupported or any other non-Pass, non-Fail status (a warning, a crash): a lost supported case is NON-PASS, it does not leave the denominator (ruling: integrator 2026-09-23 after codex closeout review - under the contract's literal Pass / (Pass + Fail), 123 of 124 BASE-Pass cases turning NotSupported would still read 100%). NotSupported, warnings and crashes that were not a BASE Pass stay out of both rates. gate: delta >= -0.5 pp on this rate. Both rates are taken over the cases that have a BASE and an AFTER result (a case BASE lacks prints a WARN, stays out of both rates and still counts for new crashes). AFTER without a single Pass/Fail/L where BASE has some is FAIL (the rate cannot be formed); BASE without one is INCOMPLETE; * every L case is listed (Pass -> its AFTER status), and a block with one that cts/adjudication.tsv ('') does not list is FAIL. An ADJUDICATED one is printed 'ADJUDICATED' and leaves that block's rate on both sides - no longer an L case in AFTER's denominator, one Pass fewer in BASE's (ruling: integrator 2026-09-23 after the int3 critic: else a single ruled-legitimate loss in the 124-case ssbo block was -0.806 pp that nothing could clear); the literal and pre-ruling info rates still count it; * crashes (Crash/Timeout/InternalError/ResourceError/DeviceHang/Incomplete, plus hung.txt) are counted in their own column and are non-Pass; one that BASE does not already have is a NEW crash, a hard red of its own that no adjudication waives (a Fail that became a Crash is a new crash; a Pass that became a Crash is a new crash AND an L case); * a block whose cases are not the tools tree's full p7--gl46.txt (a --limit run) makes an all-PASS CTS reading 'PASS-SUBSET', never PASS; * information only, never gating: the contract's literal Pass / (Pass + Fail) of BASE and AFTER and its delta, the pre-ruling Pass / (results - NotSupported) and its delta, and a WARN when the NotSupported count moved. Exit status: 0 only when GATE 3 is PASS (the canonical 36, >= 3 repeats, four arms, reboot-clean, one session, every cross-arm mismatch adjudicated) and CTS is PASS (five full blocks); 1 otherwise (a section that did not run, a PASS-SUBSET / PASS-NOT-REBOOT-CLEAN / PASS-NEEDS-ADJUDICATION reading, INCOMPLETE / INVALID-SESSION, FAIL-REPEATS / FAIL-ARMS / FAIL-CASESET, or FAIL); 2 on bad input. --check-block RUN_DIR REPORT_JSON: the completeness test 50-cts-after.sh applies before it writes a block's .done (the same code the verdict uses): exit 0 when the report has a result for every non-skipped case of the run's caselist and unrun.txt is empty, 3 when not (the reason is printed). """ import argparse import hashlib import json import os import re import sys from collections import Counter from pathlib import Path TOL = 0.0005 # CONTRACT-P7 7.1/7.2's constants: the gate is held to these, never to what gate3/arms.txt, # gate3/cases.txt or gate3/excluded.txt record about the run (ruling: integrator 2026-09-23 after # codex closeout review). GATE3_DENOMINATOR = 36 CI_SPLIT_DIRECTVULKAN = 39 CONTRACT_EXCLUDED = ("minecraft-1.21.1-neoforge-create-indirect-in-world", # OQ-17 "minecraft-1.21.11-main-menu", # UBO offset alignment "minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world") # VK_ERROR_UNKNOWN CONTRACT_REPEATS = 3 REQUIRED_ARMS = ("monolith", "inproc", "spawn", "inproc-ra0") CTS_TOL_PP = 0.5 HARD = ("Crash", "Timeout", "InternalError", "ResourceError", "DeviceHang", "Incomplete") SPLIT_ARMS = ("inproc", "spawn") OPENRA = "OpenRA" CTS_BLOCKS = ("shader-image", "ssbo", "dsa", "texture", "packed-pixels") UBO_BLOCK = {"block": "ubo", "caselist": "tools/cts/caselists/p7-ubo-gl46.txt", "cases": "GTF-GL46.gtf31.GL3Tests.uniform_buffer_object*", "verdict": "UNRUN", "reasons": ["unrun (not in this glcts): the device glcts has no GTF module (ID-P7-16);" " recorded, not gated"]} REPEAT_DIR = re.compile(r"^repeat-(\d+)$") KV = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=(\S*)") # ConfigLoader.cpp: "Config: MOBILEGL_TRANSPORT=inproc|spawn - ..." (and its refusals) and # "Config: IPC ring=..." are printed only when a transport resolves; the monolith prints neither. NOT_MONOLITH = re.compile(r"Config: IPC|MOBILEGL_TRANSPORT=") RULING = "ruling: integrator 2026-09-23" CODEX_RULING = "ruling: integrator 2026-09-23 after codex closeout review" def sha(path): h = hashlib.sha256() with open(path, "rb") as f: for block in iter(lambda: f.read(1 << 20), b""): h.update(block) return h.hexdigest() def read_kv(path): out = {} if path.is_file(): for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): if "=" in line: k, v = line.split("=", 1) out[k.strip()] = v.strip() return out def read_record(path): """A one-line 'k=v k=v ...' record (gate3 .done, boot_id.txt) -> dict, or None if absent.""" try: return dict(KV.findall(path.read_text(encoding="utf-8", errors="replace"))) except OSError: return None def first_line(path): try: lines = path.read_text(encoding="utf-8", errors="replace").splitlines() except OSError: return "" return lines[0].strip() if lines else "" def read_adjudication(path): """gate3/adjudication.tsv, cts/adjudication.tsv: '' per line ('#' comments and blank lines skipped; a case name holds no whitespace) -> ({case: reason}, problems). A line without a reason adjudicates nothing and is reported.""" found, problems = {}, [] if not path.is_file(): return found, problems for number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1): text = line.strip() if not text or text.startswith("#"): continue parts = text.split(None, 1) reason = parts[1].strip() if len(parts) > 1 else "" if not reason: problems.append("%s/%s:%d: %s has no reason: it adjudicates nothing" % (path.parent.name, path.name, number, parts[0])) continue found.setdefault(parts[0], reason) return found, problems # ------------------------------------------------------------------------------------------------ # gate 3 # ------------------------------------------------------------------------------------------------ def gate3_session(out, g): """One boot session for every record of every arm (ruling c).""" want = first_line(out / "session" / "boot-id.txt") problems, seen, unstamped = [], {}, [] def note(value, where): if value: seen.setdefault(value, []).append(where) else: unstamped.append(where) state = g / "state" for arm_dir in sorted(p for p in state.iterdir() if p.is_dir()) if state.is_dir() else []: for done in sorted(arm_dir.glob("*.done")): rec = read_record(done) or {} where = "state/%s/%s" % (arm_dir.name, done.name) note(rec.get("boot_id"), where) if "boot_id_before" in rec: note(rec["boot_id_before"], where + " (boot_id_before)") archive = g / "archive" for arm_dir in sorted(p for p in archive.iterdir() if p.is_dir()) if archive.is_dir() else []: for case_dir in sorted(p for p in arm_dir.iterdir() if p.is_dir()): for rep in sorted(p for p in case_dir.iterdir() if p.is_dir() and REPEAT_DIR.match(p.name)): where = "archive/%s/%s/%s/boot_id.txt" % (arm_dir.name, case_dir.name, rep.name) rec = read_record(rep / "boot_id.txt") or {} note(rec.get("boot_id"), where) if "boot_id_before" in rec: note(rec["boot_id_before"], where + " (boot_id_before)") if not want: problems.append("no session/boot-id.txt: 21-preflight.sh records the session's boot_id there") if unstamped: problems.append("%d record(s) carry no boot_id (pre-ruling 30-gate3.sh, or a repeat the driver did " "not stamp), e.g. %s" % (len(unstamped), unstamped[0])) for boot in sorted(b for b in seen if b != want): problems.append("boot_id %s != session %s in %d record(s), e.g. %s" % (boot, want or "?", len(seen[boot]), seen[boot][0])) return {"session_boot_id": want, "boot_ids": sorted(seen), "records": sum(len(v) for v in seen.values()), "unstamped": len(unstamped), "problems": problems, "valid": not problems, "reboot_clean": first_line(out / "session" / "reboot-clean.txt")} def monolith_identity(row): """None when the repeat's role logs show a monolith, else what says otherwise (ruling f).""" d = Path(row["dir"]) if not (d / "mobilegl.log").is_file(): return "no mobilegl.log archived: the arm's identity is unproven" for name in ("mobilegl.log", "mobilegl.client.log", "mobilegl.server.log"): p = d / name if not p.is_file(): continue for line in p.read_text(encoding="utf-8", errors="replace").splitlines(): if NOT_MONOLITH.search(line): return "%s: %s" % (name, line.strip()[:140]) return None def canonical_gate3_cases(): """CONTRACT-P7 7.1's denominator, computed the way 30-gate3.sh computes gate3/cases.txt: the tools tree's trace_cases.json CI split cases whose ci_backends carry DirectVulkan, minus the contract's three exclusions (CONTRACT_EXCLUDED, never the run's own gate3/excluded.txt). -> (cases in manifest order, or None; problems). A problem means the manifest no longer gives the contract's 39 / 36, so no run can be scored against it.""" try: from trace_cases import ci_backends, ci_trace_cases, load_trace_cases, split_trace_cases split_dv = [c["name"] for c in split_trace_cases(ci_trace_cases(load_trace_cases())) if "DirectVulkan" in ci_backends(c)] except Exception as error: # an unreadable manifest is a finding of the reading, not a crash return None, ["the tools tree's trace_cases.json cannot be read (%s: %s)" % (type(error).__name__, error)] problems = [] if len(split_dv) != CI_SPLIT_DIRECTVULKAN: problems.append("trace_cases.json gives %d CI split DirectVulkan case(s), CONTRACT-P7 7.1 names %d" % (len(split_dv), CI_SPLIT_DIRECTVULKAN)) absent = [x for x in CONTRACT_EXCLUDED if x not in split_dv] if absent: problems.append("CONTRACT-P7 7.1 exclusion(s) not among them: " + ", ".join(absent)) canonical = [c for c in split_dv if c not in CONTRACT_EXCLUDED] if len(set(canonical)) != GATE3_DENOMINATOR: problems.append("the canonical denominator is %d case(s), CONTRACT-P7 7.1 says %d" % (len(set(canonical)), GATE3_DENOMINATOR)) return canonical, problems def px_between(ca, left, right): """Exact RGB pixel-inequality count between two archived pictures (compare_actuals' mismatch_pixels, whole frame), or why it cannot be counted; None when a picture is missing.""" if not left or not right: return None try: a, b = ca.read_png_rgba(left), ca.read_png_rgba(right) return ca.mismatch_pixels(a, b, *ca.resolve_crop(a, b)) except Exception as error: # ImageError (a size mismatch), OSError, zlib: reported, not fatal return "n/a (%s)" % error def gate3(out, ca): g = out / "gate3" cases_file = g / "cases.txt" if not cases_file.is_file(): return None listed = [c for c in cases_file.read_text().split() if c] cases = list(dict.fromkeys(listed)) excluded = (g / "excluded.txt").read_text().split() if (g / "excluded.txt").is_file() else [] meta = read_kv(g / "arms.txt") try: repeat = int(meta.get("repeat", str(CONTRACT_REPEATS))) except ValueError: repeat = 0 planned = meta.get("arms", "").split() arms_present = [a for a in REQUIRED_ARMS if (g / "archive" / a).is_dir()] summaries = {a: ca.summarize(g / "archive" / a) for a in arms_present} session = gate3_session(out, g) identity = {"monolith_repeats": 0, "monolith_proven": 0, "split_repeats": 0, "split_proofs_passed": 0} adjudicated, adjudication_problems = read_adjudication(g / "adjudication.tsv") # ---- the contract's constants (ruling: integrator 2026-09-23 after codex closeout review) ---- contract = {"repeats": [], "arms": [], "cases": []} if repeat < CONTRACT_REPEATS: contract["repeats"].append("gate3/arms.txt repeat=%s < CONTRACT-P7 7.2's %d: 30-gate3.sh froze that count " "with the case list, a resume keeps it" % (meta.get("repeat", "?"), CONTRACT_REPEATS)) for arm in REQUIRED_ARMS: if arm not in arms_present and arm not in planned: contract["arms"].append("arm %s has no archive and gate3/arms.txt's arms=[%s] does not name it: the run " "was started without it; CONTRACT-P7 7.2 needs %s on every case" % (arm, " ".join(planned), " + ".join(REQUIRED_ARMS))) canonical, canonical_problems = canonical_gate3_cases() missing = [] if canonical is None or canonical_problems: contract["cases"].append("no canonical denominator to hold gate3/cases.txt to: " + "; ".join(canonical_problems)) else: outside = [c for c in cases if c not in set(canonical)] missing = [c for c in canonical if c not in set(cases)] if outside: contract["cases"].append("%d case(s) outside CONTRACT-P7 7.1's denominator (%d CI split DirectVulkan - " "the %d exclusions): %s%s" % ( len(outside), CI_SPLIT_DIRECTVULKAN, len(CONTRACT_EXCLUDED), ", ".join(outside), "; canonical case(s) missing: " + ", ".join(missing) if missing else "")) twice = sorted(c for c, n in Counter(listed).items() if n > 1) if twice: contract["cases"].append("case(s) listed more than once in gate3/cases.txt: " + ", ".join(twice)) short = [] def rows(arm, case): s = summaries.get(arm) if not s: return [] return [r for r in s["rows"] if r["arm"] == case + "-DirectVulkan"] def finished(arm, case): return (g / "state" / arm / (case + ".done")).is_file() def proof(row): p = Path(row["dir"]) / "transport-proof.json" try: return json.loads(p.read_text(encoding="utf-8")) except (OSError, ValueError): return None def logcat_sha(row): p = Path(row["dir"]) / "logcat.txt" return sha(p) if p.is_file() else None def picture(row): return str(Path(row["dir"]) / row["actual"]) if row and row.get("actual") else None def ra0_proof(row): p = Path(row["dir"]) / "mobilegl.log" if not p.is_file(): return "no-log" lines = re.findall(r"Config: IPC[^\r\n]*", p.read_text(encoding="utf-8", errors="replace")) if lines and all(re.search(r"\brun-ahead=0\b", line) for line in lines): return "run-ahead=0" return "NOT-PROVEN(" + ";".join(lines)[:120] + ")" def not_run(arm): return "%s: not run%s" % (arm, "" if arm in planned else " (and gate3/arms.txt does not name the arm)") results = [] for case in cases: reasons, incomplete = [], [] entry = {"case": case, "arms": {}} mono = rows("monolith", case) mono_row = mono_ssim = mono_sha = None if "monolith" in summaries and not finished("monolith", case): incomplete.append("monolith: no state .done (pair unfinished; 30-gate3.sh retries it)") if len(mono) != 1 or mono[0].get("error"): incomplete.append("monolith: %s" % mono[0].get("error") if mono else not_run("monolith")) else: m = mono_row = mono[0] mono_ssim, mono_sha = m.get("ssim_vs_golden"), m.get("actual_sha256") not_mono = monolith_identity(m) identity["monolith_repeats"] += 1 identity["monolith_proven"] += not_mono is None entry["arms"]["monolith"] = {"ssim": mono_ssim, "sha": mono_sha, "identity": not_mono or "monolith", "mismatch": m.get("mismatch_pixels_vs_golden"), "passed": m.get("passed")} if not_mono: reasons.append("monolith arm is NOT proven monolith (%s)" % not_mono) if m.get("passed") is not True: reasons.append("monolith control did not pass (ssim %s)" % mono_ssim) if case == OPENRA and not (m.get("mismatch_pixels_vs_golden") == 0 and (mono_ssim or 0) >= 0.9999995): reasons.append("OpenRA monolith not 1.000000/0 (%s/%s)" % (mono_ssim, m.get("mismatch_pixels_vs_golden"))) split_rows = {} for arm in SPLIT_ARMS: rs = split_rows[arm] = rows(arm, case) a = {"repeats": len(rs), "ssim": [r.get("ssim_vs_golden") for r in rs], "sha": sorted({r.get("actual_sha256") for r in rs if r.get("actual_sha256")}), "proofs_passed": 0} entry["arms"][arm] = a if arm not in summaries: incomplete.append(not_run(arm)) continue if not finished(arm, case): incomplete.append("%s: no state .done (pair unfinished; 30-gate3.sh retries it)" % arm) if len(rs) != repeat: incomplete.append("%s: %d/%d repeats" % (arm, len(rs), repeat)) if len(rs) < CONTRACT_REPEATS and finished(arm, case): # The pair is final (30-gate3.sh wrote its .done): a resume will not add a repeat. reasons.append("%s: %d repeat(s) in a finished pair < CONTRACT-P7 7.2's %d" % (arm, len(rs), CONTRACT_REPEATS)) short.append("%s/%s (%d)" % (arm, case, len(rs))) for r in rs: if r.get("error"): incomplete.append("%s rep%d: %s" % (arm, r["repeat"], r["error"])) continue if r.get("passed") is not True: reasons.append("%s rep%d below threshold (ssim %s)" % (arm, r["repeat"], r.get("ssim_vs_golden"))) p = proof(r) identity["split_repeats"] += 1 if not p or p.get("passed") is not True: reasons.append("%s rep%d arm proof (transport-proof.json) not passed" % (arm, r["repeat"])) else: a["proofs_passed"] += 1 identity["split_proofs_passed"] += 1 if mono_ssim is not None and r.get("ssim_vs_golden") is not None: d = abs(r["ssim_vs_golden"] - mono_ssim) if d > TOL: reasons.append("%s rep%d |ssim-monolith|=%.6f > %.4f" % (arm, r["repeat"], d, TOL)) if case == OPENRA and not (r.get("mismatch_pixels_vs_golden") == 0 and (r.get("ssim_vs_golden") or 0) >= 0.9999995): reasons.append("OpenRA %s rep%d not 1.000000/0 (%s/%s)" % (arm, r["repeat"], r.get("ssim_vs_golden"), r.get("mismatch_pixels_vs_golden"))) shas = [r.get("actual_sha256") for r in rs if r.get("actual_sha256")] if len(set(shas)) > 1: reasons.append("%s repeats NOT bit-identical (%d distinct pictures)" % (arm, len(set(shas)))) lc = [logcat_sha(r) for r in rs] for i in range(1, len(lc)): if lc[i] and lc[i] == lc[i - 1]: reasons.append("%s rep%d STALE (logcat identical to rep%d)" % (arm, rs[i]["repeat"], rs[i - 1]["repeat"])) # The RUN_AHEAD=0 control arm: required on every case (CONTRACT-P7 7.2 "RUN_AHEAD=0 对照臂一遍 # 记录"), and it must be that arm; its picture is recorded, never gating. ra0 = rows("inproc-ra0", case) if "inproc-ra0" in summaries and not finished("inproc-ra0", case): incomplete.append("inproc-ra0: no state .done (pair unfinished; 30-gate3.sh retries it)") if not ra0: incomplete.append(not_run("inproc-ra0")) else: r = ra0[0] entry["arms"]["inproc-ra0"] = {"ssim": r.get("ssim_vs_golden"), "sha": r.get("actual_sha256"), "passed": r.get("passed"), "proof": ra0_proof(r), "error": r.get("error")} if r.get("error"): incomplete.append("inproc-ra0 rep%d: %s" % (r["repeat"], r["error"])) elif entry["arms"]["inproc-ra0"]["proof"] != "run-ahead=0": reasons.append("inproc-ra0 is not proven the RUN_AHEAD=0 arm (%s)" % entry["arms"]["inproc-ra0"]["proof"]) # Cross-arm identity: not a per-case gate (ruling d), but every mismatch needs an adjudication. ip = set(entry["arms"].get("inproc", {}).get("sha", [])) sp = set(entry["arms"].get("spawn", {}).get("sha", [])) entry["inproc_eq_spawn"] = bool(ip) and ip == sp entry["split_eq_monolith"] = bool(ip) and ip == sp == {mono_sha} x = entry["cross_arm"] = {"monolith": mono_sha, "inproc": sorted(ip), "spawn": sorted(sp), "identical": (entry["split_eq_monolith"] if (ip and sp and mono_sha) else None)} if x["identical"] is False: first = {arm: picture(split_rows[arm][0]) if split_rows[arm] else None for arm in SPLIT_ARMS} x["px_vs_golden"] = {"monolith": mono_row.get("mismatch_pixels_vs_golden") if mono_row else None, "inproc": [r.get("mismatch_pixels_vs_golden") for r in split_rows["inproc"]], "spawn": [r.get("mismatch_pixels_vs_golden") for r in split_rows["spawn"]]} x["px_direct"] = {"inproc-monolith": px_between(ca, first["inproc"], picture(mono_row)), "spawn-monolith": px_between(ca, first["spawn"], picture(mono_row)), "inproc-spawn": px_between(ca, first["inproc"], first["spawn"])} x["adjudication"] = adjudicated.get(case) entry["verdict"] = "INCOMPLETE" if incomplete else ("FAIL" if reasons else "PASS") entry["reasons"] = incomplete + reasons results.append(entry) if short: contract["repeats"].append("%d finished inproc/spawn pair(s) archived fewer than CONTRACT-P7 7.2's %d repeats " "(arms.txt repeat=%s): %s%s" % (len(short), CONTRACT_REPEATS, meta.get("repeat", "?"), ", ".join(short[:4]), ", ..." if len(short) > 4 else "")) mismatched = [e["case"] for e in results if e["cross_arm"]["identical"] is False] unadjudicated = [c for c in mismatched if c not in adjudicated] stale = sorted(c for c in adjudicated if c not in set(mismatched)) verdicts = [e["verdict"] for e in results] # A reading that is not the gate's own (CONTRACT-P7 7.2: the canonical 36, one reboot-clean # session, every cross-arm mismatch adjudicated) may pass, but never as PASS: exit 0 is # reserved for the gate verdict. not_a_gate = [] if missing and not contract["cases"]: not_a_gate.append("%d of the canonical %d case(s) (missing %d, e.g. %s): a subset run is NOT a gate verdict" % (len(cases), GATE3_DENOMINATOR, len(missing), missing[0])) if not session["reboot_clean"].startswith("reboot-clean:"): not_a_gate.append("session not reboot-clean: NOT a gate verdict") if unadjudicated: not_a_gate.append("%d cross-arm mismatch(es) without a gate3/adjudication.tsv line (%s): %s%s" % ( len(unadjudicated), CODEX_RULING, ", ".join(unadjudicated[:3]), ", ..." if len(unadjudicated) > 3 else "")) if not session["valid"]: overall = "INVALID-SESSION" elif contract["repeats"]: overall = "FAIL-REPEATS" elif contract["arms"]: overall = "FAIL-ARMS" elif contract["cases"]: overall = "FAIL-CASESET" elif verdicts and all(v == "PASS" for v in verdicts): overall = "PASS" if missing: overall = "PASS-SUBSET" elif not session["reboot_clean"].startswith("reboot-clean:"): overall = "PASS-NOT-REBOOT-CLEAN" elif unadjudicated: overall = "PASS-NEEDS-ADJUDICATION" elif "FAIL" in verdicts: overall = "FAIL" else: overall = "INCOMPLETE" return {"cases": results, "excluded": excluded, "repeat": repeat, "denominator": len(cases), "overall": overall, "not_a_gate": not_a_gate, "arms_present": arms_present, "arms_planned": planned, "contract": contract, "canonical": canonical, "canonical_problems": canonical_problems, "cross_arm_mismatched": mismatched, "cross_arm_unadjudicated": unadjudicated, "adjudication": {"file": str(g / "adjudication.tsv"), "cases": adjudicated, "problems": adjudication_problems, "stale": stale}, "session": session, "identity": identity} def print_gate3(g3): print("== GATE 3 (DirectVulkan x pbuffer; split repeat=%d, contract >= %d; tol |ssim-monolith| <= %.4f) ==" % (g3["repeat"], CONTRACT_REPEATS, TOL)) s = g3["session"] print("-- session: boot_id %s; %d record(s) stamped; %s" % (s["session_boot_id"] or "?", s["records"], s["reboot_clean"] or "no session/reboot-clean.txt")) for problem in s["problems"]: print(" ! INVALID-SESSION: " + problem) c = g3["contract"] held = not any(c.values()) print("-- contract (CONTRACT-P7 7.1/7.2; %s): inproc/spawn >= %d repeats per case, arms %s on every case, " "case set == the canonical %d (%d CI split DirectVulkan - %d exclusions): %s" % (CODEX_RULING, CONTRACT_REPEATS, "+".join(REQUIRED_ARMS), GATE3_DENOMINATOR, CI_SPLIT_DIRECTVULKAN, len(CONTRACT_EXCLUDED), "held" if held else "VIOLATED")) for key, label in (("repeats", "FAIL-REPEATS"), ("arms", "FAIL-ARMS"), ("cases", "FAIL-CASESET")): for problem in c[key]: print(" ! %s: %s" % (label, problem)) if sorted(g3["excluded"]) != sorted(CONTRACT_EXCLUDED): print("WARN gate3/excluded.txt [%s] is not CONTRACT-P7 7.1's [%s] (the case set above is held to the contract's)" % (" ".join(g3["excluded"]), " ".join(CONTRACT_EXCLUDED))) w = max([len(e["case"]) for e in g3["cases"]] + [4]) print("%-*s %-10s %-11s %-24s %-24s %-11s %s" % (w, "case", "verdict", "monolith", "inproc(min..max) #sha", "spawn(min..max) #sha", "ra0", "i==s==m")) for e in g3["cases"]: def cell(arm): a = e["arms"].get(arm) if not a or not a.get("ssim"): return "-" vals = [v for v in a["ssim"] if isinstance(v, (int, float))] if not vals: return "err" return "%.6f..%.6f #%d" % (min(vals), max(vals), len(a["sha"])) mono = e["arms"].get("monolith", {}).get("ssim") ra0 = e["arms"].get("inproc-ra0", {}).get("ssim") print("%-*s %-10s %-11s %-24s %-24s %-11s %s" % ( w, e["case"], e["verdict"], "%.6f" % mono if isinstance(mono, (int, float)) else "-", cell("inproc"), cell("spawn"), "%.6f" % ra0 if isinstance(ra0, (int, float)) else "-", "yes" if e["split_eq_monolith"] else ("i==s" if e["inproc_eq_spawn"] else "no"))) for reason in e["reasons"]: print("%-*s ! %s" % (w, "", reason)) ra0s = [(e["case"], e["arms"]["inproc-ra0"]) for e in g3["cases"] if "inproc-ra0" in e["arms"]] print("-- RUN_AHEAD=0 control (required on every case, picture recorded, not gating): %d/%d case(s); proofs: %s" % ( len(ra0s), len(g3["cases"]), ", ".join(sorted({r["proof"] for _, r in ra0s})) or "-")) i = g3["identity"] print("-- arm identity: monolith %d/%d repeat(s) with no Config: IPC / MOBILEGL_TRANSPORT line; " "inproc+spawn transport-proof passed %d/%d" % (i["monolith_proven"], i["monolith_repeats"], i["split_proofs_passed"], i["split_repeats"])) judged = [e for e in g3["cases"] if e["cross_arm"]["identical"] is not None] adj = g3["adjudication"] print("-- cross-arm picture identity (NOT a per-case gate, %s; every mismatch needs a gate3/adjudication.tsv " "line, %s): %d/%d case(s) inproc==spawn==monolith; %d mismatch(es), %d unadjudicated" % ( RULING, CODEX_RULING, sum(e["cross_arm"]["identical"] for e in judged), len(judged), len(g3["cross_arm_mismatched"]), len(g3["cross_arm_unadjudicated"]))) def px(v): if isinstance(v, list): nums = [n for n in v if isinstance(n, int)] return "%d..%d" % (min(nums), max(nums)) if nums else "-" return "-" if v is None else str(v) for e in judged: if not e["cross_arm"]["identical"]: x = e["cross_arm"] g, d = x["px_vs_golden"], x["px_direct"] print("WARN cross-arm %s: monolith %s | inproc %s | spawn %s" % ( e["case"], (x["monolith"] or "-")[:12], ",".join(v[:12] for v in x["inproc"]) or "-", ",".join(v[:12] for v in x["spawn"]) or "-")) print(" px!=golden (archived mismatchPixels) monolith %s, inproc %s, spawn %s; px between the arms' " "first repeats: inproc-monolith %s, spawn-monolith %s, inproc-spawn %s" % ( px(g["monolith"]), px(g["inproc"]), px(g["spawn"]), px(d["inproc-monolith"]), px(d["spawn-monolith"]), px(d["inproc-spawn"]))) print(" " + ("adjudicated: " + x["adjudication"] if x.get("adjudication") else "UNADJUDICATED: no gate3/adjudication.tsv line")) for problem in adj["problems"]: print("WARN " + problem) if adj["stale"]: print(" note: gate3/adjudication.tsv names case(s) whose arms agree (nothing to adjudicate): " + ", ".join(adj["stale"])) print("-- excluded (red on monolith, CONTRACT-P7 7.1): " + (", ".join(g3["excluded"]) or "none")) n = len(g3["cases"]) npass = sum(e["verdict"] == "PASS" for e in g3["cases"]) print("GATE3 %s: %d/%d cases PASS%s" % (g3["overall"], npass, n, "".join(" (%s)" % x for x in g3["not_a_gate"]))) # ------------------------------------------------------------------------------------------------ # bsl stats # ------------------------------------------------------------------------------------------------ def bsl(out, ca): b = out / "bsl" if not b.is_dir(): return None arms = {} for arm_dir in sorted(p for p in b.iterdir() if p.is_dir()): a = {"done": read_kv(arm_dir / ".done")} a["peaks"] = (arm_dir / "peaks.txt").read_text() if (arm_dir / "peaks.txt").is_file() else "" a["wbuf"] = (arm_dir / "wbuf-gauges.txt").read_text() if (arm_dir / "wbuf-gauges.txt").is_file() else "" if (arm_dir / "archive").is_dir(): s = ca.summarize(arm_dir / "archive") a["replay"] = [{k: r.get(k) for k in ("passed", "ssim_vs_golden", "actual_sha256", "error")} for r in s["rows"]] arms[arm_dir.name] = a return arms def print_bsl(arms): print("== BSL-ESC-MENU MOBILEGL_PIPE_STATS (maps lines / RSS peaks, wbuf[] gauges) ==") for arm, a in arms.items(): print("[%s] %s" % (arm, " ".join("%s=%s" % kv for kv in a["done"].items()))) for r in a.get("replay", []): print(" replay: passed=%s ssim=%s sha=%s %s" % (r["passed"], r["ssim_vs_golden"], (r["actual_sha256"] or "-")[:16], r.get("error") or "")) for line in a["peaks"].splitlines(): print(" " + line) for line in a["wbuf"].splitlines() or ["(no wbuf[] line in the archived logs)"]: print(" " + line.strip()) # ------------------------------------------------------------------------------------------------ # CTS AFTER vs BASE # ------------------------------------------------------------------------------------------------ def read_list(path): if not path.is_file(): return [] return [l.strip() for l in path.read_text(encoding="utf-8").splitlines() if l.strip() and not l.startswith("#")] def rate(statuses, lost=0, adjudicated=0): """The gate's block rate = Pass / (Pass + Fail + lost), `lost` = the UNADJUDICATED $BASE-Pass cases this side turned NotSupported or another non-Pass, non-Fail status (a warning, a crash): a lost supported case is NON-PASS, it does not leave the denominator (ruling: integrator 2026-09-23 after codex closeout review). An ADJUDICATED one leaves the block's rate on both sides (ruling: integrator 2026-09-23 after the int3 critic): not an L case here, and on the $BASE side `adjudicated` of its Passes are taken out. Other NotSupported / warnings / crashes stay out of the denominator; crashes get their own count. Information only: CONTRACT-P7 7.3's literal Pass / (Pass + Fail) as `literal_rate` and the pre-ruling Pass / (results - NotSupported) as `results_rate`, both over every compared case.""" c = Counter(statuses) p, f, ns = c["Pass"], c["Fail"], c["NotSupported"] x = sum(c[s] for s in HARD) n = len(statuses) g = p - adjudicated return {"pass": p, "fail": f, "ns": ns, "crash": x, "warn": n - p - f - ns - x, "results": n, "lost": lost, "adjudicated": adjudicated, "rate": (g / (g + f + lost)) if (g + f + lost) else None, "literal_rate": (p / (p + f)) if (p + f) else None, "results_rate": (p / (n - ns)) if (n - ns) else None} def load_results(path): """-> (results dict, None) or (None, why the report does not count).""" if not path.is_file(): return None, "no %s" % path.name if path.stat().st_size == 0: return None, "%s is empty" % path.name try: results = json.loads(path.read_text(encoding="utf-8"))["results"] except (OSError, ValueError, KeyError, TypeError) as error: return None, "%s unreadable (%s)" % (path.name, error) if not isinstance(results, dict) or not results: return None, "%s has no results" % path.name return results, None def caselist_of(run): """-> (cases, source, None) for the cases a block's AFTER run was asked to run, or (None, source, why) when that cannot be read. 50-cts-after.sh keeps a copy (runs//caselist.txt) beside the path it ran (caselist.path), so the window's output reduces on its own after the build tree that path names is gone; a named list that is gone or empty is NOT replaced by the report's own cases (that would make a block of 0 - or of only what ran - look complete).""" copy = run / "caselist.txt" if copy.is_file(): src = copy elif (run / "caselist.path").is_file(): src = Path((run / "caselist.path").read_text(encoding="utf-8", errors="replace").strip() or "(empty caselist.path)") else: return None, None, ("no runs/%s/caselist.txt or caselist.path: which cases the run was asked " "to run is unknown" % run.name) if not src.is_file(): return None, src, "the caselist the run used is gone (%s) and runs/%s/caselist.txt holds no copy" % (src, run.name) cases = read_list(src) if not cases: return None, src, "the caselist the run used (%s) holds no case" % src return cases, src, None def block_scope(run, after): """The one completeness test of a finished block (the verdict below, and 50-cts-after.sh through --check-block before it writes .done). -> (universe, source, missing, incomplete): universe = the caselist minus run_cts's skipped.txt; `missing` says why there is no reading at all, `incomplete` why the reading lacks cases.""" cases, src, why = caselist_of(run) if why: return [], src, [why], [] skipped = set(read_list(run / "skipped.txt")) universe = [x for x in cases if x not in skipped] if not universe: return [], src, ["every case of %s is skipped: a block of 0 cases is no reading" % src], [] unrun = read_list(run / "unrun.txt") without = [x for x in universe if x not in after] incomplete = [] if unrun or without: incomplete.append("%d case(s) without an AFTER result (unrun.txt %d)" % (len(without), len(unrun))) return universe, src, [], incomplete def cts_block(c, b, base_dir, canon_dir, adjudicated=None): adjudicated = adjudicated or {} run = c / "runs" / b if not c.is_dir(): return {"block": b, "verdict": "MISSING", "reasons": ["no cts/ directory: 50-cts-after.sh did not run"]} if not (run / ".done").is_file(): return {"block": b, "verdict": "MISSING", "reasons": ["block not finished (no runs/%s/.done)" % b]} after, why = load_results(c / ("report-%s.json" % b)) if why: return {"block": b, "verdict": "MISSING", "reasons": [why]} universe, src, missing, incomplete = block_scope(run, after) if missing: return {"block": b, "verdict": "MISSING", "reasons": missing} base, why = load_results(base_dir / ("report-%s.json" % b)) if why: return {"block": b, "verdict": "NO-BASE", "reasons": ["$BASE: " + why + " in " + str(base_dir)]} unrun = read_list(run / "unrun.txt") hung = read_list(run / "hung.txt") # Like for like: both rates over the cases that have a BASE and an AFTER result. both = [x for x in universe if x in after and x in base] # A $BASE Pass that AFTER turned NotSupported / a warning / a crash: NON-PASS in the gate rate, # listed, and FAIL unless adjudicated (ruling: integrator 2026-09-23 after codex closeout review). # An adjudicated one is out of the block's rate on BOTH sides (ruling: integrator 2026-09-23 after # the int3 critic - else one ruled-legitimate loss in a 124-case block was -0.806 pp for ever). lost = sorted(x for x in both if base[x] == "Pass" and after[x] not in ("Pass", "Fail")) unadjudicated = [x for x in lost if x not in adjudicated] waived = len(lost) - len(unadjudicated) ra, rb = rate([after[x] for x in both], lost=len(unadjudicated)), rate([base[x] for x in both], adjudicated=waived) def pp(key): return None if ra[key] is None or rb[key] is None else (ra[key] - rb[key]) * 100.0 delta, delta_literal, delta_results = pp("rate"), pp("literal_rate"), pp("results_rate") new_crash = sorted(x for x in universe if after.get(x) in HARD and base.get(x) not in HARD) new_crash += sorted(x for x in hung if x not in new_crash) regress = sorted(x for x in universe if base.get(x) == "Pass" and after.get(x) not in (None, "Pass")) fixed = sorted(x for x in universe if base.get(x) not in (None, "Pass") and after.get(x) == "Pass") missing_base = sorted(x for x in universe if x not in base) fail, notes, warnings = [], [], [] if new_crash: fail.append("%d NEW crash(es)" % len(new_crash)) if rb["rate"] is None: incomplete.append("$BASE has no Pass/Fail over this block's %d case(s): no contract rate to compare against" % len(both)) elif ra["literal_rate"] is None: fail.append("AFTER has no Pass/Fail result (NS %d, W %d, X %d of %d): the literal Pass/(Pass+Fail) cannot be " "formed where $BASE's is %.3f%%; %d $BASE Pass(es) lost" % ( ra["ns"], ra["warn"], ra["crash"], ra["results"], 100 * rb["rate"], len(lost))) if delta is not None and delta < -CTS_TOL_PP: fail.append("delta %.3f pp < -%.1f pp (Pass/(Pass+Fail+L))" % (delta, CTS_TOL_PP)) if delta is not None and delta > CTS_TOL_PP: notes.append("note: AFTER is %.3f pp ABOVE BASE (not a regression; review the fixed list)" % delta) if unadjudicated: fail.append("%d $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status with no " "cts/adjudication.tsv line (%s)" % (len(unadjudicated), CODEX_RULING)) if waived: notes.append("note: %d $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status and " "adjudicated in cts/adjudication.tsv: out of the block's rate, BASE and AFTER" % waived) if missing_base: warnings.append("%d case(s) have no $BASE result (e.g. %s): left out of both rates; a crash among them " "still counts as NEW" % (len(missing_base), missing_base[0])) if ra["ns"] != rb["ns"]: warnings.append("NotSupported $BASE %d -> AFTER %d (a $BASE Pass among them is an L case: NON-PASS in the " "rate, listed below)" % (rb["ns"], ra["ns"])) canonical = canon_dir / ("p7-%s-gl46.txt" % b) subset = set(read_list(src)) != set(read_list(canonical)) verdict = "FAIL" if fail else ("INCOMPLETE" if incomplete else "PASS") return {"block": b, "cases": len(universe), "compared": len(both), "caselist": str(src), "after": ra, "base": rb, "delta_pp": delta, "delta_pp_literal_info": delta_literal, "delta_pp_results_info": delta_results, "new_crash": new_crash, "regressed": regress, "fixed": fixed, "missing_in_base": missing_base, "lost": [{"case": x, "after": after[x], "adjudication": adjudicated.get(x)} for x in lost], "lost_unadjudicated": unadjudicated, "lost_adjudicated": [x for x in lost if x in adjudicated], "unrun": len(unrun), "hung": hung, "verdict": verdict, "reasons": incomplete + fail + notes, "warnings": warnings, "subset": subset} def cts(out, base_dir, canon_dir): c = out / "cts" adjudicated, adjudication_problems = read_adjudication(c / "adjudication.tsv") blocks = [cts_block(c, b, base_dir, canon_dir, adjudicated) for b in CTS_BLOCKS] extra = sorted(p.stem[len("report-"):] for p in c.glob("report-*.json") if p.stem[len("report-"):] not in CTS_BLOCKS) if c.is_dir() else [] verdicts = [b["verdict"] for b in blocks] if "FAIL" in verdicts: overall = "FAIL" elif all(v == "PASS" for v in verdicts): # Passing on a --limit subset (or on a list that is not the tools tree's) is not gate 5. overall = "PASS-SUBSET" if any(b.get("subset") for b in blocks) else "PASS" else: overall = "INCOMPLETE" lost = {x["case"] for b in blocks for x in b.get("lost", [])} return {"blocks": blocks, "overall": overall, "ubo": dict(UBO_BLOCK), "ignored_reports": extra, "incomplete_blocks": [b["block"] for b in blocks if b["verdict"] not in ("PASS", "FAIL")], "adjudication": {"file": str(c / "adjudication.tsv"), "cases": adjudicated, "problems": adjudication_problems, "stale": sorted(x for x in adjudicated if x not in lost)}} def print_cts(c): print("== CTS AFTER (inproc x DirectVulkan) vs BASE (monolith x DirectVulkan, p7w1) ==") print("gate (CONTRACT-P7 7.3 + %s):" % CODEX_RULING) print(" rate = Pass/(Pass+Fail+L), L = the $BASE-Pass cases AFTER turned NotSupported or another non-Pass,") print(" non-Fail status (a lost supported case is NON-PASS); other NS, warnings W and crashes X are not in the") print(" denominator. Per block: delta >= -%.1f pp, 0 NEW crash, and every L case adjudicated in" % CTS_TOL_PP) print(" cts/adjudication.tsv (else FAIL); an adjudicated one (A) is out of the block's rate, BASE and AFTER.") print(" All five blocks required. '(literal)' = the contract's literal Pass/(Pass+Fail) delta and 'info' the") print(" pre-ruling Pass/(results-NS), both over every compared case: printed for comparison, never gating.") print("%-14s %6s %-20s %8s %-20s %4s %3s %8s %9s %9s %5s %4s %5s %s" % ( "block", "cases", "BASE P/F/NS/W/X", "rate", "AFTER P/F/NS/W/X", "L", "A", "rate", "delta pp", "(literal)", "crash", "regr", "fixed", "verdict")) def counts(r): return "%d/%d/%d/%d/%d" % (r["pass"], r["fail"], r["ns"], r["warn"], r["crash"]) def pct(v): return "%.3f%%" % (100 * v) if v is not None else "n/a" def pp(v): return "%+.3f" % v if v is not None else "n/a" for b in c["blocks"]: if "after" not in b: print("%-14s %s: %s" % (b["block"], b["verdict"], "; ".join(b["reasons"]))) continue print("%-14s %6d %-20s %8s %-20s %4d %3d %8s %9s %9s %5d %4d %5d %s%s" % ( b["block"], b["cases"], counts(b["base"]), pct(b["base"]["rate"]), counts(b["after"]), b["after"]["lost"], len(b.get("lost_adjudicated", [])), pct(b["after"]["rate"]), pp(b["delta_pp"]), "(%s)" % pp(b["delta_pp_literal_info"]), len(b["new_crash"]), len(b["regressed"]), len(b["fixed"]), b["verdict"], " (SUBSET)" if b["subset"] else "")) print(" info literal Pass/(Pass+Fail): BASE %s AFTER %s delta %s pp (not gating)" % ( pct(b["base"]["literal_rate"]), pct(b["after"]["literal_rate"]), pp(b["delta_pp_literal_info"]))) print(" info pre-ruling Pass/(results-NS): BASE %s AFTER %s delta %s pp (not gating)" % ( pct(b["base"]["results_rate"]), pct(b["after"]["results_rate"]), pp(b["delta_pp_results_info"]))) for reason in b["reasons"]: print(" ! " + reason) for warning in b["warnings"]: print("WARN %s: %s" % (b["block"], warning)) for x in b["new_crash"]: print(" NEW CRASH " + x) for x in b["lost"]: if x["adjudication"]: print(" ADJUDICATED %s (Pass -> %s) out of the rate; adjudicated: %s" % ( x["case"], x["after"], x["adjudication"])) else: print(" LOST %s (Pass -> %s) UNADJUDICATED" % (x["case"], x["after"])) lost = {x["case"] for x in b["lost"]} others = [x for x in b["regressed"] if x not in lost] for x in others[:40]: print(" Pass->not " + x) if len(others) > 40: print(" ... %d more regressions in the JSON" % (len(others) - 40)) for x in b["fixed"][:20]: print(" not->Pass " + x) u = c["ubo"] print("%-14s %s (%s)" % (u["block"], u["reasons"][0], u["cases"])) if c["ignored_reports"]: print(" note: report(s) outside the five gate blocks ignored: " + ", ".join(c["ignored_reports"])) for problem in c["adjudication"]["problems"]: print("WARN " + problem) if c["adjudication"]["stale"]: print(" note: cts/adjudication.tsv names case(s) that are no $BASE-Pass loss (nothing to adjudicate): " + ", ".join(c["adjudication"]["stale"])) tail = "" if c["incomplete_blocks"]: tail += " (INCOMPLETE: %s)" % ", ".join("%s %s" % (b["block"], b["verdict"]) for b in c["blocks"] if b["block"] in c["incomplete_blocks"]) if any(b.get("subset") for b in c["blocks"]): tail += " (SUBSET run: %s not the tools tree's full caselist; not a gate verdict)" % ", ".join( b["block"] for b in c["blocks"] if b.get("subset")) print("CTS %s%s" % (c["overall"], tail)) def print_timing(out): t = out / "timing.tsv" if not t.is_file(): return print("== TIMING ==") for line in t.read_text().splitlines(): step, start, end, secs, note = (line.split("\t") + [""] * 5)[:5] print(" %-18s %6ss %s -> %s %s" % (step, secs, start, end, note)) prog = out / "gate3" / "progress.tsv" if prog.is_file(): per = {} for line in prog.read_text().splitlines(): arm, case, rc, secs, _ = (line.split("\t") + [""] * 5)[:5] per.setdefault(arm, []).append(int(secs)) for arm, v in per.items(): print(" gate3 %-11s %2d case invocations, mean %.0fs, total %ds" % (arm, len(v), sum(v) / len(v), sum(v))) def check_block(run, report): """--check-block: 0 when the block is a complete AFTER reading, 3 when not (50-cts-after.sh).""" after, why = load_results(report) if why: print("INCOMPLETE: " + why) return 3 universe, src, missing, incomplete = block_scope(run, after) if missing or incomplete: print("INCOMPLETE: " + "; ".join(missing + incomplete)) return 3 print("complete: %d/%d case(s) of %s have a result" % (len(universe), len(universe), src)) return 0 def main(): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("stamp", nargs="?") ap.add_argument("--logroot", default=os.environ.get("W2_LOGROOT", str(Path.home() / "w7/logs/p7w7"))) ap.add_argument("--tools", help="MobileGL tree holding compare_actuals.py and CTS-base (default: build.env)") ap.add_argument("--json", help="write every section as JSON here (default /verdict.json)") ap.add_argument("--check-block", nargs=2, metavar=("RUN_DIR", "REPORT_JSON"), help="only test one block's completeness (50-cts-after.sh, before it writes .done)") args = ap.parse_args() if args.check_block: return check_block(Path(args.check_block[0]), Path(args.check_block[1])) if not args.stamp: ap.print_usage(sys.stderr) return 2 out = Path(args.logroot) / args.stamp if not out.is_dir(): print("no such window output: %s" % out, file=sys.stderr) return 2 tools = args.tools or os.environ.get("W2_TOOLS_OVERRIDE") or read_kv(out / "build.env").get("BUILD_TREE") if not tools: print("no tools tree (--tools)", file=sys.stderr) return 2 sys.path.insert(0, str(Path(tools) / "tools" / "trace_replay")) sys.dont_write_bytecode = True # the tools tree is only read (it may be the read-only integration tree) import compare_actuals as ca base_dir = Path(tools) / "docs/Disaggregated/notes/p7/device-window-1/CTS-base" report = {"stamp": args.stamp, "out": str(out)} g3 = gate3(out, ca) if g3: print_gate3(g3) else: g3 = {"overall": "NOT-RUN"} print("== GATE 3 ==\nGATE3 NOT-RUN (no gate3/cases.txt)") print() report["gate3"] = g3 b = bsl(out, ca) if b: print_bsl(b); print() report["bsl"] = b c = cts(out, base_dir, Path(tools) / "tools/cts/caselists") print_cts(c); print() report["cts"] = c print_timing(out) ok = g3["overall"] == "PASS" and c["overall"] == "PASS" report["window"] = {"gate3": g3["overall"], "cts": c["overall"], "exit": 0 if ok else 1} print("\nWINDOW2 GATE3 %s | CTS %s -> exit %d" % (g3["overall"], c["overall"], 0 if ok else 1)) Path(args.json or out / "verdict.json").write_text(json.dumps(report, indent=1) + "\n", encoding="utf-8") return 0 if ok else 1 if __name__ == "__main__": sys.exit(main())