mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-20 08:21:56 +09:00
[tools] (P7): window-2 reducers enforce the contract's repeats, arms and case set, and gate Pass-to-NotSupported losses
This commit is contained in:
Executable → Regular
+332
-73
@@ -4,7 +4,22 @@
|
||||
|
||||
Device window 2 verdicts, read from the window's own output directory (<logroot>/<stamp>).
|
||||
|
||||
GATE 3 (CONTRACT-P7 7.2), per case of gate3/cases.txt:
|
||||
GATE 3 (CONTRACT-P7 7.1/7.2), per case of gate3/cases.txt:
|
||||
* THE CONTRACT'S CONSTANTS, NOT THE RUNNER'S OWN RECORDS (ruling: integrator 2026-09-23 after
|
||||
codex closeout review). gate3/arms.txt says what 30-gate3.sh was asked to run, not what the
|
||||
gate needs, so the reducer holds the run to CONTRACT-P7 7.1/7.2 itself:
|
||||
- REPEATS: inproc and spawn need >= 3 archived repeats on every case. arms.txt repeat < 3,
|
||||
or a finished pair (state .done) with fewer than 3 archived repeats, is GATE3 FAIL-REPEATS
|
||||
whatever arms.txt says;
|
||||
- ARMS: monolith, inproc, spawn AND inproc-ra0 on every case. An arm with no archive that
|
||||
arms.txt's arms= does not name (the run was started without it) is GATE3 FAIL-ARMS; a named
|
||||
arm not reached yet, or a case an arm lacks, makes that case INCOMPLETE;
|
||||
- CASE SET: gate3/cases.txt must EQUAL the canonical denominator, computed the way
|
||||
30-gate3.sh computes it - the tools tree's trace_cases.json CI split cases whose
|
||||
ci_backends carry DirectVulkan (39) - minus CONTRACT-P7 7.1's three exclusions, which are
|
||||
this file's constants (CONTRACT_EXCLUDED), not gate3/excluded.txt. A case outside it, a
|
||||
case listed twice, or a manifest that no longer gives 39 / 36 is GATE3 FAIL-CASESET; a
|
||||
proper subset (a --cases run) is PASS-SUBSET at best - never merely "count 36";
|
||||
* ONE SESSION: every gate3/state/<arm>/<case>.done and every archived repeat's boot_id.txt, of
|
||||
every arm, carries one boot_id, equal to session/boot-id.txt. A second boot_id, a record that
|
||||
carries none, or no session/boot-id.txt makes the whole gate GATE3 INVALID-SESSION, whatever
|
||||
@@ -17,11 +32,18 @@ GATE 3 (CONTRACT-P7 7.2), per case of gate3/cases.txt:
|
||||
within-arm identity is the gate), and |ssim - monolith ssim| <= 0.0005 on every repeat; a
|
||||
repeat whose logcat is byte-identical to the previous repeat's is flagged STALE;
|
||||
* OpenRA: 1.000000 and mismatch 0 on every repeat of both split arms AND on monolith;
|
||||
* cross-arm picture identity (inproc vs spawn vs monolith) is REPORTED per case and a mismatch
|
||||
prints a WARN line - information, not a gate (ruling: integrator 2026-09-23);
|
||||
* inproc-ra0 (RUN_AHEAD=0): recorded (ssim, sha, run-ahead=0 proof), never gating;
|
||||
* a reading that passes but is not the gate's - a denominator other than 36, or a session that
|
||||
is not reboot-clean - is GATE3 PASS-SUBSET / PASS-NOT-REBOOT-CLEAN, never PASS.
|
||||
* cross-arm picture identity (inproc vs spawn vs monolith) is not a per-case gate (ruling:
|
||||
integrator 2026-09-23, ID-P7-59): a mismatch prints a WARN line and leaves the case's verdict
|
||||
alone. But it is never waved through silently (ruling: integrator 2026-09-23 after codex
|
||||
closeout review): every mismatching case must be listed in gate3/adjudication.tsv
|
||||
('<case><TAB><reason>'), else an otherwise PASS gate is GATE3 PASS-NEEDS-ADJUDICATION (exit 1).
|
||||
Each mismatch is listed with the archived mismatchPixels (vs golden) of every arm and the
|
||||
direct pixel counts between the arms' first repeats;
|
||||
* inproc-ra0 (RUN_AHEAD=0): required on every case (one archived repeat with its .done) and its
|
||||
role log must prove run-ahead=0 (else that case FAIL: it is not the control arm); its picture
|
||||
and ssim are recorded, never gating;
|
||||
* a reading that passes but is not the gate's - a proper subset of the canonical 36, or a
|
||||
session that is not reboot-clean - is GATE3 PASS-SUBSET / PASS-NOT-REBOOT-CLEAN, never PASS.
|
||||
BSL STATS: per arm, maps-line / VmRSS / VmHWM peaks and the wbuf[] gauges (information).
|
||||
CTS AFTER (CONTRACT-P7 7.3), per block vs device-window-1/CTS-base/report-<block>.json, over the
|
||||
cases the AFTER run was asked to run (so a --limit subset is compared like for like):
|
||||
@@ -31,21 +53,32 @@ CTS AFTER (CONTRACT-P7 7.3), per block vs device-window-1/CTS-base/report-<block
|
||||
else the file caselist.path names); a missing one makes the block MISSING and CTS INCOMPLETE -
|
||||
a block of 0 cases is never a reading. The UBO block (GTF-GL46 uniform_buffer_object) is
|
||||
recorded as 'unrun (not in this glcts)': the device glcts carries no GTF module (ID-P7-16);
|
||||
* rate = Pass / (Pass + Fail), the contract formula: NotSupported, the warnings and the crashes
|
||||
are NOT in the denominator; gate: delta >= -0.5 pp on this rate. Both rates are taken over the
|
||||
* rate = Pass / (Pass + Fail + L), L = the cases BASE passed that AFTER turned NotSupported or
|
||||
any other non-Pass, non-Fail status (a warning, a crash): a lost supported case is NON-PASS, it
|
||||
does not leave the denominator (ruling: integrator 2026-09-23 after codex closeout review -
|
||||
under the contract's literal Pass / (Pass + Fail), 123 of 124 BASE-Pass cases turning
|
||||
NotSupported would still read 100%). NotSupported, warnings and crashes that were not a BASE
|
||||
Pass stay out of both rates. gate: delta >= -0.5 pp on this rate. Both rates are taken over the
|
||||
cases that have a BASE and an AFTER result (a case BASE lacks prints a WARN, stays out of both
|
||||
rates and still counts for new crashes). AFTER without a single Pass/Fail where BASE has some
|
||||
is FAIL (the rate cannot be formed: every BASE Pass is gone); BASE without one is INCOMPLETE;
|
||||
rates and still counts for new crashes). AFTER without a single Pass/Fail/L where BASE has some
|
||||
is FAIL (the rate cannot be formed); BASE without one is INCOMPLETE;
|
||||
* every L case is listed (Pass -> its AFTER status), and a block with one that
|
||||
cts/adjudication.tsv ('<case><TAB><reason>') does not list is FAIL. An adjudication waives
|
||||
only that rule: the case still counts as NON-PASS in the rate;
|
||||
* crashes (Crash/Timeout/InternalError/ResourceError/DeviceHang/Incomplete, plus hung.txt) are
|
||||
counted in their own column and are non-Pass; one that BASE does not already have is a NEW
|
||||
crash, a hard red of its own (a Fail that became a Crash is a new crash);
|
||||
crash, a hard red of its own that no adjudication waives (a Fail that became a Crash is a new
|
||||
crash; a Pass that became a Crash is a new crash AND an L case);
|
||||
* a block whose cases are not the tools tree's full p7-<block>-gl46.txt (a --limit run) makes an
|
||||
all-PASS CTS reading 'PASS-SUBSET', never PASS;
|
||||
* information only, never gating: the pre-ruling rate Pass / (results - NotSupported) of BASE
|
||||
and AFTER and its delta, and a WARN when the NotSupported count moved.
|
||||
Exit status: 0 only when GATE 3 is PASS (36/36, reboot-clean, one session) and CTS is PASS (five
|
||||
full blocks); 1 otherwise (a section that did not run, a PASS-SUBSET / PASS-NOT-REBOOT-CLEAN
|
||||
reading, INCOMPLETE / INVALID-SESSION, or FAIL); 2 on bad input.
|
||||
* information only, never gating: the contract's literal Pass / (Pass + Fail) of BASE and AFTER
|
||||
and its delta, the pre-ruling Pass / (results - NotSupported) and its delta, and a WARN when
|
||||
the NotSupported count moved.
|
||||
Exit status: 0 only when GATE 3 is PASS (the canonical 36, >= 3 repeats, four arms, reboot-clean,
|
||||
one session, every cross-arm mismatch adjudicated) and CTS is PASS (five full blocks); 1 otherwise
|
||||
(a section that did not run, a PASS-SUBSET / PASS-NOT-REBOOT-CLEAN / PASS-NEEDS-ADJUDICATION
|
||||
reading, INCOMPLETE / INVALID-SESSION, FAIL-REPEATS / FAIL-ARMS / FAIL-CASESET, or FAIL); 2 on
|
||||
bad input.
|
||||
--check-block RUN_DIR REPORT_JSON: the completeness test 50-cts-after.sh applies before it writes
|
||||
a block's .done (the same code the verdict uses): exit 0 when the report has a result for every
|
||||
non-skipped case of the run's caselist and unrun.txt is empty, 3 when not (the reason is printed).
|
||||
@@ -60,7 +93,16 @@ from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
TOL = 0.0005
|
||||
# CONTRACT-P7 7.1/7.2's constants: the gate is held to these, never to what gate3/arms.txt,
|
||||
# gate3/cases.txt or gate3/excluded.txt record about the run (ruling: integrator 2026-09-23 after
|
||||
# codex closeout review).
|
||||
GATE3_DENOMINATOR = 36
|
||||
CI_SPLIT_DIRECTVULKAN = 39
|
||||
CONTRACT_EXCLUDED = ("minecraft-1.21.1-neoforge-create-indirect-in-world", # OQ-17
|
||||
"minecraft-1.21.11-main-menu", # UBO offset alignment
|
||||
"minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world") # VK_ERROR_UNKNOWN
|
||||
CONTRACT_REPEATS = 3
|
||||
REQUIRED_ARMS = ("monolith", "inproc", "spawn", "inproc-ra0")
|
||||
CTS_TOL_PP = 0.5
|
||||
HARD = ("Crash", "Timeout", "InternalError", "ResourceError", "DeviceHang", "Incomplete")
|
||||
SPLIT_ARMS = ("inproc", "spawn")
|
||||
@@ -76,6 +118,7 @@ KV = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=(\S*)")
|
||||
# "Config: IPC ring=..." are printed only when a transport resolves; the monolith prints neither.
|
||||
NOT_MONOLITH = re.compile(r"Config: IPC|MOBILEGL_TRANSPORT=")
|
||||
RULING = "ruling: integrator 2026-09-23"
|
||||
CODEX_RULING = "ruling: integrator 2026-09-23 after codex closeout review"
|
||||
|
||||
|
||||
def sha(path):
|
||||
@@ -112,6 +155,27 @@ def first_line(path):
|
||||
return lines[0].strip() if lines else ""
|
||||
|
||||
|
||||
def read_adjudication(path):
|
||||
"""gate3/adjudication.tsv, cts/adjudication.tsv: '<case><TAB><reason>' per line ('#' comments
|
||||
and blank lines skipped; a case name holds no whitespace) -> ({case: reason}, problems). A line
|
||||
without a reason adjudicates nothing and is reported."""
|
||||
found, problems = {}, []
|
||||
if not path.is_file():
|
||||
return found, problems
|
||||
for number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
|
||||
text = line.strip()
|
||||
if not text or text.startswith("#"):
|
||||
continue
|
||||
parts = text.split(None, 1)
|
||||
reason = parts[1].strip() if len(parts) > 1 else ""
|
||||
if not reason:
|
||||
problems.append("%s/%s:%d: %s has no reason: it adjudicates nothing"
|
||||
% (path.parent.name, path.name, number, parts[0]))
|
||||
continue
|
||||
found.setdefault(parts[0], reason)
|
||||
return found, problems
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# gate 3
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
@@ -171,19 +235,90 @@ def monolith_identity(row):
|
||||
return None
|
||||
|
||||
|
||||
def canonical_gate3_cases():
|
||||
"""CONTRACT-P7 7.1's denominator, computed the way 30-gate3.sh computes gate3/cases.txt: the
|
||||
tools tree's trace_cases.json CI split cases whose ci_backends carry DirectVulkan, minus the
|
||||
contract's three exclusions (CONTRACT_EXCLUDED, never the run's own gate3/excluded.txt).
|
||||
-> (cases in manifest order, or None; problems). A problem means the manifest no longer gives
|
||||
the contract's 39 / 36, so no run can be scored against it."""
|
||||
try:
|
||||
from trace_cases import ci_backends, ci_trace_cases, load_trace_cases, split_trace_cases
|
||||
split_dv = [c["name"] for c in split_trace_cases(ci_trace_cases(load_trace_cases()))
|
||||
if "DirectVulkan" in ci_backends(c)]
|
||||
except Exception as error: # an unreadable manifest is a finding of the reading, not a crash
|
||||
return None, ["the tools tree's trace_cases.json cannot be read (%s: %s)" % (type(error).__name__, error)]
|
||||
problems = []
|
||||
if len(split_dv) != CI_SPLIT_DIRECTVULKAN:
|
||||
problems.append("trace_cases.json gives %d CI split DirectVulkan case(s), CONTRACT-P7 7.1 names %d"
|
||||
% (len(split_dv), CI_SPLIT_DIRECTVULKAN))
|
||||
absent = [x for x in CONTRACT_EXCLUDED if x not in split_dv]
|
||||
if absent:
|
||||
problems.append("CONTRACT-P7 7.1 exclusion(s) not among them: " + ", ".join(absent))
|
||||
canonical = [c for c in split_dv if c not in CONTRACT_EXCLUDED]
|
||||
if len(set(canonical)) != GATE3_DENOMINATOR:
|
||||
problems.append("the canonical denominator is %d case(s), CONTRACT-P7 7.1 says %d"
|
||||
% (len(set(canonical)), GATE3_DENOMINATOR))
|
||||
return canonical, problems
|
||||
|
||||
|
||||
def px_between(ca, left, right):
|
||||
"""Exact RGB pixel-inequality count between two archived pictures (compare_actuals'
|
||||
mismatch_pixels, whole frame), or why it cannot be counted; None when a picture is missing."""
|
||||
if not left or not right:
|
||||
return None
|
||||
try:
|
||||
a, b = ca.read_png_rgba(left), ca.read_png_rgba(right)
|
||||
return ca.mismatch_pixels(a, b, *ca.resolve_crop(a, b))
|
||||
except Exception as error: # ImageError (a size mismatch), OSError, zlib: reported, not fatal
|
||||
return "n/a (%s)" % error
|
||||
|
||||
|
||||
def gate3(out, ca):
|
||||
g = out / "gate3"
|
||||
cases_file = g / "cases.txt"
|
||||
if not cases_file.is_file():
|
||||
return None
|
||||
cases = [c for c in cases_file.read_text().split() if c]
|
||||
listed = [c for c in cases_file.read_text().split() if c]
|
||||
cases = list(dict.fromkeys(listed))
|
||||
excluded = (g / "excluded.txt").read_text().split() if (g / "excluded.txt").is_file() else []
|
||||
meta = read_kv(g / "arms.txt")
|
||||
repeat = int(meta.get("repeat", "3"))
|
||||
arms_present = [a for a in ("monolith", "inproc", "spawn", "inproc-ra0") if (g / "archive" / a).is_dir()]
|
||||
try:
|
||||
repeat = int(meta.get("repeat", str(CONTRACT_REPEATS)))
|
||||
except ValueError:
|
||||
repeat = 0
|
||||
planned = meta.get("arms", "").split()
|
||||
arms_present = [a for a in REQUIRED_ARMS if (g / "archive" / a).is_dir()]
|
||||
summaries = {a: ca.summarize(g / "archive" / a) for a in arms_present}
|
||||
session = gate3_session(out, g)
|
||||
identity = {"monolith_repeats": 0, "monolith_proven": 0, "split_repeats": 0, "split_proofs_passed": 0}
|
||||
adjudicated, adjudication_problems = read_adjudication(g / "adjudication.tsv")
|
||||
|
||||
# ---- the contract's constants (ruling: integrator 2026-09-23 after codex closeout review) ----
|
||||
contract = {"repeats": [], "arms": [], "cases": []}
|
||||
if repeat < CONTRACT_REPEATS:
|
||||
contract["repeats"].append("gate3/arms.txt repeat=%s < CONTRACT-P7 7.2's %d: 30-gate3.sh froze that count "
|
||||
"with the case list, a resume keeps it" % (meta.get("repeat", "?"), CONTRACT_REPEATS))
|
||||
for arm in REQUIRED_ARMS:
|
||||
if arm not in arms_present and arm not in planned:
|
||||
contract["arms"].append("arm %s has no archive and gate3/arms.txt's arms=[%s] does not name it: the run "
|
||||
"was started without it; CONTRACT-P7 7.2 needs %s on every case"
|
||||
% (arm, " ".join(planned), " + ".join(REQUIRED_ARMS)))
|
||||
canonical, canonical_problems = canonical_gate3_cases()
|
||||
missing = []
|
||||
if canonical is None or canonical_problems:
|
||||
contract["cases"].append("no canonical denominator to hold gate3/cases.txt to: " + "; ".join(canonical_problems))
|
||||
else:
|
||||
outside = [c for c in cases if c not in set(canonical)]
|
||||
missing = [c for c in canonical if c not in set(cases)]
|
||||
if outside:
|
||||
contract["cases"].append("%d case(s) outside CONTRACT-P7 7.1's denominator (%d CI split DirectVulkan - "
|
||||
"the %d exclusions): %s%s" % (
|
||||
len(outside), CI_SPLIT_DIRECTVULKAN, len(CONTRACT_EXCLUDED), ", ".join(outside),
|
||||
"; canonical case(s) missing: " + ", ".join(missing) if missing else ""))
|
||||
twice = sorted(c for c, n in Counter(listed).items() if n > 1)
|
||||
if twice:
|
||||
contract["cases"].append("case(s) listed more than once in gate3/cases.txt: " + ", ".join(twice))
|
||||
short = []
|
||||
|
||||
def rows(arm, case):
|
||||
s = summaries.get(arm)
|
||||
@@ -205,6 +340,9 @@ def gate3(out, ca):
|
||||
p = Path(row["dir"]) / "logcat.txt"
|
||||
return sha(p) if p.is_file() else None
|
||||
|
||||
def picture(row):
|
||||
return str(Path(row["dir"]) / row["actual"]) if row and row.get("actual") else None
|
||||
|
||||
def ra0_proof(row):
|
||||
p = Path(row["dir"]) / "mobilegl.log"
|
||||
if not p.is_file():
|
||||
@@ -214,18 +352,21 @@ def gate3(out, ca):
|
||||
return "run-ahead=0"
|
||||
return "NOT-PROVEN(" + ";".join(lines)[:120] + ")"
|
||||
|
||||
def not_run(arm):
|
||||
return "%s: not run%s" % (arm, "" if arm in planned else " (and gate3/arms.txt does not name the arm)")
|
||||
|
||||
results = []
|
||||
for case in cases:
|
||||
reasons, incomplete = [], []
|
||||
entry = {"case": case, "arms": {}}
|
||||
mono = rows("monolith", case)
|
||||
mono_ssim = mono_sha = None
|
||||
mono_row = mono_ssim = mono_sha = None
|
||||
if "monolith" in summaries and not finished("monolith", case):
|
||||
incomplete.append("monolith: no state .done (pair unfinished; 30-gate3.sh retries it)")
|
||||
if len(mono) != 1 or mono[0].get("error"):
|
||||
incomplete.append("monolith: %s" % (mono[0].get("error") if mono else "not run"))
|
||||
incomplete.append("monolith: %s" % mono[0].get("error") if mono else not_run("monolith"))
|
||||
else:
|
||||
m = mono[0]
|
||||
m = mono_row = mono[0]
|
||||
mono_ssim, mono_sha = m.get("ssim_vs_golden"), m.get("actual_sha256")
|
||||
not_mono = monolith_identity(m)
|
||||
identity["monolith_repeats"] += 1
|
||||
@@ -238,18 +379,23 @@ def gate3(out, ca):
|
||||
reasons.append("monolith control did not pass (ssim %s)" % mono_ssim)
|
||||
if case == OPENRA and not (m.get("mismatch_pixels_vs_golden") == 0 and (mono_ssim or 0) >= 0.9999995):
|
||||
reasons.append("OpenRA monolith not 1.000000/0 (%s/%s)" % (mono_ssim, m.get("mismatch_pixels_vs_golden")))
|
||||
split_rows = {}
|
||||
for arm in SPLIT_ARMS:
|
||||
rs = rows(arm, case)
|
||||
rs = split_rows[arm] = rows(arm, case)
|
||||
a = {"repeats": len(rs), "ssim": [r.get("ssim_vs_golden") for r in rs],
|
||||
"sha": sorted({r.get("actual_sha256") for r in rs if r.get("actual_sha256")}), "proofs_passed": 0}
|
||||
entry["arms"][arm] = a
|
||||
if arm not in summaries:
|
||||
incomplete.append(arm + ": not run")
|
||||
incomplete.append(not_run(arm))
|
||||
continue
|
||||
if not finished(arm, case):
|
||||
incomplete.append("%s: no state .done (pair unfinished; 30-gate3.sh retries it)" % arm)
|
||||
if len(rs) != repeat:
|
||||
incomplete.append("%s: %d/%d repeats" % (arm, len(rs), repeat))
|
||||
if len(rs) < CONTRACT_REPEATS and finished(arm, case):
|
||||
# The pair is final (30-gate3.sh wrote its .done): a resume will not add a repeat.
|
||||
reasons.append("%s: %d repeat(s) in a finished pair < CONTRACT-P7 7.2's %d" % (arm, len(rs), CONTRACT_REPEATS))
|
||||
short.append("%s/%s (%d)" % (arm, case, len(rs)))
|
||||
for r in rs:
|
||||
if r.get("error"):
|
||||
incomplete.append("%s rep%d: %s" % (arm, r["repeat"], r["error"]))
|
||||
@@ -276,54 +422,109 @@ def gate3(out, ca):
|
||||
for i in range(1, len(lc)):
|
||||
if lc[i] and lc[i] == lc[i - 1]:
|
||||
reasons.append("%s rep%d STALE (logcat identical to rep%d)" % (arm, rs[i]["repeat"], rs[i - 1]["repeat"]))
|
||||
# The RUN_AHEAD=0 control arm: required on every case (CONTRACT-P7 7.2 "RUN_AHEAD=0 对照臂一遍
|
||||
# 记录"), and it must be that arm; its picture is recorded, never gating.
|
||||
ra0 = rows("inproc-ra0", case)
|
||||
if ra0:
|
||||
if "inproc-ra0" in summaries and not finished("inproc-ra0", case):
|
||||
incomplete.append("inproc-ra0: no state .done (pair unfinished; 30-gate3.sh retries it)")
|
||||
if not ra0:
|
||||
incomplete.append(not_run("inproc-ra0"))
|
||||
else:
|
||||
r = ra0[0]
|
||||
entry["arms"]["inproc-ra0"] = {"ssim": r.get("ssim_vs_golden"), "sha": r.get("actual_sha256"),
|
||||
"passed": r.get("passed"), "proof": ra0_proof(r), "error": r.get("error")}
|
||||
# Cross-arm identity: reported, never gating (ruling d).
|
||||
if r.get("error"):
|
||||
incomplete.append("inproc-ra0 rep%d: %s" % (r["repeat"], r["error"]))
|
||||
elif entry["arms"]["inproc-ra0"]["proof"] != "run-ahead=0":
|
||||
reasons.append("inproc-ra0 is not proven the RUN_AHEAD=0 arm (%s)" % entry["arms"]["inproc-ra0"]["proof"])
|
||||
# Cross-arm identity: not a per-case gate (ruling d), but every mismatch needs an adjudication.
|
||||
ip = set(entry["arms"].get("inproc", {}).get("sha", []))
|
||||
sp = set(entry["arms"].get("spawn", {}).get("sha", []))
|
||||
entry["inproc_eq_spawn"] = bool(ip) and ip == sp
|
||||
entry["split_eq_monolith"] = bool(ip) and ip == sp == {mono_sha}
|
||||
entry["cross_arm"] = {"monolith": mono_sha, "inproc": sorted(ip), "spawn": sorted(sp),
|
||||
"identical": (entry["split_eq_monolith"] if (ip and sp and mono_sha) else None)}
|
||||
x = entry["cross_arm"] = {"monolith": mono_sha, "inproc": sorted(ip), "spawn": sorted(sp),
|
||||
"identical": (entry["split_eq_monolith"] if (ip and sp and mono_sha) else None)}
|
||||
if x["identical"] is False:
|
||||
first = {arm: picture(split_rows[arm][0]) if split_rows[arm] else None for arm in SPLIT_ARMS}
|
||||
x["px_vs_golden"] = {"monolith": mono_row.get("mismatch_pixels_vs_golden") if mono_row else None,
|
||||
"inproc": [r.get("mismatch_pixels_vs_golden") for r in split_rows["inproc"]],
|
||||
"spawn": [r.get("mismatch_pixels_vs_golden") for r in split_rows["spawn"]]}
|
||||
x["px_direct"] = {"inproc-monolith": px_between(ca, first["inproc"], picture(mono_row)),
|
||||
"spawn-monolith": px_between(ca, first["spawn"], picture(mono_row)),
|
||||
"inproc-spawn": px_between(ca, first["inproc"], first["spawn"])}
|
||||
x["adjudication"] = adjudicated.get(case)
|
||||
entry["verdict"] = "INCOMPLETE" if incomplete else ("FAIL" if reasons else "PASS")
|
||||
entry["reasons"] = incomplete + reasons
|
||||
results.append(entry)
|
||||
if short:
|
||||
contract["repeats"].append("%d finished inproc/spawn pair(s) archived fewer than CONTRACT-P7 7.2's %d repeats "
|
||||
"(arms.txt repeat=%s): %s%s" % (len(short), CONTRACT_REPEATS, meta.get("repeat", "?"),
|
||||
", ".join(short[:4]), ", ..." if len(short) > 4 else ""))
|
||||
mismatched = [e["case"] for e in results if e["cross_arm"]["identical"] is False]
|
||||
unadjudicated = [c for c in mismatched if c not in adjudicated]
|
||||
stale = sorted(c for c in adjudicated if c not in set(mismatched))
|
||||
verdicts = [e["verdict"] for e in results]
|
||||
# A reading that is not the gate's own (CONTRACT-P7 7.2: the 36-case denominator, one
|
||||
# reboot-clean session) may pass, but never as PASS: exit 0 is reserved for the gate verdict.
|
||||
# A reading that is not the gate's own (CONTRACT-P7 7.2: the canonical 36, one reboot-clean
|
||||
# session, every cross-arm mismatch adjudicated) may pass, but never as PASS: exit 0 is
|
||||
# reserved for the gate verdict.
|
||||
not_a_gate = []
|
||||
if len(cases) != GATE3_DENOMINATOR:
|
||||
not_a_gate.append("denominator is %d, not the gate's %d: a subset run is NOT a gate verdict"
|
||||
% (len(cases), GATE3_DENOMINATOR))
|
||||
if missing and not contract["cases"]:
|
||||
not_a_gate.append("%d of the canonical %d case(s) (missing %d, e.g. %s): a subset run is NOT a gate verdict"
|
||||
% (len(cases), GATE3_DENOMINATOR, len(missing), missing[0]))
|
||||
if not session["reboot_clean"].startswith("reboot-clean:"):
|
||||
not_a_gate.append("session not reboot-clean: NOT a gate verdict")
|
||||
if unadjudicated:
|
||||
not_a_gate.append("%d cross-arm mismatch(es) without a gate3/adjudication.tsv line (%s): %s%s" % (
|
||||
len(unadjudicated), CODEX_RULING, ", ".join(unadjudicated[:3]), ", ..." if len(unadjudicated) > 3 else ""))
|
||||
if not session["valid"]:
|
||||
overall = "INVALID-SESSION"
|
||||
elif contract["repeats"]:
|
||||
overall = "FAIL-REPEATS"
|
||||
elif contract["arms"]:
|
||||
overall = "FAIL-ARMS"
|
||||
elif contract["cases"]:
|
||||
overall = "FAIL-CASESET"
|
||||
elif verdicts and all(v == "PASS" for v in verdicts):
|
||||
overall = "PASS"
|
||||
if len(cases) != GATE3_DENOMINATOR:
|
||||
if missing:
|
||||
overall = "PASS-SUBSET"
|
||||
elif not_a_gate:
|
||||
elif not session["reboot_clean"].startswith("reboot-clean:"):
|
||||
overall = "PASS-NOT-REBOOT-CLEAN"
|
||||
elif unadjudicated:
|
||||
overall = "PASS-NEEDS-ADJUDICATION"
|
||||
elif "FAIL" in verdicts:
|
||||
overall = "FAIL"
|
||||
else:
|
||||
overall = "INCOMPLETE"
|
||||
return {"cases": results, "excluded": excluded, "repeat": repeat, "denominator": len(cases),
|
||||
"overall": overall, "not_a_gate": not_a_gate, "arms_present": arms_present,
|
||||
"overall": overall, "not_a_gate": not_a_gate, "arms_present": arms_present, "arms_planned": planned,
|
||||
"contract": contract, "canonical": canonical, "canonical_problems": canonical_problems,
|
||||
"cross_arm_mismatched": mismatched, "cross_arm_unadjudicated": unadjudicated,
|
||||
"adjudication": {"file": str(g / "adjudication.tsv"), "cases": adjudicated,
|
||||
"problems": adjudication_problems, "stale": stale},
|
||||
"session": session, "identity": identity}
|
||||
|
||||
|
||||
def print_gate3(g3):
|
||||
print("== GATE 3 (DirectVulkan x pbuffer; split repeat=%d; tol |ssim-monolith| <= %.4f) ==" % (g3["repeat"], TOL))
|
||||
print("== GATE 3 (DirectVulkan x pbuffer; split repeat=%d, contract >= %d; tol |ssim-monolith| <= %.4f) =="
|
||||
% (g3["repeat"], CONTRACT_REPEATS, TOL))
|
||||
s = g3["session"]
|
||||
print("-- session: boot_id %s; %d record(s) stamped; %s" % (s["session_boot_id"] or "?", s["records"],
|
||||
s["reboot_clean"] or "no session/reboot-clean.txt"))
|
||||
for problem in s["problems"]:
|
||||
print(" ! INVALID-SESSION: " + problem)
|
||||
c = g3["contract"]
|
||||
held = not any(c.values())
|
||||
print("-- contract (CONTRACT-P7 7.1/7.2; %s): inproc/spawn >= %d repeats per case, arms %s on every case, "
|
||||
"case set == the canonical %d (%d CI split DirectVulkan - %d exclusions): %s"
|
||||
% (CODEX_RULING, CONTRACT_REPEATS, "+".join(REQUIRED_ARMS), GATE3_DENOMINATOR, CI_SPLIT_DIRECTVULKAN,
|
||||
len(CONTRACT_EXCLUDED), "held" if held else "VIOLATED"))
|
||||
for key, label in (("repeats", "FAIL-REPEATS"), ("arms", "FAIL-ARMS"), ("cases", "FAIL-CASESET")):
|
||||
for problem in c[key]:
|
||||
print(" ! %s: %s" % (label, problem))
|
||||
if sorted(g3["excluded"]) != sorted(CONTRACT_EXCLUDED):
|
||||
print("WARN gate3/excluded.txt [%s] is not CONTRACT-P7 7.1's [%s] (the case set above is held to the contract's)"
|
||||
% (" ".join(g3["excluded"]), " ".join(CONTRACT_EXCLUDED)))
|
||||
w = max([len(e["case"]) for e in g3["cases"]] + [4])
|
||||
print("%-*s %-10s %-11s %-24s %-24s %-11s %s" % (w, "case", "verdict", "monolith", "inproc(min..max) #sha", "spawn(min..max) #sha", "ra0", "i==s==m"))
|
||||
for e in g3["cases"]:
|
||||
@@ -344,22 +545,42 @@ def print_gate3(g3):
|
||||
for reason in e["reasons"]:
|
||||
print("%-*s ! %s" % (w, "", reason))
|
||||
ra0s = [(e["case"], e["arms"]["inproc-ra0"]) for e in g3["cases"] if "inproc-ra0" in e["arms"]]
|
||||
if ra0s:
|
||||
print("-- RUN_AHEAD=0 control (recorded, not gating): %d case(s); proofs: %s" % (
|
||||
len(ra0s), ", ".join(sorted({r["proof"] for _, r in ra0s}))))
|
||||
print("-- RUN_AHEAD=0 control (required on every case, picture recorded, not gating): %d/%d case(s); proofs: %s" % (
|
||||
len(ra0s), len(g3["cases"]), ", ".join(sorted({r["proof"] for _, r in ra0s})) or "-"))
|
||||
i = g3["identity"]
|
||||
print("-- arm identity: monolith %d/%d repeat(s) with no Config: IPC / MOBILEGL_TRANSPORT line; "
|
||||
"inproc+spawn transport-proof passed %d/%d" % (i["monolith_proven"], i["monolith_repeats"],
|
||||
i["split_proofs_passed"], i["split_repeats"]))
|
||||
judged = [e for e in g3["cases"] if e["cross_arm"]["identical"] is not None]
|
||||
print("-- cross-arm picture identity (reported, NOT a gate; %s): %d/%d case(s) inproc==spawn==monolith" % (
|
||||
RULING, sum(e["cross_arm"]["identical"] for e in judged), len(judged)))
|
||||
adj = g3["adjudication"]
|
||||
print("-- cross-arm picture identity (NOT a per-case gate, %s; every mismatch needs a gate3/adjudication.tsv "
|
||||
"line, %s): %d/%d case(s) inproc==spawn==monolith; %d mismatch(es), %d unadjudicated" % (
|
||||
RULING, CODEX_RULING, sum(e["cross_arm"]["identical"] for e in judged), len(judged),
|
||||
len(g3["cross_arm_mismatched"]), len(g3["cross_arm_unadjudicated"])))
|
||||
|
||||
def px(v):
|
||||
if isinstance(v, list):
|
||||
nums = [n for n in v if isinstance(n, int)]
|
||||
return "%d..%d" % (min(nums), max(nums)) if nums else "-"
|
||||
return "-" if v is None else str(v)
|
||||
for e in judged:
|
||||
if not e["cross_arm"]["identical"]:
|
||||
x = e["cross_arm"]
|
||||
g, d = x["px_vs_golden"], x["px_direct"]
|
||||
print("WARN cross-arm %s: monolith %s | inproc %s | spawn %s" % (
|
||||
e["case"], (x["monolith"] or "-")[:12], ",".join(v[:12] for v in x["inproc"]) or "-",
|
||||
",".join(v[:12] for v in x["spawn"]) or "-"))
|
||||
print(" px!=golden (archived mismatchPixels) monolith %s, inproc %s, spawn %s; px between the arms' "
|
||||
"first repeats: inproc-monolith %s, spawn-monolith %s, inproc-spawn %s" % (
|
||||
px(g["monolith"]), px(g["inproc"]), px(g["spawn"]), px(d["inproc-monolith"]),
|
||||
px(d["spawn-monolith"]), px(d["inproc-spawn"])))
|
||||
print(" " + ("adjudicated: " + x["adjudication"] if x.get("adjudication")
|
||||
else "UNADJUDICATED: no gate3/adjudication.tsv line"))
|
||||
for problem in adj["problems"]:
|
||||
print("WARN " + problem)
|
||||
if adj["stale"]:
|
||||
print(" note: gate3/adjudication.tsv names case(s) whose arms agree (nothing to adjudicate): "
|
||||
+ ", ".join(adj["stale"]))
|
||||
print("-- excluded (red on monolith, CONTRACT-P7 7.1): " + (", ".join(g3["excluded"]) or "none"))
|
||||
n = len(g3["cases"])
|
||||
npass = sum(e["verdict"] == "PASS" for e in g3["cases"])
|
||||
@@ -407,16 +628,20 @@ def read_list(path):
|
||||
return [l.strip() for l in path.read_text(encoding="utf-8").splitlines() if l.strip() and not l.startswith("#")]
|
||||
|
||||
|
||||
def rate(statuses):
|
||||
"""CONTRACT-P7 7.3: rate = Pass / (Pass + Fail). NotSupported, the warnings (Compatibility /
|
||||
Quality) and the crashes stay out of the denominator; crashes get their own count. The
|
||||
pre-ruling Pass / (results - NotSupported) is kept as `results_rate`, information only."""
|
||||
def rate(statuses, lost=0):
|
||||
"""The gate's block rate = Pass / (Pass + Fail + lost), `lost` = the $BASE-Pass cases this side
|
||||
turned NotSupported or another non-Pass, non-Fail status (a warning, a crash): a lost supported
|
||||
case is NON-PASS, it does not leave the denominator (ruling: integrator 2026-09-23 after codex
|
||||
closeout review). Other NotSupported / warnings / crashes stay out of the denominator; crashes
|
||||
get their own count. Information only: CONTRACT-P7 7.3's literal Pass / (Pass + Fail) as
|
||||
`literal_rate` and the pre-ruling Pass / (results - NotSupported) as `results_rate`."""
|
||||
c = Counter(statuses)
|
||||
p, f, ns = c["Pass"], c["Fail"], c["NotSupported"]
|
||||
x = sum(c[s] for s in HARD)
|
||||
n = len(statuses)
|
||||
return {"pass": p, "fail": f, "ns": ns, "crash": x, "warn": n - p - f - ns - x, "results": n,
|
||||
"rate": (p / (p + f)) if (p + f) else None,
|
||||
return {"pass": p, "fail": f, "ns": ns, "crash": x, "warn": n - p - f - ns - x, "results": n, "lost": lost,
|
||||
"rate": (p / (p + f + lost)) if (p + f + lost) else None,
|
||||
"literal_rate": (p / (p + f)) if (p + f) else None,
|
||||
"results_rate": (p / (n - ns)) if (n - ns) else None}
|
||||
|
||||
|
||||
@@ -477,7 +702,8 @@ def block_scope(run, after):
|
||||
return universe, src, [], incomplete
|
||||
|
||||
|
||||
def cts_block(c, b, base_dir, canon_dir):
|
||||
def cts_block(c, b, base_dir, canon_dir, adjudicated=None):
|
||||
adjudicated = adjudicated or {}
|
||||
run = c / "runs" / b
|
||||
if not c.is_dir():
|
||||
return {"block": b, "verdict": "MISSING", "reasons": ["no cts/ directory: 50-cts-after.sh did not run"]}
|
||||
@@ -496,48 +722,63 @@ def cts_block(c, b, base_dir, canon_dir):
|
||||
hung = read_list(run / "hung.txt")
|
||||
# Like for like: both rates over the cases that have a BASE and an AFTER result.
|
||||
both = [x for x in universe if x in after and x in base]
|
||||
ra, rb = rate([after[x] for x in both]), rate([base[x] for x in both])
|
||||
delta = None if ra["rate"] is None or rb["rate"] is None else (ra["rate"] - rb["rate"]) * 100.0
|
||||
delta_results = (None if ra["results_rate"] is None or rb["results_rate"] is None
|
||||
else (ra["results_rate"] - rb["results_rate"]) * 100.0)
|
||||
# A $BASE Pass that AFTER turned NotSupported / a warning / a crash: NON-PASS in the gate rate,
|
||||
# listed, and FAIL unless adjudicated (ruling: integrator 2026-09-23 after codex closeout review).
|
||||
lost = sorted(x for x in both if base[x] == "Pass" and after[x] not in ("Pass", "Fail"))
|
||||
ra, rb = rate([after[x] for x in both], lost=len(lost)), rate([base[x] for x in both])
|
||||
|
||||
def pp(key):
|
||||
return None if ra[key] is None or rb[key] is None else (ra[key] - rb[key]) * 100.0
|
||||
delta, delta_literal, delta_results = pp("rate"), pp("literal_rate"), pp("results_rate")
|
||||
new_crash = sorted(x for x in universe if after.get(x) in HARD and base.get(x) not in HARD)
|
||||
new_crash += sorted(x for x in hung if x not in new_crash)
|
||||
regress = sorted(x for x in universe if base.get(x) == "Pass" and after.get(x) not in (None, "Pass"))
|
||||
fixed = sorted(x for x in universe if base.get(x) not in (None, "Pass") and after.get(x) == "Pass")
|
||||
missing_base = sorted(x for x in universe if x not in base)
|
||||
unadjudicated = [x for x in lost if x not in adjudicated]
|
||||
fail, notes, warnings = [], [], []
|
||||
if new_crash:
|
||||
fail.append("%d NEW crash(es)" % len(new_crash))
|
||||
if rb["rate"] is None:
|
||||
incomplete.append("$BASE has no Pass/Fail over this block's %d case(s): no contract rate to compare against"
|
||||
% len(both))
|
||||
elif ra["rate"] is None:
|
||||
fail.append("AFTER has no Pass/Fail result (NS %d, W %d, X %d of %d): the contract rate cannot be formed "
|
||||
"where $BASE's is %.3f%% - every $BASE Pass is lost" % (
|
||||
ra["ns"], ra["warn"], ra["crash"], ra["results"], 100 * rb["rate"]))
|
||||
elif ra["literal_rate"] is None:
|
||||
fail.append("AFTER has no Pass/Fail result (NS %d, W %d, X %d of %d): the literal Pass/(Pass+Fail) cannot be "
|
||||
"formed where $BASE's is %.3f%%; %d $BASE Pass(es) lost" % (
|
||||
ra["ns"], ra["warn"], ra["crash"], ra["results"], 100 * rb["rate"], len(lost)))
|
||||
if delta is not None and delta < -CTS_TOL_PP:
|
||||
fail.append("delta %.3f pp < -%.1f pp (Pass/(Pass+Fail))" % (delta, CTS_TOL_PP))
|
||||
fail.append("delta %.3f pp < -%.1f pp (Pass/(Pass+Fail+L))" % (delta, CTS_TOL_PP))
|
||||
if delta is not None and delta > CTS_TOL_PP:
|
||||
notes.append("note: AFTER is %.3f pp ABOVE BASE (not a regression; review the fixed list)" % delta)
|
||||
if unadjudicated:
|
||||
fail.append("%d $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status with no "
|
||||
"cts/adjudication.tsv line (%s)" % (len(unadjudicated), CODEX_RULING))
|
||||
elif lost:
|
||||
notes.append("note: %d $BASE-Pass case(s) lost to NotSupported / another non-Pass, non-Fail status, each "
|
||||
"adjudicated in cts/adjudication.tsv (still NON-PASS in the rate)" % len(lost))
|
||||
if missing_base:
|
||||
warnings.append("%d case(s) have no $BASE result (e.g. %s): left out of both rates; a crash among them "
|
||||
"still counts as NEW" % (len(missing_base), missing_base[0]))
|
||||
if ra["ns"] != rb["ns"]:
|
||||
warnings.append("NotSupported $BASE %d -> AFTER %d (not in the contract rate; a Pass that became NS is in "
|
||||
"the Pass->not list)" % (rb["ns"], ra["ns"]))
|
||||
warnings.append("NotSupported $BASE %d -> AFTER %d (a $BASE Pass among them is an L case: NON-PASS in the "
|
||||
"rate, listed below)" % (rb["ns"], ra["ns"]))
|
||||
canonical = canon_dir / ("p7-%s-gl46.txt" % b)
|
||||
subset = set(read_list(src)) != set(read_list(canonical))
|
||||
verdict = "FAIL" if fail else ("INCOMPLETE" if incomplete else "PASS")
|
||||
return {"block": b, "cases": len(universe), "compared": len(both), "caselist": str(src),
|
||||
"after": ra, "base": rb, "delta_pp": delta, "delta_pp_results_info": delta_results,
|
||||
"after": ra, "base": rb, "delta_pp": delta, "delta_pp_literal_info": delta_literal,
|
||||
"delta_pp_results_info": delta_results,
|
||||
"new_crash": new_crash, "regressed": regress, "fixed": fixed, "missing_in_base": missing_base,
|
||||
"lost": [{"case": x, "after": after[x], "adjudication": adjudicated.get(x)} for x in lost],
|
||||
"lost_unadjudicated": unadjudicated,
|
||||
"unrun": len(unrun), "hung": hung, "verdict": verdict, "reasons": incomplete + fail + notes,
|
||||
"warnings": warnings, "subset": subset}
|
||||
|
||||
|
||||
def cts(out, base_dir, canon_dir):
|
||||
c = out / "cts"
|
||||
blocks = [cts_block(c, b, base_dir, canon_dir) for b in CTS_BLOCKS]
|
||||
adjudicated, adjudication_problems = read_adjudication(c / "adjudication.tsv")
|
||||
blocks = [cts_block(c, b, base_dir, canon_dir, adjudicated) for b in CTS_BLOCKS]
|
||||
extra = sorted(p.stem[len("report-"):] for p in c.glob("report-*.json")
|
||||
if p.stem[len("report-"):] not in CTS_BLOCKS) if c.is_dir() else []
|
||||
verdicts = [b["verdict"] for b in blocks]
|
||||
@@ -548,17 +789,23 @@ def cts(out, base_dir, canon_dir):
|
||||
overall = "PASS-SUBSET" if any(b.get("subset") for b in blocks) else "PASS"
|
||||
else:
|
||||
overall = "INCOMPLETE"
|
||||
lost = {x["case"] for b in blocks for x in b.get("lost", [])}
|
||||
return {"blocks": blocks, "overall": overall, "ubo": dict(UBO_BLOCK), "ignored_reports": extra,
|
||||
"incomplete_blocks": [b["block"] for b in blocks if b["verdict"] not in ("PASS", "FAIL")]}
|
||||
"incomplete_blocks": [b["block"] for b in blocks if b["verdict"] not in ("PASS", "FAIL")],
|
||||
"adjudication": {"file": str(c / "adjudication.tsv"), "cases": adjudicated,
|
||||
"problems": adjudication_problems, "stale": sorted(x for x in adjudicated if x not in lost)}}
|
||||
|
||||
|
||||
def print_cts(c):
|
||||
print("== CTS AFTER (inproc x DirectVulkan) vs BASE (monolith x DirectVulkan, p7w1) ==")
|
||||
print("gate (CONTRACT-P7 7.3): rate = Pass/(Pass+Fail) [NS, warnings W and crashes X are not in the denominator];")
|
||||
print(" per block delta >= -%.1f pp and 0 NEW crash; all five blocks required. 'info' = the pre-ruling" % CTS_TOL_PP)
|
||||
print(" Pass/(results-NS) delta (its rates on each block's 'info' line), for comparison only, never gating.")
|
||||
print("%-14s %6s %-20s %8s %-20s %8s %9s %9s %5s %4s %5s %s" % (
|
||||
"block", "cases", "BASE P/F/NS/W/X", "rate", "AFTER P/F/NS/W/X", "rate", "delta pp", "(info)",
|
||||
print("gate (CONTRACT-P7 7.3 + %s):" % CODEX_RULING)
|
||||
print(" rate = Pass/(Pass+Fail+L), L = the $BASE-Pass cases AFTER turned NotSupported or another non-Pass,")
|
||||
print(" non-Fail status (a lost supported case is NON-PASS); other NS, warnings W and crashes X are not in the")
|
||||
print(" denominator. Per block: delta >= -%.1f pp, 0 NEW crash, and every L case adjudicated in" % CTS_TOL_PP)
|
||||
print(" cts/adjudication.tsv (else FAIL); all five blocks required. '(literal)' = the contract's literal")
|
||||
print(" Pass/(Pass+Fail) delta and 'info' the pre-ruling Pass/(results-NS): printed for comparison, never gating.")
|
||||
print("%-14s %6s %-20s %8s %-20s %4s %8s %9s %9s %5s %4s %5s %s" % (
|
||||
"block", "cases", "BASE P/F/NS/W/X", "rate", "AFTER P/F/NS/W/X", "L", "rate", "delta pp", "(literal)",
|
||||
"crash", "regr", "fixed", "verdict"))
|
||||
|
||||
def counts(r):
|
||||
@@ -573,11 +820,13 @@ def print_cts(c):
|
||||
if "after" not in b:
|
||||
print("%-14s %s: %s" % (b["block"], b["verdict"], "; ".join(b["reasons"])))
|
||||
continue
|
||||
print("%-14s %6d %-20s %8s %-20s %8s %9s %9s %5d %4d %5d %s%s" % (
|
||||
print("%-14s %6d %-20s %8s %-20s %4d %8s %9s %9s %5d %4d %5d %s%s" % (
|
||||
b["block"], b["cases"], counts(b["base"]), pct(b["base"]["rate"]), counts(b["after"]),
|
||||
pct(b["after"]["rate"]), pp(b["delta_pp"]), "(%s)" % pp(b["delta_pp_results_info"]),
|
||||
b["after"]["lost"], pct(b["after"]["rate"]), pp(b["delta_pp"]), "(%s)" % pp(b["delta_pp_literal_info"]),
|
||||
len(b["new_crash"]), len(b["regressed"]), len(b["fixed"]), b["verdict"],
|
||||
" (SUBSET)" if b["subset"] else ""))
|
||||
print(" info literal Pass/(Pass+Fail): BASE %s AFTER %s delta %s pp (not gating)" % (
|
||||
pct(b["base"]["literal_rate"]), pct(b["after"]["literal_rate"]), pp(b["delta_pp_literal_info"])))
|
||||
print(" info pre-ruling Pass/(results-NS): BASE %s AFTER %s delta %s pp (not gating)" % (
|
||||
pct(b["base"]["results_rate"]), pct(b["after"]["results_rate"]), pp(b["delta_pp_results_info"])))
|
||||
for reason in b["reasons"]:
|
||||
@@ -586,16 +835,26 @@ def print_cts(c):
|
||||
print("WARN %s: %s" % (b["block"], warning))
|
||||
for x in b["new_crash"]:
|
||||
print(" NEW CRASH " + x)
|
||||
for x in b["regressed"][:40]:
|
||||
for x in b["lost"]:
|
||||
print(" LOST %s (Pass -> %s) %s" % (x["case"], x["after"], "adjudicated: " + x["adjudication"]
|
||||
if x["adjudication"] else "UNADJUDICATED"))
|
||||
lost = {x["case"] for x in b["lost"]}
|
||||
others = [x for x in b["regressed"] if x not in lost]
|
||||
for x in others[:40]:
|
||||
print(" Pass->not " + x)
|
||||
if len(b["regressed"]) > 40:
|
||||
print(" ... %d more regressions in the JSON" % (len(b["regressed"]) - 40))
|
||||
if len(others) > 40:
|
||||
print(" ... %d more regressions in the JSON" % (len(others) - 40))
|
||||
for x in b["fixed"][:20]:
|
||||
print(" not->Pass " + x)
|
||||
u = c["ubo"]
|
||||
print("%-14s %s (%s)" % (u["block"], u["reasons"][0], u["cases"]))
|
||||
if c["ignored_reports"]:
|
||||
print(" note: report(s) outside the five gate blocks ignored: " + ", ".join(c["ignored_reports"]))
|
||||
for problem in c["adjudication"]["problems"]:
|
||||
print("WARN " + problem)
|
||||
if c["adjudication"]["stale"]:
|
||||
print(" note: cts/adjudication.tsv names case(s) that are no $BASE-Pass loss (nothing to adjudicate): "
|
||||
+ ", ".join(c["adjudication"]["stale"]))
|
||||
tail = ""
|
||||
if c["incomplete_blocks"]:
|
||||
tail += " (INCOMPLETE: %s)" % ", ".join("%s %s" % (b["block"], b["verdict"]) for b in c["blocks"]
|
||||
|
||||
Reference in New Issue
Block a user