swung0x48/MobileGL
1
0
Fork 0
mirror of https://github.com/MobileGL-Dev/MobileGL synced 2026-09-07 19:58:32 +09:00
Code Issues Packages Projects Releases Wiki Activity

[Build] (CI): add the pipe-gen check, the stdio-instrumentation gate and the citation lint

Browse Source 
- Three P0 gates from plan B section 11, in one job that needs no build: a broken build
  must not be able to hide a drifted interface.
- pipe-gen-check regenerates G1-G7 and runs git diff --exit-code over
  MobileGL/MG_Pipe/generated. Since all seven generators read the same .def files, this is
  what makes drift between them impossible rather than merely unlikely.
- The stdio gate refuses fprintf(stderr and printf( under MG_Backend and MG_State. Nothing
  there matches today, so it lands with NO whitelist - verified with a negative control
  that fprintf(stderr and std::printf both trip it while snprintf does not. Per-draw
  instrumentation has been committed by accident before, once inside a mutex critical
  section, and MGLOG_D is the channel these trees are allowed to use because it compiles
  out in INFO builds.
- scripts/gen_pipe_dirty_surface.py reports the frontend mutation surface corollary 4
  needs covered: 926 mutator calls under MG_Impl/GLImpl over 73 distinct mutators, of
  which only 92 sit in a function that also reaches the backend. The other 834 are
  published by the NEXT verb, which is precisely the population that needs an aggregate
  generation. Informational in P0; it becomes a gate in P1 when there is a mapping file to
  diff against.
- scripts/check_doc_citations.py resolves every `path:line` citation against a git
  revision. It reproduces the failure that motivated it - the plan's first draft cited
  SamplerObject.h:468-492 in a 160-line file - and reports 84 unresolved citations out of
  1028 in docs/Disaggregated today, which is why CI runs it warning-only until those
  documents settle. --strict exits 1, verified in both directions.
This commit is contained in:
swung0x48
2026-09-05 20:16:49 -04:00
parent 2f8d0f0d51
commit 9bbf71990c
3 changed files with 377 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/test.yml
+52
View File
@@ -778,3 +778,55 @@ jobs:
)
echo "Deleted ${deleted} intermediate Linux artifact(s); retained ${retained} failed-retrace fixture(s)."
pipe-gates:
name: MGPipe generators and hygiene gates
runs-on: ubuntu-latest
# Deliberately independent of build-linux: these are source-level gates, they take
# seconds, and a broken build must not hide a drifted interface.
steps:
- name: Checkout repo
uses: actions/checkout@v6
# The seven generators all read MG_Pipe/*.def, so regenerating and diffing is what
# keeps the two interface tables, the wire records, the verify comparators, the
# PipeInputs field ids, the read-inventory coverage and the render-state member list
# from drifting apart from the catalogue. The generated files are committed
# deliberately: the build must not depend on python.
- name: Regenerate the MGPipe interface (G1-G7)
run: |
python3 scripts/gen_pipe.py
git diff --exit-code -- MobileGL/MG_Pipe/generated
# Per-draw fprintf/printf instrumentation has repeatedly been committed by accident,
# once inside a mutex critical section. Nothing under these two trees prints to a
# stdio stream today - MGLOG_D compiles out in INFO builds and is the only channel
# they are allowed to use - so this gate starts with no exceptions, and any addition
# to it needs a reason in the pull request rather than a quiet whitelist entry.
- name: No stdio instrumentation in MG_Backend or MG_State
run: |
if grep -rnE 'fprintf[[:space:]]*\(stderr|(^|[^[:alnum:]_>.])printf[[:space:]]*\(' \
MobileGL/MG_Backend MobileGL/MG_State; then
echo "::error::stdio instrumentation found; use MGLOG_D (compiled out in INFO builds)"
exit 1
fi
echo "no fprintf(stderr / printf( under MobileGL/MG_Backend or MobileGL/MG_State"
# Informational: the frontend mutation surface an MGPipe aggregate generation has to
# cover. It becomes a gate in P1, when the mapping file exists to diff against.
- name: MGPipe dirty-surface report
run: python3 scripts/gen_pipe_dirty_surface.py --summary
# Warning only for now: the disaggregation documents are still being written, and a
# lint that fails a rewrite in progress teaches people to ignore it. It becomes
# --strict when the documents settle.
- name: Documentation citation lint
run: |
shopt -s nullglob
documents=(docs/Disaggregated/*.md)
if [ ${#documents[@]} -eq 0 ]; then
echo "no disaggregation documents to check"
exit 0
fi
python3 scripts/check_doc_citations.py "${documents[@]}" || true
scripts/check_doc_citations.py
+123
View File
@@ -0,0 +1,123 @@
#!/usr/bin/env python3
# MobileGL - scripts/check_doc_citations.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""Resolve every `path:line` citation in a set of markdown documents.
A design document that cites the code is only as good as its line numbers, and a wrong one
is worse than none: it sends the next reader to a function that does something else. The
disaggregation plan's first draft cited SamplerObject.h:468-492 for a struct that lives at
:72-96 in a 160-line file, and nothing caught it.
So every `File.h:123` and `File.cpp:123-456` in the given documents is resolved against a
git revision - the file must exist there and must have at least that many lines. Bare file
names are resolved by basename, which is how the plan spells most of its citations; an
ambiguous basename is reported rather than guessed.
python3 scripts/check_doc_citations.py docs/Disaggregated/*.md
python3 scripts/check_doc_citations.py --rev 81b17c0b --strict docs/Disaggregated/*.md
Exits non-zero only with --strict, so it can be wired into CI as a warning first.
"""
import argparse
import os
import re
import subprocess
import sys
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# `Managers.cpp:4340-4390`, `MobileGL/MG_State/.../RenderState.h:263`, `:12-13` is NOT
# matched on purpose: a citation with no file name cannot be checked, only guessed.
CITATION_RE = re.compile(
r"(?<![\w/.-])((?:[\w.-]+/)*[\w.-]+\.(?:h|hpp|cpp|cc|c|py|def|inc|md|json|yml|yaml|txt|fbs))"
r":(\d+)(?:-(\d+))?")
def git(args, rev_root=REPO_ROOT):
return subprocess.run(["git", "-C", rev_root] + args, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, check=False).stdout.decode("utf-8", "replace")
class Tree(object):
def __init__(self, rev):
self.Rev = rev
listing = git(["ls-tree", "-r", "--name-only", rev]).splitlines()
if not listing:
sys.exit("check_doc_citations: revision %s has no files (bad revision?)" % rev)
self.Paths = set(listing)
self.ByBasename = {}
for path in listing:
self.ByBasename.setdefault(os.path.basename(path), []).append(path)
self.LineCounts = {}
def Resolve(self, cited):
if cited in self.Paths:
return [cited]
candidates = self.ByBasename.get(os.path.basename(cited), [])
if len(candidates) > 1 and "/" in cited:
candidates = [p for p in candidates if p.endswith(cited)] or candidates
return candidates
def LineCount(self, path):
if path not in self.LineCounts:
blob = git(["show", "%s:%s" % (self.Rev, path)])
# A file with no trailing newline still has that last line.
count = blob.count("\n") + (1 if blob and not blob.endswith("\n") else 0)
self.LineCounts[path] = count
return self.LineCounts[path]
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("documents", nargs="+", help="markdown files to check")
parser.add_argument("--rev", default="HEAD", help="git revision the citations point into")
parser.add_argument("--strict", action="store_true", help="exit 1 when a citation does not resolve")
args = parser.parse_args()
tree = Tree(args.rev)
checked = 0
problems = []
for document in args.documents:
if not os.path.exists(document):
problems.append("%s: no such document" % document)
continue
with open(document, "r", encoding="utf-8", errors="replace") as handle:
lines = handle.read().splitlines()
for number, line in enumerate(lines, start=1):
for match in CITATION_RE.finditer(line):
cited, first, last = match.group(1), int(match.group(2)), match.group(3)
last = int(last) if last else first
checked += 1
where = "%s:%d: `%s`" % (document, number, match.group(0))
candidates = tree.Resolve(cited)
if not candidates:
problems.append("%s -> no such file at %s" % (where, args.rev))
continue
if len(candidates) > 1:
problems.append("%s -> ambiguous: %s" % (where, ", ".join(sorted(candidates))))
continue
if last < first:
problems.append("%s -> inverted line range" % where)
continue
count = tree.LineCount(candidates[0])
if last > count:
problems.append("%s -> %s has %d lines at %s"
% (where, candidates[0], count, args.rev))
print("check_doc_citations: %d citations in %d document(s) against %s, %d problem(s)"
% (checked, len(args.documents), args.rev, len(problems)))
for problem in problems:
print(" %s" % problem)
if problems and args.strict:
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/gen_pipe_dirty_surface.py
+202
View File
@@ -0,0 +1,202 @@
#!/usr/bin/env python3
# MobileGL - scripts/gen_pipe_dirty_surface.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""The dirty-surface scanner (plan B corollary 4, section 5.2).
MGPipe replaces "the backend rediscovers what changed" with "the frontend says what
changed", which only works if EVERY frontend mutation that a backend can observe bumps an
aggregate generation. The failure mode is silent and one-directional: a mutation that
forgets to bump renders stale, and no purity gate can see it.
So the mutation surface has to be enumerated mechanically rather than by memory. This
script reports every place in MG_Impl/GLImpl where a GL entry point BOTH mutates frontend
state through pGLContext AND reaches the backend in the same function - those are the
publish points, the ones that must map onto an aggregate generation.
P0 is the skeleton: it reports. P1 adds the mapping file and CI regenerates it with
`git diff --exit-code` and zero unmapped mutators, the same shape as gen_pipe.py's G6.
python3 scripts/gen_pipe_dirty_surface.py # human-readable report
python3 scripts/gen_pipe_dirty_surface.py --summary # counts only
"""
import argparse
import os
import re
import sys
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SCAN_ROOT = os.path.join(REPO_ROOT, "MobileGL", "MG_Impl", "GLImpl")
# The mutating half of GLContext's surface. Prefix-matched, per the plan's list.
MUTATOR_PREFIXES = ("Add", "Set", "Mark", "Bump", "Allocate", "Truncate", "Record", "Notify",
"Begin", "End")
MUTATOR_RE = re.compile(r"pGLContext->\s*((?:%s)\w*)\s*\(" % "|".join(MUTATOR_PREFIXES))
BACKEND_RE = re.compile(r"gBackendFunctionsTable\.GL\.(\w+)|pActiveBackendObject->\s*(\w+)")
FUNCTION_RE = re.compile(r"(?:^|\n)[ \t]*(?:[A-Za-z_][\w:<>,&*\s]*?)\b(\w+)\s*\([^;{}]*\)\s*"
r"(?:const\s*)?(?:noexcept\s*)?\{")
def mask_comments_and_strings(text):
"""Replace comment and string-literal bodies with spaces, keeping every offset and
newline, so the regexes below cannot match inside a comment or a literal."""
out = list(text)
i = 0
n = len(text)
while i < n:
c = text[i]
if c == "/" and i + 1 < n and text[i + 1] == "/":
while i < n and text[i] != "\n":
out[i] = " "
i += 1
elif c == "/" and i + 1 < n and text[i + 1] == "*":
out[i] = out[i + 1] = " "
i += 2
while i < n and not (text[i] == "*" and i + 1 < n and text[i + 1] == "/"):
if text[i] != "\n":
out[i] = " "
i += 1
if i < n:
out[i] = " "
if i + 1 < n:
out[i + 1] = " "
i += 2
elif c in "\"'":
quote = c
i += 1
while i < n and text[i] != quote:
if text[i] == "\\":
out[i] = " "
i += 1
if i < n and text[i] != "\n":
out[i] = " "
i += 1
if i < n:
out[i] = " "
i += 1
else:
i += 1
return "".join(out)
def function_bodies(masked):
"""Yield (name, start_offset, end_offset) for every braced function body."""
for match in FUNCTION_RE.finditer(masked):
name = match.group(1)
start = masked.index("{", match.end() - 1) if masked[match.end() - 1] != "{" else match.end() - 1
depth = 0
i = start
while i < len(masked):
if masked[i] == "{":
depth += 1
elif masked[i] == "}":
depth -= 1
if depth == 0:
yield name, start, i
break
i += 1
def line_of(text, offset):
return text.count("\n", 0, offset) + 1
def scan_file(path):
with open(path, "r", encoding="utf-8", errors="replace") as handle:
text = handle.read()
masked = mask_comments_and_strings(text)
findings = []
# Every mutator in the file, whether or not it shares a function with a backend call.
# The difference between this and the publish points below is the whole point of the
# report: a mutation that does NOT reach the backend in the same function is published
# by the NEXT verb, and it is exactly those that need an aggregate generation rather
# than an inline push.
all_mutators = [(m.group(1), line_of(masked, m.start())) for m in MUTATOR_RE.finditer(masked)]
for name, start, end in function_bodies(masked):
body = masked[start:end]
mutators = [(m.group(1), line_of(masked, start + m.start())) for m in MUTATOR_RE.finditer(body)]
if not mutators:
continue
backend = sorted(set(m.group(1) or m.group(2) for m in BACKEND_RE.finditer(body)))
if not backend:
continue
findings.append({
"function": name,
"line": line_of(masked, start),
"mutators": mutators,
"backend": backend,
})
return findings, all_mutators
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--summary", action="store_true", help="print the counts only")
args = parser.parse_args()
if not os.path.isdir(SCAN_ROOT):
sys.exit("missing %s" % SCAN_ROOT)
sources = []
for root, _, files in os.walk(SCAN_ROOT):
for name in sorted(files):
if name.endswith((".cpp", ".h")):
sources.append(os.path.join(root, name))
sources.sort()
total_functions = 0
total_mutators = 0
deferred_mutators = 0
distinct_mutators = {}
distinct_all = {}
for path in sources:
findings, all_mutators = scan_file(path)
for mutator, _ in all_mutators:
distinct_all[mutator] = distinct_all.get(mutator, 0) + 1
deferred_mutators += len(all_mutators)
if not findings:
continue
relative = os.path.relpath(path, REPO_ROOT).replace(os.sep, "/")
if not args.summary:
print("\n%s" % relative)
for finding in findings:
total_functions += 1
total_mutators += len(finding["mutators"])
for mutator, _ in finding["mutators"]:
distinct_mutators[mutator] = distinct_mutators.get(mutator, 0) + 1
if args.summary:
continue
print(" %s (line %d) -> backend: %s" % (finding["function"], finding["line"],
", ".join(finding["backend"][:4])))
for mutator, line in finding["mutators"]:
print(" %-44s :%d UNMAPPED" % (mutator, line))
print("\ndirty-surface: %d files scanned under MG_Impl/GLImpl" % len(sources))
print("dirty-surface: %d mutator calls in total, %d distinct mutators" % (deferred_mutators,
len(distinct_all)))
print("dirty-surface: %d of them sit in %d IMMEDIATE PUBLISH POINTS - functions that also "
"reach the backend - across %d distinct mutators"
% (total_mutators, total_functions, len(distinct_mutators)))
print("dirty-surface: the remaining %d are DEFERRED: nothing reaches the backend in the same "
"function, so the next verb publishes them, and each one needs an aggregate generation"
% (deferred_mutators - total_mutators))
print("dirty-surface: distinct mutators, by call count")
for mutator in sorted(distinct_all, key=lambda k: (-distinct_all[k], k)):
print(" %5d %s%s" % (distinct_all[mutator], mutator,
" (immediate)" if mutator in distinct_mutators else ""))
print("dirty-surface: every mutator above is UNMAPPED - the aggregate-generation mapping file "
"lands in P1, and this report is what it has to cover.")
print("dirty-surface: known limits of this scanner - it matches braced function bodies "
"textually, so a mutator inside a lambda is attributed to the enclosing function, and a "
"mutation published through a helper the entry point calls reads as deferred here.")
return 0
if __name__ == "__main__":
sys.exit(main())