mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-09-12 14:18:31 +09:00
[Build] (CI): add the pipe-gen check, the stdio-instrumentation gate and the citation lint
- Three P0 gates from plan B section 11, in one job that needs no build: a broken build must not be able to hide a drifted interface. - pipe-gen-check regenerates G1-G7 and runs git diff --exit-code over MobileGL/MG_Pipe/generated. Since all seven generators read the same .def files, this is what makes drift between them impossible rather than merely unlikely. - The stdio gate refuses fprintf(stderr and printf( under MG_Backend and MG_State. Nothing there matches today, so it lands with NO whitelist - verified with a negative control that fprintf(stderr and std::printf both trip it while snprintf does not. Per-draw instrumentation has been committed by accident before, once inside a mutex critical section, and MGLOG_D is the channel these trees are allowed to use because it compiles out in INFO builds. - scripts/gen_pipe_dirty_surface.py reports the frontend mutation surface corollary 4 needs covered: 926 mutator calls under MG_Impl/GLImpl over 73 distinct mutators, of which only 92 sit in a function that also reaches the backend. The other 834 are published by the NEXT verb, which is precisely the population that needs an aggregate generation. Informational in P0; it becomes a gate in P1 when there is a mapping file to diff against. - scripts/check_doc_citations.py resolves every `path:line` citation against a git revision. It reproduces the failure that motivated it - the plan's first draft cited SamplerObject.h:468-492 in a 160-line file - and reports 84 unresolved citations out of 1028 in docs/Disaggregated today, which is why CI runs it warning-only until those documents settle. --strict exits 1, verified in both directions.
This commit is contained in:
@@ -778,3 +778,55 @@ jobs:
|
|||||||
)
|
)
|
||||||
|
|
||||||
echo "Deleted ${deleted} intermediate Linux artifact(s); retained ${retained} failed-retrace fixture(s)."
|
echo "Deleted ${deleted} intermediate Linux artifact(s); retained ${retained} failed-retrace fixture(s)."
|
||||||
|
|
||||||
|
pipe-gates:
|
||||||
|
name: MGPipe generators and hygiene gates
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
# Deliberately independent of build-linux: these are source-level gates, they take
|
||||||
|
# seconds, and a broken build must not hide a drifted interface.
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout repo
|
||||||
|
uses: actions/checkout@v6
|
||||||
|
|
||||||
|
# The seven generators all read MG_Pipe/*.def, so regenerating and diffing is what
|
||||||
|
# keeps the two interface tables, the wire records, the verify comparators, the
|
||||||
|
# PipeInputs field ids, the read-inventory coverage and the render-state member list
|
||||||
|
# from drifting apart from the catalogue. The generated files are committed
|
||||||
|
# deliberately: the build must not depend on python.
|
||||||
|
- name: Regenerate the MGPipe interface (G1-G7)
|
||||||
|
run: |
|
||||||
|
python3 scripts/gen_pipe.py
|
||||||
|
git diff --exit-code -- MobileGL/MG_Pipe/generated
|
||||||
|
|
||||||
|
# Per-draw fprintf/printf instrumentation has repeatedly been committed by accident,
|
||||||
|
# once inside a mutex critical section. Nothing under these two trees prints to a
|
||||||
|
# stdio stream today - MGLOG_D compiles out in INFO builds and is the only channel
|
||||||
|
# they are allowed to use - so this gate starts with no exceptions, and any addition
|
||||||
|
# to it needs a reason in the pull request rather than a quiet whitelist entry.
|
||||||
|
- name: No stdio instrumentation in MG_Backend or MG_State
|
||||||
|
run: |
|
||||||
|
if grep -rnE 'fprintf[[:space:]]*\(stderr|(^|[^[:alnum:]_>.])printf[[:space:]]*\(' \
|
||||||
|
MobileGL/MG_Backend MobileGL/MG_State; then
|
||||||
|
echo "::error::stdio instrumentation found; use MGLOG_D (compiled out in INFO builds)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "no fprintf(stderr / printf( under MobileGL/MG_Backend or MobileGL/MG_State"
|
||||||
|
|
||||||
|
# Informational: the frontend mutation surface an MGPipe aggregate generation has to
|
||||||
|
# cover. It becomes a gate in P1, when the mapping file exists to diff against.
|
||||||
|
- name: MGPipe dirty-surface report
|
||||||
|
run: python3 scripts/gen_pipe_dirty_surface.py --summary
|
||||||
|
|
||||||
|
# Warning only for now: the disaggregation documents are still being written, and a
|
||||||
|
# lint that fails a rewrite in progress teaches people to ignore it. It becomes
|
||||||
|
# --strict when the documents settle.
|
||||||
|
- name: Documentation citation lint
|
||||||
|
run: |
|
||||||
|
shopt -s nullglob
|
||||||
|
documents=(docs/Disaggregated/*.md)
|
||||||
|
if [ ${#documents[@]} -eq 0 ]; then
|
||||||
|
echo "no disaggregation documents to check"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
python3 scripts/check_doc_citations.py "${documents[@]}" || true
|
||||||
|
|||||||
@@ -0,0 +1,123 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# MobileGL - scripts/check_doc_citations.py
|
||||||
|
# Copyright (c) 2025-2026 MobileGL-Dev
|
||||||
|
# Licensed under the GNU Lesser General Public License v3.0:
|
||||||
|
# https://www.gnu.org/licenses/gpl-3.0.txt
|
||||||
|
# https://www.gnu.org/licenses/lgpl-3.0.txt
|
||||||
|
# SPDX-License-Identifier: LGPL-3.0-only
|
||||||
|
# End of Source File Header
|
||||||
|
"""Resolve every `path:line` citation in a set of markdown documents.
|
||||||
|
|
||||||
|
A design document that cites the code is only as good as its line numbers, and a wrong one
|
||||||
|
is worse than none: it sends the next reader to a function that does something else. The
|
||||||
|
disaggregation plan's first draft cited SamplerObject.h:468-492 for a struct that lives at
|
||||||
|
:72-96 in a 160-line file, and nothing caught it.
|
||||||
|
|
||||||
|
So every `File.h:123` and `File.cpp:123-456` in the given documents is resolved against a
|
||||||
|
git revision - the file must exist there and must have at least that many lines. Bare file
|
||||||
|
names are resolved by basename, which is how the plan spells most of its citations; an
|
||||||
|
ambiguous basename is reported rather than guessed.
|
||||||
|
|
||||||
|
python3 scripts/check_doc_citations.py docs/Disaggregated/*.md
|
||||||
|
python3 scripts/check_doc_citations.py --rev 81b17c0b --strict docs/Disaggregated/*.md
|
||||||
|
|
||||||
|
Exits non-zero only with --strict, so it can be wired into CI as a warning first.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
# `Managers.cpp:4340-4390`, `MobileGL/MG_State/.../RenderState.h:263`, `:12-13` is NOT
|
||||||
|
# matched on purpose: a citation with no file name cannot be checked, only guessed.
|
||||||
|
CITATION_RE = re.compile(
|
||||||
|
r"(?<![\w/.-])((?:[\w.-]+/)*[\w.-]+\.(?:h|hpp|cpp|cc|c|py|def|inc|md|json|yml|yaml|txt|fbs))"
|
||||||
|
r":(\d+)(?:-(\d+))?")
|
||||||
|
|
||||||
|
|
||||||
|
def git(args, rev_root=REPO_ROOT):
|
||||||
|
return subprocess.run(["git", "-C", rev_root] + args, stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE, check=False).stdout.decode("utf-8", "replace")
|
||||||
|
|
||||||
|
|
||||||
|
class Tree(object):
|
||||||
|
def __init__(self, rev):
|
||||||
|
self.Rev = rev
|
||||||
|
listing = git(["ls-tree", "-r", "--name-only", rev]).splitlines()
|
||||||
|
if not listing:
|
||||||
|
sys.exit("check_doc_citations: revision %s has no files (bad revision?)" % rev)
|
||||||
|
self.Paths = set(listing)
|
||||||
|
self.ByBasename = {}
|
||||||
|
for path in listing:
|
||||||
|
self.ByBasename.setdefault(os.path.basename(path), []).append(path)
|
||||||
|
self.LineCounts = {}
|
||||||
|
|
||||||
|
def Resolve(self, cited):
|
||||||
|
if cited in self.Paths:
|
||||||
|
return [cited]
|
||||||
|
candidates = self.ByBasename.get(os.path.basename(cited), [])
|
||||||
|
if len(candidates) > 1 and "/" in cited:
|
||||||
|
candidates = [p for p in candidates if p.endswith(cited)] or candidates
|
||||||
|
return candidates
|
||||||
|
|
||||||
|
def LineCount(self, path):
|
||||||
|
if path not in self.LineCounts:
|
||||||
|
blob = git(["show", "%s:%s" % (self.Rev, path)])
|
||||||
|
# A file with no trailing newline still has that last line.
|
||||||
|
count = blob.count("\n") + (1 if blob and not blob.endswith("\n") else 0)
|
||||||
|
self.LineCounts[path] = count
|
||||||
|
return self.LineCounts[path]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("documents", nargs="+", help="markdown files to check")
|
||||||
|
parser.add_argument("--rev", default="HEAD", help="git revision the citations point into")
|
||||||
|
parser.add_argument("--strict", action="store_true", help="exit 1 when a citation does not resolve")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
tree = Tree(args.rev)
|
||||||
|
checked = 0
|
||||||
|
problems = []
|
||||||
|
for document in args.documents:
|
||||||
|
if not os.path.exists(document):
|
||||||
|
problems.append("%s: no such document" % document)
|
||||||
|
continue
|
||||||
|
with open(document, "r", encoding="utf-8", errors="replace") as handle:
|
||||||
|
lines = handle.read().splitlines()
|
||||||
|
for number, line in enumerate(lines, start=1):
|
||||||
|
for match in CITATION_RE.finditer(line):
|
||||||
|
cited, first, last = match.group(1), int(match.group(2)), match.group(3)
|
||||||
|
last = int(last) if last else first
|
||||||
|
checked += 1
|
||||||
|
where = "%s:%d: `%s`" % (document, number, match.group(0))
|
||||||
|
candidates = tree.Resolve(cited)
|
||||||
|
if not candidates:
|
||||||
|
problems.append("%s -> no such file at %s" % (where, args.rev))
|
||||||
|
continue
|
||||||
|
if len(candidates) > 1:
|
||||||
|
problems.append("%s -> ambiguous: %s" % (where, ", ".join(sorted(candidates))))
|
||||||
|
continue
|
||||||
|
if last < first:
|
||||||
|
problems.append("%s -> inverted line range" % where)
|
||||||
|
continue
|
||||||
|
count = tree.LineCount(candidates[0])
|
||||||
|
if last > count:
|
||||||
|
problems.append("%s -> %s has %d lines at %s"
|
||||||
|
% (where, candidates[0], count, args.rev))
|
||||||
|
|
||||||
|
print("check_doc_citations: %d citations in %d document(s) against %s, %d problem(s)"
|
||||||
|
% (checked, len(args.documents), args.rev, len(problems)))
|
||||||
|
for problem in problems:
|
||||||
|
print(" %s" % problem)
|
||||||
|
if problems and args.strict:
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# MobileGL - scripts/gen_pipe_dirty_surface.py
|
||||||
|
# Copyright (c) 2025-2026 MobileGL-Dev
|
||||||
|
# Licensed under the GNU Lesser General Public License v3.0:
|
||||||
|
# https://www.gnu.org/licenses/gpl-3.0.txt
|
||||||
|
# https://www.gnu.org/licenses/lgpl-3.0.txt
|
||||||
|
# SPDX-License-Identifier: LGPL-3.0-only
|
||||||
|
# End of Source File Header
|
||||||
|
"""The dirty-surface scanner (plan B corollary 4, section 5.2).
|
||||||
|
|
||||||
|
MGPipe replaces "the backend rediscovers what changed" with "the frontend says what
|
||||||
|
changed", which only works if EVERY frontend mutation that a backend can observe bumps an
|
||||||
|
aggregate generation. The failure mode is silent and one-directional: a mutation that
|
||||||
|
forgets to bump renders stale, and no purity gate can see it.
|
||||||
|
|
||||||
|
So the mutation surface has to be enumerated mechanically rather than by memory. This
|
||||||
|
script reports every place in MG_Impl/GLImpl where a GL entry point BOTH mutates frontend
|
||||||
|
state through pGLContext AND reaches the backend in the same function - those are the
|
||||||
|
publish points, the ones that must map onto an aggregate generation.
|
||||||
|
|
||||||
|
P0 is the skeleton: it reports. P1 adds the mapping file and CI regenerates it with
|
||||||
|
`git diff --exit-code` and zero unmapped mutators, the same shape as gen_pipe.py's G6.
|
||||||
|
|
||||||
|
python3 scripts/gen_pipe_dirty_surface.py # human-readable report
|
||||||
|
python3 scripts/gen_pipe_dirty_surface.py --summary # counts only
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
SCAN_ROOT = os.path.join(REPO_ROOT, "MobileGL", "MG_Impl", "GLImpl")
|
||||||
|
|
||||||
|
# The mutating half of GLContext's surface. Prefix-matched, per the plan's list.
|
||||||
|
MUTATOR_PREFIXES = ("Add", "Set", "Mark", "Bump", "Allocate", "Truncate", "Record", "Notify",
|
||||||
|
"Begin", "End")
|
||||||
|
|
||||||
|
MUTATOR_RE = re.compile(r"pGLContext->\s*((?:%s)\w*)\s*\(" % "|".join(MUTATOR_PREFIXES))
|
||||||
|
BACKEND_RE = re.compile(r"gBackendFunctionsTable\.GL\.(\w+)|pActiveBackendObject->\s*(\w+)")
|
||||||
|
FUNCTION_RE = re.compile(r"(?:^|\n)[ \t]*(?:[A-Za-z_][\w:<>,&*\s]*?)\b(\w+)\s*\([^;{}]*\)\s*"
|
||||||
|
r"(?:const\s*)?(?:noexcept\s*)?\{")
|
||||||
|
|
||||||
|
|
||||||
|
def mask_comments_and_strings(text):
|
||||||
|
"""Replace comment and string-literal bodies with spaces, keeping every offset and
|
||||||
|
newline, so the regexes below cannot match inside a comment or a literal."""
|
||||||
|
out = list(text)
|
||||||
|
i = 0
|
||||||
|
n = len(text)
|
||||||
|
while i < n:
|
||||||
|
c = text[i]
|
||||||
|
if c == "/" and i + 1 < n and text[i + 1] == "/":
|
||||||
|
while i < n and text[i] != "\n":
|
||||||
|
out[i] = " "
|
||||||
|
i += 1
|
||||||
|
elif c == "/" and i + 1 < n and text[i + 1] == "*":
|
||||||
|
out[i] = out[i + 1] = " "
|
||||||
|
i += 2
|
||||||
|
while i < n and not (text[i] == "*" and i + 1 < n and text[i + 1] == "/"):
|
||||||
|
if text[i] != "\n":
|
||||||
|
out[i] = " "
|
||||||
|
i += 1
|
||||||
|
if i < n:
|
||||||
|
out[i] = " "
|
||||||
|
if i + 1 < n:
|
||||||
|
out[i + 1] = " "
|
||||||
|
i += 2
|
||||||
|
elif c in "\"'":
|
||||||
|
quote = c
|
||||||
|
i += 1
|
||||||
|
while i < n and text[i] != quote:
|
||||||
|
if text[i] == "\\":
|
||||||
|
out[i] = " "
|
||||||
|
i += 1
|
||||||
|
if i < n and text[i] != "\n":
|
||||||
|
out[i] = " "
|
||||||
|
i += 1
|
||||||
|
if i < n:
|
||||||
|
out[i] = " "
|
||||||
|
i += 1
|
||||||
|
else:
|
||||||
|
i += 1
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def function_bodies(masked):
|
||||||
|
"""Yield (name, start_offset, end_offset) for every braced function body."""
|
||||||
|
for match in FUNCTION_RE.finditer(masked):
|
||||||
|
name = match.group(1)
|
||||||
|
start = masked.index("{", match.end() - 1) if masked[match.end() - 1] != "{" else match.end() - 1
|
||||||
|
depth = 0
|
||||||
|
i = start
|
||||||
|
while i < len(masked):
|
||||||
|
if masked[i] == "{":
|
||||||
|
depth += 1
|
||||||
|
elif masked[i] == "}":
|
||||||
|
depth -= 1
|
||||||
|
if depth == 0:
|
||||||
|
yield name, start, i
|
||||||
|
break
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
|
||||||
|
def line_of(text, offset):
|
||||||
|
return text.count("\n", 0, offset) + 1
|
||||||
|
|
||||||
|
|
||||||
|
def scan_file(path):
|
||||||
|
with open(path, "r", encoding="utf-8", errors="replace") as handle:
|
||||||
|
text = handle.read()
|
||||||
|
masked = mask_comments_and_strings(text)
|
||||||
|
findings = []
|
||||||
|
# Every mutator in the file, whether or not it shares a function with a backend call.
|
||||||
|
# The difference between this and the publish points below is the whole point of the
|
||||||
|
# report: a mutation that does NOT reach the backend in the same function is published
|
||||||
|
# by the NEXT verb, and it is exactly those that need an aggregate generation rather
|
||||||
|
# than an inline push.
|
||||||
|
all_mutators = [(m.group(1), line_of(masked, m.start())) for m in MUTATOR_RE.finditer(masked)]
|
||||||
|
for name, start, end in function_bodies(masked):
|
||||||
|
body = masked[start:end]
|
||||||
|
mutators = [(m.group(1), line_of(masked, start + m.start())) for m in MUTATOR_RE.finditer(body)]
|
||||||
|
if not mutators:
|
||||||
|
continue
|
||||||
|
backend = sorted(set(m.group(1) or m.group(2) for m in BACKEND_RE.finditer(body)))
|
||||||
|
if not backend:
|
||||||
|
continue
|
||||||
|
findings.append({
|
||||||
|
"function": name,
|
||||||
|
"line": line_of(masked, start),
|
||||||
|
"mutators": mutators,
|
||||||
|
"backend": backend,
|
||||||
|
})
|
||||||
|
return findings, all_mutators
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--summary", action="store_true", help="print the counts only")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if not os.path.isdir(SCAN_ROOT):
|
||||||
|
sys.exit("missing %s" % SCAN_ROOT)
|
||||||
|
|
||||||
|
sources = []
|
||||||
|
for root, _, files in os.walk(SCAN_ROOT):
|
||||||
|
for name in sorted(files):
|
||||||
|
if name.endswith((".cpp", ".h")):
|
||||||
|
sources.append(os.path.join(root, name))
|
||||||
|
sources.sort()
|
||||||
|
|
||||||
|
total_functions = 0
|
||||||
|
total_mutators = 0
|
||||||
|
deferred_mutators = 0
|
||||||
|
distinct_mutators = {}
|
||||||
|
distinct_all = {}
|
||||||
|
for path in sources:
|
||||||
|
findings, all_mutators = scan_file(path)
|
||||||
|
for mutator, _ in all_mutators:
|
||||||
|
distinct_all[mutator] = distinct_all.get(mutator, 0) + 1
|
||||||
|
deferred_mutators += len(all_mutators)
|
||||||
|
if not findings:
|
||||||
|
continue
|
||||||
|
relative = os.path.relpath(path, REPO_ROOT).replace(os.sep, "/")
|
||||||
|
if not args.summary:
|
||||||
|
print("\n%s" % relative)
|
||||||
|
for finding in findings:
|
||||||
|
total_functions += 1
|
||||||
|
total_mutators += len(finding["mutators"])
|
||||||
|
for mutator, _ in finding["mutators"]:
|
||||||
|
distinct_mutators[mutator] = distinct_mutators.get(mutator, 0) + 1
|
||||||
|
if args.summary:
|
||||||
|
continue
|
||||||
|
print(" %s (line %d) -> backend: %s" % (finding["function"], finding["line"],
|
||||||
|
", ".join(finding["backend"][:4])))
|
||||||
|
for mutator, line in finding["mutators"]:
|
||||||
|
print(" %-44s :%d UNMAPPED" % (mutator, line))
|
||||||
|
|
||||||
|
print("\ndirty-surface: %d files scanned under MG_Impl/GLImpl" % len(sources))
|
||||||
|
print("dirty-surface: %d mutator calls in total, %d distinct mutators" % (deferred_mutators,
|
||||||
|
len(distinct_all)))
|
||||||
|
print("dirty-surface: %d of them sit in %d IMMEDIATE PUBLISH POINTS - functions that also "
|
||||||
|
"reach the backend - across %d distinct mutators"
|
||||||
|
% (total_mutators, total_functions, len(distinct_mutators)))
|
||||||
|
print("dirty-surface: the remaining %d are DEFERRED: nothing reaches the backend in the same "
|
||||||
|
"function, so the next verb publishes them, and each one needs an aggregate generation"
|
||||||
|
% (deferred_mutators - total_mutators))
|
||||||
|
print("dirty-surface: distinct mutators, by call count")
|
||||||
|
for mutator in sorted(distinct_all, key=lambda k: (-distinct_all[k], k)):
|
||||||
|
print(" %5d %s%s" % (distinct_all[mutator], mutator,
|
||||||
|
" (immediate)" if mutator in distinct_mutators else ""))
|
||||||
|
print("dirty-surface: every mutator above is UNMAPPED - the aggregate-generation mapping file "
|
||||||
|
"lands in P1, and this report is what it has to cover.")
|
||||||
|
print("dirty-surface: known limits of this scanner - it matches braced function bodies "
|
||||||
|
"textually, so a mutator inside a lambda is attributed to the enclosing function, and a "
|
||||||
|
"mutation published through a helper the entry point calls reads as deferred here.")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Reference in New Issue
Block a user