mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-09-17 00:28:31 +09:00
[Fix, Test] (workflows, scripts/ci): make the three CI negative controls assert their own failure reason instead of accepting any non-zero ctest exit, count the arming baseline by PASSED rather than by not-skipped, and put both control bodies in files a smoke test can drive against a stubbed ctest
This commit is contained in:
+54
-76
@@ -1074,57 +1074,32 @@ jobs:
|
|||||||
# Split entries SKIP and ctest reports green whatever the knob says, so an unconditional
|
# Split entries SKIP and ctest reports green whatever the knob says, so an unconditional
|
||||||
# control would be red for the whole of P5 for a reason that is not a defect.
|
# control would be red for the whole of P5 for a reason that is not a defect.
|
||||||
#
|
#
|
||||||
# So the expected state is DERIVED rather than assumed, from the same fact the lanes derive
|
# So the expected state is DERIVED FROM BEHAVIOUR rather than assumed. The first version read
|
||||||
# it from: MG_IntegrationTest/CMakeLists.txt puts MGITEST_REMOTE_CLIENT_PRESENT=1 into the
|
# MGITEST_REMOTE_CLIENT_PRESENT out of the generated *_tests.cmake, which was a restatement of
|
||||||
# Split entries' ENVIRONMENT exactly when MG_Remote carries no c0 signature stub, and that
|
# the CMake source probe review finding M-1 falsified; the arming condition is a runtime fact
|
||||||
# string is in the generated ctest include files this artifact ships. When it is there the
|
# inside each test process (MG_Config::Transport, ClientSession::Active() and
|
||||||
# controls MUST fire; when it is not, the step says so loudly and does not pretend.
|
# ImplementedVerbCount(), read by Harness/SplitRuntimePeek), so the only honest way to ask it
|
||||||
|
# from a shell is to look at what the entries DID. When entries passed, the controls MUST
|
||||||
|
# fire; when every one of them skipped, the step says so loudly and does not pretend.
|
||||||
|
#
|
||||||
|
# THE BODY OF THIS STEP IS scripts/ci/split_negative_controls.sh, and the move is the point
|
||||||
|
# rather than tidiness. A `run:` block executes nowhere but on a runner, so these lines were
|
||||||
|
# unreviewable and untestable: when the wave-1 cross-family review said they were broken,
|
||||||
|
# CONFIRMING it needed a hand-made copy of them (wave1-codex-verify.md 8), and a copy is not
|
||||||
|
# the thing. scripts/ci/control_smoke_test.sh now drives the very file this step runs.
|
||||||
|
#
|
||||||
|
# What that smoke test pins, and what ID-46 finding 8 found missing: each control asserts its
|
||||||
|
# OWN failure reason. A non-zero ctest exit used to be enough, so a timeout, a setup abort or
|
||||||
|
# any unrelated assertion printed "turned N selected entries red, as it must" and this step
|
||||||
|
# went green. The arming run's `|| true` had the matching defect - it counted a case that ran
|
||||||
|
# and FAILED as evidence the lane was live, so the controls could be measured against a
|
||||||
|
# baseline that was already red.
|
||||||
- name: Negative controls - the verb barrier and the persistent-map push must be load-bearing
|
- name: Negative controls - the verb barrier and the persistent-map push must be load-bearing
|
||||||
working-directory: build-split
|
working-directory: build-split
|
||||||
env:
|
env:
|
||||||
MOBILEGL_ITEST_REQUIRE_GPU: "1"
|
MOBILEGL_ITEST_REQUIRE_GPU: "1"
|
||||||
run: |
|
CONTROL_TMPDIR: ${{ runner.temp }}
|
||||||
# THE ARMED STATE IS DERIVED FROM BEHAVIOUR, not from a marker string in the generated
|
run: bash "${GITHUB_WORKSPACE}/scripts/ci/split_negative_controls.sh"
|
||||||
# ctest files. The first version read MGITEST_REMOTE_CLIENT_PRESENT out of
|
|
||||||
# *_tests.cmake, which was a restatement of the CMake source probe review finding M-1
|
|
||||||
# falsified; the arming condition is now a runtime fact inside each test process, so the
|
|
||||||
# only honest way to ask it from a shell is to look at what the entries DID.
|
|
||||||
ctest -L integration-split -j 4 --no-tests=error --output-junit "${RUNNER_TEMP}/isplit.xml" || true
|
|
||||||
armed=$(python3 - "${RUNNER_TEMP}/isplit.xml" <<'PY'
|
|
||||||
import sys, xml.etree.ElementTree as ET
|
|
||||||
ran = 0
|
|
||||||
for case in ET.parse(sys.argv[1]).getroot().iter('testcase'):
|
|
||||||
if case.find('skipped') is None and case.get('status') not in ('notrun', 'disabled'):
|
|
||||||
ran += 1
|
|
||||||
print(ran)
|
|
||||||
PY
|
|
||||||
)
|
|
||||||
echo "split entries that actually ran: ${armed}"
|
|
||||||
if [ "${armed}" -lt 1 ]; then
|
|
||||||
echo "::warning::every DirectGLES.Split. entry SKIPPED, so neither negative control can fire. The arming condition is a runtime fact - MG_Config::Transport, ClientSession::Active() and ImplementedVerbCount(), read by Harness/SplitRuntimePeek - and it becomes true on the commit that lands the last of c1/s1/v1. This step becomes a gate then, with no edit; it is not a green that asserted anything today."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
run_control() {
|
|
||||||
name="$1"; filter="$2"; shift 2
|
|
||||||
matched=$(ctest -N -L integration-split -R "${filter}" | grep -cE '^ *Test *#[0-9]+:')
|
|
||||||
if [ "${matched}" -lt 1 ]; then
|
|
||||||
echo "::error::${name} selected ${matched} tests; its filter no longer matches anything"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
if env "$@" ctest --output-on-failure -L integration-split -R "${filter}" --no-tests=error; then
|
|
||||||
echo "::error::${name} left ${matched} split entries GREEN, so the knob it turns is not load-bearing and the gate it controls proves nothing."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "${name} turned ${matched} selected entries red, as it must"
|
|
||||||
}
|
|
||||||
# E1: R-1's lockstep verb barrier. Without it the client keeps pulling fields from a live
|
|
||||||
# GLContext while the server runs ahead, so the server reads future values.
|
|
||||||
run_control "negative control E1 (MOBILEGL_IPC_VERB_BARRIER=0)" \
|
|
||||||
'DirectGLES\.Split\.(Triangle|ClearThenReadPixels)' MOBILEGL_IPC_VERB_BARRIER=0
|
|
||||||
# E3(a): the persistent-map push. 0 is admitted by ConfigLoader on purpose and is
|
|
||||||
# documented there as this control.
|
|
||||||
run_control "negative control E3(a) (MOBILEGL_IPC_PERSISTENT_BLOCK_KB=0)" \
|
|
||||||
'DirectGLES\.Split\.PersistentCoherentMapScenario' MOBILEGL_IPC_PERSISTENT_BLOCK_KB=0
|
|
||||||
|
|
||||||
- name: Upload split lane logs
|
- name: Upload split lane logs
|
||||||
if: always()
|
if: always()
|
||||||
@@ -1959,37 +1934,25 @@ jobs:
|
|||||||
# The rerun replays into the same case directory, so the good run's images are put aside and
|
# The rerun replays into the same case directory, so the good run's images are put aside and
|
||||||
# restored whichever way the control goes; "Upload actual image" below runs `if: always()`
|
# restored whichever way the control goes; "Upload actual image" below runs `if: always()`
|
||||||
# and would otherwise ship the deliberately-wrong run's output under the good run's name.
|
# and would otherwise ship the deliberately-wrong run's output under the good run's name.
|
||||||
|
# THE BODY OF THIS STEP IS scripts/ci/retrace_pull_library_control.sh, for the reason the
|
||||||
|
# split lane's control gives: a `run:` block cannot be executed off a runner, so these lines
|
||||||
|
# could not be tested until they ran in CI. scripts/ci/control_smoke_test.sh drives that file.
|
||||||
|
#
|
||||||
|
# Two holes ID-46 finding 8(b) found in this block, both CONFIRMED against the REAL ctest in a
|
||||||
|
# REAL build tree, both closed in the script: it had NO selection guard at all - unlike the
|
||||||
|
# split lane's run_control - so a case/backend regex matching nothing exited 8 through
|
||||||
|
# `--no-tests=error` and was read as "the pull library turned it red"; and only "non-zero
|
||||||
|
# ctest" was checked after the nm identity check, so a loader failure, a missing fixture or a
|
||||||
|
# timeout passed it. The red must now carry run_trace_case.cmake's own sentence.
|
||||||
- name: Negative control - the PULL library must red this split retrace
|
- name: Negative control - the PULL library must red this split retrace
|
||||||
working-directory: build-retrace/tools/trace_replay
|
working-directory: build-retrace/tools/trace_replay
|
||||||
run: |
|
env:
|
||||||
set +e
|
CONTROL_TMPDIR: ${{ runner.temp }}
|
||||||
GOOD_OUTPUT="${RUNNER_TEMP}/split-verified-output"
|
PULL_LIBRARY: ${{ github.workspace }}/pull-runtime/build-linux/libMobileGL.so
|
||||||
rm -rf "${GOOD_OUTPUT}"
|
FROZEN_LIBRARY: ${{ github.workspace }}/build-linux/libMobileGL.so
|
||||||
if [ -d "${{ matrix.case }}" ]; then cp -a "${{ matrix.case }}" "${GOOD_OUTPUT}"; fi
|
run: >-
|
||||||
# The pull library, unpacked from build-linux's artifact, over the frozen path every
|
bash "${GITHUB_WORKSPACE}/scripts/ci/retrace_pull_library_control.sh"
|
||||||
# case has baked in. It defines no MG_Remote symbol, so ConfigLoader has no transport
|
'${{ matrix.case }}' '${{ matrix.backend }}'
|
||||||
# parser and MOBILEGL_TRANSPORT=inproc is accepted and ignored - the exact shape of "the
|
|
||||||
# split lane ran monolith".
|
|
||||||
cp "${GITHUB_WORKSPACE}/pull-runtime/build-linux/libMobileGL.so" \
|
|
||||||
"${GITHUB_WORKSPACE}/build-linux/libMobileGL.so"
|
|
||||||
if nm --defined-only "${GITHUB_WORKSPACE}/build-linux/libMobileGL.so" | grep -q -i MG_Remote; then
|
|
||||||
echo "::error::the control's own library defines MG_Remote symbols, so it is not a pull build and this control would prove nothing"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
export MOBILEGL_TRANSPORT=inproc
|
|
||||||
ctest -V --no-tests=error --timeout 10800 \
|
|
||||||
-R "^MobileGLTraceReplay\.${{ matrix.case }}\.${{ matrix.backend }}$"
|
|
||||||
control_rc=$?
|
|
||||||
set -e
|
|
||||||
if [ -d "${GOOD_OUTPUT}" ]; then
|
|
||||||
rm -rf "${{ matrix.case }}"; mv "${GOOD_OUTPUT}" "${{ matrix.case }}"
|
|
||||||
echo "restored the verified run's output over the control's"
|
|
||||||
fi
|
|
||||||
if [ "${control_rc}" -eq 0 ]; then
|
|
||||||
echo "::error::a PULL library passed the split retrace. OpenRA scores ssim 1.000000 under a monolith library too (measured), so the picture is not and cannot be this lane's gate - run_trace_case.cmake's transport-resolution assertion is, and it has stopped working. Every green in this job is then a monolith run under a name that says split."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
echo "the pull library turned the split retrace red, as it must (ctest exit ${control_rc})"
|
|
||||||
|
|
||||||
# The refusal census, recorded rather than gated. run_trace_case.cmake already REDS the case
|
# The refusal census, recorded rather than gated. run_trace_case.cmake already REDS the case
|
||||||
# on any Fatal{, so reaching here means the count is zero - but the number and the distinct
|
# on any Fatal{, so reaching here means the count is zero - but the number and the distinct
|
||||||
@@ -2314,6 +2277,21 @@ jobs:
|
|||||||
python3 scripts/gen_pipe_field_ownership.py --check
|
python3 scripts/gen_pipe_field_ownership.py --check
|
||||||
python3 scripts/gen_pipe_field_ownership.py --self-test
|
python3 scripts/gen_pipe_field_ownership.py --self-test
|
||||||
|
|
||||||
|
# R-16 APPLIED TO THE NEGATIVE CONTROLS THEMSELVES. The split lane's E1/E3(a) controls and the
|
||||||
|
# retrace lane's pull-library control are gates, and until ID-46 finding 8 neither could be
|
||||||
|
# made red by anyone: their bodies were `run:` blocks, which execute only on a runner. Both
|
||||||
|
# bodies now live in scripts/ci/, and this step runs them against a stubbed ctest that
|
||||||
|
# reproduces the finding - a NON-EMPTY selection failing with UNRELATED_CONTROL_FAILURE, and a
|
||||||
|
# case/backend regex matching no tests - and requires each control to report FAILED. The same
|
||||||
|
# stub, failing with the diagnostics the scenarios really emit, must make them report PASSED.
|
||||||
|
#
|
||||||
|
# NO BRANCH GUARD: this asks "do the negative controls still reject a red that is not theirs",
|
||||||
|
# which is a question every branch can answer and none of which depends on the TEMPORARY
|
||||||
|
# feat/disaggregated trigger at the top of this file. It costs a couple of seconds and needs
|
||||||
|
# no build.
|
||||||
|
- name: The split and retrace negative controls still reject a red that is not theirs (R-16)
|
||||||
|
run: bash scripts/ci/control_smoke_test.sh
|
||||||
|
|
||||||
# A GATE as of P3a (G5). "pool 与延迟释放原样搬" (ROADMAP.md:19) is meant literally: the
|
# A GATE as of P3a (G5). "pool 与延迟释放原样搬" (ROADMAP.md:19) is meant literally: the
|
||||||
# buffer pool, the deferred-release drain and the three persistently mapped rings move
|
# buffer pool, the deferred-release drain and the three persistently mapped rings move
|
||||||
# VERBATIM, and ARCHITECTURE.md:515 says why - their retire happens only inside Present, so
|
# VERBATIM, and ARCHITECTURE.md:515 says why - their retire happens only inside Present, so
|
||||||
|
|||||||
Executable
+106
@@ -0,0 +1,106 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# R-16 FOR THE CI NEGATIVE CONTROLS THEMSELVES: a control-run smoke test.
|
||||||
|
#
|
||||||
|
# BRIEF-P5 13 (R-16) says a negative control must assert its own failure reason and that every gate
|
||||||
|
# carries a line saying "I made it red once, by doing X". The two controls this file exercises ARE
|
||||||
|
# gates, and until ID-46 finding 8 nobody could make either of them red, because a workflow `run:`
|
||||||
|
# block only executes on a runner. The wave-1 verification agent had to hand-copy the blocks into
|
||||||
|
# throwaway harnesses to show they were broken (wave1-codex-verify.md 8). This file is that
|
||||||
|
# experiment, kept: it runs the REAL control scripts - the same files .github/workflows/test.yml
|
||||||
|
# invokes, not copies of them - against a stubbed ctest, and checks that each one passes exactly
|
||||||
|
# when it should.
|
||||||
|
#
|
||||||
|
# The case that matters is the first one. A stubbed ctest reports a NON-EMPTY selection and then
|
||||||
|
# fails with UNRELATED_CONTROL_FAILURE: a reason that has nothing to do with the knob the control
|
||||||
|
# turns. Before ID-48's fix both controls printed their success message and the step exited 0. They
|
||||||
|
# must now report FAILED.
|
||||||
|
#
|
||||||
|
# usage: control_smoke_test.sh
|
||||||
|
set -u
|
||||||
|
|
||||||
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
WORK="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
STUB_DIR="${WORK}/stub"
|
||||||
|
mkdir -p "${STUB_DIR}"
|
||||||
|
cp "${HERE}/testdata/stub_ctest.sh" "${STUB_DIR}/ctest"
|
||||||
|
chmod +x "${STUB_DIR}/ctest"
|
||||||
|
|
||||||
|
passes=0
|
||||||
|
failures=0
|
||||||
|
|
||||||
|
# expect <expected: PASSED|FAILED> <label> -- <command...>
|
||||||
|
expect() {
|
||||||
|
want="$1"; label="$2"; shift 3 # shift past the literal "--"
|
||||||
|
outfile="${WORK}/run.out"
|
||||||
|
"$@" > "${outfile}" 2>&1
|
||||||
|
rc=$?
|
||||||
|
if [ "${rc}" -eq 0 ]; then got="PASSED"; else got="FAILED"; fi
|
||||||
|
if [ "${got}" = "${want}" ]; then
|
||||||
|
passes=$((passes + 1))
|
||||||
|
printf 'ok %-58s %s (rc=%d)\n' "${label}" "${got}" "${rc}"
|
||||||
|
else
|
||||||
|
failures=$((failures + 1))
|
||||||
|
printf 'NOT OK %-58s expected %s, got %s (rc=%d)\n' "${label}" "${want}" "${got}" "${rc}"
|
||||||
|
sed 's/^/ | /' "${outfile}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
run_split() { # $1 = STUB_MODE
|
||||||
|
env -i PATH="${STUB_DIR}:/usr/bin:/bin" STUB_MODE="$1" \
|
||||||
|
CTEST=ctest CONTROL_TMPDIR="${WORK}/tmp-$1" \
|
||||||
|
bash "${HERE}/split_negative_controls.sh"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_retrace() { # $1 = STUB_MODE
|
||||||
|
cd "${WORK}" || return 127
|
||||||
|
mkdir -p "${WORK}/OpenRA"
|
||||||
|
env -i PATH="${STUB_DIR}:/usr/bin:/bin" STUB_MODE="$1" \
|
||||||
|
CTEST=ctest CONTROL_TMPDIR="${WORK}/tmp-$1" \
|
||||||
|
PULL_LIBRARY="${WORK}/pull.so" FROZEN_LIBRARY="${WORK}/frozen.so" \
|
||||||
|
bash "${HERE}/retrace_pull_library_control.sh" OpenRA DirectGLES
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "=== the split lane's E1 / E3(a) controls (scripts/ci/split_negative_controls.sh)"
|
||||||
|
# THE FINDING, REPRODUCED. Non-empty selection, green baseline, and a red that is not the knob's.
|
||||||
|
expect FAILED "unrelated failure with a non-empty selection" -- run_split unrelated
|
||||||
|
# ... and the same control on the same stub, failing for its own reason: it must PASS.
|
||||||
|
expect PASSED "the scenarios' own diagnostic" -- run_split evidence
|
||||||
|
# The pre-existing half of the control, which was never broken: a knob that reds nothing.
|
||||||
|
expect FAILED "the knob leaves the selection green" -- run_split green
|
||||||
|
# The arming counter's half of the finding: a baseline that is already red cannot arm anything.
|
||||||
|
expect FAILED "the baseline is already red" -- run_split red-baseline
|
||||||
|
# The disarmed lane, which is a legitimate exit 0 while c1/s1/v1 are landing.
|
||||||
|
expect PASSED "every split entry skipped (lane not armed)" -- run_split all-skipped
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== the retrace lane's pull-library control (scripts/ci/retrace_pull_library_control.sh)"
|
||||||
|
# A pull-shaped library the nm identity check accepts: a real ELF .so defining no MG_Remote symbol.
|
||||||
|
if command -v cc > /dev/null 2>&1; then
|
||||||
|
printf '%s\n' 'int mobilegl_pull_only(void) { return 1; }' > "${WORK}/pull.c"
|
||||||
|
cc -shared -fPIC -o "${WORK}/pull.so" "${WORK}/pull.c" || { echo "cannot build the stand-in library"; exit 1; }
|
||||||
|
else
|
||||||
|
echo "no cc available; the retrace half of this smoke test needs one" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
: > "${WORK}/frozen.so"
|
||||||
|
|
||||||
|
# THE FINDING, part (b): a regex matching no tests. --no-tests=error exits non-zero and the old
|
||||||
|
# control read that as "the pull library turned it red".
|
||||||
|
expect FAILED "empty selection (--no-tests=error exit)" -- run_retrace retrace-noselect
|
||||||
|
# A red that never names the transport: a fixture failure, a loader failure, a timeout.
|
||||||
|
expect FAILED "red without the transport-resolution message" -- run_retrace retrace-unrelated
|
||||||
|
# The real thing - and note the stub emits it CMake-wrapped across two lines, which a line-oriented
|
||||||
|
# grep for the literal sentence would miss.
|
||||||
|
expect PASSED "run_trace_case.cmake's own sentence, wrapped" -- run_retrace retrace-evidence
|
||||||
|
# The pull library replaying green is the failure this control exists to catch.
|
||||||
|
expect FAILED "a pull library passed the split retrace" -- run_retrace retrace-green
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "smoke test: ${passes} passed, ${failures} failed"
|
||||||
|
if [ "${failures}" -gt 0 ]; then
|
||||||
|
echo "CONTROL_SMOKE_TEST_FAILED"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "CONTROL_SMOKE_TEST_OK"
|
||||||
Executable
+49
@@ -0,0 +1,49 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Tally a ctest --output-junit file as "passed failed skipped".
|
||||||
|
|
||||||
|
Split out of .github/workflows/test.yml's negative-control step so that the workflow, the local
|
||||||
|
gate and the control smoke test all count a run the same way.
|
||||||
|
|
||||||
|
WHY THIS EXISTS AT ALL (ID-46 finding 8, second half). The counter this replaces lived inline in
|
||||||
|
the workflow and counted a case as having "run" when it was merely not <skipped/>:
|
||||||
|
|
||||||
|
if case.find('skipped') is None and case.get('status') not in ('notrun', 'disabled'):
|
||||||
|
ran += 1
|
||||||
|
|
||||||
|
so a case that RAN AND FAILED armed the negative controls below it. Combined with the `|| true`
|
||||||
|
that hid the baseline's exit code, a lane in which every split entry was already red reported
|
||||||
|
itself armed, and a control that turns an already-red entry red then "passed". Passed, failed and
|
||||||
|
skipped are three different answers and the caller needs all three.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import xml.etree.ElementTree as ET
|
||||||
|
|
||||||
|
|
||||||
|
def tally(path):
|
||||||
|
passed = failed = skipped = 0
|
||||||
|
for case in ET.parse(path).getroot().iter('testcase'):
|
||||||
|
if case.find('failure') is not None or case.find('error') is not None:
|
||||||
|
failed += 1
|
||||||
|
elif case.find('skipped') is not None or case.get('status') in ('notrun', 'disabled'):
|
||||||
|
skipped += 1
|
||||||
|
else:
|
||||||
|
passed += 1
|
||||||
|
return passed, failed, skipped
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
if len(sys.argv) != 2:
|
||||||
|
print("usage: junit_tally.py <junit.xml>", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
try:
|
||||||
|
passed, failed, skipped = tally(sys.argv[1])
|
||||||
|
except Exception as exc: # a malformed file is not "zero of everything"
|
||||||
|
print(f"junit_tally: cannot parse {sys.argv[1]}: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
print(f"{passed} {failed} {skipped}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
raise SystemExit(main())
|
||||||
Executable
+71
@@ -0,0 +1,71 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# R-16's "I made it red once, by doing X" for scripts/ci/control_smoke_test.sh, mechanised so the
|
||||||
|
# claim can be re-checked rather than believed.
|
||||||
|
#
|
||||||
|
# X = revert the message check in each control, i.e. put the controls back in the state ID-46
|
||||||
|
# finding 8 found them in: a non-zero ctest exit is accepted whatever the failure was.
|
||||||
|
#
|
||||||
|
# The smoke test must then FAIL, and it must fail on the two cases that exist for this defect -
|
||||||
|
# "unrelated failure with a non-empty selection" and "red without the transport-resolution
|
||||||
|
# message" - and not merely somewhere. A smoke test that goes red for any other reason when the
|
||||||
|
# evidence check is removed would not be pinning the evidence check.
|
||||||
|
set -u
|
||||||
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
WORK="$(mktemp -d)"
|
||||||
|
trap 'cp "${WORK}/split.orig" "${HERE}/split_negative_controls.sh"; cp "${WORK}/retrace.orig" "${HERE}/retrace_pull_library_control.sh"; rm -rf "${WORK}"' EXIT
|
||||||
|
|
||||||
|
cp "${HERE}/split_negative_controls.sh" "${WORK}/split.orig"
|
||||||
|
cp "${HERE}/retrace_pull_library_control.sh" "${WORK}/retrace.orig"
|
||||||
|
|
||||||
|
echo "=== baseline: the smoke test must be GREEN before anything is perturbed"
|
||||||
|
if ! bash "${HERE}/control_smoke_test.sh" > "${WORK}/before.log" 2>&1; then
|
||||||
|
echo "the smoke test is ALREADY RED; the red-check below would prove nothing"
|
||||||
|
cat "${WORK}/before.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
tail -1 "${WORK}/before.log"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== perturbation: remove the evidence check from both controls"
|
||||||
|
python3 - "${HERE}/split_negative_controls.sh" "${HERE}/retrace_pull_library_control.sh" <<'PY'
|
||||||
|
import sys
|
||||||
|
split, retrace = sys.argv[1], sys.argv[2]
|
||||||
|
for path, needle in ((split, 'grep -qE "${evidence}"'), (retrace, 'grep -qF "${EVIDENCE}"')):
|
||||||
|
text = open(path).read()
|
||||||
|
out, hit = [], 0
|
||||||
|
for line in text.splitlines(keepends=True):
|
||||||
|
if needle in line and line.lstrip().startswith('if ! '):
|
||||||
|
indent = line[:len(line) - len(line.lstrip())]
|
||||||
|
out.append(f"{indent}if false; then\n")
|
||||||
|
hit += 1
|
||||||
|
else:
|
||||||
|
out.append(line)
|
||||||
|
if hit != 1:
|
||||||
|
raise SystemExit(f"expected exactly one evidence check in {path}, found {hit}")
|
||||||
|
open(path, 'w').write(''.join(out))
|
||||||
|
print("both evidence checks reverted to 'any non-zero ctest exit is accepted'")
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== the smoke test on the reverted controls (it MUST be red, on those two cases)"
|
||||||
|
bash "${HERE}/control_smoke_test.sh" > "${WORK}/after.log" 2>&1
|
||||||
|
rc=$?
|
||||||
|
cat "${WORK}/after.log"
|
||||||
|
|
||||||
|
if [ "${rc}" -eq 0 ]; then
|
||||||
|
echo
|
||||||
|
echo "RED-CHECK FAILED: the controls accept an unrelated failure again and the smoke test still passed."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
missed=0
|
||||||
|
grep -q "NOT OK unrelated failure with a non-empty selection" "${WORK}/after.log" || missed=1
|
||||||
|
grep -q "NOT OK red without the transport-resolution message" "${WORK}/after.log" || missed=1
|
||||||
|
if [ "${missed}" -ne 0 ]; then
|
||||||
|
echo
|
||||||
|
echo "RED-CHECK FAILED: the smoke test went red, but not on the two cases the evidence check exists for."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "P5_T1_CONTROL_SMOKE_REDCHECK_OK - removing the evidence check reds exactly the two cases that pin it"
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# THE RETRACE-SPLIT LANE'S NEGATIVE CONTROL: a PULL library must red this split retrace.
|
||||||
|
#
|
||||||
|
# This file is the body of .github/workflows/test.yml's "Negative control - the PULL library must
|
||||||
|
# red this split retrace" step, extracted for the reason given at the top of
|
||||||
|
# scripts/ci/split_negative_controls.sh: a `run:` block is unreviewable and untestable off a
|
||||||
|
# runner, and scripts/ci/control_smoke_test.sh now runs THIS file rather than a hand-made copy.
|
||||||
|
#
|
||||||
|
# WHAT THE CONTROL IS FOR. The retrace-split job replays a trace against the SPLIT runtime under
|
||||||
|
# MOBILEGL_TRANSPORT=inproc. OpenRA scores ssim 1.000000 against a MONOLITH library too - measured -
|
||||||
|
# so the picture is not and cannot be this lane's gate. What stands between the job and a green that
|
||||||
|
# ran monolith end to end is run_trace_case.cmake's transport-resolution assertion
|
||||||
|
# (run_trace_case.cmake:265-289): the library must have logged
|
||||||
|
# "MOBILEGL_TRANSPORT=inproc - the MGPipe record stream", which exists only in
|
||||||
|
# ConfigLoader::InitTransport's InProcess arm, which exists only under MOBILEGL_BUILD_DISAGGREGATED.
|
||||||
|
# This control swaps the pull library over the frozen path and requires the same replay to fail FOR
|
||||||
|
# THAT REASON.
|
||||||
|
#
|
||||||
|
# WHAT THE REVIEW FOUND (ID-46 finding 8 part (b), CONFIRMED by execution against the REAL ctest in
|
||||||
|
# a REAL build tree; ID-48 assigns it here). Two holes, both of which let the control pass while
|
||||||
|
# asserting nothing:
|
||||||
|
#
|
||||||
|
# 1. NO SELECTION GUARD AT ALL - unlike the split lane's run_control, which has had one since
|
||||||
|
# review finding M-4. With a case/backend regex matching no tests, `--no-tests=error` exits 8,
|
||||||
|
# and the old `if [ "${control_rc}" -eq 0 ]` accepted 8 as "the pull library turned it red".
|
||||||
|
# The verifier measured exactly that: "real ctest exit for a regex matching NO tests: 8",
|
||||||
|
# HARNESS_EXIT=0. An empty selection was the one thing this control could not tell apart from
|
||||||
|
# a working transport-identity assertion.
|
||||||
|
# 2. ONLY "non-zero ctest" WAS CHECKED after the nm identity check. The nm check establishes that
|
||||||
|
# the library IS a pull build; it says nothing about why the replay failed. A loader failure, a
|
||||||
|
# missing fixture or a timeout all passed the control.
|
||||||
|
#
|
||||||
|
# Both are closed below: the selection is counted before the run, and the red must carry
|
||||||
|
# run_trace_case.cmake's own words.
|
||||||
|
#
|
||||||
|
# Usage: retrace_pull_library_control.sh <case> <backend>
|
||||||
|
# CTEST ctest binary (default: ctest)
|
||||||
|
# CONTROL_TMPDIR scratch dir (default: ${RUNNER_TEMP:-/tmp})
|
||||||
|
# PULL_LIBRARY the pull libMobileGL.so to swap in
|
||||||
|
# FROZEN_LIBRARY the path every case has baked in, which PULL_LIBRARY is copied over
|
||||||
|
set -u
|
||||||
|
|
||||||
|
CASE="${1:?usage: retrace_pull_library_control.sh <case> <backend>}"
|
||||||
|
BACKEND="${2:?usage: retrace_pull_library_control.sh <case> <backend>}"
|
||||||
|
|
||||||
|
CTEST="${CTEST:-ctest}"
|
||||||
|
CONTROL_TMPDIR="${CONTROL_TMPDIR:-${RUNNER_TEMP:-/tmp}}"
|
||||||
|
PULL_LIBRARY="${PULL_LIBRARY:?PULL_LIBRARY must name the pull build libMobileGL.so}"
|
||||||
|
FROZEN_LIBRARY="${FROZEN_LIBRARY:?FROZEN_LIBRARY must name the path the cases have baked in}"
|
||||||
|
mkdir -p "${CONTROL_TMPDIR}"
|
||||||
|
|
||||||
|
# run_trace_case.cmake's own sentence for "this library never resolved the transport". Anchored on
|
||||||
|
# the distinctive clause rather than on the whole paragraph, which carries substituted paths.
|
||||||
|
EVIDENCE='never reported resolving it'
|
||||||
|
|
||||||
|
selector="^MobileGLTraceReplay\.${CASE}\.${BACKEND}$"
|
||||||
|
|
||||||
|
# The rerun replays into the same case directory, so the good run's images are put aside and
|
||||||
|
# restored whichever way the control goes; "Upload actual image" runs `if: always()` and would
|
||||||
|
# otherwise ship the deliberately-wrong run's output under the good run's name.
|
||||||
|
GOOD_OUTPUT="${CONTROL_TMPDIR}/split-verified-output"
|
||||||
|
rm -rf "${GOOD_OUTPUT}"
|
||||||
|
if [ -d "${CASE}" ]; then cp -a "${CASE}" "${GOOD_OUTPUT}"; fi
|
||||||
|
|
||||||
|
restore_good_output() {
|
||||||
|
if [ -d "${GOOD_OUTPUT}" ]; then
|
||||||
|
rm -rf "${CASE}"; mv "${GOOD_OUTPUT}" "${CASE}"
|
||||||
|
echo "restored the verified run's output over the control's"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# HOLE 1: COUNT THE SELECTION FIRST. `--no-tests=error` turns an empty selection into a non-zero
|
||||||
|
# exit, which is indistinguishable from a working control unless the selection is counted.
|
||||||
|
matched=$("${CTEST}" -N -R "${selector}" | grep -cE '^ *Test *#[0-9]+:')
|
||||||
|
if [ "${matched}" -lt 1 ]; then
|
||||||
|
restore_good_output
|
||||||
|
echo "::error::the control selected ${matched} tests with -R '${selector}', so there is nothing for the pull library to red. --no-tests=error would have exited non-zero on the empty selection and this control used to read that as success (ID-46 finding 8b, measured: ctest exit 8, step green)."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The pull library, unpacked from build-linux's artifact, over the frozen path every case has baked
|
||||||
|
# in. It defines no MG_Remote symbol, so ConfigLoader has no transport parser and
|
||||||
|
# MOBILEGL_TRANSPORT=inproc is accepted and ignored - the exact shape of "the split lane ran
|
||||||
|
# monolith".
|
||||||
|
cp "${PULL_LIBRARY}" "${FROZEN_LIBRARY}"
|
||||||
|
if nm --defined-only "${FROZEN_LIBRARY}" | grep -q -i MG_Remote; then
|
||||||
|
restore_good_output
|
||||||
|
echo "::error::the control's own library defines MG_Remote symbols, so it is not a pull build and this control would prove nothing"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
out="${CONTROL_TMPDIR}/retrace-control-output.txt"
|
||||||
|
export MOBILEGL_TRANSPORT=inproc
|
||||||
|
"${CTEST}" -V --no-tests=error --timeout 10800 -R "${selector}" > "${out}" 2>&1
|
||||||
|
control_rc=$?
|
||||||
|
cat "${out}"
|
||||||
|
|
||||||
|
restore_good_output
|
||||||
|
|
||||||
|
if [ "${control_rc}" -eq 0 ]; then
|
||||||
|
echo "::error::a PULL library passed the split retrace. OpenRA scores ssim 1.000000 under a monolith library too (measured), so the picture is not and cannot be this lane's gate - run_trace_case.cmake's transport-resolution assertion is, and it has stopped working. Every green in this job is then a monolith run under a name that says split."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# HOLE 2: THE RED MUST BE THIS CONTROL'S RED.
|
||||||
|
#
|
||||||
|
# Whitespace is normalised across the WHOLE file before the match, newlines included, because the
|
||||||
|
# sentence is emitted by CMake's message(FATAL_ERROR ...) and CMake re-wraps that text to its own
|
||||||
|
# width: "never reported resolving it" arrives split over two lines with a two-space continuation
|
||||||
|
# indent, and a line-oriented grep for the literal finds nothing. That is not hypothetical - it is
|
||||||
|
# the shape the stub reproduces in scripts/ci/testdata/stub_ctest.sh.
|
||||||
|
if ! tr -s '[:space:]' ' ' < "${out}" | grep -qF "${EVIDENCE}"; then
|
||||||
|
echo "::error::the split retrace went red (ctest exit ${control_rc}) with the pull library in place, but the failure never says the library did not resolve the transport - run_trace_case.cmake's \"${EVIDENCE}\" is absent from the output. A loader failure, a missing fixture, a timeout or an SSIM drop all land here, and none of them establishes that the transport-identity assertion is what caught the pull library. Only 'non-zero ctest' used to be checked (ID-46 finding 8b)."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "the pull library turned the split retrace red for its own reason (ctest exit ${control_rc}): ${matched} selected case(s) named the transport, not the picture"
|
||||||
Executable
+151
@@ -0,0 +1,151 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# EXIT GATE E1's NEGATIVE CONTROL and EXIT GATE E3(a)'s.
|
||||||
|
#
|
||||||
|
# This file is the body of .github/workflows/test.yml's "Negative controls - the verb barrier and
|
||||||
|
# the persistent-map push must be load-bearing" step. It lives in the repository rather than inline
|
||||||
|
# in the workflow for one reason: a workflow `run:` block cannot be executed anywhere except on a
|
||||||
|
# runner, so the logic below was unreviewable and untestable until it ran in CI - and when the
|
||||||
|
# wave-1 cross-family review claimed it was broken, confirming the claim needed a hand-made copy of
|
||||||
|
# these lines with their inputs stubbed (wave1-codex-verify.md 8). A copy is not the thing. The
|
||||||
|
# smoke test at scripts/ci/control_smoke_test.sh now runs THIS file, so the lines CI executes and
|
||||||
|
# the lines the smoke test proves are the same lines.
|
||||||
|
#
|
||||||
|
# WHAT THE REVIEW FOUND (ID-46 finding 8, CONFIRMED by execution; ID-48 assigns it here).
|
||||||
|
# The previous version accepted ANY non-zero ctest exit as "the knob is load-bearing". A timeout, a
|
||||||
|
# setup abort, an unrelated assertion, a harness that died before it read the knob at all - every
|
||||||
|
# one of them printed "turned N selected entries red, as it must" and the step went green. The
|
||||||
|
# verifier demonstrated it: a stubbed ctest with a NON-EMPTY selection that failed with
|
||||||
|
# `UNRELATED_CONTROL_FAILURE` produced both controls' success messages and HARNESS_EXIT=0.
|
||||||
|
#
|
||||||
|
# So each control now has to say WHY the red is its own:
|
||||||
|
#
|
||||||
|
# 1. THE BASELINE MUST BE GREEN. The arming run below used to end in `|| true` and count every
|
||||||
|
# case that was not <skipped/> as "ran" - so a case that RAN AND FAILED armed the controls,
|
||||||
|
# and a control that turns an already-red entry red proves nothing at all. It now counts
|
||||||
|
# PASSED cases, and a baseline with any failure in it is a hard error rather than an arming
|
||||||
|
# signal.
|
||||||
|
# 2. THE RED MUST CARRY THE SELECTED CASE'S OWN FAILURE TEXT. Each control names a regex of the
|
||||||
|
# diagnostics its scenarios emit when that knob is off, and the red is refused if the output
|
||||||
|
# carries none of them.
|
||||||
|
#
|
||||||
|
# WHY THE EVIDENCE IS THE SCENARIO'S ASSERTION TEXT AND NOT THE KNOB'S OWN LOG LINE.
|
||||||
|
# ConfigLoader logs a named line for both knobs (ConfigLoader.cpp:385-393, "is the R-1 NEGATIVE
|
||||||
|
# CONTROL", "is the E3(a) NEGATIVE CONTROL"), and it is tempting to grep for that. It is not
|
||||||
|
# evidence: it is written at config load, by every process in the run, whatever happens next. A
|
||||||
|
# setup abort would carry it too. It proves the knob was READ, never that the knob caused the red.
|
||||||
|
# Only the failing case's own diagnostic does that. (It is also unreachable from here: the three
|
||||||
|
# DirectGLES.Split. lanes set no MOBILEGL_LOG_FILE_PATH, and the library's console sink is compiled
|
||||||
|
# out of this configuration, so no MGLOG_ output of any level reaches ctest's transcript. Measured:
|
||||||
|
# ~/w7/p5-v1-joint-isplit-barrier0.log, 18 aborted entries, zero occurrences of the string "Fatal".)
|
||||||
|
#
|
||||||
|
# Usage: split_negative_controls.sh
|
||||||
|
# CTEST ctest binary (default: ctest)
|
||||||
|
# CONTROL_TMPDIR scratch dir for the junit + output (default: ${RUNNER_TEMP:-/tmp})
|
||||||
|
set -u
|
||||||
|
|
||||||
|
CTEST="${CTEST:-ctest}"
|
||||||
|
CONTROL_TMPDIR="${CONTROL_TMPDIR:-${RUNNER_TEMP:-/tmp}}"
|
||||||
|
mkdir -p "${CONTROL_TMPDIR}"
|
||||||
|
|
||||||
|
junit="${CONTROL_TMPDIR}/isplit.xml"
|
||||||
|
|
||||||
|
# ---- the baseline ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# THE ARMED STATE IS DERIVED FROM BEHAVIOUR, not from a marker string in the generated ctest files.
|
||||||
|
# The first version read MGITEST_REMOTE_CLIENT_PRESENT out of *_tests.cmake, which was a
|
||||||
|
# restatement of the CMake source probe review finding M-1 falsified; the arming condition is a
|
||||||
|
# runtime fact inside each test process (MG_Config::Transport, ClientSession::Active(),
|
||||||
|
# ImplementedVerbCount(), read by Harness/SplitRuntimePeek), so the only honest way to ask it from a
|
||||||
|
# shell is to look at what the entries DID.
|
||||||
|
"${CTEST}" -L integration-split -j 4 --no-tests=error --output-junit "${junit}"
|
||||||
|
baseline_rc=$?
|
||||||
|
|
||||||
|
if [ ! -f "${junit}" ]; then
|
||||||
|
echo "::error::the baseline run wrote no ${junit} (ctest exit ${baseline_rc}), so nothing below can tell an armed lane from a broken one"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
tally=$(python3 "$(dirname "$0")/junit_tally.py" "${junit}")
|
||||||
|
if [ -z "${tally}" ]; then
|
||||||
|
echo "::error::could not tally ${junit}; a run whose result cannot be read is not an arming signal"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
baseline_passed=$(echo "${tally}" | cut -d' ' -f1)
|
||||||
|
baseline_failed=$(echo "${tally}" | cut -d' ' -f2)
|
||||||
|
baseline_skipped=$(echo "${tally}" | cut -d' ' -f3)
|
||||||
|
|
||||||
|
echo "split entries - passed: ${baseline_passed}, failed: ${baseline_failed}, skipped: ${baseline_skipped} (ctest exit ${baseline_rc})"
|
||||||
|
|
||||||
|
# A RED BASELINE DISARMS THE CONTROLS RATHER THAN ARMING THEM (review finding 8, second half).
|
||||||
|
# `|| true` plus a "not skipped" counter used to treat a case that ran and FAILED as evidence the
|
||||||
|
# lane was live. Turning an already-red entry red is not a measurement.
|
||||||
|
if [ "${baseline_failed}" -gt 0 ]; then
|
||||||
|
echo "::error::${baseline_failed} DirectGLES.Split. entries are ALREADY RED with both knobs at their defaults, so neither negative control below can attribute its red to the knob it turns. Fix the lane first; a control measured against a red baseline is not a control. (This used to be swallowed by an unconditional '|| true' and counted as 'the lane is armed'.)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${baseline_passed}" -lt 1 ]; then
|
||||||
|
echo "::warning::every DirectGLES.Split. entry SKIPPED, so neither negative control can fire. The arming condition is a runtime fact - MG_Config::Transport, ClientSession::Active() and ImplementedVerbCount(), read by Harness/SplitRuntimePeek - and it becomes true on the commit that lands the last of c1/s1/v1. This step becomes a gate then, with no edit; it is not a green that asserted anything today."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---- the controls ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# run_control <name> <filter> <evidence regex> <VAR=VALUE>...
|
||||||
|
run_control() {
|
||||||
|
name="$1"; filter="$2"; evidence="$3"; shift 3
|
||||||
|
|
||||||
|
matched=$("${CTEST}" -N -L integration-split -R "${filter}" | grep -cE '^ *Test *#[0-9]+:')
|
||||||
|
if [ "${matched}" -lt 1 ]; then
|
||||||
|
echo "::error::${name} selected ${matched} tests; its filter no longer matches anything"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
out="${CONTROL_TMPDIR}/control-output.txt"
|
||||||
|
env "$@" "${CTEST}" --output-on-failure -L integration-split -R "${filter}" --no-tests=error > "${out}" 2>&1
|
||||||
|
control_rc=$?
|
||||||
|
cat "${out}"
|
||||||
|
|
||||||
|
if [ "${control_rc}" -eq 0 ]; then
|
||||||
|
echo "::error::${name} left ${matched} split entries GREEN, so the knob it turns is not load-bearing and the gate it controls proves nothing."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# THE HALF THAT WAS MISSING. A non-zero exit is necessary and nowhere near sufficient.
|
||||||
|
# Whitespace is normalised across the whole file first, for the reason given in
|
||||||
|
# retrace_pull_library_control.sh: a diagnostic that arrives wrapped is still the diagnostic.
|
||||||
|
if ! tr -s '[:space:]' ' ' < "${out}" | grep -qE "${evidence}"; then
|
||||||
|
echo "::error::${name} turned ${matched} selected entries red, but the red carries NONE of the diagnostics those scenarios emit when this knob is off, so it is not this control's red. Required one of: ${evidence}. A timeout, a setup abort, a harness that died before it read the knob, or any unrelated assertion lands here - and every one of them used to print the success message below and leave this step green (ID-46 finding 8). If the entries aborted with no output at all, that is the barrier path having no named diagnostic of its own: see t1-v2.md, it is a debt on the server package, not a reason to accept the red."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "${name} turned ${matched} selected entries red, and the red carries the scenario's own diagnostic, as it must"
|
||||||
|
}
|
||||||
|
|
||||||
|
# E1: R-1's lockstep verb barrier. Without it the client keeps pulling fields from a live GLContext
|
||||||
|
# while the server runs ahead, so the server reads future values.
|
||||||
|
#
|
||||||
|
# THE SELECTION INCLUDES THE SmallRing LANE, and that is not cosmetic. Measured on v1's joint tree
|
||||||
|
# (~/w7/p5-v1-joint-isplit-barrier0.log): with the barrier off, every entry the OLD filter selected
|
||||||
|
# aborted with no output whatsoever, and the one entry in the whole run that failed with a readable
|
||||||
|
# assertion - ClearThenReadPixelsScenario.cpp:290/:295, reading back 0 where >200 was cleared - was
|
||||||
|
# a DirectGLES.Split.SmallRing. entry, which the old filter excluded. A control whose selection
|
||||||
|
# contains no case able to say why it failed cannot assert its own failure reason. The SmallRing
|
||||||
|
# entries are the same two scenarios under the same transport with SEG_CMD/SEG_STAGE at their floor,
|
||||||
|
# so including them widens E1's selection strictly within E1's charter.
|
||||||
|
run_control "negative control E1 (MOBILEGL_IPC_VERB_BARRIER=0)" \
|
||||||
|
'DirectGLES\.Split\.(SmallRing\.)?(Triangle|ClearThenReadPixels)' \
|
||||||
|
'ClearThenReadPixelsScenario\.cpp:(290|295)|the bottom band should be red after the resolve|the top band should be blue after the resolve|TriangleScenario\.cpp:[0-9]+: Failure' \
|
||||||
|
MOBILEGL_IPC_VERB_BARRIER=0
|
||||||
|
|
||||||
|
# E3(a): the persistent-map push. 0 is admitted by ConfigLoader on purpose and is documented there
|
||||||
|
# as this control. The evidence is the scenario's own wording for "the second write never arrived":
|
||||||
|
# with the push off, the write that no GL call announces cannot reach its draw, which is exactly
|
||||||
|
# what TwoWritesThroughTheCoherentPointerEachReachTheirOwnDraw and
|
||||||
|
# AWriteAfterAFrameBoundaryReachesTheNextFramesDraw read back
|
||||||
|
# (PersistentCoherentMapScenario.cpp:414-417, :442-443). The counting case's pmap= assertion
|
||||||
|
# (:531-541) is listed too, for the lane where it is the one that runs.
|
||||||
|
run_control "negative control E3(a) (MOBILEGL_IPC_PERSISTENT_BLOCK_KB=0)" \
|
||||||
|
'DirectGLES\.Split\.(SmallRing\.)?PersistentCoherentMapScenario' \
|
||||||
|
"the SECOND write through the same mapping, announced by nothing|frame 1's write through the SAME mapping|cannot have pushed|PersistentCoherentMapScenario\.cpp:[0-9]+: Failure" \
|
||||||
|
MOBILEGL_IPC_PERSISTENT_BLOCK_KB=0
|
||||||
+119
@@ -0,0 +1,119 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# A stubbed `ctest` for scripts/ci/control_smoke_test.sh.
|
||||||
|
#
|
||||||
|
# Descended from the verification agent's stub (wave1-codex-verify.md 8, ~/w7/p5-verify-f8-stub/ctest),
|
||||||
|
# which is what CONFIRMED that the negative controls accepted an unrelated failure. Every mode below
|
||||||
|
# is deliberately the BEST case for the control under test: the selection is never empty except in
|
||||||
|
# the mode that exists to test the empty-selection guard, and the baseline is green except in the
|
||||||
|
# mode that exists to test the red-baseline guard. If a control passes here it is because the
|
||||||
|
# control's logic is wrong, not because the stub starved it.
|
||||||
|
#
|
||||||
|
# STUB_MODE:
|
||||||
|
# unrelated baseline green; the control's own run fails with UNRELATED_CONTROL_FAILURE
|
||||||
|
# evidence baseline green; the control's own run fails with the scenarios' own wording
|
||||||
|
# green baseline green; the control's own run PASSES (the knob is not load-bearing)
|
||||||
|
# red-baseline the baseline itself has a failed entry
|
||||||
|
# all-skipped the baseline is entirely skipped (the disarmed lane, a legitimate exit 0)
|
||||||
|
# retrace-noselect `ctest -N` matches nothing; the run exits 8 the way --no-tests=error does
|
||||||
|
# retrace-unrelated one match; the run fails without naming the transport
|
||||||
|
# retrace-evidence one match; the run fails with run_trace_case.cmake's own sentence
|
||||||
|
# retrace-green one match; the run PASSES
|
||||||
|
set -u
|
||||||
|
|
||||||
|
mode="${STUB_MODE:?STUB_MODE must be set}"
|
||||||
|
|
||||||
|
listing=1
|
||||||
|
junit=""
|
||||||
|
prev=""
|
||||||
|
for a in "$@"; do
|
||||||
|
[ "$a" = "-N" ] && listing_requested=1
|
||||||
|
if [ "$prev" = "--output-junit" ]; then junit="$a"; fi
|
||||||
|
prev="$a"
|
||||||
|
done
|
||||||
|
listing_requested="${listing_requested:-0}"
|
||||||
|
|
||||||
|
emit_listing() {
|
||||||
|
echo "Test project /stub"
|
||||||
|
if [ "${mode}" = "retrace-noselect" ]; then
|
||||||
|
echo "Total Tests: 0"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
echo " Test #1: DirectGLES.Split.ClearThenReadPixelsScenario.ClearWithNoDrawIsVisibleToDefaultFramebufferReadPixels"
|
||||||
|
echo "Total Tests: 1"
|
||||||
|
}
|
||||||
|
|
||||||
|
write_junit() {
|
||||||
|
case "${mode}" in
|
||||||
|
red-baseline)
|
||||||
|
body='<testcase name="DirectGLES.Split.TriangleScenario.AVboBackedTriangleReachesReadPixels" status="failed"><failure message="already red"/></testcase>'
|
||||||
|
;;
|
||||||
|
all-skipped)
|
||||||
|
body='<testcase name="DirectGLES.Split.TriangleScenario.AVboBackedTriangleReachesReadPixels" status="notrun"><skipped/></testcase>'
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
body='<testcase name="DirectGLES.Split.TriangleScenario.AVboBackedTriangleReachesReadPixels" status="run" time="0.3"/>'
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
printf '%s\n' '<?xml version="1.0" encoding="UTF-8"?>' "<testsuite name=\"stub\">" " ${body}" '</testsuite>' > "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ "${listing_requested}" = "1" ]; then
|
||||||
|
emit_listing
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -n "${junit}" ]; then
|
||||||
|
write_junit "${junit}"
|
||||||
|
case "${mode}" in
|
||||||
|
red-baseline) echo "1/1 Test #1: ... ***Failed"; exit 8 ;;
|
||||||
|
*) echo "100% tests passed, 0 tests failed out of 1"; exit 0 ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The control's own run.
|
||||||
|
case "${mode}" in
|
||||||
|
unrelated)
|
||||||
|
echo "1/1 Test #1: DirectGLES.Split.ClearThenReadPixelsScenario.ClearWithNoDrawIsVisibleToDefaultFramebufferReadPixels ...***Failed"
|
||||||
|
echo "UNRELATED_CONTROL_FAILURE: the harness aborted in setup before the knob was read"
|
||||||
|
exit 8
|
||||||
|
;;
|
||||||
|
evidence)
|
||||||
|
# Both controls' required wording, so one stub serves E1 and E3(a). Copied from the real
|
||||||
|
# diagnostics: ~/w7/p5-v1-joint-isplit-barrier0.log for the first, and
|
||||||
|
# PersistentCoherentMapScenario.cpp:414-417 for the second.
|
||||||
|
echo "1/1 Test #1: DirectGLES.Split.ClearThenReadPixelsScenario.ClearWithNoDrawIsVisibleToDefaultFramebufferReadPixels ...***Failed"
|
||||||
|
echo "../MobileGL/MG_IntegrationTest/Scenarios/ClearThenReadPixelsScenario.cpp:290: Failure"
|
||||||
|
echo "Expected: (bottom.r) > (200), actual: '\\0' vs 200"
|
||||||
|
echo "the SECOND write through the same mapping, announced by nothing: this is exit gate E3(b)"
|
||||||
|
exit 8
|
||||||
|
;;
|
||||||
|
green)
|
||||||
|
echo "100% tests passed, 0 tests failed out of 1"
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
retrace-noselect)
|
||||||
|
echo "No tests were found!!!"
|
||||||
|
exit 8
|
||||||
|
;;
|
||||||
|
retrace-unrelated)
|
||||||
|
echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed"
|
||||||
|
echo "CMake Error: the fixture could not be unpacked"
|
||||||
|
exit 8
|
||||||
|
;;
|
||||||
|
retrace-evidence)
|
||||||
|
echo "1/1 Test #1: MobileGLTraceReplay.OpenRA.DirectGLES ...***Failed"
|
||||||
|
echo "CMake Error at run_trace_case.cmake:279 (message):"
|
||||||
|
echo " MOBILEGL_TRANSPORT=inproc is set for OpenRA DirectGLES and the library never"
|
||||||
|
echo " reported resolving it: mobilegl.log carries no"
|
||||||
|
echo ' "MOBILEGL_TRANSPORT=inproc - the MGPipe record stream".'
|
||||||
|
exit 8
|
||||||
|
;;
|
||||||
|
retrace-green)
|
||||||
|
echo "100% tests passed, 0 tests failed out of 1"
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "stub_ctest: unknown STUB_MODE '${mode}'" >&2
|
||||||
|
exit 127
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Reference in New Issue
Block a user