[MG_Remote, MG_Pipe, MG_Test, CI, tools, docs] (P6.5): the first wave completed and reviewed - archive v2, publication boundary, half-close, AcceptPair/role fixes, device re-validation

This commit is contained in:
2026-09-22 10:56:19 -04:00
parent 018fab0f5b
commit fe28bdb5f3
75 changed files with 4887 additions and 254 deletions
+107
View File
@@ -403,3 +403,110 @@ the ANGLE the Android lane runs. The CI lane uses a pinned build
extension set differ from the SDK copy (`GL_EXT_texture_buffer` support, ES 3.2
entry points). Compare `GL_RENDERER` and the relevant extension lists on both
stacks before treating a local result as a statement about CI.
## TCP matrix driver (Linux / WSL)
`run_tcp_matrix.py` uses a CTest JSON catalog plus the **current** `trace_cases.json`.
It does not build, install an APK, or launch the remote server. For the formal
P6.5 run-ahead matrix, use `--require-run-ahead`. It forces `MOBILEGL_IPC_RUN_AHEAD=1`
and `MOBILEGL_IPC_VERB_BARRIER=1`, then requires `run-ahead ARMED` in each private
client log and rejects any `running lockstep` / `DISARMED` marker. The checkpoint
records `requested_arm` and `actual_arm`; strict resume cannot reuse an earlier
lockstep image pass. Start the intended server first, then run serially:
```bash
ctest --test-dir /path/to/retrace-build --show-only=json-v1 > /tmp/trace-catalog.json
python3 tools/trace_replay/run_tcp_matrix.py \
--catalog /tmp/trace-catalog.json \
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
--endpoint tcp://192.168.21.181:40613 --token devtoken \
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix
# Continue the same frozen library/runner/peer:
python3 tools/trace_replay/run_tcp_matrix.py \
--catalog /tmp/trace-catalog.json \
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
--endpoint tcp://192.168.21.181:40613 --token devtoken \
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix --resume
```
`--build-dir` can replace `--catalog` to obtain the JSON without building.
`--runner` overrides the catalog runner; fixture paths default to the catalog's
fixture directory and can be overridden with `--fixtures`. Repeat `--case` or
`--backend` to select a subset. The default is the manifest's CI split subset
(currently 39 cases), honoring each case's backend list. Explicit `--case
minecraft-1.21.4-rd12-odinlite-in-world` permits that non-CI workload separately;
an explicit `split:false` always remains excluded.
The watchdog defaults to **300 seconds without log progress** and a separate
**7200-second absolute ceiling** (`--idle-seconds`, `--max-seconds`). It observes
runner output and trace/client/server log changes. It deliberately ignores the
old local CTest/manifest `timeout_seconds`: TCP main-menu kept progressing beyond
its old local 180-second budget. A growing log does not bypass the explicit absolute ceiling.
On timeout the driver terminates and reaps the entire case process group,
including a child that ignores TERM after CMake exits, and stops the matrix to
avoid cascading Busy failures. Check the peer before resuming. Ctrl-C/TERM/HUP
also clean up the group and save a cancelled checkpoint.
Each attempt has a new directory. `checkpoint.json` is written atomically, and
only a zero exit plus current-attempt successful result, matching backend/call/
golden, adequate SSIM, an actual image, and the requested TCP arm marker count
as passed. Resume also checks request/artifact identity and stored evidence
hashes. Failed, timed-out, interrupted, changed, or missing evidence is rerun;
old scratch `results.json` files are retained but never silently promoted into
successful checkpoints. Replacing the remote deployment requires a new output
root unless it is the same frozen peer. `results.json` summarizes the currently
selected runs and marks reused evidence with `resumed:true`.
`--wake-adb-serial 2f7cbe2e` explicitly opts into `adb shell input keyevent
KEYCODE_WAKEUP` every 15 seconds during a case. It does not install anything or
change global power/idle settings. Its own adb log is excluded from the progress
watchdog, so wakeup messages cannot conceal a stalled trace.
Tool regression tests use short process stubs, never a GPU/device/trace:
```bash
python3 tools/trace_replay/test_run_tcp_matrix.py
```
## TCP credit and Stage measurements
`benchmark_tcp_credits.py` runs the complete OpenRA and rd12 traces at credits
1, 2, and 3, then selects matching client/server tail frame IDs (100 and 200
frames respectively). Extract the unchanged trace files as
`inputs/OpenRA/openra.trace` and `inputs/rd12/trace.trace`. Keep the server and
library frozen, enable server `MOBILEGL_PIPE_STATS=1` and
`MOBILEGL_PIPE_STATS_PERIOD=1`, and leave the device connection free:
```bash
python3 tools/trace_replay/benchmark_tcp_credits.py \
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
--inputs /path/to/inputs --out /tmp/tcp-credits \
--endpoint tcp://192.168.21.181:40613 --token devtoken --serial 2f7cbe2e
python3 tools/trace_replay/measure_tcp_stage_burst.py \
--library /path/to/libMobileGL.so --out /tmp/tcp-stage-burst \
--endpoint tcp://192.168.21.181:40613 --token devtoken
```
The credit runner serializes all six runs, sends the same explicit awake key as
the matrix driver, and preserves complete benchmark and role logs. It requires
the requested credit and continuous run-ahead ARMED proof in the client log,
matching complete tail frames, and positive per-thread CPU observations.
`--case` and `--credit` select a subset; `--resume` reuses only matching identity
and intact evidence. FPS, `kWaitReply` counts, bucketed RTT distribution, Stage
bytes, and the two role CPU clocks come from the selected steady window.
Completing this benchmark does not perform or replace a golden image comparison.
The Stage tool uploads exactly 64 MiB after one warmup frame, consumes the dirty
buffer in a draw, and waits for apply. It requires actual TCP run-ahead and at
least 64 MiB of observed Stage bytes in frame 2, preserves artifact hashes, and
refuses to overwrite an existing output directory. Its throughput includes
client copies and backend work; it is not raw Wi-Fi bandwidth. Divide steady
frame Stage traffic by this separately measured throughput only with that
interpretation. Device matrices, benchmarks, and the burst must run one at a time.
Offline evidence controls (no device):
```bash
python3 tools/trace_replay/test_benchmark_tcp_credits.py
```
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env python3
"""Record complete TCP benchmark runs for credit 1/2/3 with exact common tail frames."""
import argparse
import json
import os
from pathlib import Path
import re
import subprocess
import time
import uuid
from measure_transport_cpu import CASES, digest
from measure_loopback_cpu import frame_rows, METRICS
from run_tcp_matrix import supervise, atomic_json, fingerprint, actual_arm_from_text
def validate_measurement(work, endpoint, credit, tail):
bench = json.loads((work / 'benchmark.json').read_text())
client_path, server_path = work / 'mobilegl.client.log', work / 'mobilegl.server.log'
text = client_path.read_text(errors='replace')
actual_arm = actual_arm_from_text(text)
if (actual_arm != {'armed': True, 'lockstep': False, 'disarmed': False}
or not re.search(r'paced by a present credit of ' + str(credit) + r'\b', text)
or 'control=tcp data=stream server=' + endpoint.removeprefix('tcp://') not in text):
raise ValueError('runtime does not prove TCP and the requested armed credit')
client = frame_rows(client_path, 'P65LinkMetrics ')
server = frame_rows(server_path, 'P65ServerMetrics ', server=True)
ids = sorted(client)
if len(ids) != bench['totalFrames'] or len(ids) <= tail:
raise ValueError('benchmark and frame metrics differ, or tail contains setup')
selected = ids[-tail:]
if selected != list(range(selected[0], selected[-1] + 1)) or any(n not in server for n in selected):
raise ValueError('no matching complete client/server frame tail')
for number in selected:
if int(client[number]['client_thread_cpu_ns']) <= 0 or int(server[number]['apply_thread_cpu_ns']) <= 0:
raise ValueError('nonpositive CPU measurement in selected frame ' + str(number))
summary = METRICS.summarize(client_path, selected[0] - 1)
summary.update(METRICS.summarize_server(server_path, selected[0] - 1))
if summary['frames'] != tail or summary['server_frames'] != tail:
raise ValueError('unexpected frames beyond the chosen tail')
return {'run_ahead_armed': True, 'actual_arm': actual_arm,
'first_frame': selected[0], 'last_frame': selected[-1], 'benchmark': bench,
'benchmark_sha256': digest(work / 'benchmark.json'),
'client_log_sha256': digest(client_path), 'server_log_sha256': digest(server_path), **summary}
def measure(args, case, credit, work):
_, trace_name, width, height, tail = CASES[case]
trace = args.inputs / case / trace_name
work.mkdir(parents=True)
env = dict(os.environ, MOBILEGL_TRANSPORT='spawn', MOBILEGL_IPC_CONTROL=args.endpoint,
MOBILEGL_IPC_DATA='stream', MOBILEGL_IPC_TOKEN=args.token, MOBILEGL_IPC_REQUIRE_SAME_BUILD='1',
MOBILEGL_IPC_LOG_FORWARD='1', MOBILEGL_PIPE_STATS='1', MOBILEGL_PIPE_STATS_PERIOD='1',
MOBILEGL_IPC_PRESENT_CREDIT=str(credit), MOBILEGL_IPC_RUN_AHEAD='1', MOBILEGL_IPC_VERB_BARRIER='1',
MOBILEGL_IPC_BATCH_WAITS='1', MOBILEGL_BACKEND_TYPE='DirectGLES',
MOBILEGL_IPC_STRICT_ERRORS='0', MOBILEGL_IPC_ROLE_SPLIT_STATE='0', MOBILEGL_IPC_AUDIT='0')
for name in ('MOBILEGL_IPC_DIAL', 'MOBILEGL_IPC_SERVER_PATH'):
env.pop(name, None)
command = [str(args.runner), '--trace', str(trace), '--mobilegl-library', str(args.library),
'--backend', 'DirectGLES', '--width', str(width), '--height', str(height),
'--output', str(work), '--benchmark', '--benchmark-finish=0', '--benchmark-tail-frames=' + str(tail)]
def wake():
# Keep maintenance output outside the progress scan: a failed adb wake
# must not make a stalled replay look active.
with (work.parent / 'wake-adb.log').open('ab') as log:
try:
subprocess.run([args.adb, '-s', args.serial, 'shell', 'input', 'keyevent', 'KEYCODE_WAKEUP'],
stdout=log, stderr=log, timeout=5, check=False)
except (OSError, subprocess.TimeoutExpired) as error:
log.write((str(error) + '\n').encode())
result = supervise(command, env, str(args.inputs), work, work / 'runner.log',
idle_seconds=300, max_seconds=7200, wake=wake)
if result['returncode'] or result['timeout_kind'] or result['remaining_processes'] or result['leftovers_after_parent_exit']:
return {**result, 'status': 'failed', 'output': str(work), 'command': command}
actual_arm = actual_arm_from_text((work / 'mobilegl.client.log').read_text(errors='replace'))
try:
values = validate_measurement(work, args.endpoint, credit, tail)
except (OSError, ValueError, KeyError) as error:
return {**result, 'status': 'failed', 'actual_arm': actual_arm,
'error': str(error), 'output': str(work), 'command': command}
return {**result, 'status': 'passed', 'output': str(work), 'command': command, **values}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--runner', type=Path, required=True)
parser.add_argument('--library', type=Path, required=True)
parser.add_argument('--inputs', type=Path, required=True)
parser.add_argument('--out', type=Path, required=True)
parser.add_argument('--endpoint', required=True)
parser.add_argument('--token', default=os.environ.get('MOBILEGL_IPC_TOKEN', ''))
parser.add_argument('--serial', required=True, help='explicit adb serial to keep awake during each run')
parser.add_argument('--adb', default='adb')
parser.add_argument('--case', action='append', choices=CASES)
parser.add_argument('--credit', action='append', type=int, choices=(1, 2, 3))
parser.add_argument('--resume', action='store_true')
args = parser.parse_args()
if not args.endpoint.startswith('tcp://'):
parser.error('a TCP endpoint is required')
args.out.mkdir(parents=True, exist_ok=True)
checkpoint = args.out / 'checkpoint.json'
rows = json.loads(checkpoint.read_text()) if checkpoint.exists() else []
if rows and not args.resume:
parser.error('output already contains evidence; use --resume or a new output directory')
for case in args.case or CASES:
for credit in args.credit or (1, 2, 3):
provenance = {'case': case, 'credit': credit, 'endpoint': args.endpoint,
'token_hash': fingerprint(args.token), 'runner_sha256': digest(args.runner),
'library_sha256': digest(args.library),
'trace_sha256': digest(args.inputs / case / CASES[case][1]),
'script_sha256': digest(Path(__file__)),
'helper_sha256': {name: digest(Path(__file__).with_name(name)) for name in (
'measure_transport_cpu.py', 'measure_loopback_cpu.py', 'run_tcp_matrix.py')},
'metrics_sha256': digest(Path(METRICS.__file__))}
identity = fingerprint(provenance)
prior = next((row for row in reversed(rows) if row.get('identity') == identity and row.get('status') == 'passed'), None)
if args.resume and prior:
directory = Path(prior['output'])
if all((directory / filename).is_file() and digest(directory / filename) == prior[field]
for filename, field in (('benchmark.json', 'benchmark_sha256'), ('mobilegl.client.log', 'client_log_sha256'),
('mobilegl.server.log', 'server_log_sha256'))):
print(json.dumps({'case': case, 'credit': credit, 'status': 'passed', 'resumed': True}), flush=True)
continue
work = args.out / case / f'credit-{credit}' / ('attempt-' + uuid.uuid4().hex)
row = {'case': case, 'credit': credit, 'identity': identity,
'provenance': provenance, 'started_at_ns': time.time_ns()}
print(json.dumps({**row, 'status': 'running'}), flush=True)
try:
row.update(measure(args, case, credit, work))
except Exception as error:
row.update(status='failed', error=str(error), output=str(work))
rows.append(row)
atomic_json(checkpoint, rows)
print(json.dumps({key: row.get(key) for key in ('case', 'credit', 'status', 'seconds', 'fps', 'wait_replies_per_frame',
'rtt_mean_us', 'client_thread_cpu_ms_per_frame', 'error')}), flush=True)
if row['status'] != 'passed':
return 1
return 0
if __name__ == '__main__':
raise SystemExit(main())
+139
View File
@@ -0,0 +1,139 @@
#!/usr/bin/env python3
"""Compare current unix+shm and loopback TCP+stream using per-frame role CPU clocks.
Start a stats-enabled supervisor separately at --endpoint. This runner neither
contacts a device nor changes production code. It refuses partial or mismatched
client/server frame windows instead of treating missing CPU observations as zero.
"""
import argparse
import importlib.util
import json
import os
from pathlib import Path
import statistics
import subprocess
from measure_transport_cpu import CASES, digest
ROOT = Path(__file__).resolve().parents[2]
SPEC = importlib.util.spec_from_file_location('p65_metrics', ROOT / 'scripts/ci/p65_link_metrics.py')
METRICS = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(METRICS)
def frame_rows(path, marker, server=False):
rows = {}
for line in path.read_text(encoding='utf-8', errors='replace').splitlines():
if marker not in line:
continue
row = dict(METRICS.FIELDS.findall(line.split(marker, 1)[1]))
if server:
if row.get('valid') != '1':
continue
elif row.get('kind') != 'frame':
continue
number = int(row['frame'])
if number in rows:
raise ValueError(f'{path}: duplicate frame {number}')
rows[number] = row
return rows
def run(args, case, transport, repeat):
_, trace_name, width, height, tail = CASES[case]
trace = args.inputs / case / trace_name
if not trace.is_file():
raise ValueError(f'{trace}: extract the unchanged fixture before measuring')
out = args.out / case / transport / f'run-{repeat}'
if out.exists():
raise ValueError(f'{out}: choose a new output directory; evidence is never overwritten')
out.mkdir(parents=True)
env = dict(os.environ, MOBILEGL_TRANSPORT='spawn', MOBILEGL_BACKEND_TYPE='DirectGLES',
MOBILEGL_IPC_SERVER_PATH=str(args.server), MOBILEGL_PIPE_STATS='1', MOBILEGL_PIPE_STATS_PERIOD='1',
MOBILEGL_IPC_PRESENT_CREDIT='1', MOBILEGL_IPC_RUN_AHEAD='1', MOBILEGL_IPC_VERB_BARRIER='1',
MOBILEGL_IPC_BATCH_WAITS='1', MOBILEGL_IPC_SPIN_US='50', MOBILEGL_IPC_STRICT_ERRORS='0',
MOBILEGL_IPC_ROLE_SPLIT_STATE='0', MOBILEGL_IPC_AUDIT='0',
LIBGL_ALWAYS_SOFTWARE='1', EGL_PLATFORM='surfaceless',
MESA_GL_VERSION_OVERRIDE='3.3', MESA_GLSL_VERSION_OVERRIDE='330')
for key in ('MOBILEGL_IPC_CONTROL', 'MOBILEGL_IPC_DATA', 'MOBILEGL_IPC_TOKEN', 'MOBILEGL_IPC_DIAL',
'MOBILEGL_IPC_REQUIRE_SAME_BUILD', 'MOBILEGL_TEST_SERVER_COUNTERS'):
env.pop(key, None)
if transport == 'tcp':
env.update(MOBILEGL_IPC_CONTROL=args.endpoint, MOBILEGL_IPC_DATA='stream', MOBILEGL_IPC_TOKEN='',
MOBILEGL_IPC_REQUIRE_SAME_BUILD='1', MOBILEGL_IPC_LOG_FORWARD='1')
if Path('/usr/share/glvnd/egl_vendor.d/50_mesa.json').exists():
env['__EGL_VENDOR_LIBRARY_FILENAMES'] = '/usr/share/glvnd/egl_vendor.d/50_mesa.json'
command = [str(args.runner), '--trace', str(trace), '--mobilegl-library', str(args.library),
'--backend', 'DirectGLES', '--width', str(width), '--height', str(height), '--output', str(out),
'--benchmark', '--benchmark-finish=0', '--benchmark-tail-frames=' + str(tail)]
with (out / 'runner.log').open('w') as log:
result = subprocess.run(command, env=env, stdout=log, stderr=log, timeout=900)
if result.returncode:
raise RuntimeError(f'{out}: benchmark exited {result.returncode}')
bench = json.loads((out / 'benchmark.json').read_text())
client_path, server_path = out / 'mobilegl.client.log', out / 'mobilegl.server.log'
client = frame_rows(client_path, 'P65LinkMetrics ')
server = frame_rows(server_path, 'P65ServerMetrics ', server=True)
ids = sorted(client)
if len(ids) != bench['totalFrames'] or len(ids) <= tail:
raise ValueError(f'{out}: incomplete frame metrics or tail includes setup')
selected = ids[-tail:]
if selected != list(range(selected[0], selected[-1] + 1)) or any(frame not in server for frame in selected):
raise ValueError(f'{out}: client/server tail frame IDs disagree')
first, last = selected[0], selected[-1]
# The existing reducer now sees precisely the proven common tail range.
values = METRICS.summarize(client_path, first - 1)
values.update(METRICS.summarize_server(server_path, first - 1))
if values['frames'] != tail or values['server_frames'] != tail:
raise ValueError(f'{out}: unexpected frames outside the selected tail')
for frame in selected:
if int(client[frame]['client_thread_cpu_ns']) <= 0 or int(server[frame]['apply_thread_cpu_ns']) <= 0:
raise ValueError(f'{out}: nonpositive per-thread CPU in frame {frame}')
proof = client_path.read_text(errors='replace')
expected = 'control=tcp data=stream' if transport == 'tcp' else 'spawn ARMED - the server role runs in pid '
if expected not in proof:
raise ValueError(f'{out}: no proof of the requested transport')
if ('run-ahead ARMED' not in proof or 'running lockstep' in proof or 'run-ahead DISARMED' in proof):
raise ValueError(f'{out}: requested run-ahead was not continuously armed; not a pure transport CPU comparison')
row = {'case': case, 'transport': transport, 'repeat': repeat,
'first_frame': first, 'last_frame': last, 'tail_frames': tail,
'run_ahead_armed': True,
'benchmark_client_cpu_ms': statistics.mean(bench['frameCpuTimesMs'][-tail:]),
'runner_sha256': args.runner_hash, 'library_sha256': args.library_hash,
'server_sha256': args.server_hash, 'trace_sha256': digest(trace),
'command': command, 'output': str(out), **values}
(out / 'role-cpu.json').write_text(json.dumps(row, indent=2), encoding='utf-8')
return row
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--runner', type=Path, required=True)
parser.add_argument('--library', type=Path, required=True)
parser.add_argument('--server', type=Path, required=True)
parser.add_argument('--inputs', type=Path, required=True)
parser.add_argument('--out', type=Path, required=True)
parser.add_argument('--endpoint', default='tcp://127.0.0.1:40616')
parser.add_argument('--transport', action='append', choices=('spawn', 'tcp'),
help='Measure only the selected arm; default measures both')
parser.add_argument('--case', action='append', choices=CASES,
help='Measure only selected cases; default measures OpenRA and rd12')
args = parser.parse_args()
if not args.endpoint.startswith('tcp://127.0.0.1:'):
parser.error('this measurement is specifically loopback, not a device benchmark')
args.runner_hash, args.library_hash, args.server_hash = map(digest, (args.runner, args.library, args.server))
rows = []
for case, repeats in (('OpenRA', 3), ('rd12', 1)):
if args.case and case not in args.case:
continue
for repeat in range(repeats):
for transport in args.transport or ('spawn', 'tcp'):
row = run(args, case, transport, repeat)
rows.append(row)
(args.out / 'summary.json').write_text(json.dumps(rows, indent=2), encoding='utf-8')
print(json.dumps({key: row[key] for key in ('case', 'transport', 'repeat', 'first_frame', 'last_frame',
'client_thread_cpu_ms_per_frame', 'apply_thread_cpu_ms_per_frame')}), flush=True)
if __name__ == '__main__':
main()
@@ -0,0 +1,171 @@
#!/usr/bin/env python3
"""Measure one real 64 MiB buffer upload consumed by a TCP server draw.
The timed interval includes client staging, stream delivery, backend work, and
an applied fence. Its result is end-to-end staged throughput, not raw network
capacity. Start the same-build stats-enabled server before running this tool.
"""
import argparse
import ctypes as c
import json
import os
from pathlib import Path
import time
from measure_transport_cpu import digest
from measure_loopback_cpu import frame_rows
from run_tcp_matrix import actual_arm_from_text
UPLOAD_BYTES = 64 * 1024 * 1024
def run(args):
args.out.mkdir(parents=True)
os.environ.update(
MOBILEGL_TRANSPORT='spawn', MOBILEGL_IPC_CONTROL=args.endpoint,
MOBILEGL_IPC_DATA='stream', MOBILEGL_IPC_TOKEN=args.token,
MOBILEGL_IPC_REQUIRE_SAME_BUILD='1', MOBILEGL_IPC_LOG_FORWARD='1',
MOBILEGL_BACKEND_TYPE='DirectGLES', MOBILEGL_PIPE_STATS='1',
MOBILEGL_PIPE_STATS_PERIOD='1', MOBILEGL_IPC_RUN_AHEAD='1',
MOBILEGL_IPC_VERB_BARRIER='1', MOBILEGL_IPC_PRESENT_CREDIT='1',
MOBILEGL_LOG_FILE_PATH=str(args.out / 'mobilegl.log'), EGL_PLATFORM='surfaceless')
for name in ('MOBILEGL_IPC_DIAL', 'MOBILEGL_IPC_SERVER_PATH'):
os.environ.pop(name, None)
lib = c.CDLL(str(args.library), mode=c.RTLD_GLOBAL)
P, I, U, F = c.c_void_p, c.c_int, c.c_uint, c.c_float
def fn(name, result, parameters):
function = getattr(lib, name)
function.restype, function.argtypes = result, parameters
return function
get_display = fn('eglGetDisplay', P, [P])
initialize = fn('eglInitialize', U, [P, c.POINTER(I), c.POINTER(I)])
bind_api = fn('eglBindAPI', U, [U])
choose = fn('eglChooseConfig', U, [P, c.POINTER(I), c.POINTER(P), I, c.POINTER(I)])
context = fn('eglCreateContext', P, [P, P, P, c.POINTER(I)])
surface = fn('eglCreatePbufferSurface', P, [P, P, c.POINTER(I)])
current = fn('eglMakeCurrent', U, [P, P, P, P])
swap = fn('eglSwapBuffers', U, [P, P])
terminate = fn('eglTerminate', U, [P])
display = get_display(None)
major, minor = I(), I()
if not display or not initialize(display, c.byref(major), c.byref(minor)):
raise RuntimeError('eglInitialize failed')
try:
if not bind_api(0x30A2): # EGL_OPENGL_API
raise RuntimeError('eglBindAPI failed')
attributes = (I * 15)(0x3033, 1, 0x3024, 8, 0x3023, 8, 0x3022, 8,
0x3021, 8, 0x3025, 24, 0x3040, 8, 0x3038)
config, count = P(), I()
if not choose(display, attributes, c.byref(config), 1, c.byref(count)) or not count.value:
raise RuntimeError('eglChooseConfig failed')
ctx = context(display, config, None, (I * 5)(0x3098, 3, 0x30FB, 3, 0x3038))
surf = surface(display, config, (I * 5)(0x3057, 4, 0x3056, 4, 0x3038))
if not ctx or not surf or not current(display, surf, surf, ctx):
raise RuntimeError('context/surface setup failed')
create_shader = fn('glCreateShader', U, [U])
shader_source = fn('glShaderSource', None, [U, I, c.POINTER(c.c_char_p), c.POINTER(I)])
compile_shader = fn('glCompileShader', None, [U])
get_shader = fn('glGetShaderiv', None, [U, U, c.POINTER(I)])
create_program = fn('glCreateProgram', U, [])
attach = fn('glAttachShader', None, [U, U])
link = fn('glLinkProgram', None, [U])
get_program = fn('glGetProgramiv', None, [U, U, c.POINTER(I)])
use = fn('glUseProgram', None, [U])
error = fn('glGetError', U, [])
program = create_program()
for kind, source in (
(0x8B31, b'#version 330 core\nlayout(location=0) in vec2 p;void main(){gl_Position=vec4(p,0,1);}'),
(0x8B30, b'#version 330 core\nout vec4 c;void main(){c=vec4(1);}'),
):
shader, data, ok = create_shader(kind), c.c_char_p(source), I()
shader_source(shader, 1, c.byref(data), None)
compile_shader(shader)
get_shader(shader, 0x8B81, c.byref(ok))
if not ok.value:
raise RuntimeError('shader compilation failed')
attach(program, shader)
link(program)
ok = I()
get_program(program, 0x8B82, c.byref(ok))
if not ok.value:
raise RuntimeError('program link failed')
use(program)
gen_vao = fn('glGenVertexArrays', None, [I, c.POINTER(U)])
bind_vao = fn('glBindVertexArray', None, [U])
gen_buffer = fn('glGenBuffers', None, [I, c.POINTER(U)])
bind_buffer = fn('glBindBuffer', None, [U, U])
buffer_data = fn('glBufferData', None, [U, c.c_ssize_t, P, U])
subdata = fn('glBufferSubData', None, [U, c.c_ssize_t, c.c_ssize_t, P])
enable = fn('glEnableVertexAttribArray', None, [U])
pointer = fn('glVertexAttribPointer', None, [U, I, U, c.c_ubyte, I, P])
draw = fn('glDrawArrays', None, [U, I, I])
finish = fn('glFinish', None, [])
vao, buffer = U(), U()
gen_vao(1, c.byref(vao))
bind_vao(vao)
gen_buffer(1, c.byref(buffer))
bind_buffer(0x8892, buffer)
buffer_data(0x8892, UPLOAD_BYTES, None, 0x88E8)
vertices = (F * 6)(-1, -1, 1, -1, 0, 1)
subdata(0x8892, 0, c.sizeof(vertices), vertices)
enable(0)
pointer(0, 2, 0x1406, 0, 0, None)
draw(4, 0, 3)
finish()
if not swap(display, surf) or error() != 0:
raise RuntimeError('warmup draw failed')
payload = c.create_string_buffer(b'\x3f' * UPLOAD_BYTES)
start = time.monotonic_ns()
subdata(0x8892, 0, UPLOAD_BYTES, payload)
draw(4, 0, 3)
finish()
elapsed = (time.monotonic_ns() - start) / 1e9
if error() != 0 or not swap(display, surf):
raise RuntimeError('64 MiB consuming draw failed')
fn('MGPipeSyncPeerLog', None, [])()
finally:
current(display, None, None, None)
terminate(display)
client_path = args.out / 'mobilegl.client.log'
proof = client_path.read_text(errors='replace')
actual_arm = actual_arm_from_text(proof)
if (actual_arm != {'armed': True, 'lockstep': False, 'disarmed': False}
or 'control=tcp data=stream server=' + args.endpoint.removeprefix('tcp://') not in proof):
raise ValueError('no continuously armed TCP runtime proof')
frames = frame_rows(client_path, 'P65LinkMetrics ')
if set(frames) != {1, 2} or int(frames[2]['stage_bytes']) < UPLOAD_BYTES:
raise ValueError('frame 2 does not prove that at least 64 MiB crossed Stage')
return {'application_upload_bytes': UPLOAD_BYTES, 'seconds': elapsed,
'end_to_end_staged_mib_per_second': 64 / elapsed,
'frame_2_stage_bytes': int(frames[2]['stage_bytes']),
'actual_arm': actual_arm, 'endpoint': args.endpoint,
'library_sha256': digest(args.library), 'script_sha256': digest(Path(__file__)),
'client_log_sha256': digest(client_path),
'note': 'glBufferSubData plus consuming draw and applied fence; includes client copy and backend work, not raw network capacity'}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--library', required=True, type=Path)
parser.add_argument('--endpoint', required=True)
parser.add_argument('--token', default=os.environ.get('MOBILEGL_IPC_TOKEN', ''))
parser.add_argument('--out', required=True, type=Path)
args = parser.parse_args()
if not args.endpoint.startswith('tcp://'):
parser.error('a TCP endpoint is required')
if args.out.exists():
parser.error('output already exists; choose a fresh evidence directory')
result = run(args)
(args.out / 'result.json').write_text(json.dumps(result, indent=2) + '\n', encoding='utf-8')
print(json.dumps(result), flush=True)
if __name__ == '__main__':
main()
+215
View File
@@ -0,0 +1,215 @@
#!/usr/bin/env python3
"""Record client thread CPU and sampled apply-thread CPU for unchanged trace workloads.
The client clock comes from the existing benchmark runner. Apply CPU comes from
/proc/<pid>/task/<tid>/schedstat, never process-wide CPU. The sampled tail window
is approximately anchored to benchmark.json's mtime; raw samples and this
alignment limitation are retained so the result cannot claim cycle precision.
"""
import argparse
import hashlib
import json
import os
from pathlib import Path
import statistics
import subprocess
import tarfile
import time
CASES = {
'OpenRA': ('openra.tgz', 'openra.trace', 640, 480, 100),
'rd12': ('minecraft-1.21.4-rd12-odinlite-in-world.tgz', 'trace.trace', 854, 480, 200),
}
def digest(path):
value = hashlib.sha256()
with path.open('rb') as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b''):
value.update(chunk)
return value.hexdigest()
def thread_samples(pid):
pids = [pid]
try:
pids += [int(value) for value in Path(f'/proc/{pid}/task/{pid}/children').read_text().split()]
except (FileNotFoundError, ProcessLookupError):
pass
rows = []
for process in pids:
try:
tasks = sorted(Path(f'/proc/{process}/task').iterdir(), key=lambda path: int(path.name))
except (FileNotFoundError, ProcessLookupError):
continue
for task in tasks:
try:
name = (task / 'comm').read_text().strip()
if name != 'mgl-srv-apply':
continue
cpu_ns = int((task / 'schedstat').read_text().split()[0])
rows.append({'time_ns': time.monotonic_ns(), 'pid': process, 'tid': int(task.name),
'name': name, 'cpu_ns': cpu_ns})
# Driver helper threads inherit their creator's comm. The
# ServerLoop thread exists before it calls the driver, so its
# live TID precedes those helpers; do not sum them as apply CPU.
break
except (FileNotFoundError, ProcessLookupError):
continue
return rows
def at_time(samples, timestamp):
for left, right in zip(samples, samples[1:]):
if left['time_ns'] <= timestamp <= right['time_ns']:
width = right['time_ns'] - left['time_ns']
part = (timestamp - left['time_ns']) / width if width else 0
return left['cpu_ns'] + part * (right['cpu_ns'] - left['cpu_ns'])
return None
def reduce_apply(samples, start, end, frames):
grouped = {}
for row in samples:
grouped.setdefault((row['pid'], row['tid']), []).append(row)
contributions = []
for (pid, tid), rows in grouped.items():
first = at_time(rows, start)
last = at_time(rows, end)
if first is None or last is None:
continue
inside = [row for row in rows if start <= row['time_ns'] <= end]
gaps = [(b['time_ns'] - a['time_ns']) / 1e6 for a, b in zip(inside, inside[1:])]
contributions.append({'pid': pid, 'tid': tid, 'cpu_ms': (last - first) / 1e6,
'sample_count': len(inside), 'max_sample_gap_ms': max(gaps, default=None)})
# One owning apply thread must cover the whole selected window. Refuse to
# sum unrelated driver threads that inherited an indistinguishable name.
if len(contributions) != 1:
return {'valid': False, 'reason': 'no unique apply thread spans the complete tail window',
'candidates': contributions}
return {'valid': True, **contributions[0], 'mean_cpu_ms_per_frame': contributions[0]['cpu_ms'] / frames}
def run(args, transport, case, repeat):
archive, trace_name, width, height, tail = CASES[case]
trace_dir = args.inputs / case
trace_dir.mkdir(parents=True, exist_ok=True)
trace = trace_dir / trace_name
if not trace.exists():
archive_path = args.fixtures / archive
if archive_path.stat().st_size < 1024:
raise ValueError(f'{archive_path}: fixture is an LFS pointer, not a trace archive')
with tarfile.open(archive_path, 'r:gz') as bundle:
bundle.extractall(trace_dir, filter='data')
out = args.out / transport / case / f'run-{repeat}'
out.mkdir(parents=True, exist_ok=True)
if (out / 'benchmark.json').exists():
raise ValueError(f'{out}: evidence already exists; choose a new output directory')
trace_digest = digest(trace)
env = dict(os.environ, MOBILEGL_TRANSPORT=transport, MOBILEGL_IPC_SERVER_PATH=str(args.server),
MOBILEGL_PIPE_STATS='0', MOBILEGL_IPC_PRESENT_CREDIT='1', MOBILEGL_IPC_RUN_AHEAD='1',
MOBILEGL_IPC_STRICT_ERRORS='0', MOBILEGL_IPC_ROLE_SPLIT_STATE='0', MOBILEGL_IPC_AUDIT='0',
MOBILEGL_IPC_VERB_BARRIER='1', MOBILEGL_IPC_BATCH_WAITS='1', MOBILEGL_IPC_SPIN_US='50',
LIBGL_ALWAYS_SOFTWARE='1', EGL_PLATFORM='surfaceless',
MESA_GL_VERSION_OVERRIDE='3.3', MESA_GLSL_VERSION_OVERRIDE='330')
for key in ('MOBILEGL_IPC_CONTROL', 'MOBILEGL_IPC_DATA', 'MOBILEGL_IPC_TOKEN', 'MOBILEGL_IPC_DIAL',
'MOBILEGL_IPC_REQUIRE_SAME_BUILD', 'MOBILEGL_TEST_SERVER_COUNTERS'):
env.pop(key, None)
if Path('/usr/share/glvnd/egl_vendor.d/50_mesa.json').exists():
env['__EGL_VENDOR_LIBRARY_FILENAMES'] = '/usr/share/glvnd/egl_vendor.d/50_mesa.json'
command = [str(args.runner), '--trace', str(trace), '--mobilegl-library', str(args.library),
'--backend', 'DirectGLES', '--width', str(width), '--height', str(height),
'--output', str(out), '--benchmark', '--benchmark-finish=0',
'--benchmark-tail-frames=' + str(tail), '--hold-ms', '150']
raw = []
offset_before = time.monotonic_ns() - time.time_ns()
started = time.monotonic()
with (out / 'runner.log').open('w') as log:
process = subprocess.Popen(command, env=env, stdout=log, stderr=log)
try:
while True:
raw.extend(thread_samples(process.pid))
if process.poll() is not None:
break
if time.monotonic() - started > args.timeout:
process.kill()
raise TimeoutError('benchmark exceeded its run budget')
time.sleep(args.sample_ms / 1000)
finally:
process.wait()
offset_after = time.monotonic_ns() - time.time_ns()
(out / 'apply-samples.json').write_text(json.dumps(raw))
if process.returncode:
raise RuntimeError(f'{out}: replay failed with {process.returncode}')
report_path = out / 'benchmark.json'
report = json.loads(report_path.read_text())
wall, cpu = report['frameTimesMs'], report['frameCpuTimesMs']
if len(wall) != len(cpu) or len(wall) <= tail:
raise RuntimeError('no complete CPU series or no startup-excluding tail window')
# End() precedes JSON serialization. Subtract the measured unframed suffix
# after the last frame; do not accidentally include teardown in the tail.
report_end = report_path.stat().st_mtime_ns + (offset_before + offset_after) // 2
unframed_ns = int(report['totalSeconds'] * 1e9 - sum(wall) * 1e6)
tail_end = report_end - unframed_ns
tail_start = tail_end - int(sum(wall[-tail:]) * 1e6)
apply = reduce_apply(raw, tail_start, tail_end, tail)
result = {'label': args.label, 'transport': transport, 'case': case, 'repeat': repeat,
'total_frames': len(wall), 'tail_frames': tail, 'tail_wall_ms': sum(wall[-tail:]),
'client_mean_cpu_ms': statistics.mean(cpu[-tail:]),
'client_median_cpu_ms': statistics.median(cpu[-tail:]),
'apply': apply, 'sample_period_ms': args.sample_ms,
'frame_window_start_ns': tail_start, 'frame_window_end_ns': tail_end,
'clock_offset_drift_ns': offset_after - offset_before,
'apply_clock': '/proc/<pid>/task/<tid>/schedstat execution runtime (nanoseconds)',
'apply_thread_selection': 'oldest live TID named mgl-srv-apply; excludes driver helpers inheriting its name',
'alignment_limit': 'mtime anchor includes unmeasured benchmark summary/JSON serialization latency; '
'apply endpoints are interpolated between external samples, not exact frame markers',
'runner_sha256': args.runner_digest, 'library_sha256': args.library_digest,
'server_sha256': args.server_digest, 'trace_sha256': trace_digest,
'runtime_environment': {key: value for key, value in env.items()
if key.startswith(('MOBILEGL_', 'MESA_', 'LIBGL_', '__EGL_'))},
'command': command, 'output': str(out)}
(out / 'cpu-summary.json').write_text(json.dumps(result, indent=2))
return result
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--runner', type=Path, required=True)
parser.add_argument('--library', type=Path, required=True)
parser.add_argument('--server', type=Path, required=True)
parser.add_argument('--fixtures', type=Path, required=True)
parser.add_argument('--inputs', type=Path, required=True)
parser.add_argument('--out', type=Path, required=True)
parser.add_argument('--label', required=True)
parser.add_argument('--transport', action='append', choices=('inproc', 'spawn'))
parser.add_argument('--case', action='append', choices=CASES)
parser.add_argument('--repeats', type=int, default=3)
parser.add_argument('--case-repeats', action='append', default=[], metavar='CASE=N',
help='Override repetitions for one case, e.g. rd12=2')
parser.add_argument('--sample-ms', type=float, default=5)
parser.add_argument('--timeout', type=float, default=900)
args = parser.parse_args()
if args.repeats < 1 or args.sample_ms <= 0:
parser.error('repeats and sample-ms must be positive')
repeats = {}
for override in args.case_repeats:
name, separator, count = override.partition('=')
if not separator or name not in CASES or not count.isdigit() or int(count) < 1:
parser.error('--case-repeats requires a known CASE and a positive count')
repeats[name] = int(count)
args.runner_digest, args.library_digest, args.server_digest = map(digest, (args.runner, args.library, args.server))
results = []
for case in args.case or list(CASES):
for repeat in range(repeats.get(case, args.repeats)):
for transport in args.transport or ('inproc', 'spawn'):
result = run(args, transport, case, repeat)
results.append(result)
(args.out / 'summary.json').write_text(json.dumps(results, indent=2))
print(json.dumps({key: result[key] for key in ('case', 'transport', 'repeat', 'client_mean_cpu_ms', 'apply')}), flush=True)
return 0 if all(row['apply']['valid'] for row in results) else 1
if __name__ == '__main__':
raise SystemExit(main())
+260
View File
@@ -0,0 +1,260 @@
#!/usr/bin/env python3
"""Reboot-clean, pinned Redmi TCP credit measurements with original-state restore.
This explicitly mutating session is separate from the ordinary benchmark tool.
Run only while this device has no other client. The service/idle exemption is
left available to its owner; CPU/GPU/fan settings are always restored here.
"""
import argparse
import json
import os
from pathlib import Path
import re
import shlex
import subprocess
import sys
import time
import benchmark_tcp_credits as benchmark
from measure_transport_cpu import CASES, digest
from run_tcp_matrix import atomic_json, supervise
CPU = '/sys/devices/system/cpu/cpufreq'
GPU = '/sys/class/kgsl/kgsl-3d0'
FAN = '/sys/class/xm_power/hw_monitor/pwm_fan'
POLICIES = ('policy0', 'policy6')
RESTORE_NODES = [f'{CPU}/{policy}/{field}' for policy in POLICIES
for field in ('scaling_min_freq', 'scaling_max_freq', 'scaling_governor')]
RESTORE_NODES += [f'{GPU}/min_pwrlevel', f'{GPU}/max_pwrlevel', f'{FAN}/target_level']
READ_NODES = RESTORE_NODES + [f'{CPU}/{policy}/scaling_cur_freq' for policy in POLICIES]
READ_NODES += [f'{GPU}/gpuclk', f'{FAN}/real_speed', f'{FAN}/pwm_duty']
ORDERS = ((1, 2, 3), (2, 3, 1), (3, 1, 2))
class Device:
def __init__(self, serial):
self.adb = ['adb', '-s', serial]
def shell(self, argv, timeout=30):
return subprocess.run([*self.adb, 'shell', shlex.join(argv)], check=True,
capture_output=True, text=True, timeout=timeout).stdout.strip()
def su(self, script):
return self.shell(['su', '-c', script])
def boot_id(self):
return self.shell(['cat', '/proc/sys/kernel/random/boot_id'])
def observe(self):
statements = [f'printf "%s=%s\\n" {shlex.quote(node)} "$(cat {shlex.quote(node)})"'
for node in READ_NODES]
statements.append('for node in /sys/class/thermal/thermal_zone*; do '
'if [ "$(cat "$node/type")" = cpuss-0-0 ]; then '
'printf "thermal_path=%s\\nthermal_mc=%s\\n" "$node" "$(cat "$node/temp")"; '
'break; fi; done')
values = dict(line.split('=', 1) for line in self.su('; '.join(statements)).splitlines() if '=' in line)
if any(not values.get(node) for node in READ_NODES) or not values.get('thermal_mc'):
raise RuntimeError('incomplete device state; cannot establish or restore the thermal window')
for node, value in values.items():
if node.endswith('scaling_governor'):
if not re.fullmatch(r'[A-Za-z0-9_-]+', value):
raise ValueError('invalid governor observation')
elif node != 'thermal_path':
int(value)
return {'time_ns': time.time_ns(), **values}
def restore(self, original):
statements = []
for policy in POLICIES:
base = f'{CPU}/{policy}'
statements += [f'cat {base}/cpuinfo_min_freq > {base}/scaling_min_freq',
f'echo {int(original[base + "/scaling_max_freq"])} > {base}/scaling_max_freq',
f'echo {int(original[base + "/scaling_min_freq"])} > {base}/scaling_min_freq',
f'echo {shlex.quote(original[base + "/scaling_governor"])} > {base}/scaling_governor']
statements += [f'echo {int(original[node])} > {node}' for node in RESTORE_NODES if node.startswith((GPU, FAN))]
self.su('; '.join(statements))
after = self.observe()
differences = {node: {'expected': original[node], 'actual': after[node]}
for node in RESTORE_NODES if original[node] != after[node]}
return {'after': after, 'differences': differences, 'restored': not differences}
def wait_for_boot(device, old_boot_id):
subprocess.run([*device.adb, 'reboot'], check=True, timeout=30)
deadline = time.monotonic() + 300
while time.monotonic() < deadline:
try:
if device.shell(['getprop', 'sys.boot_completed']) == '1' and device.boot_id() != old_boot_id:
time.sleep(20)
return device.boot_id()
except (OSError, subprocess.SubprocessError):
pass
time.sleep(5)
raise RuntimeError('reboot did not establish a changed, fully booted device')
def pin(args, action, output):
result = subprocess.run(['bash', str(args.pin_copy), args.serial, action],
capture_output=True, text=True, timeout=30)
output.write_text(result.stdout + result.stderr, encoding='utf-8')
return result.returncode
def prepare_arm(args, device, directory):
directory.mkdir(parents=True)
if pin(args, 'pin', directory / 'pin-request.txt') != 0:
raise RuntimeError('pin did not take')
readings, consecutive = [], 0
deadline = time.monotonic() + args.cool_timeout
while time.monotonic() < deadline:
observation = device.observe()
readings.append(observation)
atomic_json(directory / 'cooldown.json', readings)
if int(observation['thermal_mc']) <= args.cool_max_mc:
consecutive += 1
if consecutive == 2:
break
else:
consecutive = 0
time.sleep(5)
if consecutive < 2:
raise RuntimeError('cooldown expired; no valid measurement window')
if pin(args, 'check', directory / 'pin-before.txt') != 0:
raise RuntimeError('frequency drift before measurement')
if int(readings[-1][FAN + '/target_level']) != 2 or int(readings[-1][FAN + '/real_speed']) <= 0:
raise RuntimeError('fan setting was not actually established')
return readings[-1]
def start_server(args, device):
script = Path(__file__).with_name('tcp_device_server.py')
command = [sys.executable, str(script), 'start', '--serial', args.serial,
'--package', args.package, '--listen', args.listen, '--token', args.token,
'--allow-idle', '--state-file', str(args.server_idle_state), '--env',
'MOBILEGL_TEST_SERVER_COUNTERS=1;MOBILEGL_PIPE_STATS=1;MOBILEGL_PIPE_STATS_PERIOD=1;'
'MOBILEGL_LOG_FILE_PATH=/data/data/' + args.package + '/files/mgl.log']
result = subprocess.run(command, check=True, capture_output=True, text=True, timeout=30)
(args.out / 'server-start.txt').write_text(result.stdout + result.stderr, encoding='utf-8')
port = args.endpoint.rsplit(':', 1)[-1]
for _ in range(20):
sockets = device.shell(['ss', '-ltn'])
if re.search(r':' + re.escape(port) + r'\s', sockets):
return
time.sleep(1)
raise RuntimeError('restarted supervisor is not listening')
def run_session(args):
device = Device(args.serial)
original = device.observe()
session = {'status': 'prepared', 'serial': args.serial, 'endpoint': args.endpoint,
'library_sha256': digest(args.library), 'runner_sha256': digest(args.runner),
'pin_script_sha256': digest(args.pin_script), 'original_state': original,
'boot_id_before': device.boot_id(), 'credit_orders': ORDERS,
'cool_max_mc': args.cool_max_mc, 'arms': [], 'started_at_ns': time.time_ns()}
state_path = args.out / 'session.json'
atomic_json(state_path, session) # Durable before the first mutation, including reboot.
args.pin_copy = args.out / 'pin_device.sh'
args.pin_copy.write_text(args.pin_script.read_text(), encoding='utf-8', newline='\n')
try:
session['boot_id'] = wait_for_boot(device, session['boot_id_before'])
session['rebooted_clean'] = True
device.su(f'echo 2 > {FAN}/target_level')
time.sleep(3)
host = args.endpoint.removeprefix('tcp://').rsplit(':', 1)[0]
wlan = device.shell(['ip', '-f', 'inet', 'addr', 'show', 'wlan0'])
session['wlan_after_reboot'] = wlan
if host not in wlan:
raise RuntimeError('device address changed; resolve host routing before a new performance session')
for _ in range(60):
names = device.shell(['ps', '-A', '-o', 'NAME'])
if not any('dex2oat' in name for name in names.splitlines()):
break
time.sleep(5)
else:
raise RuntimeError('dex2oat did not become idle')
start_server(args, device)
session['status'] = 'running'
atomic_json(state_path, session)
index = 0
for case in args.case or CASES:
for repeat, order in enumerate(ORDERS, 1):
for credit in order:
index += 1
directory = args.out / f'{index:02d}-{case}-credit{credit}-r{repeat}'
row = {'index': index, 'case': case, 'credit': credit, 'repeat': repeat, 'status': 'preparing'}
session['arms'].append(row)
atomic_json(state_path, session)
print(json.dumps(row), flush=True)
row['before'] = prepare_arm(args, device, directory)
try:
row.update(benchmark.measure(args, case, credit, directory / 'measurement'))
finally:
row['pin_after_rc'] = pin(args, 'check', directory / 'pin-after.txt')
row['after'] = device.observe()
atomic_json(state_path, session)
if row['status'] != 'passed' or row['pin_after_rc'] != 0:
row['status'] = 'invalid'
raise RuntimeError(f'arm {index} is invalid; preserve it and stop')
print(json.dumps({key: row.get(key) for key in ('index', 'case', 'credit', 'status', 'fps')}), flush=True)
directory = args.out / 'stage-burst'
burst = {'before': prepare_arm(args, device, directory)}
session['stage_burst'] = burst
script = Path(__file__).with_name('measure_tcp_stage_burst.py')
command = [sys.executable, str(script), '--library', str(args.library), '--endpoint', args.endpoint,
'--token', args.token, '--out', str(directory / 'measurement')]
def wake():
device.shell(['input', 'keyevent', 'KEYCODE_WAKEUP'])
try:
burst.update(supervise(command, dict(os.environ), str(args.inputs), directory,
directory / 'runner.log', wake=wake))
if burst['returncode'] != 0 or burst['remaining_processes'] or burst['leftovers_after_parent_exit']:
raise RuntimeError('Stage burst failed')
burst['measurement'] = json.loads((directory / 'measurement/result.json').read_text())
finally:
burst['pin_after_rc'] = pin(args, 'check', directory / 'pin-after.txt')
burst['after'] = device.observe()
if burst['pin_after_rc'] != 0:
raise RuntimeError('Stage burst pin drifted')
session['status'] = 'passed'
except BaseException as error:
session.update(status='failed', error=repr(error))
raise
finally:
try:
session['restore'] = device.restore(original)
if not session['restore']['restored']:
session['status'] = 'restore-failed'
except BaseException as error:
session.update(status='restore-failed', restore_error=repr(error))
session['completed_at_ns'] = time.time_ns()
atomic_json(state_path, session)
return 0 if session['status'] == 'passed' else 1
def main():
parser = argparse.ArgumentParser(description=__doc__)
for name in ('runner', 'library', 'inputs', 'out', 'pin-script', 'server-idle-state'):
parser.add_argument('--' + name, required=True, type=Path)
parser.add_argument('--serial', required=True, choices=('2f7cbe2e',), help='profile validated for this Redmi only')
parser.add_argument('--endpoint', required=True)
parser.add_argument('--listen', required=True)
parser.add_argument('--token', default=os.environ.get('MOBILEGL_IPC_TOKEN', ''))
parser.add_argument('--package', default='top.mobilegl.plugin.trace')
parser.add_argument('--cool-max-mc', type=int, default=36000)
parser.add_argument('--cool-timeout', type=int, default=300)
parser.add_argument('--case', action='append', choices=CASES)
parser.add_argument('--reboot', action='store_true', required=True)
args = parser.parse_args()
args.adb = 'adb'
if args.out.exists():
parser.error('choose a new output directory; paired sessions never resume across boots')
if not args.endpoint.startswith('tcp://') or args.cool_timeout <= 0:
parser.error('TCP endpoint and a positive cooldown timeout are required')
args.out.mkdir(parents=True)
return run_session(args)
if __name__ == '__main__':
raise SystemExit(main())
+495
View File
@@ -0,0 +1,495 @@
#!/usr/bin/env python3
"""Run manifest-selected TCP retraces from a CTest JSON catalog, with resumable evidence.
Requires Linux/WSL process groups. The old local CTest/manifest TIMEOUT is not a
TCP deadline: log inactivity and an explicit absolute ceiling are separate.
"""
from __future__ import annotations
import argparse
import contextlib
import ctypes
import fcntl
import hashlib
import json
import math
import os
from pathlib import Path
import re
import signal
import shutil
import subprocess
import sys
import time
import uuid
from trace_cases import load_trace_case_manifest, ci_backends
ROOT = Path(__file__).resolve().parents[2]
OWNED_ENV = {"MOBILEGL_TRANSPORT", "MOBILEGL_IPC_CONTROL", "MOBILEGL_IPC_DATA",
"MOBILEGL_IPC_TOKEN", "MOBILEGL_IPC_DIAL", "MOBILEGL_IPC_ROLE",
"MOBILEGL_IPC_SERVER_PATH", "MOBILEGL_LOG_FILE_PATH"}
def read_json(path):
def invalid(value):
raise ValueError("non-finite JSON number: " + value)
return json.loads(Path(path).read_text(encoding="utf-8"), parse_constant=invalid)
def digest(path):
h = hashlib.sha256()
with Path(path).open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
h.update(chunk)
return h.hexdigest()
def fingerprint(value):
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
def atomic_json(path, value):
temporary = path.with_name(path.name + "." + uuid.uuid4().hex + ".tmp")
try:
with temporary.open("w", encoding="utf-8") as stream:
json.dump(value, stream, ensure_ascii=False, indent=2, allow_nan=False)
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, path)
finally:
temporary.unlink(missing_ok=True)
def select_cases(manifest_path, requested):
manifest = load_trace_case_manifest(manifest_path)
raw = read_json(manifest_path)
explicit = {case["name"]: case.get("split", raw.get("defaults", {}).get("split"))
for case in raw["cases"]}
cases = {case["name"]: case for case in manifest["cases"]}
names = requested or [case["name"] for case in cases.values() if case["split"]]
if len(names) != len(set(names)):
raise ValueError("duplicate --case")
selected = []
for name in names:
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", name) or name not in cases:
raise ValueError("unknown/unsafe case: " + name)
case = cases[name]
# An explicit non-CI workload, such as rd12, is useful for extra golden/perf runs.
# An explicit split:false remains a refusal, including in this explicit selection.
if explicit[name] is False or (not case["split"] and case.get("ci", True)):
raise ValueError("case opts out of split: " + name)
selected.append(case)
if not selected:
raise ValueError("the manifest selected no split cases")
return selected
def properties(test):
return {entry["name"]: entry["value"] for entry in test.get("properties", [])}
def case_environment(test, base, args, output=None):
env = dict(base) # A fresh copy for every case; never leak the previous case's quirks.
for pair in properties(test).get("ENVIRONMENT", []):
key, sep, value = pair.partition("=")
if not sep:
raise ValueError("malformed CTest environment entry: " + pair)
if key not in OWNED_ENV:
env[key] = value
for key in OWNED_ENV:
env.pop(key, None)
env.update(MOBILEGL_TRANSPORT="spawn", MOBILEGL_IPC_CONTROL=args.endpoint,
MOBILEGL_IPC_DATA="stream", MOBILEGL_IPC_TOKEN=args.token,
MOBILEGL_IPC_REQUIRE_SAME_BUILD="1", MOBILEGL_IPC_LOG_FORWARD="1",
MOBILEGL_PIPE_STATS="1", MOBILEGL_PIPE_STATS_PERIOD="1",
MOBILEGL_IPC_PRESENT_CREDIT=str(args.credit))
if getattr(args, "require_run_ahead", False):
env["MOBILEGL_IPC_RUN_AHEAD"] = "1"
env["MOBILEGL_IPC_VERB_BARRIER"] = "1"
if output is not None:
env["MOBILEGL_LOG_FILE_PATH"] = str(output / "output/mobilegl.log")
return env
def definitions(command):
result = {}
for arg in command:
if arg.startswith("-D") and "=" in arg:
key, value = arg[2:].split("=", 1)
result[key.split(":", 1)[0]] = value
return result
def plan_case(case, backend, catalog, args, hashes):
name = f"MobileGLTraceReplay.{case['name']}.{backend}"
test = next((catalog[key] for key in (name, name + ".TCP", name + ".SPAWN", name + ".SPLIT")
if key in catalog), None)
if test is None:
raise ValueError("CTest catalog has no entry for " + name)
command = list(test["command"])
defs = definitions(command)
if defs.get("TRACE_CASE_NAME") != case["name"] or defs.get("TRACE_BACKEND") != backend:
raise ValueError("CTest name and command disagree: " + name)
if "-P" not in command or not defs.get("TRACE_ARCHIVE"):
raise ValueError("expected a run_trace_case.cmake catalog entry: " + name)
fixtures = args.fixtures or Path(defs["TRACE_ARCHIVE"]).parent
runner = (args.runner or Path(defs["TRACE_REPLAY_EXE"])).resolve()
script = args.source / "tools/trace_replay/run_trace_case.cmake"
values = {"TRACE_REPLAY_EXE": str(runner), "MOBILEGL_LIBRARY": str(args.library),
"TRACE_CASE_NAME": case["name"], "TRACE_BACKEND": backend,
"TRACE_TRANSPORT": "spawn", "TRACE_IPC_SERVER_PATH": "",
"TRACE_IPC_CONTROL": "", "TRACE_IPC_TOKEN": ""}
# Read values from the CURRENT manifest, not stale -D values in the catalog.
keys = {"TRACE_FILE": "trace_file", "TRACE_TARGET_CALL": "target_call", "TRACE_WIDTH": "width",
"TRACE_HEIGHT": "height", "TRACE_SSIM_THRESHOLD": "ssim_threshold", "TRACE_CROP_X": "crop_x",
"TRACE_CROP_Y": "crop_y", "TRACE_CROP_WIDTH": "crop_width", "TRACE_CROP_HEIGHT": "crop_height",
"TRACE_COHERENT_AS_FLUSH": "coherent_as_flush"}
for key, field in keys.items():
value = case.get(field, False if field == "coherent_as_flush" else 0.99 if field == "ssim_threshold" else 0)
values[key] = ("ON" if value else "OFF") if isinstance(value, bool) else str(value)
inputs = []
for key, field in (("TRACE_ARCHIVE", "trace_archive"), ("TRACE_GOLDEN", "golden"),
("TRACE_ALTERNATE_GOLDEN", "alternate_golden")):
path = (fixtures / case[field]).resolve() if case.get(field) else None
values[key] = str(path) if path else ""
if path:
stat = path.stat()
inputs.append({"path": str(path), "size": stat.st_size, "mtime_ns": stat.st_mtime_ns,
**({"sha256": digest(path)} if key != "TRACE_ARCHIVE" else {})})
trace_file = Path(case["trace_file"])
if trace_file.is_absolute() or ".." in trace_file.parts:
raise ValueError("trace_file must remain inside the extracted input directory")
for path in (runner, args.library, script):
if str(path) not in hashes:
hashes[str(path)] = digest(path)
env = case_environment(test, args.base_env, args)
# Ignore shell/session bookkeeping when resuming, but retain renderer knobs and
# every case-specific catalog variable. The digest stores no token/plaintext env.
case_keys = {pair.partition("=")[0] for pair in properties(test).get("ENVIRONMENT", [])}
relevant = {key: value for key, value in env.items() if key in case_keys or
key.startswith(("MOBILEGL_", "MGITEST_", "EGL_", "LIBGL_", "MESA_", "VK_", "__GL", "__EGL", "LD_")) or
key in ("DISPLAY", "WAYLAND_DISPLAY", "XDG_RUNTIME_DIR", "DRI_PRIME", "GALLIUM_DRIVER")}
identity = fingerprint({"case": case, "backend": backend, "credit": args.credit,
"endpoint": args.endpoint, "token": args.token,
"require_run_ahead": args.require_run_ahead,
"artifacts": {str(p): hashes[str(p)] for p in (runner, args.library, script)},
"inputs": inputs, "environment": relevant, "command": command})
return {"case": case, "backend": backend, "test": test, "command": command, "values": values,
"script": script, "runner": runner, "identity": identity,
"key": f"{backend}/credit-{args.credit}/{case['name']}"}
def command_for(plan, work):
values = {**plan["values"], "TRACE_OUTPUT_DIR": str(work), "TRACE_ARTIFACT_DIR": str(work.parent / "artifacts")}
command, found = [], set()
original = plan["command"]
for index, arg in enumerate(original):
if index and original[index - 1] == "-P":
command.append(str(plan["script"]))
elif arg.startswith("-D") and arg[2:].split("=", 1)[0].split(":", 1)[0] in values:
key = arg[2:].split("=", 1)[0].split(":", 1)[0]
command.append(f"-D{key}={values[key]}")
found.add(key)
else:
command.append(arg)
position = command.index("-P")
command[position:position] = [f"-D{k}={v}" for k, v in values.items() if k not in found]
return command
def group_members(pgid):
members = []
for path in Path("/proc").glob("[0-9]*/stat"):
try:
fields = path.read_text().rsplit(")", 1)[1].split()
if int(fields[2]) == pgid:
members.append((int(path.parent.name), fields[0]))
except (OSError, ValueError, IndexError):
pass
return members
def reap_group(pgid):
while True:
try:
pid, _ = os.waitpid(-pgid, os.WNOHANG)
if pid == 0:
break
except ChildProcessError:
break
@contextlib.contextmanager
def subreaper():
libc = ctypes.CDLL(None, use_errno=True)
previous = ctypes.c_int()
if libc.prctl(37, ctypes.byref(previous), 0, 0, 0) or libc.prctl(36, 1, 0, 0, 0):
raise OSError(ctypes.get_errno(), "cannot enable Linux child subreaper")
try:
yield
finally:
libc.prctl(36, previous.value, 0, 0, 0)
def terminate_group(process, grace=2.0):
"""Reap descendants even when the wrapper exits before a TERM-ignoring child."""
pgid = process.pid
for sig, duration in ((signal.SIGTERM, grace), (signal.SIGKILL, 2.0)):
try:
os.killpg(pgid, sig)
except ProcessLookupError:
pass
until = time.monotonic() + duration
while time.monotonic() < until:
process.poll() # Let Popen reap its own child before reaping adopted descendants.
if process.returncode is not None:
reap_group(pgid)
if not group_members(pgid):
return []
time.sleep(.02)
process.wait(timeout=2)
reap_group(pgid)
return group_members(pgid)
def progress_signature(work, console):
signature = []
for path in [console, *work.rglob("*.log")]:
try:
stat = path.stat()
signature.append((str(path), stat.st_size, stat.st_mtime_ns))
except FileNotFoundError:
pass
return tuple(sorted(signature))
def supervise(command, env, cwd, work, console, idle_seconds=300, max_seconds=7200,
poll_seconds=.2, terminate_grace=2.0, wake=None):
started = time.monotonic()
last_progress = started
previous = progress_signature(work, console)
timeout_kind = None
next_wake = started
with subreaper(), console.open("wb") as output:
process = subprocess.Popen(command, env=env, cwd=cwd, stdout=output, stderr=output,
start_new_session=True)
try:
while process.poll() is None:
now = time.monotonic()
if wake is not None and now >= next_wake:
wake()
next_wake = time.monotonic() + 15
signature = progress_signature(work, console)
if signature != previous:
previous, last_progress = signature, time.monotonic()
now = time.monotonic()
if now - started >= max_seconds:
timeout_kind = "absolute"
break
if now - last_progress >= idle_seconds:
timeout_kind = "idle"
break
time.sleep(poll_seconds)
leftovers = group_members(process.pid) if process.poll() is not None else []
returncode = process.returncode
finally:
remaining = terminate_group(process, terminate_grace)
return {"returncode": 124 if timeout_kind else returncode, "timeout_kind": timeout_kind,
"seconds": round(time.monotonic() - started, 3), "pid": process.pid,
"leftovers_after_parent_exit": [pid for pid, state in leftovers if state != "Z"],
"remaining_processes": remaining}
def actual_arm_from_text(client_text):
return {"armed": "run-ahead ARMED" in client_text,
"lockstep": "running lockstep" in client_text,
"disarmed": bool(re.search(r"\bDISARMED\b", client_text))}
def validate_result(plan, work, endpoint, require_run_ahead=False):
path = work / "output/result.json"
result = read_json(path)
if result.get("passed") is not True or result.get("statusCode") != 0:
raise ValueError("result.json does not report success")
if result.get("backend") != plan["backend"] or result.get("targetCall") != int(plan["values"]["TRACE_TARGET_CALL"]):
raise ValueError("result identifies a different backend/call")
expected_trace = (work / "input" / plan["case"]["trace_file"]).resolve()
if Path(result.get("tracePath", "")).resolve() != expected_trace:
raise ValueError("result tracePath belongs to a different attempt")
actual = work / "output/actual.png"
if Path(result.get("actualPath", "")).resolve() != actual.resolve() or not actual.is_file():
raise ValueError("result has no current-attempt actual image")
goldens = {str(Path(plan["values"][key]).resolve()) for key in ("TRACE_GOLDEN", "TRACE_ALTERNATE_GOLDEN") if plan["values"][key]}
if str(Path(result.get("matchedGoldenPath", "")).resolve()) not in goldens:
raise ValueError("result matched a different golden")
score = result.get("ssim")
if isinstance(score, bool) or not isinstance(score, (float, int)) or not math.isfinite(score) or score < float(plan["values"]["TRACE_SSIM_THRESHOLD"]):
raise ValueError("result SSIM does not meet the current manifest threshold")
client_log = work / "output/mobilegl.client.log"
client_text = client_log.read_text(errors="replace")
marker = re.compile(r"control=tcp data=stream server=(\S+) pid=([1-9][0-9]*)")
if not any(match[0] == endpoint.removeprefix("tcp://") for match in marker.findall(client_text)):
raise ValueError("current-attempt log lacks the requested TCP arm proof")
actual_arm = actual_arm_from_text(client_text)
if require_run_ahead and (not actual_arm["armed"] or actual_arm["lockstep"] or actual_arm["disarmed"]):
raise ValueError("required run-ahead arm missing or contradicted in private client log: " + str(actual_arm))
return {"result": result, "result_path": str(path), "result_sha256": digest(path),
"client_log": str(client_log), "client_log_sha256": digest(client_log), "actual_arm": actual_arm}
def reusable(row, plan, out, endpoint, require_run_ahead=False):
if row is None or row.get("status") != "passed" or row.get("returncode") != 0 or row.get("timeout_kind") is not None or row.get("identity") != plan["identity"]:
return False
if require_run_ahead and row.get("require_run_ahead") is not True:
return False
try:
work = Path(row["work"])
if not work.resolve().is_relative_to(out.resolve()):
return False
proof = validate_result(plan, work, endpoint, require_run_ahead)
return all(proof[key] == row.get(key) for key in ("result_sha256", "client_log_sha256", "actual_arm"))
except (OSError, ValueError, KeyError):
return False
def run_case(plan, args, row):
work = Path(row["work"])
console = work.parent / "runner.log"
env = case_environment(plan["test"], args.base_env, args, work)
row["requested_arm"] = {key: env.get(key) for key in ("MOBILEGL_IPC_RUN_AHEAD", "MOBILEGL_IPC_VERB_BARRIER")}
props = properties(plan["test"])
cwd = props.get("WORKING_DIRECTORY", str(args.source))
def wake():
with (work.parent / "wake-adb.log").open("ab") as log:
try:
subprocess.run([args.adb, "-s", args.wake_adb_serial, "shell", "input", "keyevent", "KEYCODE_WAKEUP"],
stdout=log, stderr=log, timeout=5, check=False)
except (OSError, subprocess.TimeoutExpired) as error:
log.write((str(error) + "\n").encode())
outcome = supervise(command_for(plan, work), env, cwd, work, console, args.idle_seconds,
args.max_seconds, wake=wake if args.wake_adb_serial else None)
row.update(outcome, status="failed", console=str(console))
client_log = work / "output/mobilegl.client.log"
row["actual_arm"] = actual_arm_from_text(client_log.read_text(errors="replace")) if client_log.is_file() else None
if outcome["timeout_kind"]:
row["status"] = "timeout"
elif outcome["returncode"] == 0 and not outcome["remaining_processes"] and not outcome["leftovers_after_parent_exit"]:
try:
row.update(validate_result(plan, work, args.endpoint, args.require_run_ahead), status="passed")
except (OSError, ValueError, KeyError) as error:
row["error"] = str(error)
if row["status"] != "passed" and row["returncode"] == 0:
row["process_returncode"] = 0
row["returncode"] = 1 # A zero wrapper exit cannot turn stale/missing proof green.
return row
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__)
catalog_source = parser.add_mutually_exclusive_group(required=True)
catalog_source.add_argument("--catalog", type=Path, help="ctest --show-only=json-v1 output")
catalog_source.add_argument("--build-dir", type=Path, help="read a catalog using ctest; does not build")
parser.add_argument("--source", type=Path, default=ROOT)
parser.add_argument("--manifest", type=Path)
parser.add_argument("--fixtures", type=Path, help="default: the fixture directory recorded in the catalog")
parser.add_argument("--runner", type=Path, help="override TRACE_REPLAY_EXE from the catalog")
parser.add_argument("--library", type=Path, required=True)
parser.add_argument("--endpoint", required=True)
parser.add_argument("--token", default=os.environ.get("MOBILEGL_IPC_TOKEN", ""))
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--case", action="append")
parser.add_argument("--backend", action="append", choices=("DirectGLES", "DirectVulkan"))
parser.add_argument("--credit", type=int, choices=(1, 2, 3), default=2)
parser.add_argument("--resume", action="store_true")
parser.add_argument("--require-run-ahead", action="store_true",
help="require private-client ARMED proof, no lockstep fallback/DISARMED; force RUN_AHEAD=1 and VERB_BARRIER=1")
parser.add_argument("--idle-seconds", type=float, default=300)
parser.add_argument("--max-seconds", type=float, default=7200)
parser.add_argument("--wake-adb-serial", help="explicit opt-in: KEYCODE_WAKEUP every 15 seconds while a case runs")
parser.add_argument("--adb", default="adb")
args = parser.parse_args(argv)
if sys.platform != "linux":
parser.error("run this process-group driver in Linux/WSL")
if not args.endpoint.startswith("tcp://") or any(not math.isfinite(value) or value <= 0
for value in (args.idle_seconds, args.max_seconds)):
parser.error("a tcp:// endpoint and positive timeouts are required")
args.source, args.library, args.out = args.source.resolve(), args.library.resolve(), args.out.resolve()
args.base_env = dict(os.environ)
args.manifest = args.manifest or args.source / "tools/trace_replay/trace_cases.json"
backends = args.backend or ["DirectGLES"]
if len(backends) != len(set(backends)):
parser.error("duplicate --backend")
try:
cases = select_cases(args.manifest, args.case)
document = read_json(args.catalog) if args.catalog else json.loads(subprocess.check_output(
["ctest", "--test-dir", str(args.build_dir), "--show-only=json-v1"], text=True))
catalog = {test["name"]: test for test in document["tests"]}
hashes = {}
plans = [plan_case(case, backend, catalog, args, hashes) for case in cases
for backend in backends if backend in ci_backends(case)]
if not plans:
raise ValueError("no selected case supports the requested backend")
except (OSError, ValueError, KeyError, subprocess.CalledProcessError) as error:
parser.error(str(error))
args.out.mkdir(parents=True, exist_ok=True)
checkpoint_path = args.out / "checkpoint.json"
with (args.out / ".matrix.lock").open("w") as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
parser.error("another matrix driver owns this output directory")
if not checkpoint_path.exists() and (args.out / "results.json").is_file():
legacy = args.out / ("results.legacy-" + uuid.uuid4().hex + ".json")
shutil.copy2(args.out / "results.json", legacy)
print(json.dumps({"legacy_results_preserved": str(legacy)}), flush=True)
checkpoint = read_json(checkpoint_path) if checkpoint_path.exists() else {"schema_version": 1, "runs": []}
if checkpoint.get("schema_version") != 1 or not isinstance(checkpoint.get("runs"), list):
parser.error("unsupported/corrupt checkpoint; old scratch results are not a checkpoint")
selected = []
def interrupted(signum, frame):
raise KeyboardInterrupt
handlers = {sig: signal.signal(sig, interrupted) for sig in (signal.SIGINT, signal.SIGTERM, signal.SIGHUP)}
try:
for plan in plans:
prior = next((r for r in reversed(checkpoint["runs"]) if r.get("key") == plan["key"]), None)
if args.resume and reusable(prior, plan, args.out, args.endpoint, args.require_run_ahead):
selected.append({**prior, "resumed": True})
print(json.dumps({"case": prior["case"], "backend": prior["backend"], "status": "passed", "resumed": True}), flush=True)
continue
attempt = args.out / plan["key"] / ("attempt-" + uuid.uuid4().hex)
attempt.mkdir(parents=True)
row = {"key": plan["key"], "case": plan["case"]["name"], "backend": plan["backend"],
"credit": args.credit, "ci": plan["case"].get("ci", True), "identity": plan["identity"],
"require_run_ahead": args.require_run_ahead,
"work": str(attempt / "work"), "status": "running", "started_at_ns": time.time_ns()}
checkpoint["runs"].append(row)
atomic_json(checkpoint_path, checkpoint)
print(json.dumps({"case": row["case"], "backend": row["backend"], "status": "running"}), flush=True)
try:
run_case(plan, args, row)
except KeyboardInterrupt:
row.update(status="cancelled", returncode=130)
atomic_json(checkpoint_path, checkpoint)
return 130
except Exception as error:
row.update(status="failed", returncode=1, error=str(error))
row["completed_at_ns"] = time.time_ns()
atomic_json(checkpoint_path, checkpoint)
selected.append({**row, "resumed": False})
atomic_json(args.out / "results.json", selected)
print(json.dumps({k: row.get(k) for k in ("case", "backend", "status", "returncode", "seconds", "timeout_kind", "error")}), flush=True)
if row["status"] == "timeout" or row.get("remaining_processes"):
# Do not convert an unhealthy peer into a cascade of Busy outcomes.
# The failed checkpoint is retained; resume after checking the peer.
break
time.sleep(.3)
finally:
for sig, handler in handlers.items():
signal.signal(sig, handler)
atomic_json(args.out / "results.json", selected)
return 0 if all(row["status"] == "passed" for row in selected) else 1
if __name__ == "__main__":
raise SystemExit(main())
+57 -1
View File
@@ -1,10 +1,56 @@
#!/usr/bin/env python3
"""Start or stop the trace APK's foreground TCP supervisor through adb."""
import argparse
import json
from pathlib import Path
import re
import shlex
import subprocess
def shell(adb, command):
return subprocess.run(adb + ['shell', shlex.join(command)], check=True,
capture_output=True, text=True).stdout
def idle_whitelisted(adb, package):
entries = shell(adb, ['dumpsys', 'deviceidle', 'whitelist'])
return any(package in line.strip().split(',') for line in entries.splitlines())
def allow_idle(adb, serial, package, state_path):
if state_path.exists():
saved = json.loads(state_path.read_text())
if saved['serial'] != serial or saved['package'] != package:
raise ValueError('idle state belongs to a different device or package')
else:
saved = {'serial': serial, 'package': package,
'originally_whitelisted': idle_whitelisted(adb, package)}
state_path.parent.mkdir(parents=True, exist_ok=True)
# Save the original state before the device mutation, including if the
# command's response is lost and a later stop has to recover it.
state_path.write_text(json.dumps(saved, indent=2))
if not idle_whitelisted(adb, package):
shell(adb, ['dumpsys', 'deviceidle', 'whitelist', '+' + package])
if not idle_whitelisted(adb, package):
raise RuntimeError('the package was not added to the device idle whitelist')
print(f'Idle exemption recorded in {state_path}; stop restores the original package setting')
def restore_idle(adb, serial, package, state_path):
if not state_path.exists():
return
saved = json.loads(state_path.read_text())
if saved['serial'] != serial or saved['package'] != package:
raise ValueError('idle state belongs to a different device or package')
if not saved['originally_whitelisted']:
shell(adb, ['dumpsys', 'deviceidle', 'whitelist', '-' + package])
if idle_whitelisted(adb, package):
raise RuntimeError('the package idle exemption could not be restored')
state_path.unlink()
print('Restored the original device idle whitelist setting for ' + package)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('start', 'stop'))
@@ -15,15 +61,25 @@ def main():
parser.add_argument('--token', default='')
parser.add_argument('--env', action='append', default=[], help='KEY=VALUE;KEY overrides (repeatable)')
parser.add_argument('--forward', action='store_true', help='Forward local adb port 40613 to the device')
parser.add_argument('--allow-idle', action='store_true',
help='Temporarily exempt this package from Doze; stop restores its previous setting')
parser.add_argument('--state-file', type=Path,
help='Saved original idle setting (default: ~/.cache/mobilegl/tcp-server/<serial>-<package>.json)')
args = parser.parse_args()
adb = [args.adb, '-s', args.serial]
state_name = re.sub(r'[^A-Za-z0-9_.-]', '_', args.serial + '-' + args.package) + '.json'
state_path = args.state_file or (Path.home() / '.cache/mobilegl/tcp-server' / state_name)
if args.action == 'stop':
command = ['am', 'force-stop', args.package]
else:
if args.allow_idle:
allow_idle(adb, args.serial, args.package, state_path)
command = ['am', 'start-foreground-service', '-n', args.package + '/top.mobilegl.plugin.MobileGLServerService',
'--es', 'listen', args.listen, '--es', 'token', args.token,
'--es', 'env', ';'.join(args.env)]
subprocess.run(adb + ['shell', shlex.join(command)], check=True)
print(shell(adb, command), end='')
if args.action == 'stop':
restore_idle(adb, args.serial, args.package, state_path)
if args.action == 'start' and args.forward:
subprocess.run(adb + ['forward', 'tcp:40613', 'tcp:40613'], check=True)
if args.action == 'start':
@@ -0,0 +1,119 @@
#!/usr/bin/env python3
"""Offline controls for selecting truthful TCP benchmark windows."""
import json
from pathlib import Path
import tempfile
import unittest
from unittest import mock
import benchmark_tcp_credits as benchmark
from benchmark_tcp_credits import validate_measurement
class BenchmarkEvidenceTest(unittest.TestCase):
endpoint = 'tcp://127.0.0.1:40613'
def setUp(self):
self.temporary = tempfile.TemporaryDirectory()
self.addCleanup(self.temporary.cleanup)
self.work = Path(self.temporary.name)
self.client = ['run-ahead ARMED', 'paced by a present credit of 1',
'control=tcp data=stream server=127.0.0.1:40613']
self.server = []
for frame in range(1, 4):
self.client.append('P65LinkMetrics kind=frame frame=' + str(frame) +
' wait_replies=2 rtt_samples=1 rtt_mean_us=8 stage_bytes=1024'
' wall_ns=1000000 client_thread_cpu_ns=500000 rtt_hist_us_pow2=' +
','.join('1' if bucket == 3 else '0' for bucket in range(32)))
self.server.append('P65ServerMetrics valid=1 frame=' + str(frame) +
' wall_ns=1000000 apply_thread_cpu_ns=250000')
def validate(self, credit=1, total=3):
(self.work / 'benchmark.json').write_text(json.dumps({'totalFrames': total}))
(self.work / 'mobilegl.client.log').write_text('\n'.join(self.client))
(self.work / 'mobilegl.server.log').write_text('\n'.join(self.server))
return validate_measurement(self.work, self.endpoint, credit, 2)
def test_valid_matching_tail(self):
result = self.validate()
self.assertEqual((result['first_frame'], result['last_frame']), (2, 3))
self.assertEqual(result['wait_replies_per_frame'], 2)
self.assertEqual(result['rtt_mean_us'], 8)
self.assertEqual(result['fps'], 1000)
def test_server_marker_cannot_arm_client(self):
self.client.remove('run-ahead ARMED')
self.server.append('run-ahead ARMED')
with self.assertRaisesRegex(ValueError, 'requested armed credit'):
self.validate()
def test_demotion_or_lockstep_rejected(self):
for marker in ('running lockstep', 'run-ahead DISARMED'):
with self.subTest(marker=marker):
self.client.append(marker)
with self.assertRaisesRegex(ValueError, 'requested armed credit'):
self.validate()
self.client.pop()
def test_credit_proof_is_exact(self):
self.client[1] = 'paced by a present credit of 10'
with self.assertRaisesRegex(ValueError, 'requested armed credit'):
self.validate()
def test_missing_server_tail_frame_rejected(self):
self.server.pop()
with self.assertRaisesRegex(ValueError, 'matching complete'):
self.validate()
def test_duplicate_client_frame_rejected(self):
self.client.append(self.client[-1])
with self.assertRaisesRegex(ValueError, 'duplicate frame'):
self.validate()
def test_partial_metrics_rejected(self):
with self.assertRaisesRegex(ValueError, 'benchmark and frame metrics differ'):
self.validate(total=4)
def test_missing_cpu_is_not_zero(self):
self.server[-1] = self.server[-1].replace('apply_thread_cpu_ns=250000', 'apply_thread_cpu_ns=0')
with self.assertRaisesRegex(ValueError, 'nonpositive CPU'):
self.validate()
def test_resume_requires_unchanged_artifacts_and_library(self):
valid = self.validate()
inputs = self.work / 'inputs' / 'OpenRA'
inputs.mkdir(parents=True)
(inputs / 'openra.trace').write_bytes(b'trace fixture')
runner, library = self.work / 'runner', self.work / 'library'
runner.write_bytes(b'runner fixture')
library.write_bytes(b'library fixture')
output = self.work / 'results'
argv = ['benchmark_tcp_credits.py', '--runner', str(runner), '--library', str(library),
'--inputs', str(inputs.parent), '--out', str(output), '--endpoint', self.endpoint,
'--case', 'OpenRA', '--credit', '1', '--serial', 'offline-fixture']
def completed_measurement(args, case, credit, work):
work.mkdir(parents=True)
for filename in ('benchmark.json', 'mobilegl.client.log', 'mobilegl.server.log'):
(work / filename).write_bytes((self.work / filename).read_bytes())
return {**valid, 'status': 'passed', 'output': str(work)}
with mock.patch.object(benchmark, 'measure', side_effect=completed_measurement) as measure:
with mock.patch('sys.argv', argv), mock.patch('builtins.print'):
self.assertEqual(benchmark.main(), 0)
with mock.patch('sys.argv', [*argv, '--resume']), mock.patch('builtins.print'):
self.assertEqual(benchmark.main(), 0)
self.assertEqual(measure.call_count, 1)
log = next(output.rglob('mobilegl.client.log'))
log.write_text('tampered')
with mock.patch('sys.argv', [*argv, '--resume']), mock.patch('builtins.print'):
self.assertEqual(benchmark.main(), 0)
self.assertEqual(measure.call_count, 2)
library.write_bytes(b'changed library')
with mock.patch('sys.argv', [*argv, '--resume']), mock.patch('builtins.print'):
self.assertEqual(benchmark.main(), 0)
self.assertEqual(measure.call_count, 3)
if __name__ == '__main__':
unittest.main()
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
"""Small process/fixture stubs only: no GL, network peer, adb or production trace."""
import contextlib
import io
import json
import os
from pathlib import Path
import sys
import tempfile
import textwrap
import types
import unittest
sys.path.insert(0, str(Path(__file__).resolve().parent))
import run_tcp_matrix as matrix
STUB = r"""
import json,os,sys,time
from pathlib import Path
values = dict(arg[2:].split('=',1) for arg in sys.argv[1:] if arg.startswith('-D') and '=' in arg)
work=Path(values['TRACE_OUTPUT_DIR']);(work/'output').mkdir(parents=True,exist_ok=True);(work/'input').mkdir()
behavior=os.environ.get('CASE_BEHAVIOR','success')
if behavior=='active':
for n in range(16):
with (work/'output/progress.log').open('a') as log:log.write(str(n)+'\n')
time.sleep(.05)
if behavior!='no-result':
actual=work/'output/actual.png';actual.write_bytes(b'fixture-image')
endpoint=os.environ['MOBILEGL_IPC_CONTROL'].removeprefix('tcp://')
arm=os.environ.get('CASE_ARM','armed')
if arm=='respect-env':arm='armed' if os.environ.get('MOBILEGL_IPC_RUN_AHEAD')=='1' and os.environ.get('MOBILEGL_IPC_VERB_BARRIER')=='1' else 'lockstep'
line={'armed':'run-ahead ARMED\n','lockstep':'running lockstep\n','disarmed':'run-ahead ARMED\nrun-ahead DISARMED\n','server-only':''}[arm]
(work/'output/mobilegl.client.log').write_text('control=tcp data=stream server='+endpoint+' pid=123 dial=connect\n'+line)
if arm=='server-only':(work/'output/mobilegl.server.log').write_text('run-ahead ARMED\n')
result={'passed':True,'statusCode':0,'backend':values['TRACE_BACKEND'],
'targetCall':int(values['TRACE_TARGET_CALL']),'tracePath':str(work/'input'/values['TRACE_FILE']),
'actualPath':str(actual),'matchedGoldenPath':values['TRACE_GOLDEN'],'ssim':1.0}
(work/'output/result.json').write_text(json.dumps(result))
if behavior=='timeout':time.sleep(60)
raise SystemExit(7 if behavior=='failed' else 0)
"""
@unittest.skipUnless(sys.platform == 'linux', 'Linux/WSL process-group driver')
class MatrixTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='tcp-matrix-test-')
self.root = Path(self.temp.name)
self.source = self.root/'source'
self.tool = self.source/'tools/trace_replay'
self.tool.mkdir(parents=True)
(self.tool/'run_trace_case.cmake').write_text('# stub script')
self.fixtures = self.root/'fixtures';self.fixtures.mkdir()
(self.fixtures/'fixture.tgz').write_bytes(b'fixture archive')
(self.fixtures/'golden.png').write_bytes(b'golden image')
self.library = self.root/'libMobileGL.so';self.library.write_bytes(b'library-v1')
self.stub = self.root/'stub.py';self.stub.write_text(STUB)
self.catalog = self.root/'catalog.json'
self.out = self.root/'results'
self.manifest = self.tool/'trace_cases.json'
self.write_inputs()
def tearDown(self):
self.temp.cleanup()
def write_inputs(self, behavior='success', extra_cases=None):
cases=[{'name':'Example','trace_archive':'fixture.tgz','trace_file':'trace.trace',
'golden':'golden.png','target_call':42,'width':16,'height':16,'timeout_seconds':.05}]
if extra_cases:cases.extend(extra_cases)
self.manifest.write_text(json.dumps({'defaults':{'ssim_threshold':.99},'cases':cases}))
command=[sys.executable,str(self.stub),'-DTRACE_REPLAY_EXE='+sys.executable,
'-DMOBILEGL_LIBRARY=/old/library','-DTRACE_CASE_NAME=Example','-DTRACE_BACKEND=DirectGLES',
'-DTRACE_ARCHIVE='+str(self.fixtures/'fixture.tgz'),'-DTRACE_OUTPUT_DIR=/old/output',
'-P',str(self.tool/'run_trace_case.cmake')]
self.catalog.write_text(json.dumps({'kind':'ctestInfo','tests':[{
'name':'MobileGLTraceReplay.Example.DirectGLES','command':command,
'properties':[{'name':'ENVIRONMENT','value':['CASE_BEHAVIOR='+behavior]},
{'name':'TIMEOUT','value':.05},{'name':'WORKING_DIRECTORY','value':str(self.root)}]}]}))
def run_main(self, *extra):
args=['--catalog',str(self.catalog),'--source',str(self.source),'--library',str(self.library),
'--runner',sys.executable,'--endpoint','tcp://127.0.0.1:9','--token','not-saved-in-checkpoint',
'--out',str(self.out),'--idle-seconds','1','--max-seconds','5',*extra]
with contextlib.redirect_stdout(io.StringIO()):
return matrix.main(args)
def checkpoint(self):
return matrix.read_json(self.out/'checkpoint.json')['runs']
def set_arm(self, arm, extra=None):
catalog=matrix.read_json(self.catalog)
catalog['tests'][0]['properties'][0]['value'] += ['CASE_ARM='+arm,*(extra or [])]
self.catalog.write_text(json.dumps(catalog))
def test_required_arm_forces_both_environment_knobs_and_records_actual_arm(self):
self.set_arm('respect-env',['MOBILEGL_IPC_RUN_AHEAD=0','MOBILEGL_IPC_VERB_BARRIER=0'])
self.assertEqual(self.run_main('--require-run-ahead'),0)
row=self.checkpoint()[0]
self.assertEqual(row['requested_arm'],{'MOBILEGL_IPC_RUN_AHEAD':'1','MOBILEGL_IPC_VERB_BARRIER':'1'})
self.assertEqual(row['actual_arm'],{'armed':True,'lockstep':False,'disarmed':False})
self.assertEqual(self.run_main('--require-run-ahead','--resume'),0)
self.assertEqual(len(self.checkpoint()),1)
def test_required_arm_rejects_lockstep_disarmed_and_server_only_marker(self):
for arm in ('lockstep','disarmed','server-only'):
with self.subTest(arm=arm):
self.write_inputs()
self.set_arm(arm)
self.assertEqual(self.run_main('--require-run-ahead'),1)
row=self.checkpoint()[-1]
self.assertEqual(row['status'],'failed')
self.assertIn('required run-ahead',row['error'])
self.assertIsInstance(row['actual_arm'],dict)
def test_required_arm_does_not_resume_a_former_lockstep_image_pass(self):
self.set_arm('lockstep')
self.assertEqual(self.run_main(),0)
self.assertTrue(self.checkpoint()[0]['actual_arm']['lockstep'])
self.assertEqual(self.run_main('--require-run-ahead','--resume'),1)
self.assertEqual(len(self.checkpoint()),2)
self.assertEqual(self.checkpoint()[-1]['status'],'failed')
def test_success_is_resumed_only_while_its_evidence_is_intact(self):
self.assertEqual(self.run_main(),0)
self.assertEqual(self.run_main('--resume'),0)
self.assertEqual(len(self.checkpoint()),1)
record=self.checkpoint()[0]
self.assertNotIn('not-saved-in-checkpoint',(self.out/'checkpoint.json').read_text())
Path(record['result_path']).write_text('{"passed":false}')
self.assertEqual(self.run_main('--resume'),0)
self.assertEqual(len(self.checkpoint()),2)
self.assertNotEqual(self.checkpoint()[0]['work'],self.checkpoint()[1]['work'])
def test_nonzero_exit_with_passed_json_never_becomes_a_checkpoint_pass(self):
self.write_inputs('failed')
self.assertEqual(self.run_main(),1)
self.assertEqual(self.run_main('--resume'),1)
self.assertEqual(len(self.checkpoint()),2)
self.assertTrue(all(row['status']=='failed' and row['returncode']==7 for row in self.checkpoint()))
def test_timeout_with_passed_json_is_not_resumed_as_success(self):
self.write_inputs('timeout')
self.assertEqual(self.run_main('--idle-seconds','.2'),1)
self.assertEqual(self.run_main('--idle-seconds','.2','--resume'),1)
self.assertEqual(len(self.checkpoint()),2)
self.assertTrue(all(row['status']=='timeout' and row['returncode']==124 for row in self.checkpoint()))
def test_stale_scratch_result_cannot_satisfy_a_new_attempt(self):
old=self.out/'Example/output/result.json';old.parent.mkdir(parents=True)
old.write_text('{"passed":true,"statusCode":0}')
legacy='[{"case":"Example","returncode":0}]'
(self.out/'results.json').write_text(legacy)
self.write_inputs('no-result')
self.assertEqual(self.run_main('--resume'),1)
row=self.checkpoint()[0]
self.assertEqual(row['status'],'failed')
self.assertEqual(row['returncode'],1)
self.assertEqual(json.loads(old.read_text())['passed'],True) # Old evidence is preserved, not promoted.
backups=list(self.out.glob('results.legacy-*.json'))
self.assertEqual(len(backups),1)
self.assertEqual(backups[0].read_text(),legacy)
def test_artifact_change_invalidates_a_successful_checkpoint(self):
self.assertEqual(self.run_main(),0)
self.library.write_bytes(b'library-v2')
self.assertEqual(self.run_main('--resume'),0)
self.assertEqual(len(self.checkpoint()),2)
self.assertNotEqual(self.checkpoint()[0]['identity'],self.checkpoint()[1]['identity'])
def test_active_logs_outlive_old_local_catalog_and_manifest_timeout(self):
self.write_inputs('active')
self.assertEqual(self.run_main('--idle-seconds','.3','--max-seconds','4'),0)
row=self.checkpoint()[0]
self.assertGreater(row['seconds'],.6) # Both obsolete local timeouts are .05 seconds.
self.assertIsNone(row['timeout_kind'])
self.assertEqual(row['status'],'passed')
def test_timeout_kills_and_reaps_a_child_that_ignores_term_after_parent_exits(self):
work=self.root/'group';work.mkdir()
pidfile=work/'child.pid'
child=f"import os,signal,time;from pathlib import Path;signal.signal(signal.SIGTERM,signal.SIG_IGN);Path({str(pidfile)!r}).write_text(str(os.getpid()));time.sleep(60)"
parent=f"import subprocess,sys,time;subprocess.Popen([sys.executable,'-c',{child!r}]);time.sleep(60)"
outcome=matrix.supervise([sys.executable,'-c',parent],dict(os.environ),str(work),work,work/'runner.log',
idle_seconds=1,max_seconds=4,poll_seconds=.05,terminate_grace=.1)
self.assertEqual(outcome['timeout_kind'],'idle')
self.assertEqual(outcome['remaining_processes'],[])
pid=int(pidfile.read_text())
self.assertFalse(Path('/proc',str(pid)).exists(),f'orphan/zombie {pid} survived cleanup')
def test_case_environment_starts_from_fresh_base_and_owned_transport_wins(self):
args=types.SimpleNamespace(endpoint='tcp://127.0.0.1:9',token='x',credit=2)
first={'properties':[{'name':'ENVIRONMENT','value':['CASE_ONLY=one','MOBILEGL_IPC_CONTROL=tcp://stale:1']}]}
base={'BASE':'same'}
a=matrix.case_environment(first,base,args)
b=matrix.case_environment({'properties':[]},base,args)
self.assertEqual(a['CASE_ONLY'],'one')
self.assertNotIn('CASE_ONLY',b)
self.assertEqual(a['MOBILEGL_IPC_CONTROL'],args.endpoint)
self.assertEqual(base,{'BASE':'same'})
def test_current_manifest_default_and_explicit_non_ci_selection(self):
common={'trace_archive':'fixture.tgz','trace_file':'trace.trace','golden':'golden.png',
'target_call':42,'width':16,'height':16}
self.write_inputs(extra_cases=[dict(common,name='Extra',ci=False),dict(common,name='OptOut',split=False)])
self.assertEqual([c['name'] for c in matrix.select_cases(self.manifest,None)],['Example'])
self.assertEqual(matrix.select_cases(self.manifest,['Extra'])[0]['name'],'Extra')
with self.assertRaises(ValueError):matrix.select_cases(self.manifest,['OptOut'])
if __name__ == '__main__':
unittest.main(verbosity=2)