mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-17 15:03:25 +09:00
[MG_Remote, MG_Pipe, MG_Test, CI, tools, docs] (P6.5): the first wave completed and reviewed - archive v2, publication boundary, half-close, AcceptPair/role fixes, device re-validation
This commit is contained in:
@@ -403,3 +403,110 @@ the ANGLE the Android lane runs. The CI lane uses a pinned build
|
||||
extension set differ from the SDK copy (`GL_EXT_texture_buffer` support, ES 3.2
|
||||
entry points). Compare `GL_RENDERER` and the relevant extension lists on both
|
||||
stacks before treating a local result as a statement about CI.
|
||||
|
||||
|
||||
## TCP matrix driver (Linux / WSL)
|
||||
|
||||
`run_tcp_matrix.py` uses a CTest JSON catalog plus the **current** `trace_cases.json`.
|
||||
It does not build, install an APK, or launch the remote server. For the formal
|
||||
P6.5 run-ahead matrix, use `--require-run-ahead`. It forces `MOBILEGL_IPC_RUN_AHEAD=1`
|
||||
and `MOBILEGL_IPC_VERB_BARRIER=1`, then requires `run-ahead ARMED` in each private
|
||||
client log and rejects any `running lockstep` / `DISARMED` marker. The checkpoint
|
||||
records `requested_arm` and `actual_arm`; strict resume cannot reuse an earlier
|
||||
lockstep image pass. Start the intended server first, then run serially:
|
||||
|
||||
```bash
|
||||
ctest --test-dir /path/to/retrace-build --show-only=json-v1 > /tmp/trace-catalog.json
|
||||
python3 tools/trace_replay/run_tcp_matrix.py \
|
||||
--catalog /tmp/trace-catalog.json \
|
||||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||||
--endpoint tcp://192.168.21.181:40613 --token devtoken \
|
||||
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix
|
||||
# Continue the same frozen library/runner/peer:
|
||||
python3 tools/trace_replay/run_tcp_matrix.py \
|
||||
--catalog /tmp/trace-catalog.json \
|
||||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||||
--endpoint tcp://192.168.21.181:40613 --token devtoken \
|
||||
--backend DirectGLES --credit 2 --require-run-ahead --out /tmp/tcp-matrix --resume
|
||||
```
|
||||
|
||||
`--build-dir` can replace `--catalog` to obtain the JSON without building.
|
||||
`--runner` overrides the catalog runner; fixture paths default to the catalog's
|
||||
fixture directory and can be overridden with `--fixtures`. Repeat `--case` or
|
||||
`--backend` to select a subset. The default is the manifest's CI split subset
|
||||
(currently 39 cases), honoring each case's backend list. Explicit `--case
|
||||
minecraft-1.21.4-rd12-odinlite-in-world` permits that non-CI workload separately;
|
||||
an explicit `split:false` always remains excluded.
|
||||
|
||||
The watchdog defaults to **300 seconds without log progress** and a separate
|
||||
**7200-second absolute ceiling** (`--idle-seconds`, `--max-seconds`). It observes
|
||||
runner output and trace/client/server log changes. It deliberately ignores the
|
||||
old local CTest/manifest `timeout_seconds`: TCP main-menu kept progressing beyond
|
||||
its old local 180-second budget. A growing log does not bypass the explicit absolute ceiling.
|
||||
On timeout the driver terminates and reaps the entire case process group,
|
||||
including a child that ignores TERM after CMake exits, and stops the matrix to
|
||||
avoid cascading Busy failures. Check the peer before resuming. Ctrl-C/TERM/HUP
|
||||
also clean up the group and save a cancelled checkpoint.
|
||||
|
||||
Each attempt has a new directory. `checkpoint.json` is written atomically, and
|
||||
only a zero exit plus current-attempt successful result, matching backend/call/
|
||||
golden, adequate SSIM, an actual image, and the requested TCP arm marker count
|
||||
as passed. Resume also checks request/artifact identity and stored evidence
|
||||
hashes. Failed, timed-out, interrupted, changed, or missing evidence is rerun;
|
||||
old scratch `results.json` files are retained but never silently promoted into
|
||||
successful checkpoints. Replacing the remote deployment requires a new output
|
||||
root unless it is the same frozen peer. `results.json` summarizes the currently
|
||||
selected runs and marks reused evidence with `resumed:true`.
|
||||
|
||||
`--wake-adb-serial 2f7cbe2e` explicitly opts into `adb shell input keyevent
|
||||
KEYCODE_WAKEUP` every 15 seconds during a case. It does not install anything or
|
||||
change global power/idle settings. Its own adb log is excluded from the progress
|
||||
watchdog, so wakeup messages cannot conceal a stalled trace.
|
||||
|
||||
Tool regression tests use short process stubs, never a GPU/device/trace:
|
||||
|
||||
```bash
|
||||
python3 tools/trace_replay/test_run_tcp_matrix.py
|
||||
```
|
||||
|
||||
## TCP credit and Stage measurements
|
||||
|
||||
`benchmark_tcp_credits.py` runs the complete OpenRA and rd12 traces at credits
|
||||
1, 2, and 3, then selects matching client/server tail frame IDs (100 and 200
|
||||
frames respectively). Extract the unchanged trace files as
|
||||
`inputs/OpenRA/openra.trace` and `inputs/rd12/trace.trace`. Keep the server and
|
||||
library frozen, enable server `MOBILEGL_PIPE_STATS=1` and
|
||||
`MOBILEGL_PIPE_STATS_PERIOD=1`, and leave the device connection free:
|
||||
|
||||
```bash
|
||||
python3 tools/trace_replay/benchmark_tcp_credits.py \
|
||||
--runner /path/to/mobilegl_trace_replay --library /path/to/libMobileGL.so \
|
||||
--inputs /path/to/inputs --out /tmp/tcp-credits \
|
||||
--endpoint tcp://192.168.21.181:40613 --token devtoken --serial 2f7cbe2e
|
||||
python3 tools/trace_replay/measure_tcp_stage_burst.py \
|
||||
--library /path/to/libMobileGL.so --out /tmp/tcp-stage-burst \
|
||||
--endpoint tcp://192.168.21.181:40613 --token devtoken
|
||||
```
|
||||
|
||||
The credit runner serializes all six runs, sends the same explicit awake key as
|
||||
the matrix driver, and preserves complete benchmark and role logs. It requires
|
||||
the requested credit and continuous run-ahead ARMED proof in the client log,
|
||||
matching complete tail frames, and positive per-thread CPU observations.
|
||||
`--case` and `--credit` select a subset; `--resume` reuses only matching identity
|
||||
and intact evidence. FPS, `kWaitReply` counts, bucketed RTT distribution, Stage
|
||||
bytes, and the two role CPU clocks come from the selected steady window.
|
||||
Completing this benchmark does not perform or replace a golden image comparison.
|
||||
|
||||
The Stage tool uploads exactly 64 MiB after one warmup frame, consumes the dirty
|
||||
buffer in a draw, and waits for apply. It requires actual TCP run-ahead and at
|
||||
least 64 MiB of observed Stage bytes in frame 2, preserves artifact hashes, and
|
||||
refuses to overwrite an existing output directory. Its throughput includes
|
||||
client copies and backend work; it is not raw Wi-Fi bandwidth. Divide steady
|
||||
frame Stage traffic by this separately measured throughput only with that
|
||||
interpretation. Device matrices, benchmarks, and the burst must run one at a time.
|
||||
|
||||
Offline evidence controls (no device):
|
||||
|
||||
```bash
|
||||
python3 tools/trace_replay/test_benchmark_tcp_credits.py
|
||||
```
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Record complete TCP benchmark runs for credit 1/2/3 with exact common tail frames."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from measure_transport_cpu import CASES, digest
|
||||
from measure_loopback_cpu import frame_rows, METRICS
|
||||
from run_tcp_matrix import supervise, atomic_json, fingerprint, actual_arm_from_text
|
||||
|
||||
|
||||
def validate_measurement(work, endpoint, credit, tail):
|
||||
bench = json.loads((work / 'benchmark.json').read_text())
|
||||
client_path, server_path = work / 'mobilegl.client.log', work / 'mobilegl.server.log'
|
||||
text = client_path.read_text(errors='replace')
|
||||
actual_arm = actual_arm_from_text(text)
|
||||
if (actual_arm != {'armed': True, 'lockstep': False, 'disarmed': False}
|
||||
or not re.search(r'paced by a present credit of ' + str(credit) + r'\b', text)
|
||||
or 'control=tcp data=stream server=' + endpoint.removeprefix('tcp://') not in text):
|
||||
raise ValueError('runtime does not prove TCP and the requested armed credit')
|
||||
client = frame_rows(client_path, 'P65LinkMetrics ')
|
||||
server = frame_rows(server_path, 'P65ServerMetrics ', server=True)
|
||||
ids = sorted(client)
|
||||
if len(ids) != bench['totalFrames'] or len(ids) <= tail:
|
||||
raise ValueError('benchmark and frame metrics differ, or tail contains setup')
|
||||
selected = ids[-tail:]
|
||||
if selected != list(range(selected[0], selected[-1] + 1)) or any(n not in server for n in selected):
|
||||
raise ValueError('no matching complete client/server frame tail')
|
||||
for number in selected:
|
||||
if int(client[number]['client_thread_cpu_ns']) <= 0 or int(server[number]['apply_thread_cpu_ns']) <= 0:
|
||||
raise ValueError('nonpositive CPU measurement in selected frame ' + str(number))
|
||||
summary = METRICS.summarize(client_path, selected[0] - 1)
|
||||
summary.update(METRICS.summarize_server(server_path, selected[0] - 1))
|
||||
if summary['frames'] != tail or summary['server_frames'] != tail:
|
||||
raise ValueError('unexpected frames beyond the chosen tail')
|
||||
return {'run_ahead_armed': True, 'actual_arm': actual_arm,
|
||||
'first_frame': selected[0], 'last_frame': selected[-1], 'benchmark': bench,
|
||||
'benchmark_sha256': digest(work / 'benchmark.json'),
|
||||
'client_log_sha256': digest(client_path), 'server_log_sha256': digest(server_path), **summary}
|
||||
|
||||
|
||||
def measure(args, case, credit, work):
|
||||
_, trace_name, width, height, tail = CASES[case]
|
||||
trace = args.inputs / case / trace_name
|
||||
work.mkdir(parents=True)
|
||||
env = dict(os.environ, MOBILEGL_TRANSPORT='spawn', MOBILEGL_IPC_CONTROL=args.endpoint,
|
||||
MOBILEGL_IPC_DATA='stream', MOBILEGL_IPC_TOKEN=args.token, MOBILEGL_IPC_REQUIRE_SAME_BUILD='1',
|
||||
MOBILEGL_IPC_LOG_FORWARD='1', MOBILEGL_PIPE_STATS='1', MOBILEGL_PIPE_STATS_PERIOD='1',
|
||||
MOBILEGL_IPC_PRESENT_CREDIT=str(credit), MOBILEGL_IPC_RUN_AHEAD='1', MOBILEGL_IPC_VERB_BARRIER='1',
|
||||
MOBILEGL_IPC_BATCH_WAITS='1', MOBILEGL_BACKEND_TYPE='DirectGLES',
|
||||
MOBILEGL_IPC_STRICT_ERRORS='0', MOBILEGL_IPC_ROLE_SPLIT_STATE='0', MOBILEGL_IPC_AUDIT='0')
|
||||
for name in ('MOBILEGL_IPC_DIAL', 'MOBILEGL_IPC_SERVER_PATH'):
|
||||
env.pop(name, None)
|
||||
command = [str(args.runner), '--trace', str(trace), '--mobilegl-library', str(args.library),
|
||||
'--backend', 'DirectGLES', '--width', str(width), '--height', str(height),
|
||||
'--output', str(work), '--benchmark', '--benchmark-finish=0', '--benchmark-tail-frames=' + str(tail)]
|
||||
def wake():
|
||||
# Keep maintenance output outside the progress scan: a failed adb wake
|
||||
# must not make a stalled replay look active.
|
||||
with (work.parent / 'wake-adb.log').open('ab') as log:
|
||||
try:
|
||||
subprocess.run([args.adb, '-s', args.serial, 'shell', 'input', 'keyevent', 'KEYCODE_WAKEUP'],
|
||||
stdout=log, stderr=log, timeout=5, check=False)
|
||||
except (OSError, subprocess.TimeoutExpired) as error:
|
||||
log.write((str(error) + '\n').encode())
|
||||
result = supervise(command, env, str(args.inputs), work, work / 'runner.log',
|
||||
idle_seconds=300, max_seconds=7200, wake=wake)
|
||||
if result['returncode'] or result['timeout_kind'] or result['remaining_processes'] or result['leftovers_after_parent_exit']:
|
||||
return {**result, 'status': 'failed', 'output': str(work), 'command': command}
|
||||
actual_arm = actual_arm_from_text((work / 'mobilegl.client.log').read_text(errors='replace'))
|
||||
try:
|
||||
values = validate_measurement(work, args.endpoint, credit, tail)
|
||||
except (OSError, ValueError, KeyError) as error:
|
||||
return {**result, 'status': 'failed', 'actual_arm': actual_arm,
|
||||
'error': str(error), 'output': str(work), 'command': command}
|
||||
return {**result, 'status': 'passed', 'output': str(work), 'command': command, **values}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--runner', type=Path, required=True)
|
||||
parser.add_argument('--library', type=Path, required=True)
|
||||
parser.add_argument('--inputs', type=Path, required=True)
|
||||
parser.add_argument('--out', type=Path, required=True)
|
||||
parser.add_argument('--endpoint', required=True)
|
||||
parser.add_argument('--token', default=os.environ.get('MOBILEGL_IPC_TOKEN', ''))
|
||||
parser.add_argument('--serial', required=True, help='explicit adb serial to keep awake during each run')
|
||||
parser.add_argument('--adb', default='adb')
|
||||
parser.add_argument('--case', action='append', choices=CASES)
|
||||
parser.add_argument('--credit', action='append', type=int, choices=(1, 2, 3))
|
||||
parser.add_argument('--resume', action='store_true')
|
||||
args = parser.parse_args()
|
||||
if not args.endpoint.startswith('tcp://'):
|
||||
parser.error('a TCP endpoint is required')
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
checkpoint = args.out / 'checkpoint.json'
|
||||
rows = json.loads(checkpoint.read_text()) if checkpoint.exists() else []
|
||||
if rows and not args.resume:
|
||||
parser.error('output already contains evidence; use --resume or a new output directory')
|
||||
for case in args.case or CASES:
|
||||
for credit in args.credit or (1, 2, 3):
|
||||
provenance = {'case': case, 'credit': credit, 'endpoint': args.endpoint,
|
||||
'token_hash': fingerprint(args.token), 'runner_sha256': digest(args.runner),
|
||||
'library_sha256': digest(args.library),
|
||||
'trace_sha256': digest(args.inputs / case / CASES[case][1]),
|
||||
'script_sha256': digest(Path(__file__)),
|
||||
'helper_sha256': {name: digest(Path(__file__).with_name(name)) for name in (
|
||||
'measure_transport_cpu.py', 'measure_loopback_cpu.py', 'run_tcp_matrix.py')},
|
||||
'metrics_sha256': digest(Path(METRICS.__file__))}
|
||||
identity = fingerprint(provenance)
|
||||
prior = next((row for row in reversed(rows) if row.get('identity') == identity and row.get('status') == 'passed'), None)
|
||||
if args.resume and prior:
|
||||
directory = Path(prior['output'])
|
||||
if all((directory / filename).is_file() and digest(directory / filename) == prior[field]
|
||||
for filename, field in (('benchmark.json', 'benchmark_sha256'), ('mobilegl.client.log', 'client_log_sha256'),
|
||||
('mobilegl.server.log', 'server_log_sha256'))):
|
||||
print(json.dumps({'case': case, 'credit': credit, 'status': 'passed', 'resumed': True}), flush=True)
|
||||
continue
|
||||
work = args.out / case / f'credit-{credit}' / ('attempt-' + uuid.uuid4().hex)
|
||||
row = {'case': case, 'credit': credit, 'identity': identity,
|
||||
'provenance': provenance, 'started_at_ns': time.time_ns()}
|
||||
print(json.dumps({**row, 'status': 'running'}), flush=True)
|
||||
try:
|
||||
row.update(measure(args, case, credit, work))
|
||||
except Exception as error:
|
||||
row.update(status='failed', error=str(error), output=str(work))
|
||||
rows.append(row)
|
||||
atomic_json(checkpoint, rows)
|
||||
print(json.dumps({key: row.get(key) for key in ('case', 'credit', 'status', 'seconds', 'fps', 'wait_replies_per_frame',
|
||||
'rtt_mean_us', 'client_thread_cpu_ms_per_frame', 'error')}), flush=True)
|
||||
if row['status'] != 'passed':
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare current unix+shm and loopback TCP+stream using per-frame role CPU clocks.
|
||||
|
||||
Start a stats-enabled supervisor separately at --endpoint. This runner neither
|
||||
contacts a device nor changes production code. It refuses partial or mismatched
|
||||
client/server frame windows instead of treating missing CPU observations as zero.
|
||||
"""
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import subprocess
|
||||
|
||||
from measure_transport_cpu import CASES, digest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SPEC = importlib.util.spec_from_file_location('p65_metrics', ROOT / 'scripts/ci/p65_link_metrics.py')
|
||||
METRICS = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(METRICS)
|
||||
|
||||
|
||||
def frame_rows(path, marker, server=False):
|
||||
rows = {}
|
||||
for line in path.read_text(encoding='utf-8', errors='replace').splitlines():
|
||||
if marker not in line:
|
||||
continue
|
||||
row = dict(METRICS.FIELDS.findall(line.split(marker, 1)[1]))
|
||||
if server:
|
||||
if row.get('valid') != '1':
|
||||
continue
|
||||
elif row.get('kind') != 'frame':
|
||||
continue
|
||||
number = int(row['frame'])
|
||||
if number in rows:
|
||||
raise ValueError(f'{path}: duplicate frame {number}')
|
||||
rows[number] = row
|
||||
return rows
|
||||
|
||||
|
||||
def run(args, case, transport, repeat):
|
||||
_, trace_name, width, height, tail = CASES[case]
|
||||
trace = args.inputs / case / trace_name
|
||||
if not trace.is_file():
|
||||
raise ValueError(f'{trace}: extract the unchanged fixture before measuring')
|
||||
out = args.out / case / transport / f'run-{repeat}'
|
||||
if out.exists():
|
||||
raise ValueError(f'{out}: choose a new output directory; evidence is never overwritten')
|
||||
out.mkdir(parents=True)
|
||||
env = dict(os.environ, MOBILEGL_TRANSPORT='spawn', MOBILEGL_BACKEND_TYPE='DirectGLES',
|
||||
MOBILEGL_IPC_SERVER_PATH=str(args.server), MOBILEGL_PIPE_STATS='1', MOBILEGL_PIPE_STATS_PERIOD='1',
|
||||
MOBILEGL_IPC_PRESENT_CREDIT='1', MOBILEGL_IPC_RUN_AHEAD='1', MOBILEGL_IPC_VERB_BARRIER='1',
|
||||
MOBILEGL_IPC_BATCH_WAITS='1', MOBILEGL_IPC_SPIN_US='50', MOBILEGL_IPC_STRICT_ERRORS='0',
|
||||
MOBILEGL_IPC_ROLE_SPLIT_STATE='0', MOBILEGL_IPC_AUDIT='0',
|
||||
LIBGL_ALWAYS_SOFTWARE='1', EGL_PLATFORM='surfaceless',
|
||||
MESA_GL_VERSION_OVERRIDE='3.3', MESA_GLSL_VERSION_OVERRIDE='330')
|
||||
for key in ('MOBILEGL_IPC_CONTROL', 'MOBILEGL_IPC_DATA', 'MOBILEGL_IPC_TOKEN', 'MOBILEGL_IPC_DIAL',
|
||||
'MOBILEGL_IPC_REQUIRE_SAME_BUILD', 'MOBILEGL_TEST_SERVER_COUNTERS'):
|
||||
env.pop(key, None)
|
||||
if transport == 'tcp':
|
||||
env.update(MOBILEGL_IPC_CONTROL=args.endpoint, MOBILEGL_IPC_DATA='stream', MOBILEGL_IPC_TOKEN='',
|
||||
MOBILEGL_IPC_REQUIRE_SAME_BUILD='1', MOBILEGL_IPC_LOG_FORWARD='1')
|
||||
if Path('/usr/share/glvnd/egl_vendor.d/50_mesa.json').exists():
|
||||
env['__EGL_VENDOR_LIBRARY_FILENAMES'] = '/usr/share/glvnd/egl_vendor.d/50_mesa.json'
|
||||
command = [str(args.runner), '--trace', str(trace), '--mobilegl-library', str(args.library),
|
||||
'--backend', 'DirectGLES', '--width', str(width), '--height', str(height), '--output', str(out),
|
||||
'--benchmark', '--benchmark-finish=0', '--benchmark-tail-frames=' + str(tail)]
|
||||
with (out / 'runner.log').open('w') as log:
|
||||
result = subprocess.run(command, env=env, stdout=log, stderr=log, timeout=900)
|
||||
if result.returncode:
|
||||
raise RuntimeError(f'{out}: benchmark exited {result.returncode}')
|
||||
bench = json.loads((out / 'benchmark.json').read_text())
|
||||
client_path, server_path = out / 'mobilegl.client.log', out / 'mobilegl.server.log'
|
||||
client = frame_rows(client_path, 'P65LinkMetrics ')
|
||||
server = frame_rows(server_path, 'P65ServerMetrics ', server=True)
|
||||
ids = sorted(client)
|
||||
if len(ids) != bench['totalFrames'] or len(ids) <= tail:
|
||||
raise ValueError(f'{out}: incomplete frame metrics or tail includes setup')
|
||||
selected = ids[-tail:]
|
||||
if selected != list(range(selected[0], selected[-1] + 1)) or any(frame not in server for frame in selected):
|
||||
raise ValueError(f'{out}: client/server tail frame IDs disagree')
|
||||
first, last = selected[0], selected[-1]
|
||||
# The existing reducer now sees precisely the proven common tail range.
|
||||
values = METRICS.summarize(client_path, first - 1)
|
||||
values.update(METRICS.summarize_server(server_path, first - 1))
|
||||
if values['frames'] != tail or values['server_frames'] != tail:
|
||||
raise ValueError(f'{out}: unexpected frames outside the selected tail')
|
||||
for frame in selected:
|
||||
if int(client[frame]['client_thread_cpu_ns']) <= 0 or int(server[frame]['apply_thread_cpu_ns']) <= 0:
|
||||
raise ValueError(f'{out}: nonpositive per-thread CPU in frame {frame}')
|
||||
proof = client_path.read_text(errors='replace')
|
||||
expected = 'control=tcp data=stream' if transport == 'tcp' else 'spawn ARMED - the server role runs in pid '
|
||||
if expected not in proof:
|
||||
raise ValueError(f'{out}: no proof of the requested transport')
|
||||
if ('run-ahead ARMED' not in proof or 'running lockstep' in proof or 'run-ahead DISARMED' in proof):
|
||||
raise ValueError(f'{out}: requested run-ahead was not continuously armed; not a pure transport CPU comparison')
|
||||
row = {'case': case, 'transport': transport, 'repeat': repeat,
|
||||
'first_frame': first, 'last_frame': last, 'tail_frames': tail,
|
||||
'run_ahead_armed': True,
|
||||
'benchmark_client_cpu_ms': statistics.mean(bench['frameCpuTimesMs'][-tail:]),
|
||||
'runner_sha256': args.runner_hash, 'library_sha256': args.library_hash,
|
||||
'server_sha256': args.server_hash, 'trace_sha256': digest(trace),
|
||||
'command': command, 'output': str(out), **values}
|
||||
(out / 'role-cpu.json').write_text(json.dumps(row, indent=2), encoding='utf-8')
|
||||
return row
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--runner', type=Path, required=True)
|
||||
parser.add_argument('--library', type=Path, required=True)
|
||||
parser.add_argument('--server', type=Path, required=True)
|
||||
parser.add_argument('--inputs', type=Path, required=True)
|
||||
parser.add_argument('--out', type=Path, required=True)
|
||||
parser.add_argument('--endpoint', default='tcp://127.0.0.1:40616')
|
||||
parser.add_argument('--transport', action='append', choices=('spawn', 'tcp'),
|
||||
help='Measure only the selected arm; default measures both')
|
||||
parser.add_argument('--case', action='append', choices=CASES,
|
||||
help='Measure only selected cases; default measures OpenRA and rd12')
|
||||
args = parser.parse_args()
|
||||
if not args.endpoint.startswith('tcp://127.0.0.1:'):
|
||||
parser.error('this measurement is specifically loopback, not a device benchmark')
|
||||
args.runner_hash, args.library_hash, args.server_hash = map(digest, (args.runner, args.library, args.server))
|
||||
rows = []
|
||||
for case, repeats in (('OpenRA', 3), ('rd12', 1)):
|
||||
if args.case and case not in args.case:
|
||||
continue
|
||||
for repeat in range(repeats):
|
||||
for transport in args.transport or ('spawn', 'tcp'):
|
||||
row = run(args, case, transport, repeat)
|
||||
rows.append(row)
|
||||
(args.out / 'summary.json').write_text(json.dumps(rows, indent=2), encoding='utf-8')
|
||||
print(json.dumps({key: row[key] for key in ('case', 'transport', 'repeat', 'first_frame', 'last_frame',
|
||||
'client_thread_cpu_ms_per_frame', 'apply_thread_cpu_ms_per_frame')}), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Measure one real 64 MiB buffer upload consumed by a TCP server draw.
|
||||
|
||||
The timed interval includes client staging, stream delivery, backend work, and
|
||||
an applied fence. Its result is end-to-end staged throughput, not raw network
|
||||
capacity. Start the same-build stats-enabled server before running this tool.
|
||||
"""
|
||||
import argparse
|
||||
import ctypes as c
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import time
|
||||
|
||||
from measure_transport_cpu import digest
|
||||
from measure_loopback_cpu import frame_rows
|
||||
from run_tcp_matrix import actual_arm_from_text
|
||||
|
||||
|
||||
UPLOAD_BYTES = 64 * 1024 * 1024
|
||||
|
||||
|
||||
def run(args):
|
||||
args.out.mkdir(parents=True)
|
||||
os.environ.update(
|
||||
MOBILEGL_TRANSPORT='spawn', MOBILEGL_IPC_CONTROL=args.endpoint,
|
||||
MOBILEGL_IPC_DATA='stream', MOBILEGL_IPC_TOKEN=args.token,
|
||||
MOBILEGL_IPC_REQUIRE_SAME_BUILD='1', MOBILEGL_IPC_LOG_FORWARD='1',
|
||||
MOBILEGL_BACKEND_TYPE='DirectGLES', MOBILEGL_PIPE_STATS='1',
|
||||
MOBILEGL_PIPE_STATS_PERIOD='1', MOBILEGL_IPC_RUN_AHEAD='1',
|
||||
MOBILEGL_IPC_VERB_BARRIER='1', MOBILEGL_IPC_PRESENT_CREDIT='1',
|
||||
MOBILEGL_LOG_FILE_PATH=str(args.out / 'mobilegl.log'), EGL_PLATFORM='surfaceless')
|
||||
for name in ('MOBILEGL_IPC_DIAL', 'MOBILEGL_IPC_SERVER_PATH'):
|
||||
os.environ.pop(name, None)
|
||||
lib = c.CDLL(str(args.library), mode=c.RTLD_GLOBAL)
|
||||
P, I, U, F = c.c_void_p, c.c_int, c.c_uint, c.c_float
|
||||
|
||||
def fn(name, result, parameters):
|
||||
function = getattr(lib, name)
|
||||
function.restype, function.argtypes = result, parameters
|
||||
return function
|
||||
|
||||
get_display = fn('eglGetDisplay', P, [P])
|
||||
initialize = fn('eglInitialize', U, [P, c.POINTER(I), c.POINTER(I)])
|
||||
bind_api = fn('eglBindAPI', U, [U])
|
||||
choose = fn('eglChooseConfig', U, [P, c.POINTER(I), c.POINTER(P), I, c.POINTER(I)])
|
||||
context = fn('eglCreateContext', P, [P, P, P, c.POINTER(I)])
|
||||
surface = fn('eglCreatePbufferSurface', P, [P, P, c.POINTER(I)])
|
||||
current = fn('eglMakeCurrent', U, [P, P, P, P])
|
||||
swap = fn('eglSwapBuffers', U, [P, P])
|
||||
terminate = fn('eglTerminate', U, [P])
|
||||
display = get_display(None)
|
||||
major, minor = I(), I()
|
||||
if not display or not initialize(display, c.byref(major), c.byref(minor)):
|
||||
raise RuntimeError('eglInitialize failed')
|
||||
try:
|
||||
if not bind_api(0x30A2): # EGL_OPENGL_API
|
||||
raise RuntimeError('eglBindAPI failed')
|
||||
attributes = (I * 15)(0x3033, 1, 0x3024, 8, 0x3023, 8, 0x3022, 8,
|
||||
0x3021, 8, 0x3025, 24, 0x3040, 8, 0x3038)
|
||||
config, count = P(), I()
|
||||
if not choose(display, attributes, c.byref(config), 1, c.byref(count)) or not count.value:
|
||||
raise RuntimeError('eglChooseConfig failed')
|
||||
ctx = context(display, config, None, (I * 5)(0x3098, 3, 0x30FB, 3, 0x3038))
|
||||
surf = surface(display, config, (I * 5)(0x3057, 4, 0x3056, 4, 0x3038))
|
||||
if not ctx or not surf or not current(display, surf, surf, ctx):
|
||||
raise RuntimeError('context/surface setup failed')
|
||||
|
||||
create_shader = fn('glCreateShader', U, [U])
|
||||
shader_source = fn('glShaderSource', None, [U, I, c.POINTER(c.c_char_p), c.POINTER(I)])
|
||||
compile_shader = fn('glCompileShader', None, [U])
|
||||
get_shader = fn('glGetShaderiv', None, [U, U, c.POINTER(I)])
|
||||
create_program = fn('glCreateProgram', U, [])
|
||||
attach = fn('glAttachShader', None, [U, U])
|
||||
link = fn('glLinkProgram', None, [U])
|
||||
get_program = fn('glGetProgramiv', None, [U, U, c.POINTER(I)])
|
||||
use = fn('glUseProgram', None, [U])
|
||||
error = fn('glGetError', U, [])
|
||||
program = create_program()
|
||||
for kind, source in (
|
||||
(0x8B31, b'#version 330 core\nlayout(location=0) in vec2 p;void main(){gl_Position=vec4(p,0,1);}'),
|
||||
(0x8B30, b'#version 330 core\nout vec4 c;void main(){c=vec4(1);}'),
|
||||
):
|
||||
shader, data, ok = create_shader(kind), c.c_char_p(source), I()
|
||||
shader_source(shader, 1, c.byref(data), None)
|
||||
compile_shader(shader)
|
||||
get_shader(shader, 0x8B81, c.byref(ok))
|
||||
if not ok.value:
|
||||
raise RuntimeError('shader compilation failed')
|
||||
attach(program, shader)
|
||||
link(program)
|
||||
ok = I()
|
||||
get_program(program, 0x8B82, c.byref(ok))
|
||||
if not ok.value:
|
||||
raise RuntimeError('program link failed')
|
||||
use(program)
|
||||
|
||||
gen_vao = fn('glGenVertexArrays', None, [I, c.POINTER(U)])
|
||||
bind_vao = fn('glBindVertexArray', None, [U])
|
||||
gen_buffer = fn('glGenBuffers', None, [I, c.POINTER(U)])
|
||||
bind_buffer = fn('glBindBuffer', None, [U, U])
|
||||
buffer_data = fn('glBufferData', None, [U, c.c_ssize_t, P, U])
|
||||
subdata = fn('glBufferSubData', None, [U, c.c_ssize_t, c.c_ssize_t, P])
|
||||
enable = fn('glEnableVertexAttribArray', None, [U])
|
||||
pointer = fn('glVertexAttribPointer', None, [U, I, U, c.c_ubyte, I, P])
|
||||
draw = fn('glDrawArrays', None, [U, I, I])
|
||||
finish = fn('glFinish', None, [])
|
||||
vao, buffer = U(), U()
|
||||
gen_vao(1, c.byref(vao))
|
||||
bind_vao(vao)
|
||||
gen_buffer(1, c.byref(buffer))
|
||||
bind_buffer(0x8892, buffer)
|
||||
buffer_data(0x8892, UPLOAD_BYTES, None, 0x88E8)
|
||||
vertices = (F * 6)(-1, -1, 1, -1, 0, 1)
|
||||
subdata(0x8892, 0, c.sizeof(vertices), vertices)
|
||||
enable(0)
|
||||
pointer(0, 2, 0x1406, 0, 0, None)
|
||||
draw(4, 0, 3)
|
||||
finish()
|
||||
if not swap(display, surf) or error() != 0:
|
||||
raise RuntimeError('warmup draw failed')
|
||||
|
||||
payload = c.create_string_buffer(b'\x3f' * UPLOAD_BYTES)
|
||||
start = time.monotonic_ns()
|
||||
subdata(0x8892, 0, UPLOAD_BYTES, payload)
|
||||
draw(4, 0, 3)
|
||||
finish()
|
||||
elapsed = (time.monotonic_ns() - start) / 1e9
|
||||
if error() != 0 or not swap(display, surf):
|
||||
raise RuntimeError('64 MiB consuming draw failed')
|
||||
fn('MGPipeSyncPeerLog', None, [])()
|
||||
finally:
|
||||
current(display, None, None, None)
|
||||
terminate(display)
|
||||
|
||||
client_path = args.out / 'mobilegl.client.log'
|
||||
proof = client_path.read_text(errors='replace')
|
||||
actual_arm = actual_arm_from_text(proof)
|
||||
if (actual_arm != {'armed': True, 'lockstep': False, 'disarmed': False}
|
||||
or 'control=tcp data=stream server=' + args.endpoint.removeprefix('tcp://') not in proof):
|
||||
raise ValueError('no continuously armed TCP runtime proof')
|
||||
frames = frame_rows(client_path, 'P65LinkMetrics ')
|
||||
if set(frames) != {1, 2} or int(frames[2]['stage_bytes']) < UPLOAD_BYTES:
|
||||
raise ValueError('frame 2 does not prove that at least 64 MiB crossed Stage')
|
||||
return {'application_upload_bytes': UPLOAD_BYTES, 'seconds': elapsed,
|
||||
'end_to_end_staged_mib_per_second': 64 / elapsed,
|
||||
'frame_2_stage_bytes': int(frames[2]['stage_bytes']),
|
||||
'actual_arm': actual_arm, 'endpoint': args.endpoint,
|
||||
'library_sha256': digest(args.library), 'script_sha256': digest(Path(__file__)),
|
||||
'client_log_sha256': digest(client_path),
|
||||
'note': 'glBufferSubData plus consuming draw and applied fence; includes client copy and backend work, not raw network capacity'}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--library', required=True, type=Path)
|
||||
parser.add_argument('--endpoint', required=True)
|
||||
parser.add_argument('--token', default=os.environ.get('MOBILEGL_IPC_TOKEN', ''))
|
||||
parser.add_argument('--out', required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
if not args.endpoint.startswith('tcp://'):
|
||||
parser.error('a TCP endpoint is required')
|
||||
if args.out.exists():
|
||||
parser.error('output already exists; choose a fresh evidence directory')
|
||||
result = run(args)
|
||||
(args.out / 'result.json').write_text(json.dumps(result, indent=2) + '\n', encoding='utf-8')
|
||||
print(json.dumps(result), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Record client thread CPU and sampled apply-thread CPU for unchanged trace workloads.
|
||||
|
||||
The client clock comes from the existing benchmark runner. Apply CPU comes from
|
||||
/proc/<pid>/task/<tid>/schedstat, never process-wide CPU. The sampled tail window
|
||||
is approximately anchored to benchmark.json's mtime; raw samples and this
|
||||
alignment limitation are retained so the result cannot claim cycle precision.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import subprocess
|
||||
import tarfile
|
||||
import time
|
||||
|
||||
|
||||
CASES = {
|
||||
'OpenRA': ('openra.tgz', 'openra.trace', 640, 480, 100),
|
||||
'rd12': ('minecraft-1.21.4-rd12-odinlite-in-world.tgz', 'trace.trace', 854, 480, 200),
|
||||
}
|
||||
|
||||
|
||||
def digest(path):
|
||||
value = hashlib.sha256()
|
||||
with path.open('rb') as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b''):
|
||||
value.update(chunk)
|
||||
return value.hexdigest()
|
||||
|
||||
|
||||
def thread_samples(pid):
|
||||
pids = [pid]
|
||||
try:
|
||||
pids += [int(value) for value in Path(f'/proc/{pid}/task/{pid}/children').read_text().split()]
|
||||
except (FileNotFoundError, ProcessLookupError):
|
||||
pass
|
||||
rows = []
|
||||
for process in pids:
|
||||
try:
|
||||
tasks = sorted(Path(f'/proc/{process}/task').iterdir(), key=lambda path: int(path.name))
|
||||
except (FileNotFoundError, ProcessLookupError):
|
||||
continue
|
||||
for task in tasks:
|
||||
try:
|
||||
name = (task / 'comm').read_text().strip()
|
||||
if name != 'mgl-srv-apply':
|
||||
continue
|
||||
cpu_ns = int((task / 'schedstat').read_text().split()[0])
|
||||
rows.append({'time_ns': time.monotonic_ns(), 'pid': process, 'tid': int(task.name),
|
||||
'name': name, 'cpu_ns': cpu_ns})
|
||||
# Driver helper threads inherit their creator's comm. The
|
||||
# ServerLoop thread exists before it calls the driver, so its
|
||||
# live TID precedes those helpers; do not sum them as apply CPU.
|
||||
break
|
||||
except (FileNotFoundError, ProcessLookupError):
|
||||
continue
|
||||
return rows
|
||||
|
||||
|
||||
def at_time(samples, timestamp):
|
||||
for left, right in zip(samples, samples[1:]):
|
||||
if left['time_ns'] <= timestamp <= right['time_ns']:
|
||||
width = right['time_ns'] - left['time_ns']
|
||||
part = (timestamp - left['time_ns']) / width if width else 0
|
||||
return left['cpu_ns'] + part * (right['cpu_ns'] - left['cpu_ns'])
|
||||
return None
|
||||
|
||||
|
||||
def reduce_apply(samples, start, end, frames):
|
||||
grouped = {}
|
||||
for row in samples:
|
||||
grouped.setdefault((row['pid'], row['tid']), []).append(row)
|
||||
contributions = []
|
||||
for (pid, tid), rows in grouped.items():
|
||||
first = at_time(rows, start)
|
||||
last = at_time(rows, end)
|
||||
if first is None or last is None:
|
||||
continue
|
||||
inside = [row for row in rows if start <= row['time_ns'] <= end]
|
||||
gaps = [(b['time_ns'] - a['time_ns']) / 1e6 for a, b in zip(inside, inside[1:])]
|
||||
contributions.append({'pid': pid, 'tid': tid, 'cpu_ms': (last - first) / 1e6,
|
||||
'sample_count': len(inside), 'max_sample_gap_ms': max(gaps, default=None)})
|
||||
# One owning apply thread must cover the whole selected window. Refuse to
|
||||
# sum unrelated driver threads that inherited an indistinguishable name.
|
||||
if len(contributions) != 1:
|
||||
return {'valid': False, 'reason': 'no unique apply thread spans the complete tail window',
|
||||
'candidates': contributions}
|
||||
return {'valid': True, **contributions[0], 'mean_cpu_ms_per_frame': contributions[0]['cpu_ms'] / frames}
|
||||
|
||||
|
||||
def run(args, transport, case, repeat):
|
||||
archive, trace_name, width, height, tail = CASES[case]
|
||||
trace_dir = args.inputs / case
|
||||
trace_dir.mkdir(parents=True, exist_ok=True)
|
||||
trace = trace_dir / trace_name
|
||||
if not trace.exists():
|
||||
archive_path = args.fixtures / archive
|
||||
if archive_path.stat().st_size < 1024:
|
||||
raise ValueError(f'{archive_path}: fixture is an LFS pointer, not a trace archive')
|
||||
with tarfile.open(archive_path, 'r:gz') as bundle:
|
||||
bundle.extractall(trace_dir, filter='data')
|
||||
out = args.out / transport / case / f'run-{repeat}'
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
if (out / 'benchmark.json').exists():
|
||||
raise ValueError(f'{out}: evidence already exists; choose a new output directory')
|
||||
trace_digest = digest(trace)
|
||||
env = dict(os.environ, MOBILEGL_TRANSPORT=transport, MOBILEGL_IPC_SERVER_PATH=str(args.server),
|
||||
MOBILEGL_PIPE_STATS='0', MOBILEGL_IPC_PRESENT_CREDIT='1', MOBILEGL_IPC_RUN_AHEAD='1',
|
||||
MOBILEGL_IPC_STRICT_ERRORS='0', MOBILEGL_IPC_ROLE_SPLIT_STATE='0', MOBILEGL_IPC_AUDIT='0',
|
||||
MOBILEGL_IPC_VERB_BARRIER='1', MOBILEGL_IPC_BATCH_WAITS='1', MOBILEGL_IPC_SPIN_US='50',
|
||||
LIBGL_ALWAYS_SOFTWARE='1', EGL_PLATFORM='surfaceless',
|
||||
MESA_GL_VERSION_OVERRIDE='3.3', MESA_GLSL_VERSION_OVERRIDE='330')
|
||||
for key in ('MOBILEGL_IPC_CONTROL', 'MOBILEGL_IPC_DATA', 'MOBILEGL_IPC_TOKEN', 'MOBILEGL_IPC_DIAL',
|
||||
'MOBILEGL_IPC_REQUIRE_SAME_BUILD', 'MOBILEGL_TEST_SERVER_COUNTERS'):
|
||||
env.pop(key, None)
|
||||
if Path('/usr/share/glvnd/egl_vendor.d/50_mesa.json').exists():
|
||||
env['__EGL_VENDOR_LIBRARY_FILENAMES'] = '/usr/share/glvnd/egl_vendor.d/50_mesa.json'
|
||||
command = [str(args.runner), '--trace', str(trace), '--mobilegl-library', str(args.library),
|
||||
'--backend', 'DirectGLES', '--width', str(width), '--height', str(height),
|
||||
'--output', str(out), '--benchmark', '--benchmark-finish=0',
|
||||
'--benchmark-tail-frames=' + str(tail), '--hold-ms', '150']
|
||||
raw = []
|
||||
offset_before = time.monotonic_ns() - time.time_ns()
|
||||
started = time.monotonic()
|
||||
with (out / 'runner.log').open('w') as log:
|
||||
process = subprocess.Popen(command, env=env, stdout=log, stderr=log)
|
||||
try:
|
||||
while True:
|
||||
raw.extend(thread_samples(process.pid))
|
||||
if process.poll() is not None:
|
||||
break
|
||||
if time.monotonic() - started > args.timeout:
|
||||
process.kill()
|
||||
raise TimeoutError('benchmark exceeded its run budget')
|
||||
time.sleep(args.sample_ms / 1000)
|
||||
finally:
|
||||
process.wait()
|
||||
offset_after = time.monotonic_ns() - time.time_ns()
|
||||
(out / 'apply-samples.json').write_text(json.dumps(raw))
|
||||
if process.returncode:
|
||||
raise RuntimeError(f'{out}: replay failed with {process.returncode}')
|
||||
report_path = out / 'benchmark.json'
|
||||
report = json.loads(report_path.read_text())
|
||||
wall, cpu = report['frameTimesMs'], report['frameCpuTimesMs']
|
||||
if len(wall) != len(cpu) or len(wall) <= tail:
|
||||
raise RuntimeError('no complete CPU series or no startup-excluding tail window')
|
||||
# End() precedes JSON serialization. Subtract the measured unframed suffix
|
||||
# after the last frame; do not accidentally include teardown in the tail.
|
||||
report_end = report_path.stat().st_mtime_ns + (offset_before + offset_after) // 2
|
||||
unframed_ns = int(report['totalSeconds'] * 1e9 - sum(wall) * 1e6)
|
||||
tail_end = report_end - unframed_ns
|
||||
tail_start = tail_end - int(sum(wall[-tail:]) * 1e6)
|
||||
apply = reduce_apply(raw, tail_start, tail_end, tail)
|
||||
result = {'label': args.label, 'transport': transport, 'case': case, 'repeat': repeat,
|
||||
'total_frames': len(wall), 'tail_frames': tail, 'tail_wall_ms': sum(wall[-tail:]),
|
||||
'client_mean_cpu_ms': statistics.mean(cpu[-tail:]),
|
||||
'client_median_cpu_ms': statistics.median(cpu[-tail:]),
|
||||
'apply': apply, 'sample_period_ms': args.sample_ms,
|
||||
'frame_window_start_ns': tail_start, 'frame_window_end_ns': tail_end,
|
||||
'clock_offset_drift_ns': offset_after - offset_before,
|
||||
'apply_clock': '/proc/<pid>/task/<tid>/schedstat execution runtime (nanoseconds)',
|
||||
'apply_thread_selection': 'oldest live TID named mgl-srv-apply; excludes driver helpers inheriting its name',
|
||||
'alignment_limit': 'mtime anchor includes unmeasured benchmark summary/JSON serialization latency; '
|
||||
'apply endpoints are interpolated between external samples, not exact frame markers',
|
||||
'runner_sha256': args.runner_digest, 'library_sha256': args.library_digest,
|
||||
'server_sha256': args.server_digest, 'trace_sha256': trace_digest,
|
||||
'runtime_environment': {key: value for key, value in env.items()
|
||||
if key.startswith(('MOBILEGL_', 'MESA_', 'LIBGL_', '__EGL_'))},
|
||||
'command': command, 'output': str(out)}
|
||||
(out / 'cpu-summary.json').write_text(json.dumps(result, indent=2))
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--runner', type=Path, required=True)
|
||||
parser.add_argument('--library', type=Path, required=True)
|
||||
parser.add_argument('--server', type=Path, required=True)
|
||||
parser.add_argument('--fixtures', type=Path, required=True)
|
||||
parser.add_argument('--inputs', type=Path, required=True)
|
||||
parser.add_argument('--out', type=Path, required=True)
|
||||
parser.add_argument('--label', required=True)
|
||||
parser.add_argument('--transport', action='append', choices=('inproc', 'spawn'))
|
||||
parser.add_argument('--case', action='append', choices=CASES)
|
||||
parser.add_argument('--repeats', type=int, default=3)
|
||||
parser.add_argument('--case-repeats', action='append', default=[], metavar='CASE=N',
|
||||
help='Override repetitions for one case, e.g. rd12=2')
|
||||
parser.add_argument('--sample-ms', type=float, default=5)
|
||||
parser.add_argument('--timeout', type=float, default=900)
|
||||
args = parser.parse_args()
|
||||
if args.repeats < 1 or args.sample_ms <= 0:
|
||||
parser.error('repeats and sample-ms must be positive')
|
||||
repeats = {}
|
||||
for override in args.case_repeats:
|
||||
name, separator, count = override.partition('=')
|
||||
if not separator or name not in CASES or not count.isdigit() or int(count) < 1:
|
||||
parser.error('--case-repeats requires a known CASE and a positive count')
|
||||
repeats[name] = int(count)
|
||||
args.runner_digest, args.library_digest, args.server_digest = map(digest, (args.runner, args.library, args.server))
|
||||
results = []
|
||||
for case in args.case or list(CASES):
|
||||
for repeat in range(repeats.get(case, args.repeats)):
|
||||
for transport in args.transport or ('inproc', 'spawn'):
|
||||
result = run(args, transport, case, repeat)
|
||||
results.append(result)
|
||||
(args.out / 'summary.json').write_text(json.dumps(results, indent=2))
|
||||
print(json.dumps({key: result[key] for key in ('case', 'transport', 'repeat', 'client_mean_cpu_ms', 'apply')}), flush=True)
|
||||
return 0 if all(row['apply']['valid'] for row in results) else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reboot-clean, pinned Redmi TCP credit measurements with original-state restore.
|
||||
|
||||
This explicitly mutating session is separate from the ordinary benchmark tool.
|
||||
Run only while this device has no other client. The service/idle exemption is
|
||||
left available to its owner; CPU/GPU/fan settings are always restored here.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
import benchmark_tcp_credits as benchmark
|
||||
from measure_transport_cpu import CASES, digest
|
||||
from run_tcp_matrix import atomic_json, supervise
|
||||
|
||||
|
||||
CPU = '/sys/devices/system/cpu/cpufreq'
|
||||
GPU = '/sys/class/kgsl/kgsl-3d0'
|
||||
FAN = '/sys/class/xm_power/hw_monitor/pwm_fan'
|
||||
POLICIES = ('policy0', 'policy6')
|
||||
RESTORE_NODES = [f'{CPU}/{policy}/{field}' for policy in POLICIES
|
||||
for field in ('scaling_min_freq', 'scaling_max_freq', 'scaling_governor')]
|
||||
RESTORE_NODES += [f'{GPU}/min_pwrlevel', f'{GPU}/max_pwrlevel', f'{FAN}/target_level']
|
||||
READ_NODES = RESTORE_NODES + [f'{CPU}/{policy}/scaling_cur_freq' for policy in POLICIES]
|
||||
READ_NODES += [f'{GPU}/gpuclk', f'{FAN}/real_speed', f'{FAN}/pwm_duty']
|
||||
ORDERS = ((1, 2, 3), (2, 3, 1), (3, 1, 2))
|
||||
|
||||
|
||||
class Device:
|
||||
def __init__(self, serial):
|
||||
self.adb = ['adb', '-s', serial]
|
||||
|
||||
def shell(self, argv, timeout=30):
|
||||
return subprocess.run([*self.adb, 'shell', shlex.join(argv)], check=True,
|
||||
capture_output=True, text=True, timeout=timeout).stdout.strip()
|
||||
|
||||
def su(self, script):
|
||||
return self.shell(['su', '-c', script])
|
||||
|
||||
def boot_id(self):
|
||||
return self.shell(['cat', '/proc/sys/kernel/random/boot_id'])
|
||||
|
||||
def observe(self):
|
||||
statements = [f'printf "%s=%s\\n" {shlex.quote(node)} "$(cat {shlex.quote(node)})"'
|
||||
for node in READ_NODES]
|
||||
statements.append('for node in /sys/class/thermal/thermal_zone*; do '
|
||||
'if [ "$(cat "$node/type")" = cpuss-0-0 ]; then '
|
||||
'printf "thermal_path=%s\\nthermal_mc=%s\\n" "$node" "$(cat "$node/temp")"; '
|
||||
'break; fi; done')
|
||||
values = dict(line.split('=', 1) for line in self.su('; '.join(statements)).splitlines() if '=' in line)
|
||||
if any(not values.get(node) for node in READ_NODES) or not values.get('thermal_mc'):
|
||||
raise RuntimeError('incomplete device state; cannot establish or restore the thermal window')
|
||||
for node, value in values.items():
|
||||
if node.endswith('scaling_governor'):
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]+', value):
|
||||
raise ValueError('invalid governor observation')
|
||||
elif node != 'thermal_path':
|
||||
int(value)
|
||||
return {'time_ns': time.time_ns(), **values}
|
||||
|
||||
def restore(self, original):
|
||||
statements = []
|
||||
for policy in POLICIES:
|
||||
base = f'{CPU}/{policy}'
|
||||
statements += [f'cat {base}/cpuinfo_min_freq > {base}/scaling_min_freq',
|
||||
f'echo {int(original[base + "/scaling_max_freq"])} > {base}/scaling_max_freq',
|
||||
f'echo {int(original[base + "/scaling_min_freq"])} > {base}/scaling_min_freq',
|
||||
f'echo {shlex.quote(original[base + "/scaling_governor"])} > {base}/scaling_governor']
|
||||
statements += [f'echo {int(original[node])} > {node}' for node in RESTORE_NODES if node.startswith((GPU, FAN))]
|
||||
self.su('; '.join(statements))
|
||||
after = self.observe()
|
||||
differences = {node: {'expected': original[node], 'actual': after[node]}
|
||||
for node in RESTORE_NODES if original[node] != after[node]}
|
||||
return {'after': after, 'differences': differences, 'restored': not differences}
|
||||
|
||||
|
||||
def wait_for_boot(device, old_boot_id):
|
||||
subprocess.run([*device.adb, 'reboot'], check=True, timeout=30)
|
||||
deadline = time.monotonic() + 300
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
if device.shell(['getprop', 'sys.boot_completed']) == '1' and device.boot_id() != old_boot_id:
|
||||
time.sleep(20)
|
||||
return device.boot_id()
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
pass
|
||||
time.sleep(5)
|
||||
raise RuntimeError('reboot did not establish a changed, fully booted device')
|
||||
|
||||
|
||||
def pin(args, action, output):
|
||||
result = subprocess.run(['bash', str(args.pin_copy), args.serial, action],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
output.write_text(result.stdout + result.stderr, encoding='utf-8')
|
||||
return result.returncode
|
||||
|
||||
|
||||
def prepare_arm(args, device, directory):
|
||||
directory.mkdir(parents=True)
|
||||
if pin(args, 'pin', directory / 'pin-request.txt') != 0:
|
||||
raise RuntimeError('pin did not take')
|
||||
readings, consecutive = [], 0
|
||||
deadline = time.monotonic() + args.cool_timeout
|
||||
while time.monotonic() < deadline:
|
||||
observation = device.observe()
|
||||
readings.append(observation)
|
||||
atomic_json(directory / 'cooldown.json', readings)
|
||||
if int(observation['thermal_mc']) <= args.cool_max_mc:
|
||||
consecutive += 1
|
||||
if consecutive == 2:
|
||||
break
|
||||
else:
|
||||
consecutive = 0
|
||||
time.sleep(5)
|
||||
if consecutive < 2:
|
||||
raise RuntimeError('cooldown expired; no valid measurement window')
|
||||
if pin(args, 'check', directory / 'pin-before.txt') != 0:
|
||||
raise RuntimeError('frequency drift before measurement')
|
||||
if int(readings[-1][FAN + '/target_level']) != 2 or int(readings[-1][FAN + '/real_speed']) <= 0:
|
||||
raise RuntimeError('fan setting was not actually established')
|
||||
return readings[-1]
|
||||
|
||||
|
||||
def start_server(args, device):
|
||||
script = Path(__file__).with_name('tcp_device_server.py')
|
||||
command = [sys.executable, str(script), 'start', '--serial', args.serial,
|
||||
'--package', args.package, '--listen', args.listen, '--token', args.token,
|
||||
'--allow-idle', '--state-file', str(args.server_idle_state), '--env',
|
||||
'MOBILEGL_TEST_SERVER_COUNTERS=1;MOBILEGL_PIPE_STATS=1;MOBILEGL_PIPE_STATS_PERIOD=1;'
|
||||
'MOBILEGL_LOG_FILE_PATH=/data/data/' + args.package + '/files/mgl.log']
|
||||
result = subprocess.run(command, check=True, capture_output=True, text=True, timeout=30)
|
||||
(args.out / 'server-start.txt').write_text(result.stdout + result.stderr, encoding='utf-8')
|
||||
port = args.endpoint.rsplit(':', 1)[-1]
|
||||
for _ in range(20):
|
||||
sockets = device.shell(['ss', '-ltn'])
|
||||
if re.search(r':' + re.escape(port) + r'\s', sockets):
|
||||
return
|
||||
time.sleep(1)
|
||||
raise RuntimeError('restarted supervisor is not listening')
|
||||
|
||||
|
||||
def run_session(args):
|
||||
device = Device(args.serial)
|
||||
original = device.observe()
|
||||
session = {'status': 'prepared', 'serial': args.serial, 'endpoint': args.endpoint,
|
||||
'library_sha256': digest(args.library), 'runner_sha256': digest(args.runner),
|
||||
'pin_script_sha256': digest(args.pin_script), 'original_state': original,
|
||||
'boot_id_before': device.boot_id(), 'credit_orders': ORDERS,
|
||||
'cool_max_mc': args.cool_max_mc, 'arms': [], 'started_at_ns': time.time_ns()}
|
||||
state_path = args.out / 'session.json'
|
||||
atomic_json(state_path, session) # Durable before the first mutation, including reboot.
|
||||
args.pin_copy = args.out / 'pin_device.sh'
|
||||
args.pin_copy.write_text(args.pin_script.read_text(), encoding='utf-8', newline='\n')
|
||||
try:
|
||||
session['boot_id'] = wait_for_boot(device, session['boot_id_before'])
|
||||
session['rebooted_clean'] = True
|
||||
device.su(f'echo 2 > {FAN}/target_level')
|
||||
time.sleep(3)
|
||||
host = args.endpoint.removeprefix('tcp://').rsplit(':', 1)[0]
|
||||
wlan = device.shell(['ip', '-f', 'inet', 'addr', 'show', 'wlan0'])
|
||||
session['wlan_after_reboot'] = wlan
|
||||
if host not in wlan:
|
||||
raise RuntimeError('device address changed; resolve host routing before a new performance session')
|
||||
for _ in range(60):
|
||||
names = device.shell(['ps', '-A', '-o', 'NAME'])
|
||||
if not any('dex2oat' in name for name in names.splitlines()):
|
||||
break
|
||||
time.sleep(5)
|
||||
else:
|
||||
raise RuntimeError('dex2oat did not become idle')
|
||||
start_server(args, device)
|
||||
session['status'] = 'running'
|
||||
atomic_json(state_path, session)
|
||||
index = 0
|
||||
for case in args.case or CASES:
|
||||
for repeat, order in enumerate(ORDERS, 1):
|
||||
for credit in order:
|
||||
index += 1
|
||||
directory = args.out / f'{index:02d}-{case}-credit{credit}-r{repeat}'
|
||||
row = {'index': index, 'case': case, 'credit': credit, 'repeat': repeat, 'status': 'preparing'}
|
||||
session['arms'].append(row)
|
||||
atomic_json(state_path, session)
|
||||
print(json.dumps(row), flush=True)
|
||||
row['before'] = prepare_arm(args, device, directory)
|
||||
try:
|
||||
row.update(benchmark.measure(args, case, credit, directory / 'measurement'))
|
||||
finally:
|
||||
row['pin_after_rc'] = pin(args, 'check', directory / 'pin-after.txt')
|
||||
row['after'] = device.observe()
|
||||
atomic_json(state_path, session)
|
||||
if row['status'] != 'passed' or row['pin_after_rc'] != 0:
|
||||
row['status'] = 'invalid'
|
||||
raise RuntimeError(f'arm {index} is invalid; preserve it and stop')
|
||||
print(json.dumps({key: row.get(key) for key in ('index', 'case', 'credit', 'status', 'fps')}), flush=True)
|
||||
directory = args.out / 'stage-burst'
|
||||
burst = {'before': prepare_arm(args, device, directory)}
|
||||
session['stage_burst'] = burst
|
||||
script = Path(__file__).with_name('measure_tcp_stage_burst.py')
|
||||
command = [sys.executable, str(script), '--library', str(args.library), '--endpoint', args.endpoint,
|
||||
'--token', args.token, '--out', str(directory / 'measurement')]
|
||||
def wake():
|
||||
device.shell(['input', 'keyevent', 'KEYCODE_WAKEUP'])
|
||||
try:
|
||||
burst.update(supervise(command, dict(os.environ), str(args.inputs), directory,
|
||||
directory / 'runner.log', wake=wake))
|
||||
if burst['returncode'] != 0 or burst['remaining_processes'] or burst['leftovers_after_parent_exit']:
|
||||
raise RuntimeError('Stage burst failed')
|
||||
burst['measurement'] = json.loads((directory / 'measurement/result.json').read_text())
|
||||
finally:
|
||||
burst['pin_after_rc'] = pin(args, 'check', directory / 'pin-after.txt')
|
||||
burst['after'] = device.observe()
|
||||
if burst['pin_after_rc'] != 0:
|
||||
raise RuntimeError('Stage burst pin drifted')
|
||||
session['status'] = 'passed'
|
||||
except BaseException as error:
|
||||
session.update(status='failed', error=repr(error))
|
||||
raise
|
||||
finally:
|
||||
try:
|
||||
session['restore'] = device.restore(original)
|
||||
if not session['restore']['restored']:
|
||||
session['status'] = 'restore-failed'
|
||||
except BaseException as error:
|
||||
session.update(status='restore-failed', restore_error=repr(error))
|
||||
session['completed_at_ns'] = time.time_ns()
|
||||
atomic_json(state_path, session)
|
||||
return 0 if session['status'] == 'passed' else 1
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
for name in ('runner', 'library', 'inputs', 'out', 'pin-script', 'server-idle-state'):
|
||||
parser.add_argument('--' + name, required=True, type=Path)
|
||||
parser.add_argument('--serial', required=True, choices=('2f7cbe2e',), help='profile validated for this Redmi only')
|
||||
parser.add_argument('--endpoint', required=True)
|
||||
parser.add_argument('--listen', required=True)
|
||||
parser.add_argument('--token', default=os.environ.get('MOBILEGL_IPC_TOKEN', ''))
|
||||
parser.add_argument('--package', default='top.mobilegl.plugin.trace')
|
||||
parser.add_argument('--cool-max-mc', type=int, default=36000)
|
||||
parser.add_argument('--cool-timeout', type=int, default=300)
|
||||
parser.add_argument('--case', action='append', choices=CASES)
|
||||
parser.add_argument('--reboot', action='store_true', required=True)
|
||||
args = parser.parse_args()
|
||||
args.adb = 'adb'
|
||||
if args.out.exists():
|
||||
parser.error('choose a new output directory; paired sessions never resume across boots')
|
||||
if not args.endpoint.startswith('tcp://') or args.cool_timeout <= 0:
|
||||
parser.error('TCP endpoint and a positive cooldown timeout are required')
|
||||
args.out.mkdir(parents=True)
|
||||
return run_session(args)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,495 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run manifest-selected TCP retraces from a CTest JSON catalog, with resumable evidence.
|
||||
|
||||
Requires Linux/WSL process groups. The old local CTest/manifest TIMEOUT is not a
|
||||
TCP deadline: log inactivity and an explicit absolute ceiling are separate.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import contextlib
|
||||
import ctypes
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from trace_cases import load_trace_case_manifest, ci_backends
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
OWNED_ENV = {"MOBILEGL_TRANSPORT", "MOBILEGL_IPC_CONTROL", "MOBILEGL_IPC_DATA",
|
||||
"MOBILEGL_IPC_TOKEN", "MOBILEGL_IPC_DIAL", "MOBILEGL_IPC_ROLE",
|
||||
"MOBILEGL_IPC_SERVER_PATH", "MOBILEGL_LOG_FILE_PATH"}
|
||||
|
||||
|
||||
def read_json(path):
|
||||
def invalid(value):
|
||||
raise ValueError("non-finite JSON number: " + value)
|
||||
return json.loads(Path(path).read_text(encoding="utf-8"), parse_constant=invalid)
|
||||
|
||||
|
||||
def digest(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def fingerprint(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||
|
||||
|
||||
def atomic_json(path, value):
|
||||
temporary = path.with_name(path.name + "." + uuid.uuid4().hex + ".tmp")
|
||||
try:
|
||||
with temporary.open("w", encoding="utf-8") as stream:
|
||||
json.dump(value, stream, ensure_ascii=False, indent=2, allow_nan=False)
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def select_cases(manifest_path, requested):
|
||||
manifest = load_trace_case_manifest(manifest_path)
|
||||
raw = read_json(manifest_path)
|
||||
explicit = {case["name"]: case.get("split", raw.get("defaults", {}).get("split"))
|
||||
for case in raw["cases"]}
|
||||
cases = {case["name"]: case for case in manifest["cases"]}
|
||||
names = requested or [case["name"] for case in cases.values() if case["split"]]
|
||||
if len(names) != len(set(names)):
|
||||
raise ValueError("duplicate --case")
|
||||
selected = []
|
||||
for name in names:
|
||||
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*", name) or name not in cases:
|
||||
raise ValueError("unknown/unsafe case: " + name)
|
||||
case = cases[name]
|
||||
# An explicit non-CI workload, such as rd12, is useful for extra golden/perf runs.
|
||||
# An explicit split:false remains a refusal, including in this explicit selection.
|
||||
if explicit[name] is False or (not case["split"] and case.get("ci", True)):
|
||||
raise ValueError("case opts out of split: " + name)
|
||||
selected.append(case)
|
||||
if not selected:
|
||||
raise ValueError("the manifest selected no split cases")
|
||||
return selected
|
||||
|
||||
|
||||
def properties(test):
|
||||
return {entry["name"]: entry["value"] for entry in test.get("properties", [])}
|
||||
|
||||
|
||||
def case_environment(test, base, args, output=None):
|
||||
env = dict(base) # A fresh copy for every case; never leak the previous case's quirks.
|
||||
for pair in properties(test).get("ENVIRONMENT", []):
|
||||
key, sep, value = pair.partition("=")
|
||||
if not sep:
|
||||
raise ValueError("malformed CTest environment entry: " + pair)
|
||||
if key not in OWNED_ENV:
|
||||
env[key] = value
|
||||
for key in OWNED_ENV:
|
||||
env.pop(key, None)
|
||||
env.update(MOBILEGL_TRANSPORT="spawn", MOBILEGL_IPC_CONTROL=args.endpoint,
|
||||
MOBILEGL_IPC_DATA="stream", MOBILEGL_IPC_TOKEN=args.token,
|
||||
MOBILEGL_IPC_REQUIRE_SAME_BUILD="1", MOBILEGL_IPC_LOG_FORWARD="1",
|
||||
MOBILEGL_PIPE_STATS="1", MOBILEGL_PIPE_STATS_PERIOD="1",
|
||||
MOBILEGL_IPC_PRESENT_CREDIT=str(args.credit))
|
||||
if getattr(args, "require_run_ahead", False):
|
||||
env["MOBILEGL_IPC_RUN_AHEAD"] = "1"
|
||||
env["MOBILEGL_IPC_VERB_BARRIER"] = "1"
|
||||
if output is not None:
|
||||
env["MOBILEGL_LOG_FILE_PATH"] = str(output / "output/mobilegl.log")
|
||||
return env
|
||||
|
||||
|
||||
def definitions(command):
|
||||
result = {}
|
||||
for arg in command:
|
||||
if arg.startswith("-D") and "=" in arg:
|
||||
key, value = arg[2:].split("=", 1)
|
||||
result[key.split(":", 1)[0]] = value
|
||||
return result
|
||||
|
||||
|
||||
def plan_case(case, backend, catalog, args, hashes):
|
||||
name = f"MobileGLTraceReplay.{case['name']}.{backend}"
|
||||
test = next((catalog[key] for key in (name, name + ".TCP", name + ".SPAWN", name + ".SPLIT")
|
||||
if key in catalog), None)
|
||||
if test is None:
|
||||
raise ValueError("CTest catalog has no entry for " + name)
|
||||
command = list(test["command"])
|
||||
defs = definitions(command)
|
||||
if defs.get("TRACE_CASE_NAME") != case["name"] or defs.get("TRACE_BACKEND") != backend:
|
||||
raise ValueError("CTest name and command disagree: " + name)
|
||||
if "-P" not in command or not defs.get("TRACE_ARCHIVE"):
|
||||
raise ValueError("expected a run_trace_case.cmake catalog entry: " + name)
|
||||
fixtures = args.fixtures or Path(defs["TRACE_ARCHIVE"]).parent
|
||||
runner = (args.runner or Path(defs["TRACE_REPLAY_EXE"])).resolve()
|
||||
script = args.source / "tools/trace_replay/run_trace_case.cmake"
|
||||
values = {"TRACE_REPLAY_EXE": str(runner), "MOBILEGL_LIBRARY": str(args.library),
|
||||
"TRACE_CASE_NAME": case["name"], "TRACE_BACKEND": backend,
|
||||
"TRACE_TRANSPORT": "spawn", "TRACE_IPC_SERVER_PATH": "",
|
||||
"TRACE_IPC_CONTROL": "", "TRACE_IPC_TOKEN": ""}
|
||||
# Read values from the CURRENT manifest, not stale -D values in the catalog.
|
||||
keys = {"TRACE_FILE": "trace_file", "TRACE_TARGET_CALL": "target_call", "TRACE_WIDTH": "width",
|
||||
"TRACE_HEIGHT": "height", "TRACE_SSIM_THRESHOLD": "ssim_threshold", "TRACE_CROP_X": "crop_x",
|
||||
"TRACE_CROP_Y": "crop_y", "TRACE_CROP_WIDTH": "crop_width", "TRACE_CROP_HEIGHT": "crop_height",
|
||||
"TRACE_COHERENT_AS_FLUSH": "coherent_as_flush"}
|
||||
for key, field in keys.items():
|
||||
value = case.get(field, False if field == "coherent_as_flush" else 0.99 if field == "ssim_threshold" else 0)
|
||||
values[key] = ("ON" if value else "OFF") if isinstance(value, bool) else str(value)
|
||||
inputs = []
|
||||
for key, field in (("TRACE_ARCHIVE", "trace_archive"), ("TRACE_GOLDEN", "golden"),
|
||||
("TRACE_ALTERNATE_GOLDEN", "alternate_golden")):
|
||||
path = (fixtures / case[field]).resolve() if case.get(field) else None
|
||||
values[key] = str(path) if path else ""
|
||||
if path:
|
||||
stat = path.stat()
|
||||
inputs.append({"path": str(path), "size": stat.st_size, "mtime_ns": stat.st_mtime_ns,
|
||||
**({"sha256": digest(path)} if key != "TRACE_ARCHIVE" else {})})
|
||||
trace_file = Path(case["trace_file"])
|
||||
if trace_file.is_absolute() or ".." in trace_file.parts:
|
||||
raise ValueError("trace_file must remain inside the extracted input directory")
|
||||
for path in (runner, args.library, script):
|
||||
if str(path) not in hashes:
|
||||
hashes[str(path)] = digest(path)
|
||||
env = case_environment(test, args.base_env, args)
|
||||
# Ignore shell/session bookkeeping when resuming, but retain renderer knobs and
|
||||
# every case-specific catalog variable. The digest stores no token/plaintext env.
|
||||
case_keys = {pair.partition("=")[0] for pair in properties(test).get("ENVIRONMENT", [])}
|
||||
relevant = {key: value for key, value in env.items() if key in case_keys or
|
||||
key.startswith(("MOBILEGL_", "MGITEST_", "EGL_", "LIBGL_", "MESA_", "VK_", "__GL", "__EGL", "LD_")) or
|
||||
key in ("DISPLAY", "WAYLAND_DISPLAY", "XDG_RUNTIME_DIR", "DRI_PRIME", "GALLIUM_DRIVER")}
|
||||
identity = fingerprint({"case": case, "backend": backend, "credit": args.credit,
|
||||
"endpoint": args.endpoint, "token": args.token,
|
||||
"require_run_ahead": args.require_run_ahead,
|
||||
"artifacts": {str(p): hashes[str(p)] for p in (runner, args.library, script)},
|
||||
"inputs": inputs, "environment": relevant, "command": command})
|
||||
return {"case": case, "backend": backend, "test": test, "command": command, "values": values,
|
||||
"script": script, "runner": runner, "identity": identity,
|
||||
"key": f"{backend}/credit-{args.credit}/{case['name']}"}
|
||||
|
||||
|
||||
def command_for(plan, work):
|
||||
values = {**plan["values"], "TRACE_OUTPUT_DIR": str(work), "TRACE_ARTIFACT_DIR": str(work.parent / "artifacts")}
|
||||
command, found = [], set()
|
||||
original = plan["command"]
|
||||
for index, arg in enumerate(original):
|
||||
if index and original[index - 1] == "-P":
|
||||
command.append(str(plan["script"]))
|
||||
elif arg.startswith("-D") and arg[2:].split("=", 1)[0].split(":", 1)[0] in values:
|
||||
key = arg[2:].split("=", 1)[0].split(":", 1)[0]
|
||||
command.append(f"-D{key}={values[key]}")
|
||||
found.add(key)
|
||||
else:
|
||||
command.append(arg)
|
||||
position = command.index("-P")
|
||||
command[position:position] = [f"-D{k}={v}" for k, v in values.items() if k not in found]
|
||||
return command
|
||||
|
||||
|
||||
def group_members(pgid):
|
||||
members = []
|
||||
for path in Path("/proc").glob("[0-9]*/stat"):
|
||||
try:
|
||||
fields = path.read_text().rsplit(")", 1)[1].split()
|
||||
if int(fields[2]) == pgid:
|
||||
members.append((int(path.parent.name), fields[0]))
|
||||
except (OSError, ValueError, IndexError):
|
||||
pass
|
||||
return members
|
||||
|
||||
|
||||
def reap_group(pgid):
|
||||
while True:
|
||||
try:
|
||||
pid, _ = os.waitpid(-pgid, os.WNOHANG)
|
||||
if pid == 0:
|
||||
break
|
||||
except ChildProcessError:
|
||||
break
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def subreaper():
|
||||
libc = ctypes.CDLL(None, use_errno=True)
|
||||
previous = ctypes.c_int()
|
||||
if libc.prctl(37, ctypes.byref(previous), 0, 0, 0) or libc.prctl(36, 1, 0, 0, 0):
|
||||
raise OSError(ctypes.get_errno(), "cannot enable Linux child subreaper")
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
libc.prctl(36, previous.value, 0, 0, 0)
|
||||
|
||||
|
||||
def terminate_group(process, grace=2.0):
|
||||
"""Reap descendants even when the wrapper exits before a TERM-ignoring child."""
|
||||
pgid = process.pid
|
||||
for sig, duration in ((signal.SIGTERM, grace), (signal.SIGKILL, 2.0)):
|
||||
try:
|
||||
os.killpg(pgid, sig)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
until = time.monotonic() + duration
|
||||
while time.monotonic() < until:
|
||||
process.poll() # Let Popen reap its own child before reaping adopted descendants.
|
||||
if process.returncode is not None:
|
||||
reap_group(pgid)
|
||||
if not group_members(pgid):
|
||||
return []
|
||||
time.sleep(.02)
|
||||
process.wait(timeout=2)
|
||||
reap_group(pgid)
|
||||
return group_members(pgid)
|
||||
|
||||
|
||||
def progress_signature(work, console):
|
||||
signature = []
|
||||
for path in [console, *work.rglob("*.log")]:
|
||||
try:
|
||||
stat = path.stat()
|
||||
signature.append((str(path), stat.st_size, stat.st_mtime_ns))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
return tuple(sorted(signature))
|
||||
|
||||
|
||||
def supervise(command, env, cwd, work, console, idle_seconds=300, max_seconds=7200,
|
||||
poll_seconds=.2, terminate_grace=2.0, wake=None):
|
||||
started = time.monotonic()
|
||||
last_progress = started
|
||||
previous = progress_signature(work, console)
|
||||
timeout_kind = None
|
||||
next_wake = started
|
||||
with subreaper(), console.open("wb") as output:
|
||||
process = subprocess.Popen(command, env=env, cwd=cwd, stdout=output, stderr=output,
|
||||
start_new_session=True)
|
||||
try:
|
||||
while process.poll() is None:
|
||||
now = time.monotonic()
|
||||
if wake is not None and now >= next_wake:
|
||||
wake()
|
||||
next_wake = time.monotonic() + 15
|
||||
signature = progress_signature(work, console)
|
||||
if signature != previous:
|
||||
previous, last_progress = signature, time.monotonic()
|
||||
now = time.monotonic()
|
||||
if now - started >= max_seconds:
|
||||
timeout_kind = "absolute"
|
||||
break
|
||||
if now - last_progress >= idle_seconds:
|
||||
timeout_kind = "idle"
|
||||
break
|
||||
time.sleep(poll_seconds)
|
||||
leftovers = group_members(process.pid) if process.poll() is not None else []
|
||||
returncode = process.returncode
|
||||
finally:
|
||||
remaining = terminate_group(process, terminate_grace)
|
||||
return {"returncode": 124 if timeout_kind else returncode, "timeout_kind": timeout_kind,
|
||||
"seconds": round(time.monotonic() - started, 3), "pid": process.pid,
|
||||
"leftovers_after_parent_exit": [pid for pid, state in leftovers if state != "Z"],
|
||||
"remaining_processes": remaining}
|
||||
|
||||
|
||||
def actual_arm_from_text(client_text):
|
||||
return {"armed": "run-ahead ARMED" in client_text,
|
||||
"lockstep": "running lockstep" in client_text,
|
||||
"disarmed": bool(re.search(r"\bDISARMED\b", client_text))}
|
||||
|
||||
|
||||
def validate_result(plan, work, endpoint, require_run_ahead=False):
|
||||
path = work / "output/result.json"
|
||||
result = read_json(path)
|
||||
if result.get("passed") is not True or result.get("statusCode") != 0:
|
||||
raise ValueError("result.json does not report success")
|
||||
if result.get("backend") != plan["backend"] or result.get("targetCall") != int(plan["values"]["TRACE_TARGET_CALL"]):
|
||||
raise ValueError("result identifies a different backend/call")
|
||||
expected_trace = (work / "input" / plan["case"]["trace_file"]).resolve()
|
||||
if Path(result.get("tracePath", "")).resolve() != expected_trace:
|
||||
raise ValueError("result tracePath belongs to a different attempt")
|
||||
actual = work / "output/actual.png"
|
||||
if Path(result.get("actualPath", "")).resolve() != actual.resolve() or not actual.is_file():
|
||||
raise ValueError("result has no current-attempt actual image")
|
||||
goldens = {str(Path(plan["values"][key]).resolve()) for key in ("TRACE_GOLDEN", "TRACE_ALTERNATE_GOLDEN") if plan["values"][key]}
|
||||
if str(Path(result.get("matchedGoldenPath", "")).resolve()) not in goldens:
|
||||
raise ValueError("result matched a different golden")
|
||||
score = result.get("ssim")
|
||||
if isinstance(score, bool) or not isinstance(score, (float, int)) or not math.isfinite(score) or score < float(plan["values"]["TRACE_SSIM_THRESHOLD"]):
|
||||
raise ValueError("result SSIM does not meet the current manifest threshold")
|
||||
client_log = work / "output/mobilegl.client.log"
|
||||
client_text = client_log.read_text(errors="replace")
|
||||
marker = re.compile(r"control=tcp data=stream server=(\S+) pid=([1-9][0-9]*)")
|
||||
if not any(match[0] == endpoint.removeprefix("tcp://") for match in marker.findall(client_text)):
|
||||
raise ValueError("current-attempt log lacks the requested TCP arm proof")
|
||||
actual_arm = actual_arm_from_text(client_text)
|
||||
if require_run_ahead and (not actual_arm["armed"] or actual_arm["lockstep"] or actual_arm["disarmed"]):
|
||||
raise ValueError("required run-ahead arm missing or contradicted in private client log: " + str(actual_arm))
|
||||
return {"result": result, "result_path": str(path), "result_sha256": digest(path),
|
||||
"client_log": str(client_log), "client_log_sha256": digest(client_log), "actual_arm": actual_arm}
|
||||
|
||||
|
||||
def reusable(row, plan, out, endpoint, require_run_ahead=False):
|
||||
if row is None or row.get("status") != "passed" or row.get("returncode") != 0 or row.get("timeout_kind") is not None or row.get("identity") != plan["identity"]:
|
||||
return False
|
||||
if require_run_ahead and row.get("require_run_ahead") is not True:
|
||||
return False
|
||||
try:
|
||||
work = Path(row["work"])
|
||||
if not work.resolve().is_relative_to(out.resolve()):
|
||||
return False
|
||||
proof = validate_result(plan, work, endpoint, require_run_ahead)
|
||||
return all(proof[key] == row.get(key) for key in ("result_sha256", "client_log_sha256", "actual_arm"))
|
||||
except (OSError, ValueError, KeyError):
|
||||
return False
|
||||
|
||||
|
||||
def run_case(plan, args, row):
|
||||
work = Path(row["work"])
|
||||
console = work.parent / "runner.log"
|
||||
env = case_environment(plan["test"], args.base_env, args, work)
|
||||
row["requested_arm"] = {key: env.get(key) for key in ("MOBILEGL_IPC_RUN_AHEAD", "MOBILEGL_IPC_VERB_BARRIER")}
|
||||
props = properties(plan["test"])
|
||||
cwd = props.get("WORKING_DIRECTORY", str(args.source))
|
||||
def wake():
|
||||
with (work.parent / "wake-adb.log").open("ab") as log:
|
||||
try:
|
||||
subprocess.run([args.adb, "-s", args.wake_adb_serial, "shell", "input", "keyevent", "KEYCODE_WAKEUP"],
|
||||
stdout=log, stderr=log, timeout=5, check=False)
|
||||
except (OSError, subprocess.TimeoutExpired) as error:
|
||||
log.write((str(error) + "\n").encode())
|
||||
outcome = supervise(command_for(plan, work), env, cwd, work, console, args.idle_seconds,
|
||||
args.max_seconds, wake=wake if args.wake_adb_serial else None)
|
||||
row.update(outcome, status="failed", console=str(console))
|
||||
client_log = work / "output/mobilegl.client.log"
|
||||
row["actual_arm"] = actual_arm_from_text(client_log.read_text(errors="replace")) if client_log.is_file() else None
|
||||
if outcome["timeout_kind"]:
|
||||
row["status"] = "timeout"
|
||||
elif outcome["returncode"] == 0 and not outcome["remaining_processes"] and not outcome["leftovers_after_parent_exit"]:
|
||||
try:
|
||||
row.update(validate_result(plan, work, args.endpoint, args.require_run_ahead), status="passed")
|
||||
except (OSError, ValueError, KeyError) as error:
|
||||
row["error"] = str(error)
|
||||
if row["status"] != "passed" and row["returncode"] == 0:
|
||||
row["process_returncode"] = 0
|
||||
row["returncode"] = 1 # A zero wrapper exit cannot turn stale/missing proof green.
|
||||
return row
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
catalog_source = parser.add_mutually_exclusive_group(required=True)
|
||||
catalog_source.add_argument("--catalog", type=Path, help="ctest --show-only=json-v1 output")
|
||||
catalog_source.add_argument("--build-dir", type=Path, help="read a catalog using ctest; does not build")
|
||||
parser.add_argument("--source", type=Path, default=ROOT)
|
||||
parser.add_argument("--manifest", type=Path)
|
||||
parser.add_argument("--fixtures", type=Path, help="default: the fixture directory recorded in the catalog")
|
||||
parser.add_argument("--runner", type=Path, help="override TRACE_REPLAY_EXE from the catalog")
|
||||
parser.add_argument("--library", type=Path, required=True)
|
||||
parser.add_argument("--endpoint", required=True)
|
||||
parser.add_argument("--token", default=os.environ.get("MOBILEGL_IPC_TOKEN", ""))
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--case", action="append")
|
||||
parser.add_argument("--backend", action="append", choices=("DirectGLES", "DirectVulkan"))
|
||||
parser.add_argument("--credit", type=int, choices=(1, 2, 3), default=2)
|
||||
parser.add_argument("--resume", action="store_true")
|
||||
parser.add_argument("--require-run-ahead", action="store_true",
|
||||
help="require private-client ARMED proof, no lockstep fallback/DISARMED; force RUN_AHEAD=1 and VERB_BARRIER=1")
|
||||
parser.add_argument("--idle-seconds", type=float, default=300)
|
||||
parser.add_argument("--max-seconds", type=float, default=7200)
|
||||
parser.add_argument("--wake-adb-serial", help="explicit opt-in: KEYCODE_WAKEUP every 15 seconds while a case runs")
|
||||
parser.add_argument("--adb", default="adb")
|
||||
args = parser.parse_args(argv)
|
||||
if sys.platform != "linux":
|
||||
parser.error("run this process-group driver in Linux/WSL")
|
||||
if not args.endpoint.startswith("tcp://") or any(not math.isfinite(value) or value <= 0
|
||||
for value in (args.idle_seconds, args.max_seconds)):
|
||||
parser.error("a tcp:// endpoint and positive timeouts are required")
|
||||
args.source, args.library, args.out = args.source.resolve(), args.library.resolve(), args.out.resolve()
|
||||
args.base_env = dict(os.environ)
|
||||
args.manifest = args.manifest or args.source / "tools/trace_replay/trace_cases.json"
|
||||
backends = args.backend or ["DirectGLES"]
|
||||
if len(backends) != len(set(backends)):
|
||||
parser.error("duplicate --backend")
|
||||
try:
|
||||
cases = select_cases(args.manifest, args.case)
|
||||
document = read_json(args.catalog) if args.catalog else json.loads(subprocess.check_output(
|
||||
["ctest", "--test-dir", str(args.build_dir), "--show-only=json-v1"], text=True))
|
||||
catalog = {test["name"]: test for test in document["tests"]}
|
||||
hashes = {}
|
||||
plans = [plan_case(case, backend, catalog, args, hashes) for case in cases
|
||||
for backend in backends if backend in ci_backends(case)]
|
||||
if not plans:
|
||||
raise ValueError("no selected case supports the requested backend")
|
||||
except (OSError, ValueError, KeyError, subprocess.CalledProcessError) as error:
|
||||
parser.error(str(error))
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
checkpoint_path = args.out / "checkpoint.json"
|
||||
with (args.out / ".matrix.lock").open("w") as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
parser.error("another matrix driver owns this output directory")
|
||||
if not checkpoint_path.exists() and (args.out / "results.json").is_file():
|
||||
legacy = args.out / ("results.legacy-" + uuid.uuid4().hex + ".json")
|
||||
shutil.copy2(args.out / "results.json", legacy)
|
||||
print(json.dumps({"legacy_results_preserved": str(legacy)}), flush=True)
|
||||
checkpoint = read_json(checkpoint_path) if checkpoint_path.exists() else {"schema_version": 1, "runs": []}
|
||||
if checkpoint.get("schema_version") != 1 or not isinstance(checkpoint.get("runs"), list):
|
||||
parser.error("unsupported/corrupt checkpoint; old scratch results are not a checkpoint")
|
||||
selected = []
|
||||
def interrupted(signum, frame):
|
||||
raise KeyboardInterrupt
|
||||
handlers = {sig: signal.signal(sig, interrupted) for sig in (signal.SIGINT, signal.SIGTERM, signal.SIGHUP)}
|
||||
try:
|
||||
for plan in plans:
|
||||
prior = next((r for r in reversed(checkpoint["runs"]) if r.get("key") == plan["key"]), None)
|
||||
if args.resume and reusable(prior, plan, args.out, args.endpoint, args.require_run_ahead):
|
||||
selected.append({**prior, "resumed": True})
|
||||
print(json.dumps({"case": prior["case"], "backend": prior["backend"], "status": "passed", "resumed": True}), flush=True)
|
||||
continue
|
||||
attempt = args.out / plan["key"] / ("attempt-" + uuid.uuid4().hex)
|
||||
attempt.mkdir(parents=True)
|
||||
row = {"key": plan["key"], "case": plan["case"]["name"], "backend": plan["backend"],
|
||||
"credit": args.credit, "ci": plan["case"].get("ci", True), "identity": plan["identity"],
|
||||
"require_run_ahead": args.require_run_ahead,
|
||||
"work": str(attempt / "work"), "status": "running", "started_at_ns": time.time_ns()}
|
||||
checkpoint["runs"].append(row)
|
||||
atomic_json(checkpoint_path, checkpoint)
|
||||
print(json.dumps({"case": row["case"], "backend": row["backend"], "status": "running"}), flush=True)
|
||||
try:
|
||||
run_case(plan, args, row)
|
||||
except KeyboardInterrupt:
|
||||
row.update(status="cancelled", returncode=130)
|
||||
atomic_json(checkpoint_path, checkpoint)
|
||||
return 130
|
||||
except Exception as error:
|
||||
row.update(status="failed", returncode=1, error=str(error))
|
||||
row["completed_at_ns"] = time.time_ns()
|
||||
atomic_json(checkpoint_path, checkpoint)
|
||||
selected.append({**row, "resumed": False})
|
||||
atomic_json(args.out / "results.json", selected)
|
||||
print(json.dumps({k: row.get(k) for k in ("case", "backend", "status", "returncode", "seconds", "timeout_kind", "error")}), flush=True)
|
||||
if row["status"] == "timeout" or row.get("remaining_processes"):
|
||||
# Do not convert an unhealthy peer into a cascade of Busy outcomes.
|
||||
# The failed checkpoint is retained; resume after checking the peer.
|
||||
break
|
||||
time.sleep(.3)
|
||||
finally:
|
||||
for sig, handler in handlers.items():
|
||||
signal.signal(sig, handler)
|
||||
atomic_json(args.out / "results.json", selected)
|
||||
return 0 if all(row["status"] == "passed" for row in selected) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,10 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Start or stop the trace APK's foreground TCP supervisor through adb."""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
|
||||
|
||||
def shell(adb, command):
|
||||
return subprocess.run(adb + ['shell', shlex.join(command)], check=True,
|
||||
capture_output=True, text=True).stdout
|
||||
|
||||
|
||||
def idle_whitelisted(adb, package):
|
||||
entries = shell(adb, ['dumpsys', 'deviceidle', 'whitelist'])
|
||||
return any(package in line.strip().split(',') for line in entries.splitlines())
|
||||
|
||||
|
||||
def allow_idle(adb, serial, package, state_path):
|
||||
if state_path.exists():
|
||||
saved = json.loads(state_path.read_text())
|
||||
if saved['serial'] != serial or saved['package'] != package:
|
||||
raise ValueError('idle state belongs to a different device or package')
|
||||
else:
|
||||
saved = {'serial': serial, 'package': package,
|
||||
'originally_whitelisted': idle_whitelisted(adb, package)}
|
||||
state_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# Save the original state before the device mutation, including if the
|
||||
# command's response is lost and a later stop has to recover it.
|
||||
state_path.write_text(json.dumps(saved, indent=2))
|
||||
if not idle_whitelisted(adb, package):
|
||||
shell(adb, ['dumpsys', 'deviceidle', 'whitelist', '+' + package])
|
||||
if not idle_whitelisted(adb, package):
|
||||
raise RuntimeError('the package was not added to the device idle whitelist')
|
||||
print(f'Idle exemption recorded in {state_path}; stop restores the original package setting')
|
||||
|
||||
|
||||
def restore_idle(adb, serial, package, state_path):
|
||||
if not state_path.exists():
|
||||
return
|
||||
saved = json.loads(state_path.read_text())
|
||||
if saved['serial'] != serial or saved['package'] != package:
|
||||
raise ValueError('idle state belongs to a different device or package')
|
||||
if not saved['originally_whitelisted']:
|
||||
shell(adb, ['dumpsys', 'deviceidle', 'whitelist', '-' + package])
|
||||
if idle_whitelisted(adb, package):
|
||||
raise RuntimeError('the package idle exemption could not be restored')
|
||||
state_path.unlink()
|
||||
print('Restored the original device idle whitelist setting for ' + package)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=('start', 'stop'))
|
||||
@@ -15,15 +61,25 @@ def main():
|
||||
parser.add_argument('--token', default='')
|
||||
parser.add_argument('--env', action='append', default=[], help='KEY=VALUE;KEY overrides (repeatable)')
|
||||
parser.add_argument('--forward', action='store_true', help='Forward local adb port 40613 to the device')
|
||||
parser.add_argument('--allow-idle', action='store_true',
|
||||
help='Temporarily exempt this package from Doze; stop restores its previous setting')
|
||||
parser.add_argument('--state-file', type=Path,
|
||||
help='Saved original idle setting (default: ~/.cache/mobilegl/tcp-server/<serial>-<package>.json)')
|
||||
args = parser.parse_args()
|
||||
adb = [args.adb, '-s', args.serial]
|
||||
state_name = re.sub(r'[^A-Za-z0-9_.-]', '_', args.serial + '-' + args.package) + '.json'
|
||||
state_path = args.state_file or (Path.home() / '.cache/mobilegl/tcp-server' / state_name)
|
||||
if args.action == 'stop':
|
||||
command = ['am', 'force-stop', args.package]
|
||||
else:
|
||||
if args.allow_idle:
|
||||
allow_idle(adb, args.serial, args.package, state_path)
|
||||
command = ['am', 'start-foreground-service', '-n', args.package + '/top.mobilegl.plugin.MobileGLServerService',
|
||||
'--es', 'listen', args.listen, '--es', 'token', args.token,
|
||||
'--es', 'env', ';'.join(args.env)]
|
||||
subprocess.run(adb + ['shell', shlex.join(command)], check=True)
|
||||
print(shell(adb, command), end='')
|
||||
if args.action == 'stop':
|
||||
restore_idle(adb, args.serial, args.package, state_path)
|
||||
if args.action == 'start' and args.forward:
|
||||
subprocess.run(adb + ['forward', 'tcp:40613', 'tcp:40613'], check=True)
|
||||
if args.action == 'start':
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline controls for selecting truthful TCP benchmark windows."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import benchmark_tcp_credits as benchmark
|
||||
from benchmark_tcp_credits import validate_measurement
|
||||
|
||||
|
||||
class BenchmarkEvidenceTest(unittest.TestCase):
|
||||
endpoint = 'tcp://127.0.0.1:40613'
|
||||
|
||||
def setUp(self):
|
||||
self.temporary = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temporary.cleanup)
|
||||
self.work = Path(self.temporary.name)
|
||||
self.client = ['run-ahead ARMED', 'paced by a present credit of 1',
|
||||
'control=tcp data=stream server=127.0.0.1:40613']
|
||||
self.server = []
|
||||
for frame in range(1, 4):
|
||||
self.client.append('P65LinkMetrics kind=frame frame=' + str(frame) +
|
||||
' wait_replies=2 rtt_samples=1 rtt_mean_us=8 stage_bytes=1024'
|
||||
' wall_ns=1000000 client_thread_cpu_ns=500000 rtt_hist_us_pow2=' +
|
||||
','.join('1' if bucket == 3 else '0' for bucket in range(32)))
|
||||
self.server.append('P65ServerMetrics valid=1 frame=' + str(frame) +
|
||||
' wall_ns=1000000 apply_thread_cpu_ns=250000')
|
||||
|
||||
def validate(self, credit=1, total=3):
|
||||
(self.work / 'benchmark.json').write_text(json.dumps({'totalFrames': total}))
|
||||
(self.work / 'mobilegl.client.log').write_text('\n'.join(self.client))
|
||||
(self.work / 'mobilegl.server.log').write_text('\n'.join(self.server))
|
||||
return validate_measurement(self.work, self.endpoint, credit, 2)
|
||||
|
||||
def test_valid_matching_tail(self):
|
||||
result = self.validate()
|
||||
self.assertEqual((result['first_frame'], result['last_frame']), (2, 3))
|
||||
self.assertEqual(result['wait_replies_per_frame'], 2)
|
||||
self.assertEqual(result['rtt_mean_us'], 8)
|
||||
self.assertEqual(result['fps'], 1000)
|
||||
|
||||
def test_server_marker_cannot_arm_client(self):
|
||||
self.client.remove('run-ahead ARMED')
|
||||
self.server.append('run-ahead ARMED')
|
||||
with self.assertRaisesRegex(ValueError, 'requested armed credit'):
|
||||
self.validate()
|
||||
|
||||
def test_demotion_or_lockstep_rejected(self):
|
||||
for marker in ('running lockstep', 'run-ahead DISARMED'):
|
||||
with self.subTest(marker=marker):
|
||||
self.client.append(marker)
|
||||
with self.assertRaisesRegex(ValueError, 'requested armed credit'):
|
||||
self.validate()
|
||||
self.client.pop()
|
||||
|
||||
def test_credit_proof_is_exact(self):
|
||||
self.client[1] = 'paced by a present credit of 10'
|
||||
with self.assertRaisesRegex(ValueError, 'requested armed credit'):
|
||||
self.validate()
|
||||
|
||||
def test_missing_server_tail_frame_rejected(self):
|
||||
self.server.pop()
|
||||
with self.assertRaisesRegex(ValueError, 'matching complete'):
|
||||
self.validate()
|
||||
|
||||
def test_duplicate_client_frame_rejected(self):
|
||||
self.client.append(self.client[-1])
|
||||
with self.assertRaisesRegex(ValueError, 'duplicate frame'):
|
||||
self.validate()
|
||||
|
||||
def test_partial_metrics_rejected(self):
|
||||
with self.assertRaisesRegex(ValueError, 'benchmark and frame metrics differ'):
|
||||
self.validate(total=4)
|
||||
|
||||
def test_missing_cpu_is_not_zero(self):
|
||||
self.server[-1] = self.server[-1].replace('apply_thread_cpu_ns=250000', 'apply_thread_cpu_ns=0')
|
||||
with self.assertRaisesRegex(ValueError, 'nonpositive CPU'):
|
||||
self.validate()
|
||||
|
||||
def test_resume_requires_unchanged_artifacts_and_library(self):
|
||||
valid = self.validate()
|
||||
inputs = self.work / 'inputs' / 'OpenRA'
|
||||
inputs.mkdir(parents=True)
|
||||
(inputs / 'openra.trace').write_bytes(b'trace fixture')
|
||||
runner, library = self.work / 'runner', self.work / 'library'
|
||||
runner.write_bytes(b'runner fixture')
|
||||
library.write_bytes(b'library fixture')
|
||||
output = self.work / 'results'
|
||||
argv = ['benchmark_tcp_credits.py', '--runner', str(runner), '--library', str(library),
|
||||
'--inputs', str(inputs.parent), '--out', str(output), '--endpoint', self.endpoint,
|
||||
'--case', 'OpenRA', '--credit', '1', '--serial', 'offline-fixture']
|
||||
|
||||
def completed_measurement(args, case, credit, work):
|
||||
work.mkdir(parents=True)
|
||||
for filename in ('benchmark.json', 'mobilegl.client.log', 'mobilegl.server.log'):
|
||||
(work / filename).write_bytes((self.work / filename).read_bytes())
|
||||
return {**valid, 'status': 'passed', 'output': str(work)}
|
||||
|
||||
with mock.patch.object(benchmark, 'measure', side_effect=completed_measurement) as measure:
|
||||
with mock.patch('sys.argv', argv), mock.patch('builtins.print'):
|
||||
self.assertEqual(benchmark.main(), 0)
|
||||
with mock.patch('sys.argv', [*argv, '--resume']), mock.patch('builtins.print'):
|
||||
self.assertEqual(benchmark.main(), 0)
|
||||
self.assertEqual(measure.call_count, 1)
|
||||
log = next(output.rglob('mobilegl.client.log'))
|
||||
log.write_text('tampered')
|
||||
with mock.patch('sys.argv', [*argv, '--resume']), mock.patch('builtins.print'):
|
||||
self.assertEqual(benchmark.main(), 0)
|
||||
self.assertEqual(measure.call_count, 2)
|
||||
library.write_bytes(b'changed library')
|
||||
with mock.patch('sys.argv', [*argv, '--resume']), mock.patch('builtins.print'):
|
||||
self.assertEqual(benchmark.main(), 0)
|
||||
self.assertEqual(measure.call_count, 3)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Small process/fixture stubs only: no GL, network peer, adb or production trace."""
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import types
|
||||
import unittest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import run_tcp_matrix as matrix
|
||||
|
||||
STUB = r"""
|
||||
import json,os,sys,time
|
||||
from pathlib import Path
|
||||
values = dict(arg[2:].split('=',1) for arg in sys.argv[1:] if arg.startswith('-D') and '=' in arg)
|
||||
work=Path(values['TRACE_OUTPUT_DIR']);(work/'output').mkdir(parents=True,exist_ok=True);(work/'input').mkdir()
|
||||
behavior=os.environ.get('CASE_BEHAVIOR','success')
|
||||
if behavior=='active':
|
||||
for n in range(16):
|
||||
with (work/'output/progress.log').open('a') as log:log.write(str(n)+'\n')
|
||||
time.sleep(.05)
|
||||
if behavior!='no-result':
|
||||
actual=work/'output/actual.png';actual.write_bytes(b'fixture-image')
|
||||
endpoint=os.environ['MOBILEGL_IPC_CONTROL'].removeprefix('tcp://')
|
||||
arm=os.environ.get('CASE_ARM','armed')
|
||||
if arm=='respect-env':arm='armed' if os.environ.get('MOBILEGL_IPC_RUN_AHEAD')=='1' and os.environ.get('MOBILEGL_IPC_VERB_BARRIER')=='1' else 'lockstep'
|
||||
line={'armed':'run-ahead ARMED\n','lockstep':'running lockstep\n','disarmed':'run-ahead ARMED\nrun-ahead DISARMED\n','server-only':''}[arm]
|
||||
(work/'output/mobilegl.client.log').write_text('control=tcp data=stream server='+endpoint+' pid=123 dial=connect\n'+line)
|
||||
if arm=='server-only':(work/'output/mobilegl.server.log').write_text('run-ahead ARMED\n')
|
||||
result={'passed':True,'statusCode':0,'backend':values['TRACE_BACKEND'],
|
||||
'targetCall':int(values['TRACE_TARGET_CALL']),'tracePath':str(work/'input'/values['TRACE_FILE']),
|
||||
'actualPath':str(actual),'matchedGoldenPath':values['TRACE_GOLDEN'],'ssim':1.0}
|
||||
(work/'output/result.json').write_text(json.dumps(result))
|
||||
if behavior=='timeout':time.sleep(60)
|
||||
raise SystemExit(7 if behavior=='failed' else 0)
|
||||
"""
|
||||
|
||||
|
||||
@unittest.skipUnless(sys.platform == 'linux', 'Linux/WSL process-group driver')
|
||||
class MatrixTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(prefix='tcp-matrix-test-')
|
||||
self.root = Path(self.temp.name)
|
||||
self.source = self.root/'source'
|
||||
self.tool = self.source/'tools/trace_replay'
|
||||
self.tool.mkdir(parents=True)
|
||||
(self.tool/'run_trace_case.cmake').write_text('# stub script')
|
||||
self.fixtures = self.root/'fixtures';self.fixtures.mkdir()
|
||||
(self.fixtures/'fixture.tgz').write_bytes(b'fixture archive')
|
||||
(self.fixtures/'golden.png').write_bytes(b'golden image')
|
||||
self.library = self.root/'libMobileGL.so';self.library.write_bytes(b'library-v1')
|
||||
self.stub = self.root/'stub.py';self.stub.write_text(STUB)
|
||||
self.catalog = self.root/'catalog.json'
|
||||
self.out = self.root/'results'
|
||||
self.manifest = self.tool/'trace_cases.json'
|
||||
self.write_inputs()
|
||||
|
||||
def tearDown(self):
|
||||
self.temp.cleanup()
|
||||
|
||||
def write_inputs(self, behavior='success', extra_cases=None):
|
||||
cases=[{'name':'Example','trace_archive':'fixture.tgz','trace_file':'trace.trace',
|
||||
'golden':'golden.png','target_call':42,'width':16,'height':16,'timeout_seconds':.05}]
|
||||
if extra_cases:cases.extend(extra_cases)
|
||||
self.manifest.write_text(json.dumps({'defaults':{'ssim_threshold':.99},'cases':cases}))
|
||||
command=[sys.executable,str(self.stub),'-DTRACE_REPLAY_EXE='+sys.executable,
|
||||
'-DMOBILEGL_LIBRARY=/old/library','-DTRACE_CASE_NAME=Example','-DTRACE_BACKEND=DirectGLES',
|
||||
'-DTRACE_ARCHIVE='+str(self.fixtures/'fixture.tgz'),'-DTRACE_OUTPUT_DIR=/old/output',
|
||||
'-P',str(self.tool/'run_trace_case.cmake')]
|
||||
self.catalog.write_text(json.dumps({'kind':'ctestInfo','tests':[{
|
||||
'name':'MobileGLTraceReplay.Example.DirectGLES','command':command,
|
||||
'properties':[{'name':'ENVIRONMENT','value':['CASE_BEHAVIOR='+behavior]},
|
||||
{'name':'TIMEOUT','value':.05},{'name':'WORKING_DIRECTORY','value':str(self.root)}]}]}))
|
||||
|
||||
def run_main(self, *extra):
|
||||
args=['--catalog',str(self.catalog),'--source',str(self.source),'--library',str(self.library),
|
||||
'--runner',sys.executable,'--endpoint','tcp://127.0.0.1:9','--token','not-saved-in-checkpoint',
|
||||
'--out',str(self.out),'--idle-seconds','1','--max-seconds','5',*extra]
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
return matrix.main(args)
|
||||
|
||||
def checkpoint(self):
|
||||
return matrix.read_json(self.out/'checkpoint.json')['runs']
|
||||
|
||||
def set_arm(self, arm, extra=None):
|
||||
catalog=matrix.read_json(self.catalog)
|
||||
catalog['tests'][0]['properties'][0]['value'] += ['CASE_ARM='+arm,*(extra or [])]
|
||||
self.catalog.write_text(json.dumps(catalog))
|
||||
|
||||
def test_required_arm_forces_both_environment_knobs_and_records_actual_arm(self):
|
||||
self.set_arm('respect-env',['MOBILEGL_IPC_RUN_AHEAD=0','MOBILEGL_IPC_VERB_BARRIER=0'])
|
||||
self.assertEqual(self.run_main('--require-run-ahead'),0)
|
||||
row=self.checkpoint()[0]
|
||||
self.assertEqual(row['requested_arm'],{'MOBILEGL_IPC_RUN_AHEAD':'1','MOBILEGL_IPC_VERB_BARRIER':'1'})
|
||||
self.assertEqual(row['actual_arm'],{'armed':True,'lockstep':False,'disarmed':False})
|
||||
self.assertEqual(self.run_main('--require-run-ahead','--resume'),0)
|
||||
self.assertEqual(len(self.checkpoint()),1)
|
||||
|
||||
def test_required_arm_rejects_lockstep_disarmed_and_server_only_marker(self):
|
||||
for arm in ('lockstep','disarmed','server-only'):
|
||||
with self.subTest(arm=arm):
|
||||
self.write_inputs()
|
||||
self.set_arm(arm)
|
||||
self.assertEqual(self.run_main('--require-run-ahead'),1)
|
||||
row=self.checkpoint()[-1]
|
||||
self.assertEqual(row['status'],'failed')
|
||||
self.assertIn('required run-ahead',row['error'])
|
||||
self.assertIsInstance(row['actual_arm'],dict)
|
||||
|
||||
def test_required_arm_does_not_resume_a_former_lockstep_image_pass(self):
|
||||
self.set_arm('lockstep')
|
||||
self.assertEqual(self.run_main(),0)
|
||||
self.assertTrue(self.checkpoint()[0]['actual_arm']['lockstep'])
|
||||
self.assertEqual(self.run_main('--require-run-ahead','--resume'),1)
|
||||
self.assertEqual(len(self.checkpoint()),2)
|
||||
self.assertEqual(self.checkpoint()[-1]['status'],'failed')
|
||||
|
||||
def test_success_is_resumed_only_while_its_evidence_is_intact(self):
|
||||
self.assertEqual(self.run_main(),0)
|
||||
self.assertEqual(self.run_main('--resume'),0)
|
||||
self.assertEqual(len(self.checkpoint()),1)
|
||||
record=self.checkpoint()[0]
|
||||
self.assertNotIn('not-saved-in-checkpoint',(self.out/'checkpoint.json').read_text())
|
||||
Path(record['result_path']).write_text('{"passed":false}')
|
||||
self.assertEqual(self.run_main('--resume'),0)
|
||||
self.assertEqual(len(self.checkpoint()),2)
|
||||
self.assertNotEqual(self.checkpoint()[0]['work'],self.checkpoint()[1]['work'])
|
||||
|
||||
def test_nonzero_exit_with_passed_json_never_becomes_a_checkpoint_pass(self):
|
||||
self.write_inputs('failed')
|
||||
self.assertEqual(self.run_main(),1)
|
||||
self.assertEqual(self.run_main('--resume'),1)
|
||||
self.assertEqual(len(self.checkpoint()),2)
|
||||
self.assertTrue(all(row['status']=='failed' and row['returncode']==7 for row in self.checkpoint()))
|
||||
|
||||
def test_timeout_with_passed_json_is_not_resumed_as_success(self):
|
||||
self.write_inputs('timeout')
|
||||
self.assertEqual(self.run_main('--idle-seconds','.2'),1)
|
||||
self.assertEqual(self.run_main('--idle-seconds','.2','--resume'),1)
|
||||
self.assertEqual(len(self.checkpoint()),2)
|
||||
self.assertTrue(all(row['status']=='timeout' and row['returncode']==124 for row in self.checkpoint()))
|
||||
|
||||
def test_stale_scratch_result_cannot_satisfy_a_new_attempt(self):
|
||||
old=self.out/'Example/output/result.json';old.parent.mkdir(parents=True)
|
||||
old.write_text('{"passed":true,"statusCode":0}')
|
||||
legacy='[{"case":"Example","returncode":0}]'
|
||||
(self.out/'results.json').write_text(legacy)
|
||||
self.write_inputs('no-result')
|
||||
self.assertEqual(self.run_main('--resume'),1)
|
||||
row=self.checkpoint()[0]
|
||||
self.assertEqual(row['status'],'failed')
|
||||
self.assertEqual(row['returncode'],1)
|
||||
self.assertEqual(json.loads(old.read_text())['passed'],True) # Old evidence is preserved, not promoted.
|
||||
backups=list(self.out.glob('results.legacy-*.json'))
|
||||
self.assertEqual(len(backups),1)
|
||||
self.assertEqual(backups[0].read_text(),legacy)
|
||||
|
||||
def test_artifact_change_invalidates_a_successful_checkpoint(self):
|
||||
self.assertEqual(self.run_main(),0)
|
||||
self.library.write_bytes(b'library-v2')
|
||||
self.assertEqual(self.run_main('--resume'),0)
|
||||
self.assertEqual(len(self.checkpoint()),2)
|
||||
self.assertNotEqual(self.checkpoint()[0]['identity'],self.checkpoint()[1]['identity'])
|
||||
|
||||
def test_active_logs_outlive_old_local_catalog_and_manifest_timeout(self):
|
||||
self.write_inputs('active')
|
||||
self.assertEqual(self.run_main('--idle-seconds','.3','--max-seconds','4'),0)
|
||||
row=self.checkpoint()[0]
|
||||
self.assertGreater(row['seconds'],.6) # Both obsolete local timeouts are .05 seconds.
|
||||
self.assertIsNone(row['timeout_kind'])
|
||||
self.assertEqual(row['status'],'passed')
|
||||
|
||||
def test_timeout_kills_and_reaps_a_child_that_ignores_term_after_parent_exits(self):
|
||||
work=self.root/'group';work.mkdir()
|
||||
pidfile=work/'child.pid'
|
||||
child=f"import os,signal,time;from pathlib import Path;signal.signal(signal.SIGTERM,signal.SIG_IGN);Path({str(pidfile)!r}).write_text(str(os.getpid()));time.sleep(60)"
|
||||
parent=f"import subprocess,sys,time;subprocess.Popen([sys.executable,'-c',{child!r}]);time.sleep(60)"
|
||||
outcome=matrix.supervise([sys.executable,'-c',parent],dict(os.environ),str(work),work,work/'runner.log',
|
||||
idle_seconds=1,max_seconds=4,poll_seconds=.05,terminate_grace=.1)
|
||||
self.assertEqual(outcome['timeout_kind'],'idle')
|
||||
self.assertEqual(outcome['remaining_processes'],[])
|
||||
pid=int(pidfile.read_text())
|
||||
self.assertFalse(Path('/proc',str(pid)).exists(),f'orphan/zombie {pid} survived cleanup')
|
||||
|
||||
def test_case_environment_starts_from_fresh_base_and_owned_transport_wins(self):
|
||||
args=types.SimpleNamespace(endpoint='tcp://127.0.0.1:9',token='x',credit=2)
|
||||
first={'properties':[{'name':'ENVIRONMENT','value':['CASE_ONLY=one','MOBILEGL_IPC_CONTROL=tcp://stale:1']}]}
|
||||
base={'BASE':'same'}
|
||||
a=matrix.case_environment(first,base,args)
|
||||
b=matrix.case_environment({'properties':[]},base,args)
|
||||
self.assertEqual(a['CASE_ONLY'],'one')
|
||||
self.assertNotIn('CASE_ONLY',b)
|
||||
self.assertEqual(a['MOBILEGL_IPC_CONTROL'],args.endpoint)
|
||||
self.assertEqual(base,{'BASE':'same'})
|
||||
|
||||
def test_current_manifest_default_and_explicit_non_ci_selection(self):
|
||||
common={'trace_archive':'fixture.tgz','trace_file':'trace.trace','golden':'golden.png',
|
||||
'target_call':42,'width':16,'height':16}
|
||||
self.write_inputs(extra_cases=[dict(common,name='Extra',ci=False),dict(common,name='OptOut',split=False)])
|
||||
self.assertEqual([c['name'] for c in matrix.select_cases(self.manifest,None)],['Example'])
|
||||
self.assertEqual(matrix.select_cases(self.manifest,['Extra'])[0]['name'],'Extra')
|
||||
with self.assertRaises(ValueError):matrix.select_cases(self.manifest,['OptOut'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main(verbosity=2)
|
||||
Reference in New Issue
Block a user