mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-20 16:31:57 +09:00
[CI, Tools] (android-plugin): log each shard case's settings, record and transport lines
- The monolith lane fails when the library log shows a split transport
This commit is contained in:
@@ -79,8 +79,8 @@ adb shell run-as top.mobilegl.plugin.espryt.trace cat files/trace-replay/output/
|
|||||||
|
|
||||||
The APK workflow replays its cases through `trace-replay-shard.sh`: one emulator boot, one APK
|
The APK workflow replays its cases through `trace-replay-shard.sh`: one emulator boot, one APK
|
||||||
install, then an ordered list of cases for one lane, transport and backend. Each case still runs
|
install, then an ordered list of cases for one lane, transport and backend. Each case still runs
|
||||||
through `trace-replay-ci.sh`, with the same arguments, environment and transport proof as a
|
through `trace-replay-ci.sh`, with the arguments and environment of a one-case job and the
|
||||||
one-case job.
|
transport proof of its lane (see [Transport proofs](#transport-proofs)).
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
sh android-plugin/trace-replay-shard.sh \
|
sh android-plugin/trace-replay-shard.sh \
|
||||||
@@ -136,5 +136,73 @@ A failing case does not stop the shard. Every case ends with `<case>-<backend>/c
|
|||||||
a verdict is missing. `--finalize` writes the summary inputs and a `not-run` verdict for every
|
a verdict is missing. `--finalize` writes the summary inputs and a `not-run` verdict for every
|
||||||
case without one, without a device; the workflow runs it after the replay step whatever happened.
|
case without one, without a device; the workflow runs it after the replay step whatever happened.
|
||||||
|
|
||||||
|
### Transport proofs
|
||||||
|
|
||||||
|
Every lane passes `trace-replay-ci.sh` the flag that requires the proof of its transport. The
|
||||||
|
check runs after a replay that passed, reads the library's own log (`mobilegl.log`, the client
|
||||||
|
role's log followed by the server role's), writes `transport-proof.json` and prints one
|
||||||
|
`Android retrace: ...` line. A case whose proof fails is a failed case.
|
||||||
|
|
||||||
|
| transport | flag | what the library log must show |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| monolith | `--require-monolith` | nothing a split transport writes: neither `Config: MOBILEGL_TRANSPORT=... - the MGPipe record stream` sentence, no `Config: IPC` line, no `spawn ARMED` line, no library line in a server role log |
|
||||||
|
| inproc | `--require-inproc` | the inproc transport sentence, `strict=1`, `role-split-state=1` and `run-ahead=1` on every `Config: IPC` line, no `Fatal{` record |
|
||||||
|
| spawn | `--require-spawn` | the spawn transport sentence, `spawn ARMED - the server role runs in pid N`, the same settings on the client's `Config: IPC` line, no `Fatal{` record |
|
||||||
|
|
||||||
|
The monolith proof is one of absence. The library logs a transport sentence only when it
|
||||||
|
resolves inproc or spawn, and a library built without the split transports has no such code, so
|
||||||
|
the check fails only on what a split run writes. Its line says how much the log proves:
|
||||||
|
|
||||||
|
- `Android retrace: monolith confirmed: ...` when the log shows the configuration load
|
||||||
|
(`Config loaded`), during which the transport sentence and the `Config: IPC` line are
|
||||||
|
written, and nothing of the table row above.
|
||||||
|
- `Android retrace: monolith not confirmed (...)` when the log is missing or lacks the load. A
|
||||||
|
lost log says nothing about the transport, so the case still passes; its
|
||||||
|
`transport-proof.json` has `"confirmed": false` and the reason.
|
||||||
|
|
||||||
|
The check reads a split APK run with `MOBILEGL_TRANSPORT=monolith` and an APK built without
|
||||||
|
the split transports alike. That build writes one log, `mobilegl.log` itself, which the script
|
||||||
|
pulls whenever no role log arrived.
|
||||||
|
|
||||||
|
### What the job log holds
|
||||||
|
|
||||||
|
The workflow deletes the result artifacts of a green run, so the job log repeats what the
|
||||||
|
result files say. The shard starts with one `trace-replay-shard: every run: {...}` line, and
|
||||||
|
each case's group holds:
|
||||||
|
|
||||||
|
```text
|
||||||
|
trace-replay-shard: run <case>: {"arguments": [...], "case": "<case>", "environment": {...}, "timeout": N}
|
||||||
|
(the output of trace-replay-ci.sh, ending with its "Android retrace: ..." proof line)
|
||||||
|
trace-replay-shard: library-log <case>: mobilegl.log: <a line of the library log>
|
||||||
|
trace-replay-shard: phases <case>: {"fixture-pushed": 2.5, "apk-installed": 0.0, ...}
|
||||||
|
trace-replay-shard: record <case>: {"attempts": 1, "case": "<case>", "exit_codes": [0], ...}
|
||||||
|
trace-replay-shard: emulator boots after <case>: 1
|
||||||
|
trace-replay-shard: <case>: passed
|
||||||
|
```
|
||||||
|
|
||||||
|
- `every run` is the replay command, the environment the runner adds to every case, and
|
||||||
|
under `inherited` each further variable `trace-replay-ci.sh` reads (`ADB`, `PYTHON`,
|
||||||
|
`MOBILEGL_RETRACE_USE_PBUFFER`, `MOBILEGL_TRACE_ENV`) with the value it has in the runner's
|
||||||
|
own environment, which a replay inherits, or `null` when it is not set there.
|
||||||
|
- `run` is the line `--dry-run` prints for the case: the argument list `trace-replay-ci.sh` is
|
||||||
|
started with, the case's replay environment and its time budget. The same object starts the
|
||||||
|
replay, so the two cannot differ.
|
||||||
|
- `library-log` quotes the lines of the pulled library log that say which transport ran: the
|
||||||
|
backend and `Config loaded` lines, every line naming `MOBILEGL_TRANSPORT`, the `Config: IPC`
|
||||||
|
and `spawn ARMED` lines, and `Fatal{` records. At most 12 lines are quoted per case, each cut
|
||||||
|
to 1000 characters with characters outside ASCII escaped, and a further line says how many
|
||||||
|
were left out. The log is that of the attempt that counts. A failed case is quoted too, and
|
||||||
|
a case whose replay script ended before its own pulls is quoted from the logs the runner
|
||||||
|
pulls: the two role logs, or the one log of a build without the split transports.
|
||||||
|
- `phases` is the seconds between one mark of `phase-timings.txt` and the next, named by the
|
||||||
|
later mark, for the attempt that counts.
|
||||||
|
- `record` is `case-status.json`: status, reason, attempts, exit codes, seconds and notes.
|
||||||
|
- The verdict line is the last line of the group.
|
||||||
|
|
||||||
|
The shard ends with `trace-replay-shard: shard-status: {...}`, the content of
|
||||||
|
`shard-status.json`, and then the tally line. `--finalize` prints the same line for the file as
|
||||||
|
it leaves it.
|
||||||
|
|
||||||
`python3 android-plugin/test_trace_replay_shard.py` drives the runner with a fake adb, replay
|
`python3 android-plugin/test_trace_replay_shard.py` drives the runner with a fake adb, replay
|
||||||
script and AVD script.
|
script and AVD script, and runs `trace-replay-ci.sh` itself against an adb that serves a host
|
||||||
|
directory.
|
||||||
|
|||||||
@@ -10,8 +10,11 @@
|
|||||||
"""Drive the emulator shard runner with a fake adb, replay script and AVD script.
|
"""Drive the emulator shard runner with a fake adb, replay script and AVD script.
|
||||||
|
|
||||||
No device is needed. The fakes record every call, so the tests pin what each case is run
|
No device is needed. The fakes record every call, so the tests pin what each case is run
|
||||||
with, that every case ends with a verdict, and what a failure, an infrastructure retry, a
|
with, that every case ends with a verdict, what the job log says about it, and what a
|
||||||
timeout or a dead emulator does to the rest of the shard.
|
failure, an infrastructure retry, a timeout or a dead emulator does to the rest of the shard.
|
||||||
|
|
||||||
|
The last two classes run the transport proofs of trace-replay-ci.sh itself: the monolith
|
||||||
|
check on log files, and the whole script against an adb that serves a host directory.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import contextlib
|
import contextlib
|
||||||
@@ -21,15 +24,19 @@ import json
|
|||||||
import os
|
import os
|
||||||
import pathlib
|
import pathlib
|
||||||
import re
|
import re
|
||||||
|
import shlex
|
||||||
|
import shutil
|
||||||
import signal
|
import signal
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import tarfile
|
||||||
import tempfile
|
import tempfile
|
||||||
import time
|
import time
|
||||||
import unittest
|
import unittest
|
||||||
from unittest import mock
|
from unittest import mock
|
||||||
|
|
||||||
HERE = pathlib.Path(__file__).resolve().parent
|
HERE = pathlib.Path(__file__).resolve().parent
|
||||||
|
REPLAY_SCRIPT = HERE / "trace-replay-ci.sh"
|
||||||
|
|
||||||
|
|
||||||
def load(name):
|
def load(name):
|
||||||
@@ -86,6 +93,12 @@ elif args[:3] == ["shell", "ps", "-A"]:
|
|||||||
print("root 1 init")
|
print("root 1 init")
|
||||||
if count("ps") < plan.get("ps_calls_with_leftover", 0):
|
if count("ps") < plan.get("ps_calls_with_leftover", 0):
|
||||||
print("u0_a77 4242 libMobileGLServer.so")
|
print("u0_a77 4242 libMobileGLServer.so")
|
||||||
|
elif args[:2] == ["exec-out", "run-as"] and args[3:4] == ["cat"] and "device_files" in plan:
|
||||||
|
name = args[4].rsplit("/", 1)[-1]
|
||||||
|
if name not in plan["device_files"]:
|
||||||
|
print(f"cat: {args[4]}: No such file or directory")
|
||||||
|
sys.exit(1)
|
||||||
|
sys.stdout.write(plan["device_files"][name])
|
||||||
'''
|
'''
|
||||||
|
|
||||||
FAKE_REPLAY = FAKE_COMMON + r'''
|
FAKE_REPLAY = FAKE_COMMON + r'''
|
||||||
@@ -112,9 +125,11 @@ if code == 75:
|
|||||||
if step.get("result", code in (0, 1)):
|
if step.get("result", code in (0, 1)):
|
||||||
(directory / "result.json").write_text(json.dumps(
|
(directory / "result.json").write_text(json.dumps(
|
||||||
{"passed": step.get("passed", code == 0), "ssim": 0.999, "message": step.get("message", "retrace completed")}))
|
{"passed": step.get("passed", code == 0), "ssim": 0.999, "message": step.get("message", "retrace completed")}))
|
||||||
transport = "inproc" if "--require-inproc" in args else "spawn" if "--require-spawn" in args else ""
|
transport = next((name for name in ("monolith", "inproc", "spawn") if f"--require-{name}" in args), "")
|
||||||
if transport and code == 0 and step.get("proof", True):
|
if transport and code == 0 and step.get("proof", True):
|
||||||
(directory / "transport-proof.json").write_text(json.dumps({"passed": True, "required_transport": transport}))
|
(directory / "transport-proof.json").write_text(json.dumps({"passed": True, "required_transport": transport}))
|
||||||
|
for name, text in step.get("files", {}).items():
|
||||||
|
(directory / name).write_text(text, encoding="utf-8")
|
||||||
sys.exit(code)
|
sys.exit(code)
|
||||||
'''
|
'''
|
||||||
|
|
||||||
@@ -127,6 +142,46 @@ if args[:1] == ["start"]:
|
|||||||
sys.exit(exits[index] if index < len(exits) else 0)
|
sys.exit(exits[index] if index < len(exits) else 0)
|
||||||
'''
|
'''
|
||||||
|
|
||||||
|
# Library log lines as the emulator jobs write them.
|
||||||
|
CLIENT = "[02:55:51] [Android MobileGLTraceRe/INFO]: "
|
||||||
|
SERVER = "[02:55:51] [Android mgl-srv-apply/INFO]: "
|
||||||
|
BACKEND_LINE = CLIENT + "Config: Active backend type set to DirectVulkan"
|
||||||
|
LOADED_LINE = CLIENT + "Config loaded"
|
||||||
|
INPROC_LINE = CLIENT + ("Config: MOBILEGL_TRANSPORT=inproc - the MGPipe record stream crosses a real ring to an "
|
||||||
|
"apply thread")
|
||||||
|
SPAWN_LINE = CLIENT + ("Config: MOBILEGL_TRANSPORT=spawn - the MGPipe record stream crosses a real ring to an "
|
||||||
|
"apply thread in ANOTHER PROCESS")
|
||||||
|
IPC_LINE = CLIENT + ("Config: IPC ring=8MiB stage=32MiB wire-deferred=64MiB spin=50us event-wait=2000ms "
|
||||||
|
"create-window=10 pbo-readback-sync=0 poll-escalate=64 adopt-tier=unset(T0) allow-t0=1 "
|
||||||
|
"run-ahead=1 present-credit=1 control-timeout=5000ms cold-start=20000ms strict=1 audit=0 "
|
||||||
|
"role-split-state=1 affinity='auto' surface=offscreen")
|
||||||
|
ARMED_LINE = CLIENT + ('MG_Remote client: spawn ARMED - the server role runs in pid 2252, reached at '
|
||||||
|
'"@mgl-2134-124880560801840"')
|
||||||
|
SERVER_LINE = SERVER + "MG_Remote server: mgl-srv-apply started."
|
||||||
|
REST_OF_LOG = [CLIENT + "MobileGL Backend Info:", CLIENT + " Renderer Name: Magma"]
|
||||||
|
|
||||||
|
|
||||||
|
def log_text(*lines):
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def library_logs(transport):
|
||||||
|
"""The files a replay on the transport leaves in the app's output directory, by name."""
|
||||||
|
start = [CLIENT + "Initializing MobileGL...", BACKEND_LINE]
|
||||||
|
if transport == "monolith":
|
||||||
|
return {"mobilegl.client.log": log_text(*start, LOADED_LINE, *REST_OF_LOG)}
|
||||||
|
if transport == "inproc":
|
||||||
|
return {"mobilegl.client.log": log_text(*start, INPROC_LINE, IPC_LINE, LOADED_LINE, *REST_OF_LOG),
|
||||||
|
"mobilegl.server.log": log_text(SERVER_LINE)}
|
||||||
|
return {"mobilegl.client.log": log_text(*start, SPAWN_LINE, IPC_LINE, LOADED_LINE, *REST_OF_LOG, ARMED_LINE),
|
||||||
|
"mobilegl.server.log": log_text(SERVER_LINE)}
|
||||||
|
|
||||||
|
|
||||||
|
def said(output, kind, name):
|
||||||
|
"""What the runner's `kind` lines say about the case."""
|
||||||
|
prefix = f"trace-replay-shard: {kind} {name}: "
|
||||||
|
return [line[len(prefix):] for line in output.splitlines() if line.startswith(prefix)]
|
||||||
|
|
||||||
|
|
||||||
def case(name, **extra):
|
def case(name, **extra):
|
||||||
base = {"name": name, "trace_archive": f"fixtures/{name}.tgz", "trace_file": "trace.trace",
|
base = {"name": name, "trace_archive": f"fixtures/{name}.tgz", "trace_file": "trace.trace",
|
||||||
@@ -209,7 +264,7 @@ class ShardTest(unittest.TestCase):
|
|||||||
"--crop-x", "0", "--crop-y", "0", "--crop-width", "0", "--crop-height", "0",
|
"--crop-x", "0", "--crop-y", "0", "--crop-width", "0", "--crop-height", "0",
|
||||||
"--timeout-seconds", "900"]
|
"--timeout-seconds", "900"]
|
||||||
tails = {
|
tails = {
|
||||||
"monolith": ["--env", "MOBILEGL_TRANSPORT=monolith;MOBILEGL_GPU_HANG_BUDGET_MS=0"],
|
"monolith": ["--env", "MOBILEGL_TRANSPORT=monolith;MOBILEGL_GPU_HANG_BUDGET_MS=0", "--require-monolith"],
|
||||||
"inproc": ["--env", f"MOBILEGL_TRANSPORT=inproc;{SPLIT};MOBILEGL_GPU_HANG_BUDGET_MS=0", "--require-inproc"],
|
"inproc": ["--env", f"MOBILEGL_TRANSPORT=inproc;{SPLIT};MOBILEGL_GPU_HANG_BUDGET_MS=0", "--require-inproc"],
|
||||||
"spawn": ["--env", f"MOBILEGL_TRANSPORT=spawn;{SPLIT};MOBILEGL_GPU_HANG_BUDGET_MS=0",
|
"spawn": ["--env", f"MOBILEGL_TRANSPORT=spawn;{SPLIT};MOBILEGL_GPU_HANG_BUDGET_MS=0",
|
||||||
"--use-pbuffer", "--require-spawn"],
|
"--use-pbuffer", "--require-spawn"],
|
||||||
@@ -229,7 +284,8 @@ class ShardTest(unittest.TestCase):
|
|||||||
vulkan = {plan["case"]: plan for plan in self.planned(cases, "DirectVulkan", "monolith")}
|
vulkan = {plan["case"]: plan for plan in self.planned(cases, "DirectVulkan", "monolith")}
|
||||||
|
|
||||||
def flags(plan):
|
def flags(plan):
|
||||||
return [argument for argument in plan["arguments"][plan["arguments"].index("--env") + 2:]]
|
# What follows the transport's own arguments, which end with its proof.
|
||||||
|
return plan["arguments"][plan["arguments"].index("--require-monolith") + 1:]
|
||||||
|
|
||||||
self.assertEqual(gles[BLISS]["environment"]["MOBILEGL_TRACE_ANGLE_VARIANT"], "90a62123d794")
|
self.assertEqual(gles[BLISS]["environment"]["MOBILEGL_TRACE_ANGLE_VARIANT"], "90a62123d794")
|
||||||
self.assertEqual(vulkan[BLISS]["environment"]["MOBILEGL_TRACE_ANGLE_VARIANT"], "90a62123d794")
|
self.assertEqual(vulkan[BLISS]["environment"]["MOBILEGL_TRACE_ANGLE_VARIANT"], "90a62123d794")
|
||||||
@@ -324,6 +380,167 @@ class ShardTest(unittest.TestCase):
|
|||||||
self.assertIn("without a passed result.json", self.status("a")["reason"])
|
self.assertIn("without a passed result.json", self.status("a")["reason"])
|
||||||
self.assertIn("without a passed spawn transport proof", self.status("b")["reason"])
|
self.assertIn("without a passed spawn transport proof", self.status("b")["reason"])
|
||||||
|
|
||||||
|
def test_the_monolith_lane_needs_its_proof_too(self):
|
||||||
|
status, _ = self.run_shard([case("a"), case("b")], scenario={"replay": {"a": [{"exit": 0, "proof": False}]}})
|
||||||
|
self.assertEqual(status, 1)
|
||||||
|
self.assertEqual(self.status("a")["reason"], "the replay exited 0 without a passed monolith transport proof")
|
||||||
|
self.assertEqual(self.status("b")["status"], "passed")
|
||||||
|
|
||||||
|
# ---- what the job log says ---------------------------------------------------------
|
||||||
|
|
||||||
|
def test_the_log_says_what_each_case_is_run_with(self):
|
||||||
|
cases = [case("a", coherent_as_flush=True), case(BLISS)]
|
||||||
|
shared = {"MOBILEGL_TRACE_SKIP_INSTALL": "1", "MOBILEGL_TRACE_POLL_INTERVAL": "0.5"}
|
||||||
|
planned = self.planned(cases, "DirectGLES", "spawn")
|
||||||
|
# The test decides what the runner inherits, not the job the test runs in.
|
||||||
|
with mock.patch.dict(os.environ, {"MOBILEGL_RETRACE_USE_PBUFFER": "1"}):
|
||||||
|
for name in ("PYTHON", "MOBILEGL_TRACE_ENV"):
|
||||||
|
os.environ.pop(name, None)
|
||||||
|
status, output = self.run_shard(cases, backend="DirectGLES", transport="spawn")
|
||||||
|
self.assertEqual(status, 0, output)
|
||||||
|
lines = output.splitlines()
|
||||||
|
for index, plan in enumerate(planned):
|
||||||
|
name = plan["case"]
|
||||||
|
run, = said(output, "run", name)
|
||||||
|
# The line is the case's --dry-run line, and it opens the case's group.
|
||||||
|
self.assertEqual(run, json.dumps(plan, sort_keys=True))
|
||||||
|
self.assertEqual(lines[lines.index(f"::group::{name} ({index + 1}/2)") + 1],
|
||||||
|
f"trace-replay-shard: run {name}: {run}")
|
||||||
|
# It is also what the replay script was started with.
|
||||||
|
started = self.log("replay.log")[index]
|
||||||
|
self.assertEqual(started["arguments"], plan["arguments"])
|
||||||
|
self.assertEqual(started["environment"], {**plan["environment"], **shared})
|
||||||
|
every, = [line for line in lines if line.startswith("trace-replay-shard: every run: ")]
|
||||||
|
self.assertEqual(json.loads(every.split(": ", 2)[2]), {
|
||||||
|
"command": shlex.split(self.environment["MOBILEGL_SHARD_REPLAY_COMMAND"]), "environment": shared,
|
||||||
|
# What the replay script reads from the runner's own environment, set or not.
|
||||||
|
"inherited": {"ADB": self.environment["ADB"], "PYTHON": None, "MOBILEGL_RETRACE_USE_PBUFFER": "1",
|
||||||
|
"MOBILEGL_TRACE_ENV": None}})
|
||||||
|
|
||||||
|
def test_the_log_names_every_setting_the_replay_script_reads(self):
|
||||||
|
# Each variable trace-replay-ci.sh takes from its environment is in the case's run line
|
||||||
|
# or in the every-run line. The two it assigns itself are not settings.
|
||||||
|
script = REPLAY_SCRIPT.read_text(encoding="utf-8")
|
||||||
|
read = set(re.findall(r"\$\{([A-Z][A-Z0-9_]*)", script)) - {"IFS", "INFRASTRUCTURE_FAILURE_EXIT_CODE"}
|
||||||
|
plan, = self.planned([case("a")], "DirectVulkan", "monolith")
|
||||||
|
named = set(plan["environment"]) | {"MOBILEGL_TRACE_SKIP_INSTALL", "MOBILEGL_TRACE_POLL_INTERVAL"}
|
||||||
|
self.assertEqual(read - named, set(shard.INHERITED_REPLAY_SETTINGS))
|
||||||
|
|
||||||
|
def test_the_log_holds_every_record_and_the_shard_status(self):
|
||||||
|
phases = "script-started 1000\nfixture-pushed 3500\napk-installed 3500\napp-started 4700\nwait-ended 52700\n"
|
||||||
|
cases = [case("a"), case("b"), case("c")]
|
||||||
|
status, output = self.run_shard(cases, scenario={"replay": {
|
||||||
|
"a": [{"exit": 75}, {"files": {"phase-timings.txt": phases}}], "b": [{"exit": 1, "message": "ssim=0.5"}]}})
|
||||||
|
self.assertEqual(status, 1)
|
||||||
|
lines = output.splitlines()
|
||||||
|
# a was retried on a second boot, and c starts on a third because b did not pass.
|
||||||
|
verdicts = {"a": ("trace-replay-shard: a: passed", 2),
|
||||||
|
"b": ("trace-replay-shard: b: failed (exit 1: ssim=0.5)", 2),
|
||||||
|
"c": ("trace-replay-shard: c: passed", 3)}
|
||||||
|
for name, (verdict, boots) in verdicts.items():
|
||||||
|
record, = said(output, "record", name)
|
||||||
|
self.assertEqual(json.loads(record), self.status(name))
|
||||||
|
# The record and the boots come before the verdict line, which still ends the group.
|
||||||
|
at = lines.index(verdict)
|
||||||
|
self.assertEqual(lines[at - 2:at + 2], [
|
||||||
|
f"trace-replay-shard: record {name}: {record}",
|
||||||
|
f"trace-replay-shard: emulator boots after {name}: {boots}", verdict, "::endgroup::"])
|
||||||
|
self.assertEqual(json.loads(said(output, "record", "a")[0])["exit_codes"], [75, 0])
|
||||||
|
# The phases are those of the attempt that counts; a replay that left no timings has none.
|
||||||
|
self.assertEqual(said(output, "phases", "a"),
|
||||||
|
['{"fixture-pushed": 2.5, "apk-installed": 0.0, "app-started": 1.2, "wait-ended": 48.0}'])
|
||||||
|
self.assertEqual(said(output, "phases", "b"), [])
|
||||||
|
self.assertEqual(lines[-1], "trace-replay-shard: 2 passed, 1 not passed, 3 verdict(s) for 3 case(s)")
|
||||||
|
self.assertTrue(lines[-2].startswith("trace-replay-shard: shard-status: "), lines[-2])
|
||||||
|
self.assertEqual(json.loads(lines[-2].split(": ", 2)[2]), self.shard_status())
|
||||||
|
# The step that runs afterwards prints the file as it leaves it.
|
||||||
|
status, output = self.run_shard(cases, "--finalize")
|
||||||
|
self.assertEqual(status, 0)
|
||||||
|
self.assertEqual(json.loads(output.splitlines()[-1].split(": ", 2)[2]), self.shard_status())
|
||||||
|
|
||||||
|
def test_the_log_quotes_the_transport_lines_of_each_library_log(self):
|
||||||
|
refusals = [CLIENT + f"MGPipe: Fatal{{Refused, {index}}}" for index in range(40)]
|
||||||
|
long_line = SERVER + "MGPipe: Fatal{" + "x" * 3000 + "}"
|
||||||
|
inproc_log = log_text(BACKEND_LINE, INPROC_LINE, IPC_LINE, LOADED_LINE, *REST_OF_LOG, SERVER_LINE)
|
||||||
|
status, output = self.run_shard([case(name) for name in "abcde"], transport="inproc", scenario={
|
||||||
|
"replay": {
|
||||||
|
"a": [{"files": {"mobilegl.log": inproc_log}}],
|
||||||
|
# A failed case is quoted too, and a log full of such lines does not flood the job log.
|
||||||
|
"b": [{"exit": 1, "files": {"mobilegl.log": log_text(BACKEND_LINE, LOADED_LINE, *refusals)}}],
|
||||||
|
# Only the role logs: what the runner pulls itself when the replay script was killed.
|
||||||
|
"c": [{"files": {"mobilegl.client.log": log_text(SPAWN_LINE, *REST_OF_LOG, ARMED_LINE),
|
||||||
|
"mobilegl.server.log": log_text(SERVER_LINE, long_line)}}],
|
||||||
|
# Outside ASCII: the quote is escaped, whatever the encoding of the job log.
|
||||||
|
"d": [{"files": {"mobilegl.log": log_text(CLIENT + "MGPipe: Fatal{Refused, \u00a72.3}")}}],
|
||||||
|
"e": [{"files": {"mobilegl.log": log_text(*REST_OF_LOG)}}]}})
|
||||||
|
self.assertEqual(status, 1)
|
||||||
|
self.assertEqual(said(output, "library-log", "a"),
|
||||||
|
[f"mobilegl.log: {line}" for line in (BACKEND_LINE, INPROC_LINE, IPC_LINE, LOADED_LINE)])
|
||||||
|
quoted = said(output, "library-log", "b")
|
||||||
|
self.assertEqual(quoted[:-1], [f"mobilegl.log: {line}" for line in [BACKEND_LINE, LOADED_LINE] + refusals[:10]])
|
||||||
|
self.assertEqual(quoted[-1], "30 more such line(s) are not shown")
|
||||||
|
self.assertEqual(said(output, "library-log", "c"), [
|
||||||
|
f"mobilegl.client.log: {SPAWN_LINE}", f"mobilegl.client.log: {ARMED_LINE}",
|
||||||
|
f"mobilegl.server.log: {long_line[:1000]}"])
|
||||||
|
self.assertEqual(said(output, "library-log", "d"),
|
||||||
|
[f"mobilegl.log: {CLIENT}MGPipe: Fatal{{Refused, \\xa72.3}}"])
|
||||||
|
self.assertEqual(said(output, "library-log", "e"), ["the library log has none of the lines looked for"])
|
||||||
|
# The lines are in the case's group, before its record.
|
||||||
|
lines = output.splitlines()
|
||||||
|
quoted = [f"trace-replay-shard: library-log a: mobilegl.log: {line}" for line in (BACKEND_LINE, LOADED_LINE)]
|
||||||
|
record = next(line for line in lines if line.startswith("trace-replay-shard: record a: "))
|
||||||
|
self.assertLess(lines.index("::group::a (1/5)"), lines.index(quoted[0]))
|
||||||
|
self.assertLess(lines.index(quoted[1]), lines.index(record))
|
||||||
|
|
||||||
|
def test_the_quoted_library_log_is_that_of_the_attempt_that_counts(self):
|
||||||
|
first = log_text(BACKEND_LINE, INPROC_LINE, IPC_LINE, LOADED_LINE)
|
||||||
|
status, output = self.run_shard([case("a"), case("b")], scenario={"replay": {
|
||||||
|
"a": [{"exit": 75, "files": {"mobilegl.log": first}},
|
||||||
|
{"files": {"mobilegl.log": log_text(BACKEND_LINE, LOADED_LINE)}}],
|
||||||
|
"b": [{"exit": 75, "files": {"mobilegl.log": first}}, {}]}})
|
||||||
|
self.assertEqual(status, 0, output)
|
||||||
|
self.assertEqual(said(output, "library-log", "a"),
|
||||||
|
[f"mobilegl.log: {line}" for line in (BACKEND_LINE, LOADED_LINE)])
|
||||||
|
# A retry that left no log is not quoted from the attempt it replaced.
|
||||||
|
self.assertEqual(said(output, "library-log", "b"), ["no library log was pulled"])
|
||||||
|
for name in "ab":
|
||||||
|
kept = self.results / f"{name}-DirectVulkan" / "first-attempt" / "mobilegl.log"
|
||||||
|
self.assertEqual(kept.read_text(), first)
|
||||||
|
|
||||||
|
def test_a_replay_that_pulled_nothing_is_quoted_from_what_the_runner_pulls(self):
|
||||||
|
# The replay script ended before its own pulls. A split build has its two role logs
|
||||||
|
# on the device; a build without the split transports has the one log.
|
||||||
|
devices = {
|
||||||
|
"a": ({"mobilegl.client.log": log_text(BACKEND_LINE, INPROC_LINE, LOADED_LINE),
|
||||||
|
"mobilegl.server.log": log_text(SERVER_LINE)},
|
||||||
|
[f"mobilegl.client.log: {BACKEND_LINE}", f"mobilegl.client.log: {INPROC_LINE}",
|
||||||
|
f"mobilegl.client.log: {LOADED_LINE}"]),
|
||||||
|
"b": ({"mobilegl.log": log_text(BACKEND_LINE, LOADED_LINE)},
|
||||||
|
[f"mobilegl.log: {BACKEND_LINE}", f"mobilegl.log: {LOADED_LINE}"])}
|
||||||
|
for name, (files, quoted) in devices.items():
|
||||||
|
with self.subTest(name):
|
||||||
|
status, output = self.run_shard([case(name)], scenario={
|
||||||
|
"device_files": files, "replay": {name: [{"exit": 1, "result": False}]}})
|
||||||
|
self.assertEqual(status, 1)
|
||||||
|
self.assertEqual(said(output, "library-log", name), quoted)
|
||||||
|
pulled = sorted(path.name for path in (self.results / f"{name}-DirectVulkan").glob("mobilegl*"))
|
||||||
|
self.assertEqual(pulled, sorted(files))
|
||||||
|
|
||||||
|
def test_a_case_without_a_library_log_says_so(self):
|
||||||
|
status, output = self.run_shard([case("a")])
|
||||||
|
self.assertEqual(status, 0)
|
||||||
|
self.assertEqual(said(output, "library-log", "a"), ["no library log was pulled"])
|
||||||
|
|
||||||
|
def test_the_quoted_lines_are_the_ones_the_proofs_read(self):
|
||||||
|
# Every sentence a --require-* check of trace-replay-ci.sh looks for is one the runner quotes.
|
||||||
|
script = REPLAY_SCRIPT.read_text(encoding="utf-8")
|
||||||
|
for sentence in ("Config: MOBILEGL_TRANSPORT=inproc - the MGPipe record stream",
|
||||||
|
"Config: MOBILEGL_TRANSPORT=spawn - the MGPipe record stream",
|
||||||
|
"Config: MOBILEGL_TRANSPORT={transport} - the MGPipe record stream",
|
||||||
|
"spawn ARMED - the server role runs in pid", "Config: IPC", "Fatal\\{", "Config loaded"):
|
||||||
|
self.assertIn(sentence, script)
|
||||||
|
self.assertTrue(any(mark in sentence.replace("\\", "") for mark in shard.LIBRARY_LOG_EVIDENCE), sentence)
|
||||||
|
|
||||||
def test_an_infrastructure_failure_restarts_and_retries_once(self):
|
def test_an_infrastructure_failure_restarts_and_retries_once(self):
|
||||||
status, output = self.run_shard([case("a"), case("b")], scenario={
|
status, output = self.run_shard([case("a"), case("b")], scenario={
|
||||||
"replay": {"a": [{"exit": 75, "reason": "angle-surface-lost", "result": True, "passed": False}]}})
|
"replay": {"a": [{"exit": 75, "reason": "angle-surface-lost", "result": True, "passed": False}]}})
|
||||||
@@ -583,5 +800,296 @@ class ShardTest(unittest.TestCase):
|
|||||||
self.assertEqual(self.shard_status()["expected_cases"], ["a", "b", "c"])
|
self.assertEqual(self.shard_status()["expected_cases"], ["a", "b", "c"])
|
||||||
|
|
||||||
|
|
||||||
|
MONOLITH_CONFIRMED = ("Android retrace: monolith confirmed: the library loaded its configuration and logged no "
|
||||||
|
"inproc or spawn transport, no IPC configuration, no spawned server and no server role")
|
||||||
|
NO_LOG = "the library log is missing or empty"
|
||||||
|
NO_LOAD = "the library log does not show the configuration load"
|
||||||
|
|
||||||
|
|
||||||
|
def monolith_not_confirmed(reason):
|
||||||
|
return (f"Android retrace: monolith not confirmed ({reason}): nothing a split transport writes was found, "
|
||||||
|
"so the run is not failed")
|
||||||
|
|
||||||
|
|
||||||
|
class MonolithProofTest(unittest.TestCase):
|
||||||
|
"""The --require-monolith check of trace-replay-ci.sh, run on the log files of a result directory."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.temporary = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temporary.cleanup)
|
||||||
|
self.directory = pathlib.Path(self.temporary.name)
|
||||||
|
script = REPLAY_SCRIPT.read_text(encoding="utf-8")
|
||||||
|
start = script.index("<<'PY'\n", script.index('if [ "${require_monolith}" -eq 1 ]; then\n "${PYTHON}"'))
|
||||||
|
self.program = script[start + len("<<'PY'\n"):script.index("\nPY\n", start)]
|
||||||
|
|
||||||
|
def check(self, **logs):
|
||||||
|
"""Returns (exit status, stdout, stderr, the proof) for a result directory holding the logs."""
|
||||||
|
for name, text in logs.items():
|
||||||
|
(self.directory / name.replace("_", ".")).write_text(text, encoding="utf-8")
|
||||||
|
proof = self.directory / "transport-proof.json"
|
||||||
|
done = subprocess.run([sys.executable, "-", str(self.directory / "mobilegl.log"), str(proof)],
|
||||||
|
input=self.program, capture_output=True, text=True)
|
||||||
|
return done.returncode, done.stdout.strip(), done.stderr.strip(), json.loads(proof.read_text())
|
||||||
|
|
||||||
|
def test_a_monolith_log_passes(self):
|
||||||
|
# The split build names its only log after the client role; a build without the split
|
||||||
|
# transports writes the same lines. Either way mobilegl.log is what the check reads.
|
||||||
|
status, said_line, _, proof = self.check(mobilegl_log=library_logs("monolith")["mobilegl.client.log"])
|
||||||
|
self.assertEqual((status, said_line), (0, MONOLITH_CONFIRMED))
|
||||||
|
self.assertEqual({key: proof[key] for key in ("required_transport", "passed", "confirmed",
|
||||||
|
"configuration_loaded", "split_transport_markers",
|
||||||
|
"problems")},
|
||||||
|
{"required_transport": "monolith", "passed": True, "confirmed": True,
|
||||||
|
"configuration_loaded": True, "split_transport_markers": {"inproc": False, "spawn": False},
|
||||||
|
"problems": []})
|
||||||
|
|
||||||
|
def test_anything_a_split_run_logs_fails_it(self):
|
||||||
|
monolith = library_logs("monolith")["mobilegl.client.log"]
|
||||||
|
inproc, spawn = (log_text(*library_logs(transport).values()) for transport in ("inproc", "spawn"))
|
||||||
|
server_role = "a server role wrote a log of its own"
|
||||||
|
failures = [
|
||||||
|
("the library log has the inproc transport marker", {"mobilegl_log": inproc}),
|
||||||
|
("the library log has the spawn transport marker", {"mobilegl_log": spawn}),
|
||||||
|
("the library log names a spawned server process", {"mobilegl_log": monolith + ARMED_LINE + "\n"}),
|
||||||
|
("the library log has an IPC configuration line", {"mobilegl_log": monolith + IPC_LINE + "\n"}),
|
||||||
|
# The three headers a server role's line can have: the apply thread of an inproc
|
||||||
|
# run, a thread of a spawned server, and a line a peer forwarded.
|
||||||
|
(server_role, {"mobilegl_log": monolith, "mobilegl_server_log": SERVER_LINE}),
|
||||||
|
(server_role, {"mobilegl_log": monolith, "mobilegl_server_log":
|
||||||
|
"[08:25:29] [Android libMobileGLServ/ERROR]: MG_Remote server: pid=2252 listening\n"}),
|
||||||
|
(server_role, {"mobilegl_log": monolith, "mobilegl_server_log": "[peer/WARN]: refused\n"}),
|
||||||
|
# Without the client's log, where the transport sentence is.
|
||||||
|
(server_role, {"mobilegl_server_log": SERVER_LINE}),
|
||||||
|
]
|
||||||
|
for problem, logs in failures:
|
||||||
|
with self.subTest(problem, logs=sorted(logs.items())):
|
||||||
|
for leftover in self.directory.iterdir():
|
||||||
|
leftover.unlink()
|
||||||
|
status, said_line, error, proof = self.check(**logs)
|
||||||
|
self.assertEqual((status, said_line), (1, ""))
|
||||||
|
self.assertTrue(error.startswith("trace replay failed actual monolith/no-split-transport proof ("),
|
||||||
|
error)
|
||||||
|
self.assertIn(problem, error)
|
||||||
|
self.assertIn(problem, proof["problems"])
|
||||||
|
self.assertEqual((proof["required_transport"], proof["passed"], proof["confirmed"]),
|
||||||
|
("monolith", False, False))
|
||||||
|
|
||||||
|
def test_a_log_without_the_configuration_load_passes_as_not_confirmed(self):
|
||||||
|
# A lost or cut log says nothing about the transport, so it does not fail the run; the
|
||||||
|
# line and the proof say that monolith was not confirmed.
|
||||||
|
for reason, logs in ((NO_LOG, {}), (NO_LOG, {"mobilegl_log": "\n"}),
|
||||||
|
(NO_LOAD, {"mobilegl_log": log_text(BACKEND_LINE)})):
|
||||||
|
with self.subTest(reason, logs=logs):
|
||||||
|
for leftover in self.directory.iterdir():
|
||||||
|
leftover.unlink()
|
||||||
|
status, said_line, error, proof = self.check(**logs)
|
||||||
|
self.assertEqual((status, said_line, error), (0, monolith_not_confirmed(reason), ""))
|
||||||
|
self.assertEqual({key: proof[key] for key in ("required_transport", "passed", "confirmed",
|
||||||
|
"not_confirmed_because", "problems")},
|
||||||
|
{"required_transport": "monolith", "passed": True, "confirmed": False,
|
||||||
|
"not_confirmed_because": reason, "problems": []})
|
||||||
|
|
||||||
|
def test_what_a_device_answers_for_a_missing_server_log_is_not_a_server_role(self):
|
||||||
|
monolith = library_logs("monolith")["mobilegl.client.log"]
|
||||||
|
for answer in ("cat: /data/user/0/top.mobilegl.plugin.trace/files/mobilegl.server.log: Permission denied\n",
|
||||||
|
"run-as: unknown package: top.mobilegl.plugin.trace\n", "\n"):
|
||||||
|
with self.subTest(answer):
|
||||||
|
# trace-replay-ci.sh appends what it pulled for the server role to mobilegl.log.
|
||||||
|
status, said_line, _, proof = self.check(mobilegl_log=monolith + answer, mobilegl_server_log=answer)
|
||||||
|
self.assertEqual((status, said_line), (0, MONOLITH_CONFIRMED))
|
||||||
|
self.assertEqual((proof["server_role_log_lines"], proof["confirmed"]), (0, True))
|
||||||
|
|
||||||
|
|
||||||
|
# An adb whose device is the host directory FAKE_DEVICE: `exec-out run-as <package> cat <path>`
|
||||||
|
# prints the file of that name there, or cat's own complaint as a device folds it into stdout.
|
||||||
|
FAKE_DEVICE_ADB = r'''#!/bin/sh
|
||||||
|
case "$1 ${2:-}" in
|
||||||
|
"get-state "*) echo "${FAKE_ADB_STATE:-device}" ;;
|
||||||
|
"install "*) echo Success ;;
|
||||||
|
"exec-out run-as")
|
||||||
|
if [ -f "${FAKE_DEVICE}/${5##*/}" ]; then
|
||||||
|
cat "${FAKE_DEVICE}/${5##*/}"
|
||||||
|
else
|
||||||
|
echo "cat: $5: No such file or directory"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
"shell getprop") echo 1 ;;
|
||||||
|
"shell pm") if [ "$3" = path ]; then echo "package:/data/app/trace/base.apk"; else echo Success; fi ;;
|
||||||
|
"shell run-as") if [ "${4:-}" = id ]; then echo u0_a77; fi ;;
|
||||||
|
"shell ps") echo "USER PID NAME" ;;
|
||||||
|
shell*) case "$*" in *mobilegl-poll-result*) echo mobilegl-poll-result ;; esac ;;
|
||||||
|
esac
|
||||||
|
exit 0
|
||||||
|
'''
|
||||||
|
|
||||||
|
# The shard runner takes ADB as a command line; trace-replay-ci.sh runs it as one word.
|
||||||
|
FAKE_REPLAY_WRAPPER = r'''#!/bin/sh
|
||||||
|
ADB="${FAKE_DEVICE_ADB}"
|
||||||
|
export ADB
|
||||||
|
exec "${FAKE_SH}" "${FAKE_REPLAY_SCRIPT}" "$@"
|
||||||
|
'''
|
||||||
|
|
||||||
|
# BASH_EXE names a POSIX shell where `sh` is not one, for example Git Bash on Windows.
|
||||||
|
POSIX_SH = os.environ.get("BASH_EXE") or shutil.which("sh")
|
||||||
|
|
||||||
|
|
||||||
|
@unittest.skipUnless(POSIX_SH, "needs a POSIX sh (BASH_EXE) to run trace-replay-ci.sh")
|
||||||
|
class ReplayScriptTest(unittest.TestCase):
|
||||||
|
"""trace-replay-ci.sh itself, alone and under the shard runner, against the fake device."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.temporary = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(self.temporary.cleanup)
|
||||||
|
self.root = pathlib.Path(self.temporary.name)
|
||||||
|
self.device = self.root / "device"
|
||||||
|
for directory in (self.device, self.root / "fixtures"):
|
||||||
|
directory.mkdir()
|
||||||
|
# A checkout with CRLF line endings does not run under sh.
|
||||||
|
scripts = {"trace-replay-ci.sh": REPLAY_SCRIPT.read_bytes().replace(b"\r\n", b"\n"),
|
||||||
|
"adb": FAKE_DEVICE_ADB.encode(), "replay.sh": FAKE_REPLAY_WRAPPER.encode()}
|
||||||
|
for name, content in scripts.items():
|
||||||
|
(self.root / name).write_bytes(content)
|
||||||
|
(self.root / name).chmod(0o755)
|
||||||
|
(self.root / "trace.apk").write_bytes(b"apk")
|
||||||
|
(self.root / "fixtures" / "a.png").write_bytes(b"png")
|
||||||
|
(self.root / "trace.trace").write_bytes(b"trace")
|
||||||
|
with tarfile.open(self.root / "fixtures" / "a.tgz", "w:gz") as archive:
|
||||||
|
archive.add(self.root / "trace.trace", arcname="trace.trace")
|
||||||
|
(self.device / "result.json").write_text('{"passed": true, "ssim": 0.999, "message": "retrace completed"}\n')
|
||||||
|
sh = pathlib.Path(POSIX_SH)
|
||||||
|
# Nothing of the job that runs the tests: its replay settings, its emulator, its summary.
|
||||||
|
own = ("AVD_NAME", "EMULATOR_LOG", "EMULATOR_PID_FILE", "GITHUB_STEP_SUMMARY")
|
||||||
|
self.environment = {key: value for key, value in os.environ.items()
|
||||||
|
if not key.startswith("MOBILEGL_") and key not in own}
|
||||||
|
self.environment.update({
|
||||||
|
"GITHUB_STEP_SUMMARY": str(self.root / "summary.md"),
|
||||||
|
"PATH": str(sh.parent) + os.pathsep + os.environ.get("PATH", ""),
|
||||||
|
"PYTHON": pathlib.Path(sys.executable).as_posix(), "PYTHONDONTWRITEBYTECODE": "1",
|
||||||
|
"FAKE_DEVICE": self.device.as_posix(), "FAKE_DEVICE_ADB": (self.root / "adb").as_posix(),
|
||||||
|
"FAKE_SH": sh.as_posix(), "FAKE_REPLAY_SCRIPT": (self.root / "trace-replay-ci.sh").as_posix()})
|
||||||
|
|
||||||
|
def device_has(self, files):
|
||||||
|
for name, text in files.items():
|
||||||
|
(self.device / name).write_bytes(text.encode())
|
||||||
|
|
||||||
|
# Paths are relative to the temporary directory: tar reads a drive letter as a host name.
|
||||||
|
CASE = ["--apk-file", "trace.apk", "--package", "top.mobilegl.plugin.trace", "--backend", "DirectVulkan",
|
||||||
|
"--result-root", "results", "--fixture-root", "work", "--case", "a",
|
||||||
|
"--trace-archive", "fixtures/a.tgz", "--trace-file", "trace.trace", "--golden", "fixtures/a.png",
|
||||||
|
"--target-call", "1234", "--width", "854", "--height", "480", "--ssim-threshold", "0.99",
|
||||||
|
"--crop-x", "0", "--crop-y", "0", "--crop-width", "0", "--crop-height", "0", "--timeout-seconds", "5"]
|
||||||
|
|
||||||
|
def replay(self, *flags, adb_state="device"):
|
||||||
|
"""Returns (exit status, stdout, stderr, the proof or None) of one run of the script."""
|
||||||
|
environment = {**self.environment, "ADB": self.environment["FAKE_DEVICE_ADB"], "FAKE_ADB_STATE": adb_state}
|
||||||
|
done = subprocess.run([POSIX_SH, "trace-replay-ci.sh", *self.CASE, "--reuse-fixture", *flags],
|
||||||
|
cwd=self.root, env=environment, capture_output=True, text=True, timeout=600)
|
||||||
|
proof = self.root / "results" / "a-DirectVulkan" / "transport-proof.json"
|
||||||
|
return done.returncode, done.stdout, done.stderr, json.loads(proof.read_text()) if proof.exists() else None
|
||||||
|
|
||||||
|
def test_require_monolith_passes_on_a_monolith_run_of_the_split_build(self):
|
||||||
|
self.device_has(library_logs("monolith"))
|
||||||
|
status, output, error, proof = self.replay("--require-monolith")
|
||||||
|
self.assertEqual(status, 0, error)
|
||||||
|
self.assertIn(MONOLITH_CONFIRMED, output.splitlines())
|
||||||
|
self.assertEqual((proof["required_transport"], proof["passed"]), ("monolith", True))
|
||||||
|
|
||||||
|
def test_require_monolith_passes_on_a_build_without_the_split_transports(self):
|
||||||
|
# One role, one log, under the name the role logs are derived from.
|
||||||
|
self.device_has({"mobilegl.log": library_logs("monolith")["mobilegl.client.log"]})
|
||||||
|
status, output, error, proof = self.replay("--require-monolith")
|
||||||
|
self.assertEqual(status, 0, error)
|
||||||
|
self.assertIn(MONOLITH_CONFIRMED, output.splitlines())
|
||||||
|
self.assertTrue(proof["passed"])
|
||||||
|
|
||||||
|
def test_require_monolith_fails_a_run_that_was_not_monolith(self):
|
||||||
|
for transport in ("inproc", "spawn"):
|
||||||
|
with self.subTest(transport):
|
||||||
|
self.device_has(library_logs(transport))
|
||||||
|
status, output, error, proof = self.replay("--require-monolith")
|
||||||
|
self.assertEqual(status, 1)
|
||||||
|
self.assertNotIn(MONOLITH_CONFIRMED, output)
|
||||||
|
self.assertIn("trace replay failed actual monolith/no-split-transport proof", error)
|
||||||
|
self.assertIn(f"the library log has the {transport} transport marker", error)
|
||||||
|
self.assertFalse(proof["passed"])
|
||||||
|
|
||||||
|
def test_a_passed_monolith_replay_without_a_library_log_stays_passed(self):
|
||||||
|
# Whether the device still answers or went away before the log was pulled: a passed
|
||||||
|
# replay is not failed or retried for a log that says nothing about the transport.
|
||||||
|
for adb_state in ("device", "offline"):
|
||||||
|
with self.subTest(adb_state):
|
||||||
|
status, output, error, proof = self.replay("--require-monolith", adb_state=adb_state)
|
||||||
|
self.assertEqual(status, 0, error)
|
||||||
|
self.assertIn(monolith_not_confirmed(NO_LOG), output.splitlines())
|
||||||
|
self.assertEqual((proof["required_transport"], proof["passed"], proof["confirmed"]),
|
||||||
|
("monolith", True, False))
|
||||||
|
|
||||||
|
def test_a_split_log_lost_with_the_device_asks_for_the_infrastructure_retry(self):
|
||||||
|
# The inproc and spawn proofs need the log, so a device that went away before it was
|
||||||
|
# pulled is the emulator's failure and not the proof's.
|
||||||
|
status, _, error, proof = self.replay("--require-inproc", adb_state="offline")
|
||||||
|
self.assertEqual(status, shard.INFRASTRUCTURE_FAILURE_EXIT_CODE, error)
|
||||||
|
self.assertIsNone(proof)
|
||||||
|
|
||||||
|
def test_a_failed_replay_of_a_build_without_the_split_transports_keeps_its_library_log(self):
|
||||||
|
log = library_logs("monolith")["mobilegl.client.log"]
|
||||||
|
self.device_has({"mobilegl.log": log,
|
||||||
|
"result.json": '{"passed": false, "ssim": 0.5, "message": "retrace completed"}\n'})
|
||||||
|
status, _, error, proof = self.replay("--require-monolith")
|
||||||
|
self.assertEqual(status, 1)
|
||||||
|
self.assertIsNone(proof)
|
||||||
|
self.assertEqual((self.root / "results" / "a-DirectVulkan" / "mobilegl.log").read_bytes(), log.encode())
|
||||||
|
self.assertIn("trace-replay-ci.sh: tail of mobilegl.log:", error)
|
||||||
|
self.assertIn(LOADED_LINE, error)
|
||||||
|
|
||||||
|
def shard_through_the_script(self, transport, logs=None):
|
||||||
|
"""Runs one case through the runner and the real script; returns (exit status, the job log)."""
|
||||||
|
self.device_has(library_logs(transport) if logs is None else logs)
|
||||||
|
sh, adb = self.environment["FAKE_SH"], self.environment["FAKE_DEVICE_ADB"]
|
||||||
|
environment = {**self.environment, "ADB": f'"{sh}" "{adb}"',
|
||||||
|
"MOBILEGL_SHARD_REPLAY_COMMAND": f'"{sh}" "{(self.root / "replay.sh").as_posix()}"',
|
||||||
|
"MOBILEGL_GPU_HANG_BUDGET_MS": "0", "MOBILEGL_SHARD_JOB_ID": "1234"}
|
||||||
|
one_case = case("a", golden="fixtures/a.png", timeout_seconds=5)
|
||||||
|
done = subprocess.run(
|
||||||
|
[sys.executable, str(HERE / "trace_replay_shard.py"), "--apk-file", "trace.apk",
|
||||||
|
"--backend", "DirectVulkan", "--transport", transport, "--lane", f"{transport}-lane",
|
||||||
|
"--cases-json", json.dumps([one_case]), "--result-root", "results", "--fixture-root", "work"],
|
||||||
|
cwd=self.root, env=environment, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, timeout=900)
|
||||||
|
return done.returncode, done.stdout
|
||||||
|
|
||||||
|
def test_a_shard_reaches_the_proof_of_its_transport_through_the_script(self):
|
||||||
|
proofs = {"monolith": MONOLITH_CONFIRMED,
|
||||||
|
"inproc": "Android retrace: real inproc transport, strict role state and zero Fatal records "
|
||||||
|
"confirmed",
|
||||||
|
"spawn": "Android retrace: real spawn transport, server role in pid 2252, and zero Fatal records "
|
||||||
|
"confirmed"}
|
||||||
|
marks = {"monolith": LOADED_LINE, "inproc": INPROC_LINE, "spawn": ARMED_LINE}
|
||||||
|
for transport, proof in proofs.items():
|
||||||
|
with self.subTest(transport):
|
||||||
|
shutil.rmtree(self.root / "results", ignore_errors=True)
|
||||||
|
for leftover in self.device.glob("mobilegl*"):
|
||||||
|
leftover.unlink()
|
||||||
|
status, output = self.shard_through_the_script(transport)
|
||||||
|
self.assertEqual(status, 0, output)
|
||||||
|
lines = output.splitlines()
|
||||||
|
# The script took the runner's arguments, its own proof line is in the log, and
|
||||||
|
# the runner quoted the library's lines from the log the script pulled.
|
||||||
|
self.assertIn(proof, lines)
|
||||||
|
self.assertIn(f"mobilegl.log: {marks[transport]}", said(output, "library-log", "a"))
|
||||||
|
self.assertEqual(json.loads(said(output, "record", "a")[0])["status"], "passed")
|
||||||
|
self.assertIn("artifacts-copied", json.loads(said(output, "phases", "a")[0]))
|
||||||
|
self.assertEqual(lines[-1], "trace-replay-shard: 1 passed, 0 not passed, 1 verdict(s) for 1 case(s)")
|
||||||
|
written = json.loads((self.root / "results" / "a-DirectVulkan" / "transport-proof.json").read_text())
|
||||||
|
self.assertEqual((written["required_transport"], written["passed"]), (transport, True))
|
||||||
|
|
||||||
|
def test_a_monolith_shard_stays_green_without_a_library_log(self):
|
||||||
|
status, output = self.shard_through_the_script("monolith", logs={})
|
||||||
|
self.assertEqual(status, 0, output)
|
||||||
|
self.assertIn(monolith_not_confirmed(NO_LOG), output.splitlines())
|
||||||
|
self.assertEqual(said(output, "library-log", "a"), ["no library log was pulled"])
|
||||||
|
self.assertEqual(json.loads(said(output, "record", "a")[0])["status"], "passed")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ Usage:
|
|||||||
[--env "K=V;K=V"] \
|
[--env "K=V;K=V"] \
|
||||||
[--require-inproc] \
|
[--require-inproc] \
|
||||||
[--require-spawn] \
|
[--require-spawn] \
|
||||||
|
[--require-monolith] \
|
||||||
[--benchmark] \
|
[--benchmark] \
|
||||||
[--benchmark-tail-frames N] \
|
[--benchmark-tail-frames N] \
|
||||||
[--benchmark-finish 0|1] \
|
[--benchmark-finish 0|1] \
|
||||||
@@ -75,6 +76,11 @@ sentence a monolith fallback can never write: the pid of the server process.
|
|||||||
Pass --require-inproc to require the library's real transport-resolution log,
|
Pass --require-inproc to require the library's real transport-resolution log,
|
||||||
strict errors and separate role state, plus zero Fatal records. This checks the
|
strict errors and separate role state, plus zero Fatal records. This checks the
|
||||||
APK process after replay; a host environment echo or an SSIM-only pass is not proof.
|
APK process after replay; a host environment echo or an SSIM-only pass is not proof.
|
||||||
|
Pass --require-monolith to fail a run whose library log shows a split transport:
|
||||||
|
either transport sentence, an IPC configuration line, a spawned server or a log of
|
||||||
|
a server role. A log with the configuration load and none of these is reported as
|
||||||
|
monolith confirmed; a run whose log is missing or lacks the load passes and is
|
||||||
|
reported as not confirmed.
|
||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -152,6 +158,7 @@ texture_2d_dumps=""
|
|||||||
env_overrides="${MOBILEGL_TRACE_ENV:-}"
|
env_overrides="${MOBILEGL_TRACE_ENV:-}"
|
||||||
require_inproc=0
|
require_inproc=0
|
||||||
require_spawn=0
|
require_spawn=0
|
||||||
|
require_monolith=0
|
||||||
benchmark=0
|
benchmark=0
|
||||||
benchmark_tail_frames=200
|
benchmark_tail_frames=200
|
||||||
benchmark_finish=1
|
benchmark_finish=1
|
||||||
@@ -200,6 +207,7 @@ while [ "$#" -gt 0 ]; do
|
|||||||
--env) env_overrides="$(next_arg "$@")"; shift 2 ;;
|
--env) env_overrides="$(next_arg "$@")"; shift 2 ;;
|
||||||
--require-inproc) require_inproc=1; shift 1 ;;
|
--require-inproc) require_inproc=1; shift 1 ;;
|
||||||
--require-spawn) require_spawn=1; shift 1 ;;
|
--require-spawn) require_spawn=1; shift 1 ;;
|
||||||
|
--require-monolith) require_monolith=1; shift 1 ;;
|
||||||
--benchmark) benchmark=1; shift 1 ;;
|
--benchmark) benchmark=1; shift 1 ;;
|
||||||
--benchmark-tail-frames) benchmark_tail_frames="$(next_arg "$@")"; shift 2 ;;
|
--benchmark-tail-frames) benchmark_tail_frames="$(next_arg "$@")"; shift 2 ;;
|
||||||
--benchmark-finish) benchmark_finish="$(next_arg "$@")"; shift 2 ;;
|
--benchmark-finish) benchmark_finish="$(next_arg "$@")"; shift 2 ;;
|
||||||
@@ -371,7 +379,16 @@ collect_role_logs() {
|
|||||||
rm -f "${role_dst}"
|
rm -f "${role_dst}"
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
[ "${wrote_any}" -eq 1 ] || rm -f "${dest}"
|
if [ "${wrote_any}" -eq 1 ]; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
# A library built without the split transports has one role and writes one log, at the base
|
||||||
|
# path itself. Pulled only when no role log arrived, and judged by content as above, so a
|
||||||
|
# replay of such a build leaves its library log in <dest> whether it passed or failed.
|
||||||
|
adb_device_path exec-out run-as "${package_name}" cat "${base_source}" > "${dest}" 2>/dev/null || true
|
||||||
|
if [ ! -s "${dest}" ] || head -1 "${dest}" | grep -q '^cat: .*: No such file'; then
|
||||||
|
rm -f "${dest}"
|
||||||
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
copy_texture_2d_dumps() {
|
copy_texture_2d_dumps() {
|
||||||
@@ -771,6 +788,66 @@ if not passed:
|
|||||||
sys.exit("trace replay failed actual spawn/second-process/zero-Fatal proof: " + json.dumps(proof))
|
sys.exit("trace replay failed actual spawn/second-process/zero-Fatal proof: " + json.dumps(proof))
|
||||||
print("Android retrace: real spawn transport, server role in pid "
|
print("Android retrace: real spawn transport, server role in pid "
|
||||||
f"{proof['server_pid']}, and zero Fatal records confirmed")
|
f"{proof['server_pid']}, and zero Fatal records confirmed")
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
if [ "${require_monolith}" -eq 1 ]; then
|
||||||
|
"${PYTHON}" - "$(host_path_for_adb "${result_dir}/mobilegl.log")" "$(host_path_for_adb "${result_dir}/transport-proof.json")" <<'PY'
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
log_path, proof_path = map(Path, sys.argv[1:])
|
||||||
|
text = log_path.read_text(encoding="utf-8", errors="replace") if log_path.is_file() else ""
|
||||||
|
# A monolith run has no sentence of its own: ConfigLoader logs a transport line only when it
|
||||||
|
# resolves inproc or spawn, and a library built without the split transports has no such
|
||||||
|
# code. So the check fails only on what a split run writes: either transport sentence, an IPC
|
||||||
|
# configuration line (logged for every transport but monolith), a spawned server, or library
|
||||||
|
# lines in a server role's log. The sentence and the IPC line are written during the
|
||||||
|
# configuration load, so a log that shows the load and none of these confirms monolith. A log
|
||||||
|
# that is missing or lacks the load says nothing about the transport: the run passes and is
|
||||||
|
# reported as not confirmed.
|
||||||
|
loaded = "Config loaded" in text
|
||||||
|
markers = {transport: f"Config: MOBILEGL_TRANSPORT={transport} - the MGPipe record stream" in text
|
||||||
|
for transport in ("inproc", "spawn")}
|
||||||
|
configs = re.findall(r"Config: IPC[^\r\n]*", text)
|
||||||
|
armed = re.search(r"spawn ARMED - the server role runs in pid (\d+)", text)
|
||||||
|
server_log = log_path.with_name(log_path.stem + ".server" + log_path.suffix)
|
||||||
|
server_text = server_log.read_text(encoding="utf-8", errors="replace") if server_log.is_file() else ""
|
||||||
|
# Library lines only, known by the "[...thread/LEVEL]: " header each one starts with, so that
|
||||||
|
# what a device answers when the file is not there is never read as a server role.
|
||||||
|
server_lines = len(re.findall(r"^\[[^\r\n]*?/(?:DEBUG|INFO|WARN|ERROR|FATAL)\]: ", server_text, re.M))
|
||||||
|
|
||||||
|
problems = [f"the library log has the {transport} transport marker"
|
||||||
|
for transport, present in markers.items() if present]
|
||||||
|
if configs:
|
||||||
|
problems.append("the library log has an IPC configuration line")
|
||||||
|
if armed:
|
||||||
|
problems.append("the library log names a spawned server process")
|
||||||
|
if server_lines:
|
||||||
|
problems.append("a server role wrote a log of its own")
|
||||||
|
if loaded:
|
||||||
|
unconfirmed = ""
|
||||||
|
elif text.strip():
|
||||||
|
unconfirmed = "the library log does not show the configuration load"
|
||||||
|
else:
|
||||||
|
unconfirmed = "the library log is missing or empty"
|
||||||
|
|
||||||
|
proof = {"required_transport": "monolith", "library_log": str(log_path),
|
||||||
|
"configuration_loaded": loaded, "split_transport_markers": markers,
|
||||||
|
"ipc_config_lines": configs, "spawned_server_pid": int(armed.group(1)) if armed else None,
|
||||||
|
"server_role_log_lines": server_lines, "problems": problems, "passed": not problems,
|
||||||
|
"confirmed": not problems and loaded, "not_confirmed_because": unconfirmed}
|
||||||
|
proof_path.write_text(json.dumps(proof, indent=2), encoding="utf-8")
|
||||||
|
if problems:
|
||||||
|
sys.exit("trace replay failed actual monolith/no-split-transport proof (" + "; ".join(problems) + "): "
|
||||||
|
+ json.dumps(proof))
|
||||||
|
if unconfirmed:
|
||||||
|
print(f"Android retrace: monolith not confirmed ({unconfirmed}): nothing a split transport writes "
|
||||||
|
"was found, so the run is not failed")
|
||||||
|
else:
|
||||||
|
print("Android retrace: monolith confirmed: the library loaded its configuration and logged no inproc "
|
||||||
|
"or spawn transport, no IPC configuration, no spawned server and no server role")
|
||||||
PY
|
PY
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,6 +22,10 @@ Layout under --result-root (the directory the workflow uploads):
|
|||||||
<case>-<backend>/first-attempt/ files of an attempt that was retried
|
<case>-<backend>/first-attempt/ files of an attempt that was retried
|
||||||
shard-status.json every case of the shard with its verdict
|
shard-status.json every case of the shard with its verdict
|
||||||
diagnostics/ emulator logs kept across restarts
|
diagnostics/ emulator logs kept across restarts
|
||||||
|
|
||||||
|
The job log repeats what those files say, since the workflow deletes them after a green run:
|
||||||
|
what each case is run with, the transport lines of its library log, its phase timings and its
|
||||||
|
record, and shard-status.json at the end.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
@@ -63,6 +67,18 @@ BOOT_TIMEOUT_SECONDS = 300
|
|||||||
# cold boot that follows a case that did not pass.
|
# cold boot that follows a case that did not pass.
|
||||||
CASE_OVERHEAD_SECONDS = 180
|
CASE_OVERHEAD_SECONDS = 180
|
||||||
|
|
||||||
|
# What marks the lines of a library log that say which transport ran: the configuration load
|
||||||
|
# every run logs, and what the --require-* checks of trace-replay-ci.sh read. The job log
|
||||||
|
# quotes at most LIBRARY_LOG_LINE_LIMIT of them per case, cut to LIBRARY_LOG_LINE_LENGTH.
|
||||||
|
LIBRARY_LOG_EVIDENCE = ("Config: Active backend type set to", "Config loaded", "MOBILEGL_TRANSPORT",
|
||||||
|
"Config: IPC", "spawn ARMED", "Fatal{")
|
||||||
|
LIBRARY_LOG_LINE_LIMIT = 12
|
||||||
|
LIBRARY_LOG_LINE_LENGTH = 1000
|
||||||
|
|
||||||
|
# What trace-replay-ci.sh reads from its environment and the runner does not set. A replay
|
||||||
|
# inherits the runner's environment, so the job log names the value each one has there.
|
||||||
|
INHERITED_REPLAY_SETTINGS = ("ADB", "PYTHON", "MOBILEGL_RETRACE_USE_PBUFFER", "MOBILEGL_TRACE_ENV")
|
||||||
|
|
||||||
|
|
||||||
def say(message):
|
def say(message):
|
||||||
print(message, flush=True)
|
print(message, flush=True)
|
||||||
@@ -145,7 +161,7 @@ def transport_arguments(transport, hang_budget_ms):
|
|||||||
# required: a spawned server cannot be handed the client's window.
|
# required: a spawned server cannot be handed the client's window.
|
||||||
return ["--env", f"MOBILEGL_TRANSPORT=spawn;{SPLIT_CLIENT_SETTINGS};{hang_budget}",
|
return ["--env", f"MOBILEGL_TRANSPORT=spawn;{SPLIT_CLIENT_SETTINGS};{hang_budget}",
|
||||||
"--use-pbuffer", "--require-spawn"]
|
"--use-pbuffer", "--require-spawn"]
|
||||||
return ["--env", f"MOBILEGL_TRANSPORT=monolith;{hang_budget}"]
|
return ["--env", f"MOBILEGL_TRANSPORT=monolith;{hang_budget}", "--require-monolith"]
|
||||||
|
|
||||||
|
|
||||||
def replay_arguments(options, case, hang_budget_ms):
|
def replay_arguments(options, case, hang_budget_ms):
|
||||||
@@ -251,6 +267,22 @@ class Shard:
|
|||||||
write_json(self.result_dir(case) / "case-status.json", record)
|
write_json(self.result_dir(case) / "case-status.json", record)
|
||||||
self.write_shard_status()
|
self.write_shard_status()
|
||||||
|
|
||||||
|
def plan(self, case):
|
||||||
|
"""What a replay of the case is run with. --dry-run prints it; a run prints it and uses it."""
|
||||||
|
return {"case": case["name"],
|
||||||
|
"environment": replay_environment(self.options.backend, case["name"]),
|
||||||
|
"arguments": replay_arguments(self.options, case, self.hang_budget_ms),
|
||||||
|
"timeout": int(as_text(case["timeout_seconds"])) + REPLAY_GRACE_SECONDS}
|
||||||
|
|
||||||
|
def shared_environment(self):
|
||||||
|
"""What every replay of the shard gets on top of its case's environment."""
|
||||||
|
# The shard installs the APK once per boot.
|
||||||
|
return {"MOBILEGL_TRACE_SKIP_INSTALL": "1", "MOBILEGL_TRACE_POLL_INTERVAL": self.poll_interval}
|
||||||
|
|
||||||
|
def say_shard_status(self):
|
||||||
|
status = read_json(self.result_root / "shard-status.json")
|
||||||
|
say(f"trace-replay-shard: shard-status: {json.dumps(status, sort_keys=True)}")
|
||||||
|
|
||||||
def write_shard_status(self, complete=False):
|
def write_shard_status(self, complete=False):
|
||||||
self.result_root.mkdir(parents=True, exist_ok=True)
|
self.result_root.mkdir(parents=True, exist_ok=True)
|
||||||
records = [self.records.get(case["name"]) or
|
records = [self.records.get(case["name"]) or
|
||||||
@@ -477,20 +509,16 @@ class Shard:
|
|||||||
except OSError:
|
except OSError:
|
||||||
return []
|
return []
|
||||||
|
|
||||||
def run_attempt(self, case):
|
def run_attempt(self, plan):
|
||||||
"""Returns (exit status, seconds, how the shard deadline cut the attempt short or "")."""
|
"""Returns (exit status, seconds, how the shard deadline cut the attempt short or "")."""
|
||||||
case_budget = int(as_text(case["timeout_seconds"])) + REPLAY_GRACE_SECONDS
|
case_budget = plan["timeout"]
|
||||||
budget = case_budget
|
budget = case_budget
|
||||||
remaining = self.remaining_seconds()
|
remaining = self.remaining_seconds()
|
||||||
if remaining is not None and remaining < case_budget:
|
if remaining is not None and remaining < case_budget:
|
||||||
budget = max(remaining, 1.0)
|
budget = max(remaining, 1.0)
|
||||||
environment = replay_environment(self.options.backend, case["name"])
|
|
||||||
# The shard installs the APK once per boot.
|
|
||||||
environment["MOBILEGL_TRACE_SKIP_INSTALL"] = "1"
|
|
||||||
environment["MOBILEGL_TRACE_POLL_INTERVAL"] = self.poll_interval
|
|
||||||
started = time.monotonic()
|
started = time.monotonic()
|
||||||
status, _ = self.run(self.replay_command + replay_arguments(self.options, case, self.hang_budget_ms),
|
status, _ = self.run(self.replay_command + plan["arguments"], timeout=budget,
|
||||||
timeout=budget, extra_env=environment)
|
extra_env={**plan["environment"], **self.shared_environment()})
|
||||||
elapsed = round(time.monotonic() - started, 1)
|
elapsed = round(time.monotonic() - started, 1)
|
||||||
cut = ""
|
cut = ""
|
||||||
if status == TIMEOUT_EXIT_CODE and budget < case_budget:
|
if status == TIMEOUT_EXIT_CODE and budget < case_budget:
|
||||||
@@ -520,11 +548,10 @@ class Shard:
|
|||||||
result = read_json(directory / "result.json")
|
result = read_json(directory / "result.json")
|
||||||
if not isinstance(result, dict) or result.get("passed") is not True:
|
if not isinstance(result, dict) or result.get("passed") is not True:
|
||||||
return FAILED, "the replay exited 0 without a passed result.json"
|
return FAILED, "the replay exited 0 without a passed result.json"
|
||||||
if self.options.transport != "monolith":
|
proof = read_json(directory / "transport-proof.json")
|
||||||
proof = read_json(directory / "transport-proof.json")
|
if not (isinstance(proof, dict) and proof.get("passed") is True and
|
||||||
if not (isinstance(proof, dict) and proof.get("passed") is True and
|
proof.get("required_transport") == self.options.transport):
|
||||||
proof.get("required_transport") == self.options.transport):
|
return FAILED, f"the replay exited 0 without a passed {self.options.transport} transport proof"
|
||||||
return FAILED, f"the replay exited 0 without a passed {self.options.transport} transport proof"
|
|
||||||
return PASSED, ""
|
return PASSED, ""
|
||||||
if status == INFRASTRUCTURE_FAILURE_EXIT_CODE:
|
if status == INFRASTRUCTURE_FAILURE_EXIT_CODE:
|
||||||
return FAILED, "the emulator infrastructure failed again on the retry"
|
return FAILED, "the emulator infrastructure failed again on the retry"
|
||||||
@@ -547,7 +574,9 @@ class Shard:
|
|||||||
_, output = self.adb("logcat", "-d", "-t", "20000", timeout=30)
|
_, output = self.adb("logcat", "-d", "-t", "20000", timeout=30)
|
||||||
logcat.write_text(output, encoding="utf-8", errors="replace")
|
logcat.write_text(output, encoding="utf-8", errors="replace")
|
||||||
app_output = f"/data/user/0/{self.options.package}/files/trace-replay/output"
|
app_output = f"/data/user/0/{self.options.package}/files/trace-replay/output"
|
||||||
for name in ("retrace.log", "mobilegl.client.log", "mobilegl.server.log"):
|
# The two role logs of a split build, and the one log of a build without the split
|
||||||
|
# transports; the device has one kind or the other.
|
||||||
|
for name in ("retrace.log", "mobilegl.client.log", "mobilegl.server.log", "mobilegl.log"):
|
||||||
status, output = self.adb("exec-out", "run-as", self.options.package, "cat",
|
status, output = self.adb("exec-out", "run-as", self.options.package, "cat",
|
||||||
f"{app_output}/{name}", timeout=30)
|
f"{app_output}/{name}", timeout=30)
|
||||||
if status == 0 and output and not output.startswith("cat: "):
|
if status == 0 and output and not output.startswith("cat: "):
|
||||||
@@ -561,6 +590,50 @@ class Shard:
|
|||||||
_, dmesg = self.run(["sudo", "-n", "dmesg", "-T"], timeout=30, capture=True)
|
_, dmesg = self.run(["sudo", "-n", "dmesg", "-T"], timeout=30, capture=True)
|
||||||
(directory / "host-dmesg.txt").write_text("\n".join(dmesg.splitlines()[-300:]) + "\n", encoding="utf-8")
|
(directory / "host-dmesg.txt").write_text("\n".join(dmesg.splitlines()[-300:]) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
def say_library_log(self, case):
|
||||||
|
"""Quote the lines of the case's pulled library log that say which transport ran."""
|
||||||
|
directory = self.result_dir(case)
|
||||||
|
label = f"trace-replay-shard: library-log {case['name']}:"
|
||||||
|
# mobilegl.log is what the proofs of trace-replay-ci.sh read: the client role's log and
|
||||||
|
# then the server role's, or the one log of a build without the split transports. A
|
||||||
|
# split build has only its role logs when the runner pulled them itself.
|
||||||
|
sources = [directory / "mobilegl.log"]
|
||||||
|
if not sources[0].is_file():
|
||||||
|
sources = [directory / "mobilegl.client.log", directory / "mobilegl.server.log"]
|
||||||
|
read = shown = matched = 0
|
||||||
|
for source in sources:
|
||||||
|
try:
|
||||||
|
lines = source.read_text(encoding="utf-8", errors="replace").splitlines()
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
read += 1
|
||||||
|
for line in lines:
|
||||||
|
if any(mark in line for mark in LIBRARY_LOG_EVIDENCE):
|
||||||
|
matched += 1
|
||||||
|
if shown < LIBRARY_LOG_LINE_LIMIT:
|
||||||
|
shown += 1
|
||||||
|
# Escaped, so that no character of a device log can fail the job log's encoding.
|
||||||
|
quoted = line[:LIBRARY_LOG_LINE_LENGTH].encode("ascii", "backslashreplace").decode("ascii")
|
||||||
|
say(f"{label} {source.name}: {quoted}")
|
||||||
|
if not read:
|
||||||
|
say(f"{label} no library log was pulled")
|
||||||
|
elif not matched:
|
||||||
|
say(f"{label} the library log has none of the lines looked for")
|
||||||
|
elif matched > shown:
|
||||||
|
say(f"{label} {matched - shown} more such line(s) are not shown")
|
||||||
|
|
||||||
|
def say_phases(self, case):
|
||||||
|
"""Seconds from each mark of the last attempt's phase-timings.txt to the next, named by the later."""
|
||||||
|
try:
|
||||||
|
text = (self.result_dir(case) / "phase-timings.txt").read_text(encoding="utf-8", errors="replace")
|
||||||
|
except OSError:
|
||||||
|
return
|
||||||
|
marks = [line.split() for line in text.splitlines()]
|
||||||
|
marks = [(mark[0], int(mark[1])) for mark in marks if len(mark) == 2 and mark[1].isdecimal()]
|
||||||
|
phases = {later[0]: round((later[1] - earlier[1]) / 1000, 1) for earlier, later in zip(marks, marks[1:])}
|
||||||
|
if phases:
|
||||||
|
say(f"trace-replay-shard: phases {case['name']}: {json.dumps(phases)}")
|
||||||
|
|
||||||
def run_case(self, index, case):
|
def run_case(self, index, case):
|
||||||
name = case["name"]
|
name = case["name"]
|
||||||
notes = []
|
notes = []
|
||||||
@@ -569,6 +642,8 @@ class Shard:
|
|||||||
|
|
||||||
if self.deadline_passed():
|
if self.deadline_passed():
|
||||||
return self.record_for(case, NOT_RUN, "the shard deadline passed before this case started")
|
return self.record_for(case, NOT_RUN, "the shard deadline passed before this case started")
|
||||||
|
plan = self.plan(case)
|
||||||
|
say(f"trace-replay-shard: run {name}: {json.dumps(plan, sort_keys=True)}")
|
||||||
|
|
||||||
restarted_for = ""
|
restarted_for = ""
|
||||||
if self.restart_each_case and index > 0:
|
if self.restart_each_case and index > 0:
|
||||||
@@ -591,7 +666,7 @@ class Shard:
|
|||||||
reasons_before = len(self.reason_lines())
|
reasons_before = len(self.reason_lines())
|
||||||
exit_codes = []
|
exit_codes = []
|
||||||
replay_seconds = []
|
replay_seconds = []
|
||||||
status, elapsed, cut = self.run_attempt(case)
|
status, elapsed, cut = self.run_attempt(plan)
|
||||||
exit_codes.append(status)
|
exit_codes.append(status)
|
||||||
replay_seconds.append(elapsed)
|
replay_seconds.append(elapsed)
|
||||||
surface_lost = 0
|
surface_lost = 0
|
||||||
@@ -610,7 +685,7 @@ class Shard:
|
|||||||
cut = "the shard deadline passed before the infrastructure retry could start"
|
cut = "the shard deadline passed before the infrastructure retry could start"
|
||||||
else:
|
else:
|
||||||
notes += self.reset_app_state()
|
notes += self.reset_app_state()
|
||||||
status, elapsed, cut = self.run_attempt(case)
|
status, elapsed, cut = self.run_attempt(plan)
|
||||||
exit_codes.append(status)
|
exit_codes.append(status)
|
||||||
replay_seconds.append(elapsed)
|
replay_seconds.append(elapsed)
|
||||||
seconds["replay"] = replay_seconds
|
seconds["replay"] = replay_seconds
|
||||||
@@ -624,6 +699,8 @@ class Shard:
|
|||||||
verdict, reason = self.verdict(case, status)
|
verdict, reason = self.verdict(case, status)
|
||||||
if verdict != PASSED:
|
if verdict != PASSED:
|
||||||
self.collect_failure_diagnostics(case)
|
self.collect_failure_diagnostics(case)
|
||||||
|
self.say_library_log(case)
|
||||||
|
self.say_phases(case)
|
||||||
if not self.aborted:
|
if not self.aborted:
|
||||||
self.remove_case_leftovers(case)
|
self.remove_case_leftovers(case)
|
||||||
seconds["finish"] = round(time.monotonic() - finish_started, 1)
|
seconds["finish"] = round(time.monotonic() - finish_started, 1)
|
||||||
@@ -643,6 +720,9 @@ class Shard:
|
|||||||
self.write_shard_status()
|
self.write_shard_status()
|
||||||
say(f"trace-replay-shard: {len(self.cases)} case(s) on {self.options.backend} / "
|
say(f"trace-replay-shard: {len(self.cases)} case(s) on {self.options.backend} / "
|
||||||
f"{self.options.transport} ({self.options.lane})")
|
f"{self.options.transport} ({self.options.lane})")
|
||||||
|
say("trace-replay-shard: every run: " + json.dumps(
|
||||||
|
{"command": self.replay_command, "environment": self.shared_environment(),
|
||||||
|
"inherited": {name: self.environ.get(name) for name in INHERITED_REPLAY_SETTINGS}}, sort_keys=True))
|
||||||
if not self.prepare_booted_device():
|
if not self.prepare_booted_device():
|
||||||
if not self.restart_emulator("the trace APK did not install"):
|
if not self.restart_emulator("the trace APK did not install"):
|
||||||
self.aborted = "the trace APK could not be installed"
|
self.aborted = "the trace APK could not be installed"
|
||||||
@@ -657,6 +737,8 @@ class Shard:
|
|||||||
record = self.record_for(case, FAILED, f"the shard runner failed: {one_line(error)}")
|
record = self.record_for(case, FAILED, f"the shard runner failed: {one_line(error)}")
|
||||||
self.unclean_case = "" if record["status"] == PASSED else case["name"]
|
self.unclean_case = "" if record["status"] == PASSED else case["name"]
|
||||||
self.save_record(case, record)
|
self.save_record(case, record)
|
||||||
|
say(f"trace-replay-shard: record {case['name']}: {json.dumps(record, sort_keys=True)}")
|
||||||
|
say(f"trace-replay-shard: emulator boots after {case['name']}: {self.boots}")
|
||||||
say(f"trace-replay-shard: {case['name']}: {record['status']}"
|
say(f"trace-replay-shard: {case['name']}: {record['status']}"
|
||||||
+ (f" ({record['reason']})" if record["reason"] else ""))
|
+ (f" ({record['reason']})" if record["reason"] else ""))
|
||||||
say("::endgroup::")
|
say("::endgroup::")
|
||||||
@@ -698,6 +780,7 @@ class Shard:
|
|||||||
if verdicts != len(self.cases):
|
if verdicts != len(self.cases):
|
||||||
say(f"::error title=Android retrace shard incomplete ({label})::"
|
say(f"::error title=Android retrace shard incomplete ({label})::"
|
||||||
f"{verdicts} verdict(s) for {len(self.cases)} case(s)")
|
f"{verdicts} verdict(s) for {len(self.cases)} case(s)")
|
||||||
|
self.say_shard_status()
|
||||||
say(f"trace-replay-shard: {len(self.cases) - len(failed)} passed, {len(failed)} not passed, "
|
say(f"trace-replay-shard: {len(self.cases) - len(failed)} passed, {len(failed)} not passed, "
|
||||||
f"{verdicts} verdict(s) for {len(self.cases)} case(s)")
|
f"{verdicts} verdict(s) for {len(self.cases)} case(s)")
|
||||||
return 0 if not failed and verdicts == len(self.cases) else 1
|
return 0 if not failed and verdicts == len(self.cases) else 1
|
||||||
@@ -721,16 +804,12 @@ class Shard:
|
|||||||
self.install_seconds = previous.get("install_seconds", self.install_seconds)
|
self.install_seconds = previous.get("install_seconds", self.install_seconds)
|
||||||
self.write_shard_status(
|
self.write_shard_status(
|
||||||
complete=all(self.records[case["name"]]["status"] in (PASSED, FAILED) for case in self.cases))
|
complete=all(self.records[case["name"]]["status"] in (PASSED, FAILED) for case in self.cases))
|
||||||
|
self.say_shard_status()
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
def dry_run(self):
|
def dry_run(self):
|
||||||
for case in self.cases:
|
for case in self.cases:
|
||||||
say(json.dumps({
|
say(json.dumps(self.plan(case), sort_keys=True))
|
||||||
"case": case["name"],
|
|
||||||
"environment": replay_environment(self.options.backend, case["name"]),
|
|
||||||
"arguments": replay_arguments(self.options, case, self.hang_budget_ms),
|
|
||||||
"timeout": int(as_text(case["timeout_seconds"])) + REPLAY_GRACE_SECONDS,
|
|
||||||
}, sort_keys=True))
|
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user