docs(perf,benchmark-infra): authorize realtime audio benchmark + advisory CI (SysRS-307..309 / SysDes-156..158 / SRS-216..219 / SAD-088..091 / SDD-120)
Author the full SysRS -> SysDes -> SRS -> SAD -> SDD chain for the benchmark infrastructure authorized by the Option B product decision (Dimensions 1 + 2-advisory; Dimension 3 telemetry export deferred to P1; build-failing hard CI gate deferred until baseline maturity). SysRS v0.9.10 adds: - SysRS-307: maintained numeric performance baselines for the realtime audio path (allocations per callback after warmup, callback wall-clock, Opus encode/decode latency, resampler throughput). - SysRS-308: advisory CI regression reporting on PR + merge to default; non-blocking semantics. - SysRS-309: explicit declared tolerance window. SysDes v0.9.8 adds: - SysDes-156: benchmark coverage allocated to SE-13 (Audio Subsystem). - SysDes-157: CI advisory-reporting integration allocated to SE-18 (Deployment). - SysDes-158: per-metric tolerance table (zero / +20% p95 / +15% mean / -10% throughput). SRS v0.9.9 adds: - SRS-216: realtime audio benchmark instrumentation in chanora_audio/benches/. - SRS-217: baseline storage format (JSON with metric/value/unit/host_arch/toolchain/git_sha/timestamp). - SRS-218: CI advisory workflow with non-blocking semantics. - SRS-219: tolerance window binding + merge-base comparison methodology. SAD v0.9.9 adds: - SAD-088: chanora_audio criterion bench harness (extends SAD-034). - SAD-089: baseline JSON path pinned to crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json. - SAD-090: advisory CI workflow file (.github/workflows/bench-advisory.yml). - SAD-091: manual-trigger baseline-update workflow (sole writer of SAD-089). - Yellow-marker semantics pinned at SAD: 50%-of-tolerance trending detection. SDD v0.9.15 adds: - SDD-120: criterion 0.5 + dhat 0.3 dev-deps; three bench files (realtime_capture, opus_codec, resampler) + common.rs; two post-processor binaries (emit_baseline, compare_baseline); two GitHub Actions workflow YAMLs; simpler-form yellow-marker realization (baseline-only comparator). Out of scope (deferred): - Dimension 3 production telemetry export (P1). - Build-failing hard CI gate (post-baseline-maturity). - Multi-host benchmarking (Linux x86_64 only). - Dart-side flutter_rust_bridge round-trip benchmark. Implementation follows in a separate commit per the no-huge-commit guideline.
This commit is contained in:
@@ -1434,6 +1434,50 @@ A single SRS item may be intentionally allocated to more than one SAD item when
|
||||
- Source SysDes: SysDes-155 (macOS runtime baseline — deployment target, universal-binary packaging, `.framework` `Versions/A` layout, CocoaPods podspec automation)
|
||||
- Verification method: Platform Test (macOS), Release Inspection, Architecture Review
|
||||
|
||||
**SAD-088**: The software architecture shall add a `criterion`-based realtime-audio benchmark harness as a first-class architectural element of the `chanora_audio` crate, extending the existing audio-subsystem allocation in SAD-034. The harness lives in `crates/chanora_audio/benches/` as a Cargo-convention `benches/` directory (a sibling of the crate's `src/`, not a public seam of the crate's library API). The harness shall use the `criterion` crate as its measurement framework (selected at this layer; SDD pins the exact version and feature flags). Rationale for `criterion`: it is the de-facto stable-toolchain Rust microbenchmark harness, produces stable statistical output (mean / p50 / p95 / p99) without requiring a nightly compiler, and integrates with `cargo bench` so the same invocation works in developer shells and on CI runners. The harness shall expose one bench function (or one criterion `BenchmarkGroup`) per SysDes-156 metric family, with the following architectural assignment of metric → bench element: (a) `bench_capture_alloc_count` — counts heap allocations on the steady-state `CaptureState::ingest` path after a warmup phase, using a per-bench allocation-counting mechanism (a dev-dependency allocation tracker such as `dhat`, or a custom global-allocator counter — selection deferred to SDD); (b) `bench_capture_callback_wall_clock` — measures wall-clock duration of `CaptureState::ingest` and reports p50 / p95 / p99 from criterion's sample distribution; (c) `bench_opus_encode_latency` — directly times an Opus encoder over a fixed 960-sample / 20 ms input frame; (d) `bench_opus_decode_latency` — directly times the Opus decode path; if the upstream `tsclientlib` `AudioHandler` is opaque to the crate, the bench shall time the smallest decode wrapper the crate owns and the SDD shall name that wrapper; (e) `bench_resampler_throughput` — measures samples-per-second on the three canonical rate pairs (44.1→48, 16→48, 48→48 kHz). The harness shall emit, for each bench function, the SRS-217 baseline JSON record (one record per metric, in the SRS-217 schema); the emission mechanism (a custom `criterion` measurement reporter, a post-bench JSON-aggregation step, or equivalent) is delegated to SDD. The harness shall not introduce any new public seam on the `chanora_audio` crate's library API surface — benches consume the same internal entry points that the realtime callback consumes, so SAD-034's isolation property is preserved.
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: Software Architecture Item
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: Audio Subsystem (`chanora_audio` crate, `benches/` directory; extends SAD-034)
|
||||
- Source SRS: SRS-216, SRS-217
|
||||
- Source SysDes: SysDes-156 (realtime-audio benchmark surface, SE-13; five metric families; baseline-JSON storage)
|
||||
- Verification method: Inspection (bench-function presence and metric coverage), Bench Run (criterion executes on the host runner), Architecture Review
|
||||
|
||||
**SAD-089**: The software architecture shall pin the baseline-JSON on-disk path for the SAD-088 harness to `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json`, a single deterministic file per host architecture under the crate's `benches/` directory. The filename is keyed on Rust target triple (`x86_64-unknown-linux-gnu`) so future multi-host expansion can add sibling files (`aarch64-apple-darwin.json`, `x86_64-pc-windows-msvc.json`, etc.) without restructuring the directory layout; for the present revision, exactly one such file exists because SysDes-157 fixes the host to Linux x86_64 (`ubuntu-latest`). The file shall be committed to the default branch so that the SAD-090 advisory workflow has a deterministic comparison target at every PR's merge-base commit. The file contents shall conform to the SRS-217 record schema (one record per metric, fields `metric`, `value`, `unit`, `host_arch`, `toolchain`, `git_sha`, `timestamp` at minimum). The file is the sole on-disk output sink of the SAD-088 harness; it is written by exactly one workflow (SAD-091) and read by exactly one workflow (SAD-090) — see SAD-090 / SAD-091 for the workflow isolation property. Multi-host baselines (additional sibling files keyed on alternate target triples) are explicitly out of scope for this revision per SysDes-157 point 2.
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: Software Architecture Item
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: Audio Subsystem (`chanora_audio` crate, `benches/baselines/` subdirectory; output sink of SAD-088)
|
||||
- Source SRS: SRS-217
|
||||
- Source SysDes: SysDes-156 (baseline-JSON storage; deterministic repo location)
|
||||
- Verification method: Inspection (file path, schema), Bench Run (file is regenerable end-to-end)
|
||||
|
||||
**SAD-090**: The software architecture shall add an advisory CI workflow file `.github/workflows/bench-advisory.yml` to the build/release/operations allocation (extending SAD-037), realizing the SRS-218 advisory-reporting workflow. Triggers: `pull_request` against the default branch, plus `push` to the default branch (matching SRS-218 clause 1; tag pushes and non-default branch pushes are excluded). Runner: `ubuntu-latest` (the sole host in scope per SysDes-157 point 2). Step shape (high-level; SDD authors the exact YAML): (1) checkout the PR head; (2) install the Rust toolchain pinned by the repo's `rust-toolchain.toml`; (3) for each bench function defined under SAD-088, invoke `cargo bench` on the `chanora_audio` crate; (4) read the baseline JSON at SAD-089's path *as of the PR's merge-base commit* (using `git show <merge-base>:crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json` or an equivalent git-plumbing read) — never the default-branch-tip baseline, per SRS-219's anti-retroactive-reclassification rule; (5) compare current measurements against that merge-base baseline using the SRS-219 per-metric tolerance window; (6) render the comparison as a markdown table with one row per metric and a 🟢 / 🟡 / 🔴 marker per the SAD-determined semantics (see below); (7) post the table as a PR comment via `actions/github-script@v7` or an equivalent comment-posting action (exact action choice delegated to SDD); (8) the workflow's status check shall always exit `success` (exit code 0) regardless of any 🔴 marker, per SRS-218 clause 4 — regression visibility is delivered through the markdown table, not through the check status. The named status check shall follow existing repo workflow-naming convention (e.g. `bench-advisory`; SDD reads existing workflow files and matches the convention). This workflow shall never write to the baseline JSON file authored under SAD-089 — write authority is exclusively SAD-091's, enforcing the SRS-218 clause 5 workflow-isolation property at the architecture layer. Yellow-marker semantics is pinned at this SAD layer (see §24.1 paragraph below). The workflow is allocated to the CI surface (SE-18) and is the realization of the SE-18 anchor previously named only at the SysDes layer (no prior SAD item exists for the SE-18 surface; this is the first such allocation).
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: Software Architecture Item
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: Build, Release, Operations (`.github/workflows/bench-advisory.yml`; extends SAD-037 — CI surface, SE-18 realization)
|
||||
- Source SRS: SRS-218, SRS-219 (tolerance binding consumed by comparator step)
|
||||
- Source SysDes: SysDes-157 (advisory CI integration), SysDes-158 (tolerance window and merge-base-snapshot comparator)
|
||||
- Verification method: Inspection (workflow trigger / runner / non-blocking exit), CI Dry Run (synthetic PR exercises markdown rendering end-to-end), Architecture Review
|
||||
|
||||
**SAD-091**: The software architecture shall add a separate baseline-update CI workflow file `.github/workflows/bench-baseline-update.yml` to the build/release/operations allocation (extending SAD-037), realizing the SRS-218 clause 5 baseline-update-isolation property. Trigger: `workflow_dispatch` only — the workflow is manually invoked from the GitHub Actions tab on the default branch and shall not fire on `pull_request`, `push`, or any tag event. Runner: `ubuntu-latest` (matching SAD-090's host for byte-comparability of measurements). Step shape: (1) checkout the default branch; (2) install the Rust toolchain pinned by `rust-toolchain.toml`; (3) execute the same `cargo bench` invocations as SAD-090 across the SAD-088 bench functions; (4) write the resulting JSON, conformant to SRS-217's schema, to the SAD-089 path `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json` (overwriting the prior file); (5) commit and push the updated file to the default branch — the commit mechanism (a direct push using a GitHub-bot token, the `peter-evans/create-pull-request` action, or an equivalent push action) is delegated to SDD. This workflow is the **only** writer of the SAD-089 baseline JSON. The SAD-090 PR-triggered workflow is strictly a reader. This single-writer property is the architectural realization of SRS-218 clause 5 and shall be invariant under future revisions (any future revision adding a second writer requires a SAD revision, not an SDD revision).
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: Software Architecture Item
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: Build, Release, Operations (`.github/workflows/bench-baseline-update.yml`; extends SAD-037 — CI surface, SE-18 realization; sole writer of SAD-089)
|
||||
- Source SRS: SRS-218 (workflow-isolation clause)
|
||||
- Source SysDes: SysDes-157 (baseline-update isolation via `workflow_dispatch`)
|
||||
- Verification method: Inspection (trigger is `workflow_dispatch` only; commit destination is SAD-089's path), CI Dry Run (manual invocation regenerates baseline end-to-end), Architecture Review
|
||||
|
||||
### Yellow-marker semantics (pinned at SAD)
|
||||
|
||||
Pinned at SAD layer to remove a downstream ambiguity that would otherwise surface as an open question in the SAD-090 SDD unit: the 🟡 "within tolerance but trending" marker (SRS-218 clause 3, third element of the marker set) shall be defined as follows: a metric is rendered 🟡 if and only if (a) the metric is within the SRS-219 per-metric tolerance window relative to the merge-base baseline (i.e. it is not 🔴), AND (b) the absolute delta between the PR's current measurement and the **previous run of the SAD-090 workflow on the default branch** (i.e. the most recent default-branch CI bench result, not the baseline JSON itself) is greater than 50% of the SRS-219 tolerance window for that metric. Metrics whose SRS-219 tolerance is exactly zero (the heap-allocation-count metric, SRS-219 clause a) cannot be 🟡 because the 50% trigger is undefined when the window has zero width; for those metrics only 🟢 (zero allocations) and 🔴 (any non-zero allocation count) apply. Rationale for pinning at SAD rather than deferring to SDD: the question "is yellow relative to baseline or relative to previous-run-on-default" is an architectural choice about what state the workflow consumes (baseline JSON only, vs baseline JSON + a previous-runs cache), not an implementation detail; pinning it here fixes the workflow's required inputs at the architecture layer so SDD has a fully determined input set. Operationally, the "previous-run-on-default" datum may be sourced from the SAD-091 baseline (treating the committed baseline as the latest default-branch run, which is the simpler and recommended SDD realization) or from a separate run-history artifact; SDD selects which.
|
||||
|
||||
## 27. Updated SRS-to-SAD Coverage Matrix
|
||||
|
||||
| SRS Range | SAD Coverage |
|
||||
@@ -1447,6 +1491,10 @@ A single SRS item may be intentionally allocated to more than one SAD item when
|
||||
| SRS-209 | Covered by `SAD-085` |
|
||||
| SRS-111 (Android foreground voice service) | Covered by `SAD-086` (Android-specific allocation; SAD-036 retains the cross-cutting diagnostics allocation for the SRS-111–124 range per §24.1 dual-allocation pattern) |
|
||||
| SRS-013 (multi-platform support — macOS runtime baseline aspect) | Covered by `SAD-087` (macOS runtime baseline; counterpart to SAD-061 / SAD-063) |
|
||||
| SRS-216 (realtime-audio benchmark instrumentation) | Covered by `SAD-088` (criterion-based bench harness in `chanora_audio/benches/`; extends SAD-034 audio-subsystem allocation). Cross-cutting note: this is a cross-cutting allocation per §24.1 in the sense that the harness sits in the Audio Subsystem (SAD-034 extension) but its CI consumers sit in Build/Release/Operations (SAD-037 extensions); there is no platform-specific variation because SysDes-157 fixes the host to Linux x86_64. |
|
||||
| SRS-217 (baseline-JSON storage format and on-disk path) | Covered by `SAD-088` (the harness emits the JSON) and `SAD-089` (the on-disk path is pinned to `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json`) |
|
||||
| SRS-218 (CI advisory workflow + baseline-update isolation) | Covered by `SAD-090` (`.github/workflows/bench-advisory.yml`, advisory reader, non-blocking exit) and `SAD-091` (`.github/workflows/bench-baseline-update.yml`, `workflow_dispatch`, sole writer of SAD-089); both extend SAD-037 build/release/operations allocation |
|
||||
| SRS-219 (per-metric tolerance window and merge-base-snapshot comparison methodology) | Covered by `SAD-090` (comparator step consumes the SRS-219 tolerance and the merge-base-snapshot rule; tolerance-window evaluation is the workflow's comparator-step responsibility) |
|
||||
|
||||
|
||||
## Baseline Candidate 0.9.1 Update
|
||||
@@ -1497,3 +1545,4 @@ A single SRS item may be intentionally allocated to more than one SAD item when
|
||||
|---|---|---|
|
||||
| 0.9.7 | 2026-05-18 | Formalized the cross-cutting + platform-specific SAD allocation pattern flagged by the traceability-auditor: added §24.1 narrative subsection describing the deliberate dual-allocation rule (cross-cutting concern + platform-specific realization both recorded), with SRS-111 / SAD-036 / SAD-086 as the worked example, and annotated the §24.2 matrix row for SRS-111..124 accordingly. Added SAD-087 (macOS runtime baseline: deployment target isolation, universal-binary `lipo` packaging, `.framework` `Versions/A` layout, hand-rolled CocoaPods podspec automation), parallel to SAD-061 (iOS runtime) and SAD-063 (Android runtime), closing the macOS-runtime anchor gap flagged by the SDD-119 detailed-designer. SAD-087 traces upward through SRS-013 (multi-platform support); a parallel macOS-runtime SysDes derivation does not yet exist and is recorded as an open follow-up for the system-architect (SysDes-133 covers iOS, SysDes-135 covers Android, macOS has no counterpart yet). §27 matrix updated with the SAD-087 row. Strict layered sourcing preserved (`SAD -> SRS` only). |
|
||||
| 0.9.8 | 2026-05-18 | Retargeted SAD-087's `Source SysDes` field from the "open follow-up — no dedicated macOS-runtime SysDes anchor exists" placeholder to `SysDes-155` (macOS runtime baseline — deployment target, universal-binary `lipo` packaging, `.framework` `Versions/A` layout, CocoaPods podspec automation), authored by the system-architect in SysDes v0.9.7 as the parallel macOS-runtime anchor to SysDes-133 (iOS) / SysDes-135 (Android). This closes the open follow-up that SAD-087 carried at its 0.9.7 authoring, giving SAD-087 a proper upward SysDes trace. Surgical single-field edit: no behavioral content of SAD-087 changed; `Source SRS` retained as `SRS-013` (multi-platform support — Windows, macOS, Linux, iOS, Android), which remains the correct SRS anchor (the SRS-013 vs alternate-anchor question raised during SysDes-155 authoring referred to SysRS-013 / SysRS-002 at the system-requirements layer, not to SRS at the software-requirements layer). §27 SRS-to-SAD coverage matrix row for SRS-013 unchanged because it already cites SAD-087 and remains textually accurate. Strict layered sourcing preserved (`SAD -> SRS`, with SysDes cross-reference now resolved). |
|
||||
| 0.9.9 | 2026-05-18 | Added realtime-audio benchmark-infrastructure software-architecture items SAD-088 through SAD-091 sourced from SRS-216 / SRS-217 / SRS-218 / SRS-219 (the Option B benchmark-infrastructure scope decision propagated through the Wave 1.5 benchmark chain SysRS-307/308/309 → SysDes-156/157/158 → SRS-216..219). SAD-088 allocates a `criterion`-based bench harness to the `chanora_audio` crate's `benches/` directory, extending SAD-034 (audio subsystem allocation); the harness covers the five SysDes-156 metric families (heap allocation count, per-callback wall-clock at p50/p95/p99, Opus encode latency on 960-sample/20 ms frame, Opus decode latency on the same shape, resampler throughput at 44.1→48 / 16→48 / 48→48 kHz) and emits SRS-217 baseline records. SAD-089 pins the baseline-JSON on-disk path to `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json` (single deterministic file, single host architecture in scope, target-triple-keyed filename to permit future multi-host expansion without directory restructuring). SAD-090 adds the advisory CI workflow `.github/workflows/bench-advisory.yml` extending SAD-037 (build/release/operations allocation): triggers on PR-against-default + push-to-default, runs on `ubuntu-latest`, reads the merge-base baseline (never the default-tip baseline, per SRS-219 anti-retroactive-reclassification), compares using SRS-219 tolerances, posts a markdown table with 🟢/🟡/🔴 markers, and always exits success (non-blocking per SRS-218 clause 4). SAD-091 adds the separate baseline-update workflow `.github/workflows/bench-baseline-update.yml` extending SAD-037: `workflow_dispatch` trigger only, sole writer of the SAD-089 baseline file, realizing the SRS-218 clause 5 workflow-isolation property at the architecture layer (single-writer invariant). Yellow-marker semantics pinned at this SAD layer in a §26 subsection: 🟡 means "within SRS-219 tolerance relative to merge-base baseline AND delta-from-previous-default-branch-run exceeds 50% of the tolerance window"; metrics with zero-width tolerance (heap allocations) cannot be 🟡. Rationale for pinning at SAD: the choice between baseline-only vs baseline-plus-run-history as workflow input is an architectural input-set decision, not an implementation detail. §27 SRS-to-SAD coverage matrix updated with four new rows (SRS-216, SRS-217, SRS-218, SRS-219) citing SAD-088..091; the SRS-216 row also annotates the cross-cutting allocation pattern per §24.1 (harness sits in Audio Subsystem via SAD-034 extension while CI consumers sit in Build/Release/Operations via SAD-037 extensions, with no platform-specific variation because SysDes-157 fixes the host to Linux x86_64). Intentionally NOT authored consistent with the SysRS/SysDes/SRS deferral chain: (a) Dimension 3 production telemetry export (deferred to P1; SysRS-295 active); (b) build-failing hard CI gate (SysRS-308 advisory-only rule active; SAD-090 status-check always exits success); (c) multi-host benchmarking (SysDes-157 host-scope clause active; SAD-089 holds a single target-triple-keyed file); (d) IDE integration (e.g. `cargo bench` invocation from rust-analyzer in VS Code — tooling polish, not P0 architecture). Strict layered sourcing preserved (`SAD -> SRS` only per the SAD §2 rule, with SysDes cross-references recorded for completeness on each new item). |
|
||||
|
||||
Reference in New Issue
Block a user