docs(perf,benchmark-infra): authorize realtime audio benchmark + advisory CI (SysRS-307..309 / SysDes-156..158 / SRS-216..219 / SAD-088..091 / SDD-120)
Author the full SysRS -> SysDes -> SRS -> SAD -> SDD chain for the benchmark infrastructure authorized by the Option B product decision (Dimensions 1 + 2-advisory; Dimension 3 telemetry export deferred to P1; build-failing hard CI gate deferred until baseline maturity). SysRS v0.9.10 adds: - SysRS-307: maintained numeric performance baselines for the realtime audio path (allocations per callback after warmup, callback wall-clock, Opus encode/decode latency, resampler throughput). - SysRS-308: advisory CI regression reporting on PR + merge to default; non-blocking semantics. - SysRS-309: explicit declared tolerance window. SysDes v0.9.8 adds: - SysDes-156: benchmark coverage allocated to SE-13 (Audio Subsystem). - SysDes-157: CI advisory-reporting integration allocated to SE-18 (Deployment). - SysDes-158: per-metric tolerance table (zero / +20% p95 / +15% mean / -10% throughput). SRS v0.9.9 adds: - SRS-216: realtime audio benchmark instrumentation in chanora_audio/benches/. - SRS-217: baseline storage format (JSON with metric/value/unit/host_arch/toolchain/git_sha/timestamp). - SRS-218: CI advisory workflow with non-blocking semantics. - SRS-219: tolerance window binding + merge-base comparison methodology. SAD v0.9.9 adds: - SAD-088: chanora_audio criterion bench harness (extends SAD-034). - SAD-089: baseline JSON path pinned to crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json. - SAD-090: advisory CI workflow file (.github/workflows/bench-advisory.yml). - SAD-091: manual-trigger baseline-update workflow (sole writer of SAD-089). - Yellow-marker semantics pinned at SAD: 50%-of-tolerance trending detection. SDD v0.9.15 adds: - SDD-120: criterion 0.5 + dhat 0.3 dev-deps; three bench files (realtime_capture, opus_codec, resampler) + common.rs; two post-processor binaries (emit_baseline, compare_baseline); two GitHub Actions workflow YAMLs; simpler-form yellow-marker realization (baseline-only comparator). Out of scope (deferred): - Dimension 3 production telemetry export (P1). - Build-failing hard CI gate (post-baseline-maturity). - Multi-host benchmarking (Linux x86_64 only). - Dart-side flutter_rust_bridge round-trip benchmark. Implementation follows in a separate commit per the no-huge-commit guideline.
This commit is contained in:
@@ -3,7 +3,7 @@
|
||||
**Product name:** Chanora
|
||||
**Document type:** SysDes / System Architectural Design
|
||||
**Process alignment:** Automotive SPICE SYS.3 System Architectural Design
|
||||
**Version:** 0.9.7
|
||||
**Version:** 0.9.8
|
||||
**Status:** Baseline Candidate
|
||||
**Input baseline:** Chanora SysRS v0.6
|
||||
**Target application type:** Cross-platform voice client application system
|
||||
@@ -2940,9 +2940,58 @@ This SysDes version covers all known SysRS requirements from `SysRS-001` through
|
||||
- Allocated SysRS: SysRS-002, SysRS-050, SysRS-193
|
||||
- Notes: Parallel in role to SysDes-133 (iOS runtime baseline) and SysDes-135 (Android runtime baseline); neither SysDes-133 nor SysDes-135 is modified by this item. Source SysRS selection: SysRS-002 anchors the multi-platform obligation that explicitly enumerates macOS as a target client platform, SysRS-050 anchors the macOS runtime environment obligation for native desktop integration, and SysRS-193 anchors the signed/notarized macOS build obligation that the `.framework` `Versions/A` layout and universal-binary policy must satisfy in the release pipeline. No macOS-specific deployment-target or universal-binary SysRS item currently exists; if a finer-grained macOS minimum-runtime SysRS item is required (analogous to SysRS-286 for iOS and SysRS-288 for Android), the systems-requirements owner should consider authoring it in a follow-up — this SysDes item is structured so that such a future SysRS item can be added to `Allocated SysRS` without restructuring the element. SysDes-155 cites the podspec file path only and does not embed its current deployment-target value; the value itself remains owned by the Build Configuration subsystem.
|
||||
|
||||
**SysDes-156**: The system architecture shall allocate to SE-13 (Audio Subsystem) the obligation to expose its realtime capture and playback paths to benchmark instrumentation, such that the maintained numeric performance baselines authorized by SysRS-307 are measurable as a deterministic, automated comparison surface. The metric set authored at this layer is: (i) **heap allocation count per realtime callback after warmup** — zero allocations are expected on the steady-state realtime audio thread, where warmup is defined as the first N callbacks with N implementation-specific (recommended N=100 callbacks; refined at SDD); (ii) **per-callback wall-clock time expressed as a fraction of the cpal stream's reported audio frame period**, captured and reported as p50, p95, and p99 of the callback wall-clock as a fraction of that period; (iii) **Opus encode latency** — end-to-end wall-clock time for `encoder.encode_float()` on a 960-sample (20 ms) frame, captured per call; (iv) **Opus decode latency** — same shape, decoder side; (v) **resampler throughput** in samples-per-second produced at the canonical rate pairs 44.1 kHz → 48 kHz, 16 kHz → 48 kHz, and 48 kHz → 48 kHz passthrough (the passthrough pair serves as a control point). The baseline storage format is declared at this layer as structured JSON committed to a deterministic location in the repository; the exact path is delegated to SDD. Each JSON record shall include the metric value, the host architecture, the toolchain version, the git SHA of the measurement, and a timestamp, so that a baseline snapshot is reproducible and traceable to the commit that produced it. This SysDes item does not authorize off-device transmission of any measured baseline data and is consistent with SysRS-295.
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: System Element Allocation / Performance Verification Surface
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: SE-13 (Audio Subsystem) — owns the realtime path being measured; the realtime callback, the Opus encode/decode paths, and the resampler are all SE-13 surfaces
|
||||
- Downstream artifact: SRS, SAD, SDD, Verification
|
||||
- Verification method: Automated Benchmark Execution (criterion crate per SDD); Inspection of the JSON baseline records committed to the default branch
|
||||
- Verification owner: Audio / Platform QA
|
||||
- ASPICE SYS.3 alignment: Element allocation, dynamic architecture, resource and performance design, verification handoff
|
||||
- Allocated SysRS: SysRS-307 (primary derivation); refines SysRS-180 / SysRS-181 / SysRS-186 from prescriptive intent into a measurable contract surface
|
||||
- Notes: The numeric warmup constant (recommended N=100), the exact JSON repository path, the benchmark crate selection (recommendation: criterion), and the harness wiring against the cpal stream are SDD concerns and are not authored here. The cpal stream's reported period referenced in metric (ii) is the stream period reported by the existing audio capture/playback path owned by SE-13; this SysDes item does not authorize a new platform integration. The control-point passthrough at 48 kHz → 48 kHz in metric (v) provides a zero-arithmetic baseline that isolates harness overhead from resampling cost. Metrics (i) and (v) are the structural-property metrics (no acceptable variance for allocations; throughput is a steady-state measurement); metrics (ii)–(iv) are latency-distribution metrics whose tolerance characterisation is the subject of SysDes-158.
|
||||
|
||||
**SysDes-157**: The system architecture shall allocate to SE-18 (Deployment and Release Environment) the integration of the realtime-audio benchmark suite (SysDes-156) into the existing CI workflow as an **advisory** regression-reporting surface, satisfying SysRS-308. The design constraints authored at this layer are: (1) **Trigger** — the benchmark workflow shall execute on every pull request opened against the default branch and on every merge to the default branch; it shall not be triggered on tags and shall not be triggered on non-default-branch pushes. (2) **Host scope** — the benchmark workflow shall execute on the Linux x86_64 GitHub Actions runner (`ubuntu-latest`); other host architectures are explicitly out of scope for this revision and shall not be added by SDD or by implementation without a future SysRS clause authorising multi-host benchmarking. (3) **Surface** — the result shall be rendered on the PR status-check view as a named check (suggested name "Benchmark / advisory" or equivalent; the exact check name is delegated to SDD) and shall additionally be rendered as a markdown table posted to the PR conversation, comparing each metric against the most recent baseline on the default branch with the percentage delta and a clear visual marker (e.g., 🟢 within tolerance, 🟡 within tolerance but trending, 🔴 outside tolerance). (4) **Non-blocking semantics** — the status check shall **never** report a `failure` status; on regression the check shall report `success` (so it does not block merge) and the regression shall be surfaced exclusively in the PR comment for human reviewer attention. This clause is the SysDes-layer enforcement of SysRS-308's advisory-only contract; any future escalation to a build-failing hard gate requires a separate SysRS authorisation and is out of scope of SysDes-157. (5) **Baseline source** — the comparison input shall be read from the deterministic JSON location on the default branch committed under SysDes-156; on the first run after a new benchmark is added, the baseline file may not yet contain that metric, in which case the report shall state "no baseline; this run establishes the baseline candidate" and shall not produce a comparison delta for that metric. (6) **Baseline update isolation** — a separate, manually-invoked CI workflow (GitHub Actions `workflow_dispatch`) shall be the sole mechanism that writes the baseline JSON on the default branch; the PR-triggered benchmark workflow shall never write the baseline file. This isolation guarantees that baseline updates are intentional, reviewable acts and prevents silent baseline drift from PR runs.
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: System Element Allocation / CI Integration Architecture
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: SE-18 (Deployment and Release Environment) — owns CI/CD per SysDes-036; the CI workflow definitions, the GitHub Actions runner selection, the PR status-check surface, the PR comment rendering, and the baseline-update workflow all live here
|
||||
- Downstream artifact: SRS, SAD, SDD, Verification
|
||||
- Verification method: Demo (a CI run on a synthetic PR demonstrates the markdown table report, the visual markers, the non-blocking `success` status on a simulated regression, and the "no baseline" fallback on a metric with no prior baseline)
|
||||
- Verification owner: Release / Operations QA, Audio / Platform QA (co-verification of the report content)
|
||||
- ASPICE SYS.3 alignment: Element allocation, deployment architecture, verification handoff
|
||||
- Allocated SysRS: SysRS-308 (primary derivation); composes with SysRS-307 (baselines being compared against) and SysRS-309 (tolerance window applied by the comparison, refined by SysDes-158); consistent with SysRS-295 (no off-device transmission beyond the existing CI provider surface visible to repository collaborators) and with the SysRS-234..239 verification-family pattern
|
||||
- Notes: The exact YAML workflow filenames, the exact named status-check string, the exact markdown layout of the comparison table, the exact emoji set, and the exact triggering criteria for the 🟡 "trending" state are SDD concerns and are not authored here. The merge-base baseline-snapshot semantics of the comparison are authored separately in SysDes-158. The Linux x86_64 host-scope clause is the active rule that explicitly defers ARM Android, macOS Apple Silicon, Windows x86_64, and any other host runner; that deferral is preserved here intentionally. Composes with SysDes-036 (CI/CD as a system element) as the SE-18 integration anchor.
|
||||
|
||||
**SysDes-158**: The system architecture shall allocate to SE-18 (Deployment and Release Environment) the per-metric tolerance window and the baseline-comparison methodology used by the advisory CI surface (SysDes-157), satisfying SysRS-309. The tolerance window is authored at this layer as a per-metric set rather than as a single global value, because the metrics defined in SysDes-156 have different statistical character and require different comparison rules:
|
||||
|
||||
| Metric (per SysDes-156) | Tolerance window | Comparison rule | Rationale |
|
||||
|---|---|---|---|
|
||||
| Heap allocation count per realtime callback after warmup | **Zero** | Any non-zero allocation count is reported as a regression (🔴) | Structural property of the code, not a statistical measurement; no acceptable variance |
|
||||
| Per-callback wall-clock as a fraction of the audio frame period | **+20% on p95** | Compare current-run p95 against baseline p95; report regression if delta exceeds +20%. p50 and p99 are reported but not compared against tolerance (advisory data only) | p95 is the operational tail; p50 is too lenient and p99 is too noisy to gate on at MVP |
|
||||
| Opus encode latency (960-sample / 20 ms frame) | **+15% on mean** | Compare current-run mean against baseline mean; report regression if delta exceeds +15% | Codec encode latency is a tight loop with low variance; mean is a stable comparator |
|
||||
| Opus decode latency (same frame shape) | **+15% on mean** | Same shape as encode | Symmetric with encode |
|
||||
| Resampler throughput (samples/second) at 44.1→48, 16→48, 48→48 passthrough | **−10% on samples/second** (lower is worse) | Compare current-run samples/sec against baseline samples/sec; report regression if delta is below −10% | Throughput regression is a *drop* in samples/sec, not an *increase*; sign convention is inverted relative to the latency metrics |
|
||||
|
||||
The **comparison methodology** authored at this layer is: each PR-triggered benchmark run shall be compared against the most recent baseline committed to the default branch *at the time of the PR's merge-base commit*, not against the current tip of the default branch. This ensures PR comparisons are stable as the default branch advances during the PR's lifetime, and it also means that rebasing a PR onto a newer default-branch base can change which baseline snapshot is used as the comparator. The +20% starting value recorded as non-binding guidance in SysRS-309 is honoured here only for the per-callback wall-clock metric (and is bound to p95); the other metric tolerances are authored independently of the SysRS-309 suggestion, as authorised by SysRS-309's "single declared value (or one declared value per metric)" allowance.
|
||||
|
||||
- Status: Baseline Candidate
|
||||
- Type: System Element Allocation / Performance Comparison Methodology
|
||||
- Stage: P0 / MVP
|
||||
- Allocated to: SE-18 (Deployment and Release Environment) — the comparison logic lives in CI tooling co-located with the workflow authored in SysDes-157
|
||||
- Downstream artifact: SRS, SAD, SDD, Verification
|
||||
- Verification method: Review (the per-metric tolerance values are subject to periodic review; SDD ratifies the values as authored here and may refine the harness implementation; future SAD/SDD revisions may re-tune the values as baseline noise characteristics are observed empirically)
|
||||
- Verification owner: Release / Operations QA, Audio / Platform QA
|
||||
- ASPICE SYS.3 alignment: Element allocation, architecture decision / rationale, resource and performance design, verification handoff
|
||||
- Allocated SysRS: SysRS-309 (primary derivation); composes with SysRS-307 (the baselines being compared) and SysRS-308 (the advisory surface that consumes this comparison)
|
||||
- Notes: The selection of p95 (rather than p99) as the gated tail for the wall-clock metric is an explicit architectural decision recorded here: p99 is reported for diagnostic value but is intentionally not tolerance-gated at MVP because the sample population of a single CI run is too small for p99 to be a stable comparator. The merge-base baseline-snapshot rule is the active comparison methodology; an alternative methodology comparing against the latest default-branch baseline (rather than the merge-base baseline) was considered and intentionally rejected because it produces unstable PR comparisons when the default branch advances during the PR's lifetime. Re-tuning of the per-metric tolerance values may be performed by future SAD or SDD revisions without requiring a new SysRS clause, provided the values remain numeric and per-metric (the structural constraints authored here); a change from the per-metric model to a global single value, or vice versa, would require a SysRS-309 revision.
|
||||
|
||||
## 17. Updated SysRS Coverage Statement
|
||||
|
||||
This SysDes version covers all known SysRS requirements from `SysRS-001` through `SysRS-306`.
|
||||
This SysDes version covers all known SysRS requirements from `SysRS-001` through `SysRS-309`.
|
||||
|
||||
| SysRS Range | SysDes Coverage |
|
||||
|---|---|
|
||||
@@ -2952,6 +3001,7 @@ This SysDes version covers all known SysRS requirements from `SysRS-001` through
|
||||
| SysRS-303 through SysRS-304 | Covered by `SysDes-149` through `SysDes-151` |
|
||||
| SysRS-305 through SysRS-306 | Covered by `SysDes-152` through `SysDes-154` (with SysDes-154 additionally refining SysRS-055 for the Android voice audio backend) |
|
||||
| SysRS-002 / SysRS-050 / SysRS-193 (macOS platform-baseline slice) | Additionally refined by `SysDes-155` (macOS runtime baseline, parallel to SysDes-133 / SysDes-135) |
|
||||
| SysRS-307 through SysRS-309 (realtime-audio benchmark infrastructure) | Covered by `SysDes-156` (audio benchmark surface, SE-13), `SysDes-157` (advisory CI integration, SE-18), and `SysDes-158` (per-metric tolerance window and comparison methodology, SE-18) |
|
||||
|
||||
|
||||
## Baseline Candidate 0.9.1 Update
|
||||
@@ -2993,4 +3043,5 @@ This SysDes version covers all known SysRS requirements from `SysRS-001` through
|
||||
|
||||
| Version | Date | Description |
|
||||
|---|---|---|
|
||||
| 0.9.8 | 2026-05-18 | Added benchmark-infrastructure SysDes derivations SysDes-156, SysDes-157, and SysDes-158 sourced from SysRS-307 / SysRS-308 / SysRS-309 (Wave 1.5 benchmark chain, Step 2). SysDes-156 allocates the realtime-audio benchmark instrumentation surface (heap allocation count per realtime callback after warmup, per-callback wall-clock as a fraction of the cpal stream period at p50/p95/p99, Opus encode latency on a 960-sample / 20 ms frame, Opus decode latency on the same shape, and resampler throughput at 44.1→48 kHz, 16→48 kHz, and 48→48 kHz passthrough as a control) to SE-13 (Audio Subsystem); declares baseline storage as structured JSON committed to a deterministic repo location (exact path delegated to SDD) capturing metric value, host architecture, toolchain version, git SHA, and timestamp. SysDes-157 allocates the advisory CI integration to SE-18 (Deployment and Release Environment): trigger on every PR against the default branch + every merge to default (not on tags, not on non-default-branch pushes), host = Linux x86_64 (`ubuntu-latest`), surface = PR status-check view rendered as a markdown table with per-metric delta and a visual marker (🟢 within tolerance, 🟡 within tolerance but trending, 🔴 outside tolerance), strict non-blocking semantics (the check shall never report `failure` status — even on regression it reports `success` and surfaces the regression in the PR comment for human reviewer attention), baseline source = JSON on the default branch with first-run-establishes-candidate fallback, and baseline-update isolation via a separate `workflow_dispatch` workflow (PR-triggered workflow shall never write the baseline file). SysDes-158 authors the per-metric tolerance window and the comparison methodology: heap allocations tolerance = zero (any non-zero count is 🔴), per-callback wall-clock tolerance = +20% on p95 (p50 and p99 are advisory data only), Opus encode/decode latency tolerance = +15% on mean, resampler throughput tolerance = −10% on samples/second; comparison is against the most recent baseline committed to the default branch at the PR's merge-base. Allocated to SE-18 (comparison logic lives in CI tooling). Strict layered sourcing preserved (`SysDes -> SysRS` only). Explicitly **NOT** authored in this update, consistent with the SysRS-307/308/309 deferral: (a) Dimension 3 production telemetry export (deferred to P1; SysRS-307/308/309 do not authorize off-device transmission and SysRS-295 still applies); (b) build-failing hard CI gate (SysRS-308's advisory-only clause is the active rule — SysDes-157's non-blocking semantics enforces it at this layer); (c) multi-host benchmarking (ARM Android, macOS Apple Silicon, etc. — SysDes-157's Linux x86_64 host-scope clause is the active rule). |
|
||||
| 0.9.7 | 2026-05-18 | Added dedicated macOS runtime baseline element allocation SysDes-155, parallel in role to SysDes-133 (iOS runtime baseline) and SysDes-135 (Android runtime baseline). SysDes-155 anchors the macOS deployment-target policy as a single-source-of-truth concern (citing the podspec file path `apps/chanora_flutter/macos/chanora_bridge.podspec` without embedding its value), the universal-binary (`lipo`'d `arm64` + `x86_64`) packaging policy, the `.framework` `Versions/A` bundle layout convention, and CocoaPods podspec integration as the auto-build mechanism for the Rust cdylib. Allocated to macOS Build Configuration, Platform Adapter Layer (macOS Platform Services), Release Pipeline, and Verification. Source SysRS: SysRS-002 (multi-platform target client platforms including macOS), SysRS-050 (macOS runtime environment / native desktop window integration), SysRS-193 (signed/notarized macOS builds). SysDes-155 is the proper SysDes anchor for SAD-087 and for downstream SRS items that allocate macOS-runtime responsibility, and is the recommended retarget for SAD-087's previously-open Source SysDes follow-up. SysDes-133 and SysDes-135 are unchanged. Strict layered sourcing preserved (`SysDes -> SysRS` only). Follow-up recommendation: systems-requirements owner may consider authoring a finer-grained macOS minimum-runtime SysRS item parallel to SysRS-286 (iOS) and SysRS-288 (Android); SysDes-155 is structured so that such a future SysRS item can be added to its `Allocated SysRS` list without restructuring the element. |
|
||||
|
||||
Reference in New Issue
Block a user