edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(perf,benchmark-infra): authorize realtime audio benchmark + advisory CI (SysRS-307..309 / SysDes-156..158 / SRS-216..219 / SAD-088..091 / SDD-120)

Browse Source 
Author the full SysRS -> SysDes -> SRS -> SAD -> SDD chain for the benchmark infrastructure authorized by the Option B product decision (Dimensions 1 + 2-advisory; Dimension 3 telemetry export deferred to P1; build-failing hard CI gate deferred until baseline maturity).

SysRS v0.9.10 adds:
- SysRS-307: maintained numeric performance baselines for the realtime audio path (allocations per callback after warmup, callback wall-clock, Opus encode/decode latency, resampler throughput).
- SysRS-308: advisory CI regression reporting on PR + merge to default; non-blocking semantics.
- SysRS-309: explicit declared tolerance window.

SysDes v0.9.8 adds:
- SysDes-156: benchmark coverage allocated to SE-13 (Audio Subsystem).
- SysDes-157: CI advisory-reporting integration allocated to SE-18 (Deployment).
- SysDes-158: per-metric tolerance table (zero / +20% p95 / +15% mean / -10% throughput).

SRS v0.9.9 adds:
- SRS-216: realtime audio benchmark instrumentation in chanora_audio/benches/.
- SRS-217: baseline storage format (JSON with metric/value/unit/host_arch/toolchain/git_sha/timestamp).
- SRS-218: CI advisory workflow with non-blocking semantics.
- SRS-219: tolerance window binding + merge-base comparison methodology.

SAD v0.9.9 adds:
- SAD-088: chanora_audio criterion bench harness (extends SAD-034).
- SAD-089: baseline JSON path pinned to crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json.
- SAD-090: advisory CI workflow file (.github/workflows/bench-advisory.yml).
- SAD-091: manual-trigger baseline-update workflow (sole writer of SAD-089).
- Yellow-marker semantics pinned at SAD: 50%-of-tolerance trending detection.

SDD v0.9.15 adds:
- SDD-120: criterion 0.5 + dhat 0.3 dev-deps; three bench files (realtime_capture, opus_codec, resampler) + common.rs; two post-processor binaries (emit_baseline, compare_baseline); two GitHub Actions workflow YAMLs; simpler-form yellow-marker realization (baseline-only comparator).

Out of scope (deferred):
- Dimension 3 production telemetry export (P1).
- Build-failing hard CI gate (post-baseline-maturity).
- Multi-host benchmarking (Linux x86_64 only).
- Dart-side flutter_rust_bridge round-trip benchmark.

Implementation follows in a separate commit per the no-huge-commit guideline.
This commit is contained in:
EdisonJwa
2026-05-18 13:36:01 +08:00
parent d13b56d379
commit 575a6cbc5c
5 changed files with 309 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/sysrs.md
+23 -1
View File
@@ -2,7 +2,7 @@
**Product name:** Chanora
**Document type:** SysRS / System Requirements Specification
**Version:** 0.9.2
**Version:** 0.9.10
**Status:** Baseline Candidate
**Product category:** Cross-platform voice client application
**Architecture:** Flutter + Rust Core
@@ -1944,6 +1944,21 @@ This section converts the baseline product decisions into auditable system-level
- Priority: P0
- Verification: Platform Test (Android), Functional Test
**SysRS-307**: The Chanora project shall maintain measured numeric performance baselines for the realtime audio capture and playback paths. The baseline set shall include, at minimum: (i) heap allocation count per realtime audio callback measured after warmup, (ii) per-callback wall-clock time expressed as a fraction of (or absolute bound relative to) the audio frame period, (iii) Opus encode latency and Opus decode latency per frame, and (iv) resampler throughput at the common rate-pair conversions exercised by the audio pipeline. Each baseline shall be expressed as a numeric threshold (or numeric range), not as free-form prose, so that regression against the baseline is deterministically detectable by automated comparison. This requirement extends the prescriptive intent of SysRS-180 ("shall minimize local audio pipeline latency") and the directional target of SysRS-181 ("should target local audio pipeline latency under 100 ms") into a measurable, contract-grade obligation, and it is consistent with the underrun-avoidance obligation in SysRS-186. The specific baseline values, the warmup definition, the rate-pair set, and the storage format for the baselines are SysDes/SAD/SDD concerns and are not authored here. This clause does not authorize automatic upload, transmission, or off-device export of any measured baseline data and is therefore consistent with SysRS-295 (no automatic telemetry / diagnostic upload in MVP).
- Priority: P0
- Verification: Test (SWE.4 benchmark assertions, delegated to verification layer; cross-references SysRS-236)
**SysRS-308**: The Chanora project's continuous-integration workflow shall execute the realtime-audio benchmark suite on every pull request against the default branch and on every merge to the default branch, on at least one host architecture (Linux x86_64 on the existing GitHub Actions runner is sufficient to satisfy this clause), and shall report the comparison of the executed run against the maintained baselines (SysRS-307) in the pull request's status-check surface such that human reviewers can see, before approving the change, whether any baseline has regressed beyond the declared tolerance window (SysRS-309). The CI regression check authorized by this clause is **advisory only**: it shall NOT fail the CI build, shall NOT block merge, and shall NOT be treated as a hard quality gate at the P0-MVP stage. Its purpose is to surface evidence for human reviewer judgement and to avoid the failure mode in which a strict build-failing gate is bypassed under release-crunch pressure (e.g., "skip CI"). Escalation of this advisory tier to a build-failing hard gate is explicitly out of scope of SysRS-308 and shall be authorized only by a separate, later SysRS clause after a period of baseline maturity sufficient to establish that the advisory signal is stable and low-noise (indicative target: 4–6 weeks of clean baseline data on the default branch, but the actual escalation criteria are to be set by the future clause). This requirement is consistent with the CI-as-quality-evidence pattern established by the SysRS-234..239 verification family and does not displace any obligation in that family. This clause does not authorize off-device transmission of baseline measurements beyond the project's existing CI provider surface (i.e., GitHub Actions logs and PR status checks visible to repository collaborators); it is therefore consistent with SysRS-295.
- Priority: P0
- Verification: Demo (CI workflow exercised on a representative PR; advisory status check appears and does not block merge on regression)
**SysRS-309**: The advisory CI regression comparison authorized by SysRS-308 shall use an explicitly declared tolerance window — a numeric percentage (or a numeric per-metric percentage set) above the maintained baseline (SysRS-307) — beyond which the advisory check shall mark the run as a regression. The tolerance shall be a single declared value (or one declared value per metric), not an ad-hoc reviewer judgement, so that the advisory signal is reproducible. The specific numeric value(s) of the tolerance window and the comparison methodology (for example, "comparison against the most recent baseline snapshot on the default branch") shall be authored at the SysDes layer and may be refined at SAD/SDD; a suggested starting value of +20% over baseline is recorded here for downstream traceability but is not binding at the SysRS layer. Cross-references: SysRS-307 (the baselines being compared against), SysRS-308 (the advisory CI surface in which the comparison runs), SysRS-180/SysRS-181 (latency intent the tolerance must not silently erode), SysRS-295 (the tolerance window value is a project-local configuration and does not authorize off-device telemetry).
- Priority: P0
- Verification: Review (SysDes/SAD declaration of tolerance value is present and is referenced by the CI workflow definition)
## 32. Change History Addendum
@@ -1977,3 +1992,10 @@ This section converts the baseline product decisions into auditable system-level
| Version | Date | Description |
|---|---|---|
| 0.9.9 | 2026-05-17 | Reconciled SysRS-288 with DEC-004 (Accepted 2026-05-14): Android minimum runtime baseline raised from API 24 to API 28 (Android 9.0); rationale and decision citation added in-line. Verified SysRS-055, SysRS-161, SysRS-195, SysRS-217, and SysRS-289 remain consistent with DEC-004 (no API-level text in any of these; no rewrite required). Added SysRS-305 (Android in-call audio mode engagement during active voice session — back-fills the v0.9.8 product-decision-register `AudioManager.setMode(MODE_IN_COMMUNICATION)` entry) and SysRS-306 (explicit Android runtime microphone permission acquisition at or before voice session activation, additive to SysRS-055). |
## Baseline Candidate 0.9.10 Update
| Version | Date | Description |
|---|---|---|
| 0.9.10 | 2026-05-18 | Authorized Option B of the benchmark-infrastructure decision for the realtime audio path. Added SysRS-307 (maintained numeric performance baselines for the realtime audio capture and playback paths — heap allocation count per callback after warmup, per-callback wall-clock budget relative to the audio frame period, Opus encode/decode latency, and resampler throughput at common rate-pair conversions; extends SysRS-180/SysRS-181 from prescriptive intent into a measurable contract; consistent with SysRS-186 and SysRS-236). Added SysRS-308 (advisory CI regression reporting executing the benchmark suite on every PR and every merge to the default branch on at least one host architecture, surfacing results in the PR status-check view; explicitly **advisory only** — does not fail the build, does not block merge; escalation to a build-failing hard gate is out of scope and deferred to a future SysRS clause after baseline maturity; consistent with the SysRS-234..239 verification-family pattern). Added SysRS-309 (explicitly declared numeric tolerance window for the advisory comparison; numeric value(s) and comparison methodology delegated to SysDes/SAD; suggested starting value +20% recorded as non-binding downstream guidance). All three new clauses are consistent with SysRS-295 (no automatic telemetry / diagnostic upload in MVP) — they neither authorize nor require off-device transmission of measurement data beyond the existing CI provider surface visible to repository collaborators. Explicitly **NOT** authored in this update: (a) Dimension 3 production telemetry export of timing histograms (deferred to P1; any future opt-in performance-evidence export through the user-initiated diagnostic-export path requires a separate P1 SysRS clause and must be reconciled with SysRS-295 at that time); (b) build-failing hard CI gate (deferred to a future SysRS clause). |