docs(perf,benchmark-infra): authorize realtime audio benchmark + advisory CI (SysRS-307..309 / SysDes-156..158 / SRS-216..219 / SAD-088..091 / SDD-120)
Author the full SysRS -> SysDes -> SRS -> SAD -> SDD chain for the benchmark infrastructure authorized by the Option B product decision (Dimensions 1 + 2-advisory; Dimension 3 telemetry export deferred to P1; build-failing hard CI gate deferred until baseline maturity). SysRS v0.9.10 adds: - SysRS-307: maintained numeric performance baselines for the realtime audio path (allocations per callback after warmup, callback wall-clock, Opus encode/decode latency, resampler throughput). - SysRS-308: advisory CI regression reporting on PR + merge to default; non-blocking semantics. - SysRS-309: explicit declared tolerance window. SysDes v0.9.8 adds: - SysDes-156: benchmark coverage allocated to SE-13 (Audio Subsystem). - SysDes-157: CI advisory-reporting integration allocated to SE-18 (Deployment). - SysDes-158: per-metric tolerance table (zero / +20% p95 / +15% mean / -10% throughput). SRS v0.9.9 adds: - SRS-216: realtime audio benchmark instrumentation in chanora_audio/benches/. - SRS-217: baseline storage format (JSON with metric/value/unit/host_arch/toolchain/git_sha/timestamp). - SRS-218: CI advisory workflow with non-blocking semantics. - SRS-219: tolerance window binding + merge-base comparison methodology. SAD v0.9.9 adds: - SAD-088: chanora_audio criterion bench harness (extends SAD-034). - SAD-089: baseline JSON path pinned to crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json. - SAD-090: advisory CI workflow file (.github/workflows/bench-advisory.yml). - SAD-091: manual-trigger baseline-update workflow (sole writer of SAD-089). - Yellow-marker semantics pinned at SAD: 50%-of-tolerance trending detection. SDD v0.9.15 adds: - SDD-120: criterion 0.5 + dhat 0.3 dev-deps; three bench files (realtime_capture, opus_codec, resampler) + common.rs; two post-processor binaries (emit_baseline, compare_baseline); two GitHub Actions workflow YAMLs; simpler-form yellow-marker realization (baseline-only comparator). Out of scope (deferred): - Dimension 3 production telemetry export (P1). - Build-failing hard CI gate (post-baseline-maturity). - Multi-host benchmarking (Linux x86_64 only). - Dart-side flutter_rust_bridge round-trip benchmark. Implementation follows in a separate commit per the no-huge-commit guideline.
This commit is contained in:
EdisonJwa
@@ -1703,6 +1703,112 @@ Notes:
|
||||
- This is a back-fill SDD. No new implementation work is created by SDD-119; the code already exists and ships. A builder agent does NOT need to act on SDD-119. The doc-drift defect in item 4 and the deployment-target single-source-of-truth improvement in item 3 are recorded as open follow-ups and are not in scope for this unit.
|
||||
- SAD-side anchoring of the build-automation pattern at the architectural layer is NOT performed by this SDD unit; a software-architect dispatch is required if the orchestrator wants a SAD-layer build-automation item.
|
||||
|
||||
**SDD-120 — Realtime audio benchmark harness and advisory CI infrastructure**
|
||||
|
||||
- Source SAD: SAD-088 (criterion-based bench harness allocated to `chanora_audio` `benches/` directory, extending SAD-034 audio-subsystem allocation), SAD-089 (baseline-JSON on-disk path pinned to `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json`), SAD-090 (advisory CI workflow `.github/workflows/bench-advisory.yml` — PR-triggered, non-blocking, comment-posting reader of SAD-089), SAD-091 (baseline-update CI workflow `.github/workflows/bench-baseline-update.yml` — `workflow_dispatch`-only, sole writer of SAD-089).
|
||||
- Allocated to: Audio Subsystem (`crates/chanora_audio/benches/`) + Build/Release/Operations (`.github/workflows/bench-advisory.yml`, `.github/workflows/bench-baseline-update.yml`).
|
||||
- Software units (new): `crates/chanora_audio/benches/realtime_capture.rs`, `crates/chanora_audio/benches/opus_codec.rs`, `crates/chanora_audio/benches/resampler.rs`, `crates/chanora_audio/benches/common.rs` (shared synthetic-input helper module), `crates/chanora_audio/benches/emit_baseline.rs` (post-bench JSON aggregator binary), `crates/chanora_audio/benches/compare_baseline.rs` (PR-vs-baseline comparator binary), `crates/chanora_audio/Cargo.toml` (dev-dependency additions and `[[bench]]` entries), `.github/workflows/bench-advisory.yml`, `.github/workflows/bench-baseline-update.yml`, `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json` (seeded SAD-089 baseline file).
|
||||
- Cross-trace: SDD-094 (the `chanora_audio::engine::AudioEngine` capture path that the §3 bench seam exercises is the same audio engine whose `ensure_running` / `shutdown_if_idle` lifecycle is specified by SDD-094; the harness measures the steady-state callback that runs while `ensure_running` has opened the cpal input stream, so the bench seam is part of the engine's verification surface). No cross-trace to SDD-076..SDD-080 / SDD-118: the bench targets the realtime audio path and not the PTT capability surface; the Android-bridge build automation is unrelated to the Linux-host bench workflow.
|
||||
|
||||
§1 — Crate selection and version pin.
|
||||
|
||||
1. Add `criterion = "0.5"` to `crates/chanora_audio/Cargo.toml` `[dev-dependencies]`. The `0.5` series is the current stable major and is compatible with the workspace `rust-version = "1.95"` pin. Default features are kept; `html_reports` is left at its default (criterion only emits HTML when invoked with `--save-baseline`/`--load-baseline` against the local user, and CI consumes the per-bench `estimates.json` files directly from `target/criterion/`, so an HTML feature toggle is not load-bearing for the workflow). Rationale: criterion is the de-facto statistical benchmark harness for Rust; it emits a stable JSON estimates file under `target/criterion/<bench>/<group>/estimates.json` that the §5 post-processor parses.
|
||||
2. Add `dhat = "0.3"` to `crates/chanora_audio/Cargo.toml` `[dev-dependencies]`. `dhat` is used inside `bench_capture_alloc_count` to install a `#[global_allocator] static ALLOC: dhat::Alloc = dhat::Alloc;` profiler at bench-binary scope and to compare `dhat::HeapStats::get().total_blocks` (or the equivalent v0.3 API name) before and after the measurement window. Rationale: `dhat` is the de-facto heap profiler for Rust, has zero-cost when disabled, and is the lowest-friction way to obtain a steady-state heap-allocation count for the SRS-219 clause-a zero-allocation invariant (which carries `tolerance = 0`).
|
||||
3. Add three `[[bench]]` entries to `crates/chanora_audio/Cargo.toml`, one per bench file enumerated in §2. Each entry shall set `harness = false` only if the bench file calls `criterion::criterion_main!` directly (criterion's `main!` macro replaces the cargo test harness); the §3 signatures use `criterion_main!` so `harness = false` is required.
|
||||
4. Add two `[[bin]]` entries pointing at `benches/emit_baseline.rs` and `benches/compare_baseline.rs` so that `cargo run --bin emit_baseline` and `cargo run --bin compare_baseline` resolve correctly. The §6 / §7 workflow steps invoke these binaries by name; placing the source files under `benches/` keeps the entire benchmark surface in one directory but requires the `path = "benches/<name>.rs"` field on each `[[bin]]` entry.
|
||||
|
||||
§2 — Bench file layout.
|
||||
|
||||
1. `crates/chanora_audio/benches/realtime_capture.rs` — declares `bench_capture_alloc_count` and `bench_capture_callback_wall_clock`. Uses `criterion_group!(realtime_capture, bench_capture_alloc_count, bench_capture_callback_wall_clock);` and `criterion_main!(realtime_capture);`.
|
||||
2. `crates/chanora_audio/benches/opus_codec.rs` — declares `bench_opus_encode_latency` and `bench_opus_decode_latency`. Uses `criterion_group!(opus_codec, bench_opus_encode_latency, bench_opus_decode_latency);` and `criterion_main!(opus_codec);`.
|
||||
3. `crates/chanora_audio/benches/resampler.rs` — declares `bench_resampler_throughput`. Uses `criterion_group!(resampler, bench_resampler_throughput);` and `criterion_main!(resampler);`.
|
||||
4. `crates/chanora_audio/benches/common.rs` — shared synthetic-input helpers per §4. Included from each bench file with `mod common;` (criterion bench files compile as standalone binaries, so `common.rs` is included by relative module declaration in each `realtime_capture.rs` / `opus_codec.rs` / `resampler.rs`; do NOT add it as a fourth `[[bench]]` entry).
|
||||
|
||||
§3 — Bench function signatures and warmup phase.
|
||||
|
||||
1. `fn bench_capture_alloc_count(c: &mut Criterion)` — the bench installs a `dhat::Profiler::builder().testing().build()` once at the start of the function, runs the §4 `synthetic_capture_buffer(960, 1)` through `CaptureState::ingest(&buf)` 100 times as a warm-up (this primes any first-call allocations from the cpal capture-state machine — ring-buffer growth, resampler-state init, opus-encoder-handle init via the existing engine code path), then samples `dhat::HeapStats::get()` to capture the `total_blocks` count `B_warm`, runs 1000 additional `ingest` calls inside `c.bench_function("capture_alloc_count", |b| b.iter(|| ingest_one()))`, samples `dhat::HeapStats::get()` again to obtain `B_final`, and the criterion-reported metric value shall be `(B_final - B_warm)` (the post-warmup allocation count). The function additionally `assert_eq!(B_final - B_warm, 0)` so a regression also fails the bench process at the local-developer surface; the CI advisory comparator in `compare_baseline.rs` carries the SRS-219 clause-a `tolerance = 0` zero-allocation rule independently. Bench seam: the function exercises the `CaptureState::ingest<T>(&mut self, buf: &[T])` signature located at `crates/chanora_audio/src/engine.rs:1207` per the §“State” reading; the bench obtains a `CaptureState` instance by calling the engine's existing test-mode constructor (the same path `tests/ptt_privacy.rs` uses today). No new public API surface is created on `CaptureState`.
|
||||
2. `fn bench_capture_callback_wall_clock(c: &mut Criterion)` — uses criterion's default `c.bench_function("capture_callback_wall_clock", |b| b.iter(|| capture_state.ingest(black_box(&buf))));` over the same `synthetic_capture_buffer(960, 1)` input. Criterion's internal warmup (default 3 s, configurable but not overridden here) precedes the measurement window; the harness emits `mean`, `median`, and the 95 % and 99 % bootstrap-confidence-interval percentiles into `target/criterion/capture_callback_wall_clock/.../estimates.json`. The §5 post-processor extracts p50 (median), p95, and p99 from that file and writes the three values into the SRS-217 record's `value` triplet.
|
||||
3. `fn bench_opus_encode_latency(c: &mut Criterion)` — constructs an `audiopus::coder::Encoder::new(SampleRate::Hz48000, Channels::Mono, Application::Voip)` once outside the measurement loop (per criterion best practice; encoder construction is dominated by libopus state allocation and is not part of the per-frame latency), pre-builds a `synthetic_opus_frame()` PCM buffer (exactly `FRAME_SAMPLES = 960` samples per SAD-088 / SysDes-156), and measures `encoder.encode_float(&pcm, &mut opus_out)` per iteration. The bench reports p50/p95/p99 as in item 2.
|
||||
4. `fn bench_opus_decode_latency(c: &mut Criterion)` — same pattern as item 3 but with `audiopus::coder::Decoder::decode_float(&opus_bytes, &mut pcm_out, false)`. The SDD explicitly pins the bench target to the `audiopus` direct decoder call rather than `tsclientlib::AudioHandler::fill_buffer`, because `AudioHandler::fill_buffer` performs jitter-buffer dequeue + Opus decode + PCM mix in a single call and the wall-clock measurement would conflate three distinct concerns. The canonical SRS-217 metric `opus_decode_latency` therefore refers to the codec-level decode of a single 20 ms / 960-sample frame, not the full AudioHandler call. (Future revision may add a separate `audio_handler_fill_buffer_latency` metric covering the composite call; that metric is out of scope at this SDD revision.)
|
||||
5. `fn bench_resampler_throughput(c: &mut Criterion)` — opens one `criterion::BenchmarkGroup` named `resampler_throughput` and registers three sub-benches: `c.bench_function("44100_to_48000", ...)`, `c.bench_function("16000_to_48000", ...)`, and `c.bench_function("48000_passthrough", ...)`. Each sub-bench feeds a 1-second synthetic-sine input (length is 44100 / 16000 / 48000 frames respectively) through the existing engine's resampler entry point and reports `samples_per_second = input_len / measured_duration_secs`. The bench uses `group.throughput(Throughput::Elements(input_len as u64))` so criterion's report exposes the throughput estimate directly in `estimates.json`.
|
||||
|
||||
§4 — Synthetic input generation.
|
||||
|
||||
1. `crates/chanora_audio/benches/common.rs` exposes two functions, both deterministic (no RNG, no time-of-day input): `pub fn synthetic_capture_buffer(frames: usize, channels: usize) -> Vec<f32>` and `pub fn synthetic_opus_frame() -> Vec<f32>` (the latter is the special case `synthetic_capture_buffer(960, 1)`).
|
||||
2. Sample generation: a 440 Hz sine at amplitude 0.5 over the 48 kHz playback rate (matching the audio engine's mixer rate per SAD-034). For frame index `n` the sample value is `0.5 * (2.0 * PI * 440.0 * (n as f32) / 48_000.0).sin()`. For multi-channel buffers the same scalar value is replicated across all channels at the same frame index (interleaved layout, matching cpal's `Stream` data layout). No randomness, no fade-in, no DC offset — the same input produces the same bytes across runs and the same bytes across hosts.
|
||||
3. Determinism is load-bearing for §5 baseline stability: if the synthetic input drifts between runs the criterion `estimates.json` would also drift and the comparator would emit false-positive 🔴 markers. The §5 emit-baseline binary and the §6 comparator both assume byte-stable input.
|
||||
|
||||
§5 — JSON post-processing.
|
||||
|
||||
1. After each bench file runs, criterion writes `target/criterion/<bench-name>/<group-name>/estimates.json` per benchmark group. The §5 binary `crates/chanora_audio/benches/emit_baseline.rs` reads the five expected estimates files (one per §3 bench function, with the resampler bench contributing three sub-bench estimate files that are aggregated into the single SRS-217 `resampler_throughput` record by averaging the three throughput estimates into a per-mode triplet, NOT a single scalar — the SRS-217 schema carries a per-mode record).
|
||||
2. The binary parses each `estimates.json` (a stable criterion-emitted JSON shape: `mean`, `median`, `std_dev`, `confidence_interval`) and projects each metric into an SRS-217 record with fields `{ metric: <name>, value: <f64 or triplet>, unit: <string>, tolerance_pct: <f64 from SRS-219>, host: "x86_64-unknown-linux-gnu", timestamp_utc: <RFC3339> }`. The full output is a single JSON document with one top-level array `metrics`.
|
||||
3. The binary writes the document to `./current.json` in the workspace root (not the SAD-089 path — that path is the comparator's input on advisory runs and the writer's output target on baseline-update runs; §6 step 7 reads SAD-089 from git, §7 step 5 writes SAD-089 by copy).
|
||||
4. Rationale for the Rust-binary approach over a `tools/bench-emit.sh` script: same toolchain as the bench files (no separate `jq` / `python` / `bash` dependency on the runner), same `serde_json` already pulled by the workspace, and the binary can share types with `compare_baseline.rs` (both crates use the same SRS-217 struct definitions).
|
||||
|
||||
§6 — `.github/workflows/bench-advisory.yml` structure (SAD-090 realization).
|
||||
|
||||
1. Triggers: `pull_request: { types: [opened, synchronize, reopened], branches: [main] }` and `push: { branches: [main] }`. (The default branch is `main` per the existing `.github/workflows/ci.yml` convention; if the repository default branch ever renames, both YAML files in this SDD unit must update in lockstep.)
|
||||
2. Job name: `bench-advisory`. `runs-on: ubuntu-latest`. `permissions: { pull-requests: write, contents: read }` — `pull-requests: write` is required by step 9 (PR-comment posting); `contents: read` is the minimum for `actions/checkout`.
|
||||
3. Step 1: `actions/checkout@v4` with `fetch-depth: 0`. Full history is required because step 6 resolves the PR merge-base via `git merge-base origin/main HEAD` and step 7 reads the baseline file from that merge-base via `git show`.
|
||||
4. Step 2: install Rust toolchain via `dtolnay/rust-toolchain@stable` (matching the existing `.github/workflows/ci.yml` pattern). The workspace `rust-version = "1.95"` floor is satisfied by `@stable` at the time of authoring; if a future workspace pin moves above the public stable channel, both this file and `ci.yml` must update.
|
||||
5. Step 3: `Swatinem/rust-cache@v2` (matching existing `.github/workflows/ci.yml`). Reuses the bench target directory across runs.
|
||||
6. Step 4: install the same system audio deps `ci.yml` installs (`libasound2-dev libpulse-dev pkg-config libopus-dev`). Required because the `chanora_audio` crate links `libopus` and the cpal Linux backend pulls ALSA headers at build time.
|
||||
7. Step 5: `cargo bench --bench realtime_capture --bench opus_codec --bench resampler`. (`cargo bench` without `--benches` would build doctests; the explicit `--bench <name>` list bypasses that and matches the three `[[bench]]` entries from §1 item 3.) No `-- --output-format json` flag is required because §5 reads `target/criterion/.../estimates.json` files directly; criterion writes those unconditionally.
|
||||
8. Step 6: `cargo run --bin emit_baseline` → writes `./current.json` to the workspace root.
|
||||
9. Step 7: `MERGE_BASE=$(git merge-base origin/main HEAD)` and `git show $MERGE_BASE:crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json > baseline.json`. If the merge-base predates the introduction of SAD-089 (i.e. the file did not exist at that commit), `git show` exits non-zero; the step shall handle the missing-baseline case by writing an empty `baseline.json` placeholder and emitting a single advisory comment `Baseline not yet established at merge-base; first run on default branch will establish.` instead of running the comparator. This is the only branch where the workflow short-circuits before the comparator step. (The `push: branches: [main]` trigger never hits this short-circuit after the first baseline lands.)
|
||||
10. Step 8: `cargo run --bin compare_baseline -- --current ./current.json --baseline ./baseline.json --output ./report.md`. The binary applies the SRS-219 per-metric tolerance table and the §8 marker logic, then writes a markdown table to `./report.md`. The binary shall always exit 0 (per SRS-218 clause 4); regression detection is rendered as a 🔴 marker inside the comment, not as a non-zero exit code.
|
||||
11. Step 9: post the markdown as a PR comment via `actions/github-script@v7`, gated on `if: github.event_name == 'pull_request'` (the `push` trigger has no PR to comment on; for `push` runs the workflow still writes `./report.md` as a workflow artifact via `actions/upload-artifact@v4` so the default-branch trend is inspectable from the Actions tab). Comment body is `body: require('fs').readFileSync('report.md', 'utf8')`. The script shall also de-duplicate prior comments by the same `github-actions[bot]` identifier carrying a sentinel marker line (e.g. `<!-- chanora-bench-advisory -->`) at the head of the body — on each run the existing comment is updated in place rather than appended, so a long-lived PR does not accumulate one comment per push. The de-duplication query uses `octokit.rest.issues.listComments` filtered on `user.login === 'github-actions[bot]'` and body-startsWith the sentinel.
|
||||
12. Step 10: `if: always()` and a final `exit 0` — no step in the job sets `continue-on-error: false` against a regression-detecting predicate, so the SRS-218 clause 4 non-blocking property is realized by absence of a failing step rather than by an explicit override.
|
||||
13. Status-check name registered with the branch-protection ruleset (if any): `bench-advisory`. The SDD does not mandate adding this to the required-checks list; SRS-218 clause 4 explicitly prohibits making it required.
|
||||
|
||||
§7 — `.github/workflows/bench-baseline-update.yml` structure (SAD-091 realization).
|
||||
|
||||
1. Trigger: `workflow_dispatch: {}` only. No `pull_request`, no `push`, no `schedule`, no tag triggers. SAD-091 is invariant on this property.
|
||||
2. Job name: `bench-baseline-update`. `runs-on: ubuntu-latest` (byte-comparable to SAD-090 per SAD-091). `permissions: { contents: write, pull-requests: write }` — write on both because step 6 opens a PR via `peter-evans/create-pull-request@v6`.
|
||||
3. Step 1: `actions/checkout@v4` with default ref (no `fetch-depth: 0` needed; the workflow does not resolve merge-bases).
|
||||
4. Step 2: install Rust toolchain (same shape as §6 step 2).
|
||||
5. Step 3: install audio system deps (same shape as §6 step 4).
|
||||
6. Step 4: `cargo bench --bench realtime_capture --bench opus_codec --bench resampler` (same shape as §6 step 5).
|
||||
7. Step 5: `cargo run --bin emit_baseline` (same shape as §6 step 6).
|
||||
8. Step 6: `cp ./current.json crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json`.
|
||||
9. Step 7: `peter-evans/create-pull-request@v6` with `title: "chore(bench): update baselines on $(date -u +%Y-%m-%d)"`, `branch: bench/baseline-update-$(date -u +%Y%m%d)`, `commit-message: "chore(bench): refresh x86_64-unknown-linux-gnu baseline"`, `body: <generated diff summary>`, and `add-paths: crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json`. Rationale for opening a PR rather than direct-pushing to `main`: baseline updates should be reviewable in case of accidental triggering (a contributor clicks the dispatch button by mistake), runner pollution (an unrelated process on the GitHub-hosted runner skewed the measurement), or a legitimate regression that should not silently move the floor. The PR is then merged by a reviewer using the normal branch-protection flow; this preserves the SAD-091 invariant that this workflow is the **sole writer** of the SAD-089 file (the PR-merge commit is also written by this workflow's PR, not by a human directly editing the JSON).
|
||||
|
||||
§8 — Yellow-marker realization (simpler-form selection per SAD-090).
|
||||
|
||||
1. Per SAD-090's yellow-marker pin, the SDD selects the simpler operationalization: the committed SAD-089 baseline JSON IS treated as the most recent default-branch run, and is the comparator for both the green/red tolerance evaluation AND the yellow trending evaluation. No separate run-history artifact is maintained.
|
||||
2. `compare_baseline.rs` logic — for each metric `m` in the SRS-217 record set:
|
||||
- Let `c = current.metrics[m].value`, `b = baseline.metrics[m].value`, `t = SRS-219 tolerance for m`.
|
||||
- If `m` is a zero-tolerance metric (SRS-219 clause a — `capture_alloc_count`): emit 🔴 if `c != 0`, 🟢 if `c == 0`. Yellow does not apply.
|
||||
- Otherwise (`t > 0`): compute `delta_pct = (c - b) / b` when `b > 0`, or `delta_pct = 0.0` when `b == 0` (a degenerate baseline). Then:
|
||||
- 🔴 if `delta_pct > t` (regression beyond tolerance).
|
||||
- 🟡 if `0.5 * t < delta_pct ≤ t` (within tolerance but trending up; the SAD-090 yellow-marker semantics).
|
||||
- 🟢 otherwise (`delta_pct ≤ 0.5 * t`, including all improvements where `delta_pct < 0`).
|
||||
3. The markdown report's table carries columns: `metric | unit | baseline | current | delta_pct | tolerance | marker`. Marker column uses the literal emoji characters 🟢 / 🟡 / 🔴. A header line cites the merge-base SHA from §6 step 7 so the reviewer can confirm which baseline the comparison used.
|
||||
|
||||
§9 — Cross-traces (informational, not derivation).
|
||||
|
||||
1. SDD-094 (cross-platform audio engine `voice_join` / `voice_leave` lifecycle): the §3 bench seam targets `CaptureState::ingest`, which is part of the engine's running-state surface exposed by `AudioEngine::ensure_running`. The harness measures the steady-state callback path that runs while the engine is in the `ensure_running == true` state; this places the bench inside SDD-094's verification surface. The bench does NOT construct a full `AudioEngine` (no cpal stream is opened — the bench drives `CaptureState::ingest` directly with synthetic input), so the §3 bench seam is isolated from cpal device enumeration and from JNI / oboe-rs concerns.
|
||||
2. SDD-118 (Android `chanora_bridge` build automation): explicitly NOT cross-traced. The bench harness runs only on the `ubuntu-latest` host with the `x86_64-unknora-linux-gnu` target per SysDes-157; Android cross-compilation is not exercised by either workflow file in this SDD unit.
|
||||
3. SDD-076..SDD-080 (PTT capability surface): explicitly NOT cross-traced. The realtime callback being benched is downstream of these — a PTT key event ultimately gates the audio engine's transmit path — but the bench specifically targets the audio capture / codec / resampler path, not the PTT event-routing path. Adding PTT events would conflate the measurement.
|
||||
|
||||
§10 — Out of scope (deferrals preserved through Wave 1.5).
|
||||
|
||||
1. No production telemetry export — the bench is a CI-only artifact; no runtime metric is exported from a shipping `chanora_audio` build.
|
||||
2. No hard CI gate — SRS-218 clause 4 is invariant; the advisory workflow never fails a check on regression.
|
||||
3. No multi-host benchmarking — SysDes-157 pins the host to `x86_64-unknown-linux-gnu`; the SAD-089 filename is target-triple-keyed to permit future multi-host expansion without directory restructuring, but adding a second host (e.g. `aarch64-apple-darwin` for macOS measurement) is out of scope at this revision.
|
||||
4. No IDE integration — no VS Code task, no IntelliJ run configuration, no flutter-tool wrapper is authored.
|
||||
5. No benchmark of the Dart-side `flutter_rust_bridge` round-trip — deferred (researcher Tier B4). Measuring the FRB serialization overhead would require a separate `package:benchmark_harness` setup on the Flutter side and a synthetic Dart harness; the Rust-side bench in this SDD unit does not exercise that path.
|
||||
|
||||
§11 — Verification matrix (forward-allocation, not authoritative).
|
||||
|
||||
1. The new SDD unit traces to suggested SWE.4 verification IDs `SWE4-UV-058` through `SWE4-UV-062` covering the five §3 bench functions (`bench_capture_alloc_count`, `bench_capture_callback_wall_clock`, `bench_opus_encode_latency`, `bench_opus_decode_latency`, `bench_resampler_throughput`). These IDs are SUGGESTED forward allocations only; verification-engineer authors the actual SWE.4 entries in a follow-up dispatch per the SAD-090 verification-method field (CI Dry Run + Inspection). The SDD does not author SWE.4 entries.
|
||||
2. Verification method per SAD-088 / SAD-089 / SAD-090 / SAD-091: Inspection of the bench-file structure and YAML structure against this SDD unit; Test (manual local `cargo bench --bench …` invocation) for the §3 bench functions; CI Dry Run (manual dispatch of `bench-baseline-update.yml`) for the §7 baseline-update path; PR-comment-posting validation via the §6 step 9 sentinel-marker round-trip on a synthetic PR.
|
||||
|
||||
Notes:
|
||||
|
||||
- This is a forward-looking SDD unit. The bench files, the Cargo.toml additions, the two workflow YAML files, the `emit_baseline.rs` / `compare_baseline.rs` binaries, and the initial seed of `crates/chanora_audio/benches/baselines/x86_64-unknown-linux-gnu.json` are NEW implementation work for a builder agent. The seed baseline file is generated by manually dispatching `bench-baseline-update.yml` once after the builder lands the rest of the unit; that first dispatch is the bootstrap that the §6 step 9 short-circuit branch will then no longer take on subsequent PR runs.
|
||||
- The `tsclientlib::AudioHandler::fill_buffer` decode-side bench was explicitly NOT selected as the canonical `opus_decode_latency` target (see §3 item 4 rationale). A future SDD revision may add `audio_handler_fill_buffer_latency` as a separate composite-call metric without modifying the canonical Opus codec metric.
|
||||
|
||||
## 11. Updated SAD-to-SDD Coverage Matrix
|
||||
|
||||
| SAD Range | SDD Coverage |
|
||||
@@ -1715,6 +1821,7 @@ Notes:
|
||||
| SAD-077, SAD-081, SAD-084, SAD-085, SAD-086 | Covered by `SDD-111` through `SDD-116` (Android voice audio backend — SRS-210..SRS-215). No dedicated SAD anchor exists for the platform-abstracted mobile-voice-audio-backend trait itself; SDD-111 is sourced from SAD-077 / SAD-081 pending a SAD revision that adds a dedicated allocation item. |
|
||||
| SAD-063 | Covered by `SDD-118` (Android `chanora_bridge` cdylib build automation — Gradle + `cargo-ndk` + per-ABI jniLibs staging). Cross-trace: SDD-073, SDD-105, SDD-109. |
|
||||
| SAD-061, SAD-062, SAD-087 | Covered by `SDD-119` (iOS / macOS `chanora_bridge` cdylib build automation — CocoaPods podspec + `cargo` + `lipo` + framework-layout shell, back-fill of existing code). Source SAD allocation: iOS half anchored to SAD-061 (iOS runtime) and SAD-062 (App Store / packaging); macOS half anchored to SAD-087 (macOS runtime baseline — deployment target isolation, universal-binary `lipo` packaging, `.framework` `Versions/A` layout, hand-rolled CocoaPods podspec automation), authored in SAD v0.9.7 to close the macOS-runtime anchor gap that this SDD unit had previously bridged by parallelism from SAD-061 / SAD-062. SDD-117 remains reserved-but-unauthored for the deferred `ios_voice_unit` trait back-fill noted by SDD-111. |
|
||||
| SAD-088, SAD-089, SAD-090, SAD-091 | Covered by `SDD-120` (realtime-audio benchmark harness and advisory CI infrastructure — criterion-based bench harness, dhat-backed heap-allocation-count metric, SAD-089 baseline JSON post-processor, SAD-090 advisory PR-comment workflow with merge-base baseline read, SAD-091 `workflow_dispatch`-only baseline-update workflow that opens a PR rather than direct-pushing; simpler-form yellow-marker realization treats the SAD-089 baseline as the comparator for both red/green tolerance evaluation and yellow trending detection). Cross-trace: SDD-094 (the bench seam exercises the same `chanora_audio::engine` capture path whose lifecycle is specified by SDD-094). Suggested SWE.4 forward allocation: SWE4-UV-058 through SWE4-UV-062 (verification-engineer follow-up; not authored by this SDD unit). |
|
||||
|
||||
|
||||
## Baseline Candidate 0.9.1 Update
|
||||
@@ -1776,6 +1883,12 @@ Notes:
|
||||
| 0.9.8 | 2026-05-18 | Reconciled Android P0 detailed design after Wave 1.5 SysRS / SRS / SAD propagation. SDD-073 (`AndroidBuildConfig`) updated in place (ID preserved) to record `minSdk = 28` per DEC-004 (superseding the prior API 24 baseline), `targetSdk` policy bound to SRS-188 Google Play floor, NDK ABI set pinned to `arm64-v8a / armeabi-v7a / x86_64`, release-signing CI assertion, R8/ProGuard stance, and AAB output per SRS-119 (delegating packaging pipeline detail to SDD-109). SDD-028 (`BackIntentService`) expanded from a one-line stub to an implementable specification covering API 33+ `OnBackInvokedCallback` registration, pre-33 `OnBackPressedDispatcher` fallback, deterministic route-pop ordering, and edge cases (root route → exit candidate, modal/dialog → close-only, active PTT → ignore). Added six new Android P0 SDD units: SDD-105 (`AndroidJniBootstrap` — back-fill for `JNI_OnLoad` / `initChanoraContext` / library-load location, sourced from SAD-081 / SAD-086); SDD-106 (`AndroidPermissionRequester` — RECORD_AUDIO runtime flow with listen-only fallback, settings deep-link for permanent denial, revocation handling, BridgeEvent surface, TransmitModeSelector clamp, sourced from SAD-085); SDD-107 (`AndroidVoiceForegroundService` — service class, manifest entry, notification channel id `chanora.voice.session`, `foregroundServiceType="microphone"` on API 30+, POST_NOTIFICATIONS request on API 33+, lifecycle bound to voice_join / voice_leave / shutdown_if_idle, START_NOT_STICKY on process death, ongoing-notification re-post on dismissal, sourced from SAD-086); SDD-108 (`AndroidAudioModeController` — `setMode(MODE_IN_COMMUNICATION)` with prior-mode snapshot/restore, ref-count semantics composable for P1, JNI surface from Rust audio engine, error paths including AudioManager-unavailable fallback, sourced from SAD-084 — back-fills the v0.9.8 product-decision-register entry); SDD-109 (`AndroidAaBuildPipeline` — bundletool config, ABI splits matching SDD-073, CI signing assertion, versionCode/versionName derivation, sourced from SAD-037 / SAD-063); SDD-110 (`AndroidPttCapability` — pins Android P0 to FocusedPttBackend only, no global key hook, no accessibility-service PTT, capability badge text under `ptt.capability.android.l0`, sourced from SAD-077). Updated §11 SAD-to-SDD coverage matrix to list the new Android P0 SDD range. Strict layered sourcing preserved (`SDD -> SAD` only). |
|
||||
|
||||
|
||||
## Baseline Candidate 0.9.15 Update
|
||||
|
||||
| Version | Date | Description |
|
||||
| --- | --- | --- |
|
||||
| 0.9.15 | 2026-05-18 | Added realtime-audio benchmark-infrastructure detailed-design unit SDD-120 (`RealtimeAudioBenchmarkHarnessAndAdvisoryCI`) sourced from SAD-088 / SAD-089 / SAD-090 / SAD-091 (the Wave 1.5 benchmark-infrastructure software-architecture chain authored at SAD v0.9.9 from the Option B scope decision propagated through SysRS-307/308/309 → SysDes-156/157/158 → SRS-216..219). SDD-120 is a single cohesive 11-section unit specifying: §1 crate selection (`criterion = "0.5"`, `dhat = "0.3"` dev-deps, three `[[bench]]` entries and two `[[bin]]` entries in `crates/chanora_audio/Cargo.toml`); §2 bench-file layout (three files `realtime_capture.rs` / `opus_codec.rs` / `resampler.rs` plus a shared `common.rs` synthetic-input module); §3 bench function signatures with explicit warmup (100-call pre-warm into `CaptureState::ingest` at `crates/chanora_audio/src/engine.rs:1207` before 1000-call dhat measurement window; criterion default warmup for the wall-clock / encode / decode / throughput benches), with the canonical `opus_decode_latency` metric pinned to `audiopus::coder::Decoder::decode_float` directly rather than `tsclientlib::AudioHandler::fill_buffer` because the latter conflates Opus decode with jitter-buffer dequeue and PCM mix; §4 deterministic synthetic-input generation (440 Hz sine at amplitude 0.5, no RNG, byte-stable across runs); §5 JSON post-processing via a Rust binary `crates/chanora_audio/benches/emit_baseline.rs` that parses criterion's `target/criterion/.../estimates.json` files and projects them into the SRS-217 schema; §6 `.github/workflows/bench-advisory.yml` (PR + push triggers, `dtolnay/rust-toolchain@stable` + `Swatinem/rust-cache@v2` matching the existing `ci.yml` pattern, audio system deps `libasound2-dev libpulse-dev pkg-config libopus-dev`, `fetch-depth: 0` for merge-base resolution, `git show $MERGE_BASE:…/x86_64-unknown-linux-gnu.json` baseline read with a missing-baseline short-circuit, `actions/github-script@v7` PR comment with sentinel-marker de-duplication and `pull_request`-only gating, `actions/upload-artifact@v4` for push-trigger reports, always exit 0 per SRS-218 clause 4); §7 `.github/workflows/bench-baseline-update.yml` (`workflow_dispatch` only — sole writer of SAD-089 per SAD-091, `peter-evans/create-pull-request@v6` opens a PR rather than direct-pushing so baseline updates are review-gated against accidental triggering or runner pollution); §8 yellow-marker realization selecting the SAD-090-permitted simpler form where the SAD-089 baseline JSON is the comparator for both red/green tolerance evaluation and yellow trending detection (no separate run-history artifact), with the comparator algorithm (`compare_baseline.rs`) pinned: 🔴 if `delta_pct > t`, 🟡 if `0.5*t < delta_pct ≤ t`, 🟢 otherwise; zero-tolerance metrics (heap allocation, SRS-219 clause a) collapse to 🔴/🟢 only; §9 cross-traces (SDD-094 audio-engine lifecycle as the bench-seam container; explicit NOT-cross-traces to SDD-118 Android bridge build automation and SDD-076..080 PTT capability surface, both correctly out of scope); §10 deferrals preserved (no telemetry export, no hard CI gate, no multi-host bench, no IDE integration, no Dart-side `flutter_rust_bridge` bench — researcher Tier B4); §11 suggested-forward SWE.4 allocation `SWE4-UV-058` through `SWE4-UV-062` (NOT authored at this layer — verification-engineer follow-up). Updated §11 SAD-to-SDD coverage matrix with a new SAD-088..SAD-091 row mapping to SDD-120. Strict layered sourcing preserved (`SDD -> SAD` only); no SRS, SysDes, SysRS, or SAD edits performed. Recommended next agent: `builder` (Step 6 of the benchmark chain — implements the three bench files, the `common.rs` helper, the two binary post-processors, the `Cargo.toml` dev-dependency + `[[bench]]` + `[[bin]]` additions, and the two workflow YAML files; strictly per this SDD unit). |
|
||||
|
||||
## Baseline Candidate 0.9.14 Update
|
||||
|
||||
| Version | Date | Description |
|
||||
|
||||
Reference in New Issue
Block a user