edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(verification,traceability): SWE4-UV-058..062 for SDD-120 benches + matrix absorption

Browse Source 
Closes the SDD-120 §11 verification-engineer follow-up and refreshes the
traceability matrix to incorporate the full benchmark-infrastructure
chain landed in commits 3a7750a / 8e95972 / 75b04f0.

swe4-unit-verification-plan.md v0.9.14 → v0.9.15:
- SWE4-UV-058: bench_capture_alloc_count (SDD-120 §3; SRS-216 metric 1;
  SRS-219 clause a zero-tolerance).
- SWE4-UV-059: bench_capture_callback_wall_clock (metric 2; SRS-219 b
  +20% p95).
- SWE4-UV-060: bench_opus_encode_latency (metric 3; SRS-219 c +15% mean).
- SWE4-UV-061: bench_opus_decode_latency (metric 4; SRS-219 c +15% mean).
- SWE4-UV-062: bench_resampler_throughput (metric 5; SRS-219 d -10%
  samples/sec).
All five status PENDING_BASELINE until the first manual
bench-baseline-update.yml dispatch establishes baselines.

traceability-matrix.md v0.9.9 → v0.9.10:
- 9 new chain rows binding SysRS-307..309 → SysDes-156..158 →
  SRS-216..219 → SAD-088..091 → SDD-120 sections → code anchors →
  SWE4-UV-058..062.
- 10 new code-anchor rows for the benchmark files (benches/{common,
  realtime_capture,opus_codec,resampler}.rs + examples/{emit,compare}
  _baseline.rs + Cargo.toml + bench-advisory.yml +
  bench-baseline-update.yml + .gitignore).
- 5 §A markers reconciled to PENDING_BASELINE (the SWE.4 IDs are now
  authored; what remains pending is the baseline measurement).
- DEC-032 (abiFilters reduction) status unchanged; exit criteria
  pending.
- All 5 prior open issues from earlier wave audits confirmed closed.

Verdict: TRACEABILITY_OK (engineering chain closed end-to-end). Only
remaining gap: PENDING_BASELINE, gated on CI minutes return + manual
workflow_dispatch on bench-baseline-update.yml.
This commit is contained in:
EdisonJwa
2026-05-18 14:15:04 +08:00
parent ed3c01bcb9
commit 92087d066a
2 changed files with 220 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/verification/swe4-unit-verification-plan.md
+82
View File
@@ -563,6 +563,7 @@ Unit verification results shall be summarized in a unit verification summary rep
| 0.9.4 | 2026-05-18 | Added Android P0 unit-verification items SWE4-UV-040 through SWE4-UV-046 covering SDD-028 (expanded back-intent branches), SDD-073 (refreshed build config + AAB), SDD-105 (JNI bootstrap), SDD-106 (permission state machine + transmit clamp), SDD-107 (foreground service lifecycle), SDD-108 (audio mode controller refcount + restore), SDD-110 (PTT capability pin). |
| 0.9.5 | 2026-05-18 | Added Android voice audio backend unit-verification items SWE4-UV-047 through SWE4-UV-052 covering SDD-111 (trait surface), SDD-112 (config requested vs achieved), SDD-113 (per-effect hardware engagement + software fallback) and feeding SRS-210/211/213/214 unit-level classification. |
| 0.9.14 | 2026-05-18 | Wave 3 reviewer follow-up: allocated real SWE.4 IDs for two test sets that had been cited under provisional `SWE4-UV-NNN` placeholders. SWE4-UV-053..055 anchor the three `TransmitModeSelector` permission-state clamp tests in `crates/chanora_audio/src/transmit_selector.rs` (commit 635d160 was cited under the "SWE4-UV-026 family" without real anchors). SWE4-UV-056..057 anchor the two missing `DiagnosticExport::with_android_audio` rendering tests in `crates/chanora_diagnostics/src/lib.rs` (commit aea50a1 introduced the field/builder/`to_text` branch but the unit-test pair was queued for the commit-8 amend). Closes the Wave 3 traceability gap; downstream test cites in commits 4 and 8 are amended to reference these IDs. |
| 0.9.15 | 2026-05-18 | Closed SDD-120 §11 verification-engineer follow-up: allocated SWE4-UV-058 through SWE4-UV-062 anchoring the five realtime-audio bench functions landed in commit 3a7750a (`bench_capture_alloc_count` and `bench_capture_callback_wall_clock` in `crates/chanora_audio/benches/realtime_capture.rs`; `bench_opus_encode_latency` and `bench_opus_decode_latency` in `crates/chanora_audio/benches/opus_codec.rs`; `bench_resampler_throughput` in `crates/chanora_audio/benches/resampler.rs`) and feeding the SRS-216 metric families under the SRS-219 per-metric tolerance windows. Status PENDING_BASELINE: pass/fail evaluation cannot proceed until the first `bench-baseline-update.yml` `workflow_dispatch` invocation lands the SAD-089 baseline JSON; once landed, the SDD-120 §6 `bench-advisory.yml` workflow plus `examples/compare_baseline` perform automated 🔴/🟡/🟢 evaluation against the merge-base baseline snapshot. Strict-layered sourcing preserved (`SWE.4 -> SDD/SRS` only); no SDD, SRS, SAD, SysDes, or SysRS edits performed. |
## 12. Android Permission Clamp + Diagnostics Android-Audio Unit Verification Addendum (Wave 3 — v0.9.14)
@@ -625,3 +626,84 @@ This addendum allocates the five required anchors. SWE4-UV-052 was the prior hig
| SDD-106 §5 | SWE4-UV-055 |
| SDD-106 §6 | SWE4-UV-053, SWE4-UV-054 |
| SDD-116 item 3 | SWE4-UV-056, SWE4-UV-057 |
## 13. Realtime-Audio Benchmark-Harness Unit Verification Addendum (SDD-120 §11 closure — v0.9.15)
**Background.** SDD-120 (`RealtimeAudioBenchmarkHarnessAndAdvisoryCI`, v0.9.16) §11 explicitly defers the SWE.4 allocation for the five bench functions it specifies in §3, noting: "Suggested SWE.4 forward allocation: SWE4-UV-058 through SWE4-UV-062 (verification-engineer follow-up; not authored by this SDD unit)." Commit 3a7750a landed the implementation across three bench files plus two `examples/` post-processor binaries. This addendum closes the §11 follow-up by authoring the five anchors. SWE4-UV-057 was the prior high-water mark; SWE4-UV-058..062 are the next free contiguous range.
**Status conventions.** All five items are marked **PENDING_BASELINE**: pass/fail evaluation requires a committed SAD-089 baseline JSON for the active host triple (`x86_64-unknown-linux-gnu`), which is produced only by the SDD-120 §7 `bench-baseline-update.yml` `workflow_dispatch` workflow. Until that workflow has been dispatched at least once and its PR merged, the SDD-120 §6 `bench-advisory.yml` short-circuits on missing baseline (per SRS-218 clause 4 always-success) and these verification items cannot return 🟢/🟡/🔴. Automated comparison thereafter is performed by `crates/chanora_audio/examples/compare_baseline.rs` (invoked via `cargo run --example compare_baseline`) against the merge-base baseline per SRS-219.
**SWE4-UV-058**: Verify the realtime capture path performs zero heap allocations per callback after the warmup window.
- Source SDD: SDD-120 §3 (bench function `bench_capture_alloc_count` with 100-callback warmup before the dhat measurement window); SDD-120 §8 (zero-tolerance metric collapses to 🔴/🟢 only — no 🟡 band).
- Source SRS: SRS-216 metric 1 (heap allocation count per realtime callback after warmup); SRS-219 clause a (zero tolerance — "any non-zero allocation count on a PR is flagged 🔴").
- Verification method: Criterion bench with dhat global allocator. Invocation: `cargo bench --bench realtime_capture -- bench_capture_alloc_count`.
- Test code path: `crates/chanora_audio/benches/realtime_capture.rs::bench_capture_alloc_count` (line 27).
- Pass criteria: After a 100-call `CaptureState::ingest` pre-warm (per SDD-120 §3), the dhat `HeapStats.total_blocks` delta across the subsequent 1000 `CaptureState::ingest` calls **SHALL equal zero**. Any non-zero count is a 🔴 regression per SRS-219 clause a (zero tolerance).
- Acceptance evidence: dhat global-allocator `HeapStats.total_blocks` delta == 0 captured by the bench; criterion `estimates.json` projected into the SRS-217 baseline schema by `examples/emit_baseline`; advisory marker rendered by `examples/compare_baseline` against the merge-base baseline.
- Cross-trace: SAD-088 (benchmark-instrumentation seam), SDD-094 (audio-engine lifecycle as the bench-seam container).
- Status: **PENDING_BASELINE** (zero-tolerance metric — first baseline run establishes the binary 🟢/🔴 evaluation; subsequent runs gate on any non-zero allocation regardless of baseline value).
**SWE4-UV-059**: Verify the realtime capture callback wall-clock stays within the SRS-219 p95 tolerance.
- Source SDD: SDD-120 §3 (bench function `bench_capture_callback_wall_clock` with criterion default warmup; measurement reports p50/p95/p99 as a fraction of the cpal stream period).
- Source SRS: SRS-216 metric 2 (per-callback wall-clock at p50/p95/p99 as a fraction of cpal stream period); SRS-219 clause b ("per-callback wall-clock duration at p95 — tolerance is +20% relative to the baseline (p50 and p99 are advisory data only and do not drive the 🔴 marker)").
- Verification method: Criterion timing bench. Invocation: `cargo bench --bench realtime_capture -- bench_capture_callback_wall_clock`.
- Test code path: `crates/chanora_audio/benches/realtime_capture.rs::bench_capture_callback_wall_clock` (line 95).
- Pass criteria: p95 wall-clock per callback **SHALL stay within +20% of the baseline-at-merge-base** per SRS-219 clause b. p50 and p99 are reported but **advisory only** (no tolerance gate; cannot drive 🔴). Automated 🟢/🟡/🔴 evaluation by `examples/compare_baseline` per SDD-120 §8 (🔴 if `delta_pct > 20%`, 🟡 if `10% < delta_pct ≤ 20%`, 🟢 otherwise).
- Acceptance evidence: criterion p95 estimate from `target/criterion/.../estimates.json` projected through `examples/emit_baseline`; advisory marker rendered by `examples/compare_baseline`.
- Cross-trace: SAD-088, SDD-094.
- Status: **PENDING_BASELINE**.
**SWE4-UV-060**: Verify Opus encode latency on a 960-sample / 20 ms frame stays within the SRS-219 mean tolerance.
- Source SDD: SDD-120 §3 (bench function `bench_opus_encode_latency` exercising `audiopus::coder::Encoder::encode_float` **directly**, not through `tsclientlib::AudioHandler`, to isolate encode latency from higher-layer mix/jitter handling).
- Source SRS: SRS-216 metric 3 (Opus encode latency on 960-sample/20 ms frame); SRS-219 clause c ("Opus encode latency on a 960-sample / 20 ms frame — tolerance is +15% relative to the baseline on the mean").
- Verification method: Criterion timing bench. Invocation: `cargo bench --bench opus_codec -- bench_opus_encode_latency`.
- Test code path: `crates/chanora_audio/benches/opus_codec.rs::bench_opus_encode_latency` (line 21).
- Pass criteria: `encoder.encode_float()` mean latency on a 960-sample (20 ms @ 48 kHz mono) deterministic-sine frame **SHALL stay within +15% of the baseline-at-merge-base** per SRS-219 clause c. Automated 🟢/🟡/🔴 evaluation by `examples/compare_baseline` per SDD-120 §8 (🔴 if `delta_pct > 15%`, 🟡 if `7.5% < delta_pct ≤ 15%`, 🟢 otherwise).
- Acceptance evidence: criterion mean estimate from `target/criterion/.../estimates.json` projected through `examples/emit_baseline`; advisory marker rendered by `examples/compare_baseline`.
- Cross-trace: SAD-088, SDD-094.
- Status: **PENDING_BASELINE**.
**SWE4-UV-061**: Verify Opus decode latency on a 20 ms Opus frame stays within the SRS-219 mean tolerance.
- Source SDD: SDD-120 §3 (bench function `bench_opus_decode_latency` exercising `audiopus::coder::Decoder::decode_float` **directly**, not `tsclientlib::AudioHandler::fill_buffer`; SDD-120 §3 pinned this explicitly because the `AudioHandler` path conflates Opus decode with jitter-buffer dequeue and PCM mix).
- Source SRS: SRS-216 metric 4 (Opus decode latency on 960-sample/20 ms frame); SRS-219 clause c ("Opus decode latency on a 960-sample / 20 ms frame — tolerance is +15% relative to the baseline on the mean").
- Verification method: Criterion timing bench. Invocation: `cargo bench --bench opus_codec -- bench_opus_decode_latency`.
- Test code path: `crates/chanora_audio/benches/opus_codec.rs::bench_opus_decode_latency` (line 37).
- Pass criteria: `decoder.decode_float()` mean latency on a single 20 ms (960-sample @ 48 kHz mono) Opus-encoded frame **SHALL stay within +15% of the baseline-at-merge-base** per SRS-219 clause c. Automated 🟢/🟡/🔴 evaluation by `examples/compare_baseline` per SDD-120 §8 (🔴 if `delta_pct > 15%`, 🟡 if `7.5% < delta_pct ≤ 15%`, 🟢 otherwise).
- Acceptance evidence: criterion mean estimate from `target/criterion/.../estimates.json` projected through `examples/emit_baseline`; advisory marker rendered by `examples/compare_baseline`.
- Cross-trace: SAD-088, SDD-094.
- Status: **PENDING_BASELINE**.
**SWE4-UV-062**: Verify resampler throughput on the three canonical rate pairs stays within the SRS-219 samples-per-second tolerance.
- Source SDD: SDD-120 §3 (bench function `bench_resampler_throughput` structured as a criterion `BenchmarkGroup` with three sub-benches: 44.1 kHz → 48 kHz, 16 kHz → 48 kHz, and 48 kHz → 48 kHz passthrough).
- Source SRS: SRS-216 metric 5 (resampler throughput on canonical rate pairs); SRS-219 clause d ("resampler throughput on the canonical rate pairs — tolerance is −10% relative to the baseline on samples-per-second (i.e. a throughput drop greater than 10% is flagged 🔴)").
- Verification method: Criterion throughput bench. Invocation: `cargo bench --bench resampler -- bench_resampler_throughput`.
- Test code path: `crates/chanora_audio/benches/resampler.rs::bench_resampler_throughput` (line 17).
- Pass criteria: Each of the three sub-benches (44.1→48, 16→48, 48→48-passthrough) samples-per-second throughput **SHALL stay no worse than −10% of the baseline-at-merge-base** per SRS-219 clause d. A throughput drop greater than 10% on any of the three sub-benches is a 🔴 regression. Automated 🟢/🟡/🔴 evaluation by `examples/compare_baseline` per SDD-120 §8 (🔴 if `delta_pct < −10%`, 🟡 if `−10% ≤ delta_pct < −5%`, 🟢 otherwise; note the sign inversion versus latency metrics — for throughput, a *decrease* is the regression direction).
- Acceptance evidence: criterion samples-per-second estimates from `target/criterion/.../estimates.json` for each of the three sub-benches projected through `examples/emit_baseline`; advisory marker rendered by `examples/compare_baseline`.
- Cross-trace: SAD-088, SDD-094.
- Status: **PENDING_BASELINE**.
### 13.1 SDD-120 Coverage Index
| SDD Range | SWE.4 Coverage | Tolerance Source |
|---|---|---|
| SDD-120 §3 (bench `bench_capture_alloc_count`) | SWE4-UV-058 | SRS-219 clause a (zero-tolerance) |
| SDD-120 §3 (bench `bench_capture_callback_wall_clock`) | SWE4-UV-059 | SRS-219 clause b (+20% p95) |
| SDD-120 §3 (bench `bench_opus_encode_latency`) | SWE4-UV-060 | SRS-219 clause c (+15% mean) |
| SDD-120 §3 (bench `bench_opus_decode_latency`) | SWE4-UV-061 | SRS-219 clause c (+15% mean) |
| SDD-120 §3 (bench `bench_resampler_throughput`) | SWE4-UV-062 | SRS-219 clause d (−10% samples/sec) |
### 13.2 SWE4-UV-058..062 Status Index
| SWE.4 ID | Bench function | Bench file | Tolerance | Status |
|---|---|---|---|---|
| SWE4-UV-058 | `bench_capture_alloc_count` | `crates/chanora_audio/benches/realtime_capture.rs` | zero | PENDING_BASELINE |
| SWE4-UV-059 | `bench_capture_callback_wall_clock` | `crates/chanora_audio/benches/realtime_capture.rs` | +20% p95 | PENDING_BASELINE |
| SWE4-UV-060 | `bench_opus_encode_latency` | `crates/chanora_audio/benches/opus_codec.rs` | +15% mean | PENDING_BASELINE |
| SWE4-UV-061 | `bench_opus_decode_latency` | `crates/chanora_audio/benches/opus_codec.rs` | +15% mean | PENDING_BASELINE |
| SWE4-UV-062 | `bench_resampler_throughput` | `crates/chanora_audio/benches/resampler.rs` | −10% samples/sec | PENDING_BASELINE |