feat(perf,benchmark-infra): criterion bench harness + advisory CI workflows (SDD-120)
Implementation of SDD-120 §1-§8: Bench harness (crates/chanora_audio/benches/): - common.rs: deterministic synthetic audio (440 Hz sine, no RNG). - realtime_capture.rs: bench_capture_alloc_count (dhat) + bench_capture_callback_wall_clock (criterion). - opus_codec.rs: bench_opus_encode_latency + bench_opus_decode_latency (direct audiopus, not AudioHandler — SDD-120 §3 item 4). - resampler.rs: bench_resampler_throughput across 44.1->48 / 16->48 / 48->48 passthrough. CI tooling (crates/chanora_audio/examples/): - emit_baseline.rs: aggregates criterion estimates.json outputs into the SRS-217 baseline schema. - compare_baseline.rs: applies SRS-219 tolerance, renders markdown table with 🟢/🟡/🔴 markers + yellow simpler-form realization per SDD-120 §8. Deviation from SDD-120 §2 / §5 / §7 placement: these tools live under examples/, not benches/ or src/bin/. Rationale: they must consume serde_json (a dev-only dep — production builds must not pull it). Cargo only resolves dev-dependencies for [[test]], [[bench]], and [[example]] targets; [[bin]] targets under src/bin/ see only regular [dependencies]. examples/ keeps the binaries out of the production dep tree while still giving them cargo run --example invocation. An SDD-120 amendment should reflect this. Workflows (.github/workflows/): - bench-advisory.yml: PR + push triggers; runs benches; posts a sticky PR comment via actions/github-script@v7; job status is always success (SRS-218 clause 4 — non-blocking). - bench-baseline-update.yml: workflow_dispatch only; runs benches; opens PR via peter-evans/create-pull-request@v6 (sole writer of the SAD-089 baseline JSON). Cargo.toml additions ([dev-dependencies] only — verified excluded from --release builds): criterion 0.5, dhat 0.3, serde_json 1. Source-code seam: minimal pub-but-#[doc(hidden)] bench_seam module in chanora_audio (engine.rs + lib.rs re-export) so the criterion bench harness can construct a CaptureState and drive CaptureState::ingest without re-implementing the engine (SDD-120 §3). Non-iOS targets only — CaptureState itself is iOS-gated. Initial baseline seed: crates/chanora_audio/benches/baselines/ x86_64-unknown-linux-gnu.json = {}. compare_baseline handles the missing-baseline case gracefully and emits a 'no red markers' report; the first manual dispatch of bench-baseline-update.yml after merge establishes the real values. Out of scope per SDD-120 §10: production telemetry export, build-failing hard CI gate, multi-host benchmarking, IDE integration, Dart-side bridge round-trip bench. Verification: - cargo check --workspace --all-targets: PASS. - cargo bench --bench realtime_capture --no-run: PASS. - cargo bench --bench opus_codec --no-run: PASS. - cargo bench --bench resampler --no-run: PASS. - cargo build --example emit_baseline --example compare_baseline -p chanora_audio: PASS. - cargo test --workspace: 106 passed, 0 failed, 3 ignored — no regression from prior count.
This commit is contained in:
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1,57 @@
|
||||
// SDD-120 §4 — deterministic synthetic input generation.
|
||||
//
|
||||
// Shared by the three bench files (`realtime_capture.rs`,
|
||||
// `opus_codec.rs`, `resampler.rs`). Included via `mod common;` in
|
||||
// each bench file because criterion bench files compile as
|
||||
// independent binaries — there is no shared crate boundary between
|
||||
// them and `common.rs` is NOT registered as a `[[bench]]` entry.
|
||||
//
|
||||
// Determinism is load-bearing for §5 baseline stability: a 440 Hz
|
||||
// sine at amplitude 0.5 over the 48 kHz mixer rate, no RNG, no
|
||||
// fade-in, no DC offset. Same input bytes across runs and hosts.
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::f32::consts::PI;
|
||||
|
||||
/// Frame count of one 20 ms Opus frame at 48 kHz mono — matches
|
||||
/// `chanora_audio::engine::FRAME_SAMPLES` (private constant; the
|
||||
/// value `960` is fixed by the Opus codec protocol and SAD-088).
|
||||
pub const FRAME_SAMPLES: usize = 960;
|
||||
|
||||
/// 48 kHz mixer rate — matches `chanora_audio::engine::SAMPLE_RATE`.
|
||||
pub const SAMPLE_RATE: u32 = 48_000;
|
||||
|
||||
/// Interleaved synthetic capture buffer. For multi-channel layouts
|
||||
/// the same scalar value is replicated across each frame's channels
|
||||
/// (matches cpal's interleaved `Stream` data layout).
|
||||
pub fn synthetic_capture_buffer(frames: usize, channels: usize) -> Vec<f32> {
|
||||
let mut out = Vec::with_capacity(frames * channels);
|
||||
for n in 0..frames {
|
||||
let s = 0.5 * (2.0 * PI * 440.0 * (n as f32) / 48_000.0).sin();
|
||||
for _ in 0..channels {
|
||||
out.push(s);
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// One Opus-frame-sized synthetic PCM buffer (960 samples mono).
|
||||
pub fn synthetic_opus_frame() -> Vec<f32> {
|
||||
synthetic_capture_buffer(FRAME_SAMPLES, 1)
|
||||
}
|
||||
|
||||
/// Synthetic encoded Opus bytes for decode benches. Built by
|
||||
/// encoding `synthetic_opus_frame()` with a one-shot encoder.
|
||||
pub fn synthetic_opus_bytes() -> Vec<u8> {
|
||||
use audiopus::coder::Encoder;
|
||||
use audiopus::{Application, Channels, SampleRate};
|
||||
let enc = Encoder::new(SampleRate::Hz48000, Channels::Mono, Application::Voip)
|
||||
.expect("opus encoder init");
|
||||
let pcm = synthetic_opus_frame();
|
||||
let mut out = vec![0u8; 1275]; // MAX_OPUS_FRAME
|
||||
let len = enc
|
||||
.encode_float(&pcm, &mut out)
|
||||
.expect("opus encode synthetic frame");
|
||||
out.truncate(len);
|
||||
out
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
// SDD-120 §3 items 3-4 — Opus encode/decode latency benches.
|
||||
//
|
||||
// Both benches measure the codec-level call directly (audiopus
|
||||
// Encoder::encode_float / Decoder::decode_float) and not the
|
||||
// composite tsclientlib AudioHandler::fill_buffer call. Rationale
|
||||
// per SDD-120 §3 item 4: AudioHandler::fill_buffer conflates
|
||||
// jitter-buffer dequeue + Opus decode + PCM mix in a single call
|
||||
// and the wall-clock measurement would conflate three distinct
|
||||
// concerns. The canonical `opus_decode_latency` metric here is
|
||||
// codec-only.
|
||||
|
||||
use audiopus::coder::{Decoder, Encoder};
|
||||
use audiopus::packet::Packet;
|
||||
use audiopus::{Application, Channels, MutSignals, SampleRate};
|
||||
use std::convert::TryFrom;
|
||||
use criterion::{black_box, criterion_group, criterion_main, Criterion};
|
||||
|
||||
mod common;
|
||||
use common::{synthetic_opus_bytes, synthetic_opus_frame};
|
||||
|
||||
fn bench_opus_encode_latency(c: &mut Criterion) {
|
||||
let mut enc = Encoder::new(SampleRate::Hz48000, Channels::Mono, Application::Voip)
|
||||
.expect("opus encoder init");
|
||||
let pcm = synthetic_opus_frame();
|
||||
let mut out = vec![0u8; 1275];
|
||||
|
||||
c.bench_function("opus_encode_latency", |b| {
|
||||
b.iter(|| {
|
||||
let len = enc
|
||||
.encode_float(black_box(&pcm[..]), &mut out[..])
|
||||
.expect("encode");
|
||||
black_box(len);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
fn bench_opus_decode_latency(c: &mut Criterion) {
|
||||
let mut dec = Decoder::new(SampleRate::Hz48000, Channels::Mono).expect("opus decoder init");
|
||||
let bytes = synthetic_opus_bytes();
|
||||
let mut pcm_out = vec![0.0f32; 960];
|
||||
|
||||
c.bench_function("opus_decode_latency", |b| {
|
||||
b.iter(|| {
|
||||
let input = Packet::try_from(black_box(&bytes[..])).expect("packet");
|
||||
let output = MutSignals::try_from(&mut pcm_out[..]).expect("signals");
|
||||
let n = dec.decode_float(Some(input), output, false).expect("decode");
|
||||
black_box(n);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
criterion_group!(opus_codec, bench_opus_encode_latency, bench_opus_decode_latency);
|
||||
criterion_main!(opus_codec);
|
||||
@@ -0,0 +1,118 @@
|
||||
// SDD-120 §3 items 1-2 — realtime capture bench harness.
|
||||
//
|
||||
// - `bench_capture_alloc_count`: dhat-backed heap-allocation count
|
||||
// across 1000 post-warmup `CaptureState::ingest` calls. Realizes
|
||||
// the SRS-219 clause-a zero-allocation invariant. Local-developer
|
||||
// surface additionally asserts that the post-warmup count is 0
|
||||
// so a regression hard-fails locally; the CI advisory comparator
|
||||
// in `compare_baseline.rs` carries the same `tolerance = 0` rule
|
||||
// independently.
|
||||
// - `bench_capture_callback_wall_clock`: criterion default-warmup
|
||||
// wall-clock bench of `ingest` on the same synthetic buffer.
|
||||
//
|
||||
// dhat is invasive — it replaces the global allocator for the
|
||||
// whole bench binary, but only this bench binary; production
|
||||
// builds and other benches are unaffected per Cargo's per-bench
|
||||
// compilation model.
|
||||
|
||||
#[global_allocator]
|
||||
static ALLOC: dhat::Alloc = dhat::Alloc;
|
||||
|
||||
use chanora_audio::bench_seam::CaptureBenchHandle;
|
||||
use criterion::{black_box, criterion_group, criterion_main, Criterion};
|
||||
|
||||
mod common;
|
||||
use common::{synthetic_capture_buffer, FRAME_SAMPLES};
|
||||
|
||||
fn bench_capture_alloc_count(c: &mut Criterion) {
|
||||
// Build the dhat profiler in test mode so it is process-local
|
||||
// and does not write a JSON heap-dump file. Held for the
|
||||
// duration of this bench function.
|
||||
let _profiler = dhat::Profiler::builder().testing().build();
|
||||
|
||||
let mut handle = CaptureBenchHandle::new(48_000, 1);
|
||||
let buf = synthetic_capture_buffer(FRAME_SAMPLES, 1);
|
||||
|
||||
// 100-call warm-up to prime first-call allocations (ring
|
||||
// growth, opus encoder lazy state, mono_scratch / pcm_accum
|
||||
// capacity). SDD-120 §3 item 1.
|
||||
for _ in 0..100 {
|
||||
handle.ingest_f32(&buf);
|
||||
}
|
||||
|
||||
let stats_warm = dhat::HeapStats::get();
|
||||
let blocks_warm = stats_warm.total_blocks;
|
||||
|
||||
// 1000 post-warmup calls — measurement window.
|
||||
for _ in 0..1000 {
|
||||
handle.ingest_f32(&buf);
|
||||
}
|
||||
|
||||
let stats_final = dhat::HeapStats::get();
|
||||
let blocks_final = stats_final.total_blocks;
|
||||
let delta = blocks_final - blocks_warm;
|
||||
|
||||
// Local-developer surface: hard-fail on any regression.
|
||||
// The CI advisory comparator carries the same rule with a
|
||||
// markdown 🔴 marker on regression instead of a panic.
|
||||
assert_eq!(
|
||||
delta, 0,
|
||||
"post-warmup heap allocation regression: {} blocks (SRS-219 clause a)",
|
||||
delta
|
||||
);
|
||||
|
||||
// Register the metric value with criterion so it appears in
|
||||
// `target/criterion/.../estimates.json` for the §5 emitter to
|
||||
// pick up. We bench a no-op closure here because the actual
|
||||
// measurement is the delta computed above; criterion's
|
||||
// function-time-mean is uninteresting for an alloc-count
|
||||
// metric. The §5 emitter reads the `capture_alloc_count`
|
||||
// metric value out of band via a sidecar file written below.
|
||||
c.bench_function("capture_alloc_count", |b| {
|
||||
b.iter(|| {
|
||||
black_box(delta);
|
||||
});
|
||||
});
|
||||
|
||||
// Sidecar file for §5 emit_baseline: criterion's own
|
||||
// estimates.json carries the no-op closure timing, NOT the
|
||||
// alloc count, so we write the canonical alloc-count value
|
||||
// here and the emitter reads it directly.
|
||||
if let Ok(dir) = std::env::var("CARGO_TARGET_DIR")
|
||||
.map(std::path::PathBuf::from)
|
||||
.or_else(|_| {
|
||||
std::env::current_dir().map(|d| d.join("target"))
|
||||
})
|
||||
{
|
||||
let path = dir.join("criterion").join("capture_alloc_count.sidecar");
|
||||
if let Some(parent) = path.parent() {
|
||||
let _ = std::fs::create_dir_all(parent);
|
||||
}
|
||||
let _ = std::fs::write(&path, format!("{}", delta));
|
||||
}
|
||||
}
|
||||
|
||||
fn bench_capture_callback_wall_clock(c: &mut Criterion) {
|
||||
let mut handle = CaptureBenchHandle::new(48_000, 1);
|
||||
let buf = synthetic_capture_buffer(FRAME_SAMPLES, 1);
|
||||
|
||||
// Pre-warm so first-call allocations do not skew the
|
||||
// measurement window (criterion's default warmup is 3 s which
|
||||
// is more than enough; this loop is belt-and-braces).
|
||||
for _ in 0..100 {
|
||||
handle.ingest_f32(&buf);
|
||||
}
|
||||
|
||||
c.bench_function("capture_callback_wall_clock", |b| {
|
||||
b.iter(|| {
|
||||
handle.ingest_f32(black_box(&buf));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
criterion_group!(
|
||||
realtime_capture,
|
||||
bench_capture_alloc_count,
|
||||
bench_capture_callback_wall_clock
|
||||
);
|
||||
criterion_main!(realtime_capture);
|
||||
@@ -0,0 +1,43 @@
|
||||
// SDD-120 §3 item 5 — resampler throughput bench.
|
||||
//
|
||||
// The chanora_audio engine uses a hand-rolled linear resampler
|
||||
// inside CaptureState (see engine.rs `resample_into_accum`); there
|
||||
// is no rubato dependency. The bench drives the same code path
|
||||
// via the §3 bench seam by instantiating CaptureBenchHandle at
|
||||
// different `in_sample_rate` values (44_100, 16_000, 48_000) and
|
||||
// feeding a 1-second buffer per iteration. Throughput is reported
|
||||
// as samples/sec via criterion's `Throughput::Elements`.
|
||||
|
||||
use chanora_audio::bench_seam::CaptureBenchHandle;
|
||||
use criterion::{black_box, criterion_group, criterion_main, Criterion, Throughput};
|
||||
|
||||
mod common;
|
||||
use common::synthetic_capture_buffer;
|
||||
|
||||
fn bench_resampler_throughput(c: &mut Criterion) {
|
||||
let mut group = c.benchmark_group("resampler_throughput");
|
||||
|
||||
for (label, in_rate, frames) in [
|
||||
("44100_to_48000", 44_100u32, 44_100usize),
|
||||
("16000_to_48000", 16_000u32, 16_000usize),
|
||||
("48000_passthrough", 48_000u32, 48_000usize),
|
||||
] {
|
||||
let buf = synthetic_capture_buffer(frames, 1);
|
||||
let mut handle = CaptureBenchHandle::new(in_rate, 1);
|
||||
// Pre-warm so the first allocation does not dominate.
|
||||
for _ in 0..4 {
|
||||
handle.ingest_f32(&buf);
|
||||
}
|
||||
group.throughput(Throughput::Elements(frames as u64));
|
||||
group.bench_function(label, |b| {
|
||||
b.iter(|| {
|
||||
handle.ingest_f32(black_box(&buf));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
group.finish();
|
||||
}
|
||||
|
||||
criterion_group!(resampler, bench_resampler_throughput);
|
||||
criterion_main!(resampler);
|
||||
Reference in New Issue
Block a user