edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add test tool designs and iOS test plan (TODO-020,043,044,046,047)

Browse Source 
Design documents for event replay, audio processing test, audio
loopback test, and protocol probe tools. iOS audio session test
plan with 28 scenarios and 3-phase automation approach.
This commit is contained in:
Edison Jwa
2026-06-11 21:04:45 +09:00
parent c04aaf4a51
commit 72ded4e011
5 changed files with 423 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/designs/audio-loopback-test-tool.md
+70
View File
@@ -0,0 +1,70 @@
# Audio Loopback Test Tool Design
**Date:** 2026-06-11
**Status:** Design proposal
**TODO:** TODO-046
**Requirements:** SysRS-073, SRS-083
**Effort:** L
**Dependencies:** TODO-054 (audio loopback harness)
## Purpose
End-to-end audio quality verification. Sends a known test signal through the full encode→decode→playback→capture loop and measures signal quality metrics to verify the entire audio pipeline works correctly on a given platform.
## Inputs
- Known test signal (sine sweep, white noise, or chirp)
- Loopback device configuration (virtual audio device or hardware loopback)
- Test duration and sample rate
## Outputs
- Signal quality metrics: SNR (dB), latency (ms), jitter (ms), THD+N (%)
- Pass/fail against acceptance thresholds
- Captured loopback audio WAV for manual inspection
## Architecture
```text
┌──────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
│ Test │────>│ Opus │────>│ Loopback │────>│ Opus │
│ Signal │ │ Encode │ │ Device │ │ Decode │
│ Generator│ │ │ │ (virtual) │ │ │
└──────────┘ └───────────┘ └───────────┘ └─────┬─────┘
│
┌──────────┐ ┌───────────┐ ┌───────────┐ ┌────v─────┐
│ Report │<────│ Metrics │<────│ Compare │<────│ Capture │
│ (SNR, │ │ Extract │ │ (original │ │ (loopback│
│ latency)│ │ │ │ vs recv) │ │ audio) │
└──────────┘ └───────────┘ └───────────┘ └──────────┘
```
1. **Generate:** Create known test signal (e.g., 1kHz sine, sweep)
2. **Encode:** Pass through Opus encoder (matching voice pipeline config)
3. **Loopback:** Send encoded audio through virtual audio device
4. **Decode:** Capture loopback audio and decode through Opus decoder
5. **Compare:** Cross-correlate original and received signals
6. **Measure:** Extract SNR, latency (peak correlation offset), jitter, THD+N
## Implementation Plan
- New Rust binary crate: `tools/audio-loopback-test/`
- Reuse `chanora_audio` Opus encoder/decoder wrappers
- Virtual audio device: BlackHole (macOS), VB-Audio (Windows), snd-aloop (Linux)
- Cross-correlation for latency measurement
- CLI interface: `audio-loopback-test [--duration 5] [--signal sine|sweep|noise] [--device <name>]`
- Requires TODO-054 (loopback harness) for CI virtual device setup
## Dependencies
- `chanora_audio` (Opus encode/decode, audio config)
- `opus` crate (encoder/decoder)
- `hound` (WAV I/O)
- Virtual audio device (platform-specific, from TODO-054)
## Verification
- Unit test: encode→decode roundtrip without loopback, verify signal preserved
- Integration test: full loopback on macOS with BlackHole, verify SNR > threshold
- Platform test: run on each target OS with configured virtual device
- Demo: run tool on developer machine, show metrics report
docs/designs/audio-processing-test-tool.md
+65
View File
@@ -0,0 +1,65 @@
# Audio Processing Test Tool Design
**Date:** 2026-06-11
**Status:** Design proposal
**TODO:** TODO-044
**Requirements:** SysRS-074, SRS-083
**Effort:** L
## Purpose
Test the audio DSP pipeline (AEC, NS, AGC, HPF) in isolation. Measures processing latency, signal quality, and verifies each filter stage produces expected output for known input signals.
## Inputs
- WAV file (reference signal) or live microphone input
- Processing configuration (enable/disable AEC, NS, AGC, HPF)
- Optional: reference signal for AEC (far-end playback)
## Outputs
- Processed audio WAV file
- Per-stage metrics: latency (ms), signal level (dBFS), spectral changes
- Pass/fail per processing stage against acceptance thresholds
## Architecture
```text
┌──────────┐ ┌───────────────────────────────────────┐ ┌──────────┐
│ WAV / │────>│ DSP Chain: HPF → NS → AEC → AGC │────>│ Processed│
│ Mic Input│ │ (chanora_audio processors) │ │ WAV + │
└──────────┘ └────────────────────────┬──────────────┘ │ Metrics │
│ └──────────┘
┌──────v───────┐
│ Metrics │
│ Collector │
│ (latency, │
│ dBFS, SNR) │
└──────────────┘
```
1. **Load:** Read WAV file or open mic stream
2. **Process:** Feed PCM frames through each enabled DSP stage sequentially
3. **Measure:** Collect per-stage latency and signal metrics
4. **Output:** Write processed WAV and print metrics table
## Implementation Plan
- New Rust binary crate: `tools/audio-processing-test/`
- Reuse `chanora_audio` processors: `HpfProcessor`, noise suppression, AEC, AGC
- WAV I/O via `hound` crate
- CLI interface: `audio-processing-test <input.wav> [--output processed.wav] [--stages hpf,ns,aec,agc]`
- Metrics: frame-level latency, input/output RMS, spectral centroid shift
## Dependencies
- `chanora_audio` (HPF, NS, AEC, AGC processors)
- `hound` (WAV read/write)
- `chanora_audio::engine` (AudioProcessingConfig)
## Verification
- Unit test: known-tone input through HPF, verify low-frequency attenuation
- Unit test: known-noise input through NS, verify noise floor reduction
- Integration test: full pipeline on reference WAV, verify output within thresholds
- Demo: run tool on sample file, show metrics output
docs/designs/event-replay-tool.md
+63
View File
@@ -0,0 +1,63 @@
# Event Replay Tool Design
**Date:** 2026-06-11
**Status:** Design proposal
**TODO:** TODO-043
**Requirements:** SysRS-171, SRS-098
**Effort:** L
## Purpose
Replay recorded protocol events for debugging state synchronization. Enables deterministic reproduction of state bugs by replaying a captured event sequence through the state reducer and comparing the result with expected state.
## Inputs
- Event log file (JSON or protobuf) recorded in diagnostics mode (SRS-097)
- Optional: expected final state snapshot for comparison
## Outputs
- Replayed session state at each step
- Diff between replayed state and expected state (if provided)
- Reducer execution trace for debugging
## Architecture
```text
┌──────────────┐ ┌───────────────┐ ┌──────────────────┐
│ Event Log │────>│ Event Iterator│────>│ State Reducer │
│ (JSON/PB) │ │ (ordered) │ │ (chanora_state) │
└──────────────┘ └───────────────┘ └────────┬─────────┘
│
┌────────v─────────┐
│ State Comparator │
│ (expected vs │
│ actual) │
└──────────────────┘
```
1. **Load:** Parse event log file into ordered `StateEvent` sequence
2. **Iterate:** Feed events one-by-one to `chanora_state` reducer
3. **Capture:** Record state after each event for step-through debugging
4. **Compare:** If expected snapshot provided, diff final state against it
5. **Report:** Output pass/fail with reducer trace and state diff
## Implementation Plan
- New Rust binary crate: `tools/event-replay/`
- Reuse `chanora_state::ServerState` and reducer functions directly
- JSON event format matches `chanora_diagnostics` event recording output
- CLI interface: `event-replay <event-log> [--expected <snapshot>] [--trace]`
- `--trace` flag prints state after each event
## Dependencies
- `chanora_state` (reducer, ServerState)
- `chanora_diagnostics` (event log format)
- `serde_json` or `prost` (event deserialization)
## Verification
- Unit test: replay known event sequence, assert final state matches expected
- Integration test: record events from live session, replay, verify state reconstruction
- Demo: replay recorded session and show state diff
docs/designs/ios-audio-session-test-plan.md
+147
View File
@@ -0,0 +1,147 @@
# iOS Audio Session Lifecycle — Integration Test Plan
## Overview
Chanora uses Apple's **VoiceProcessingIO** (VPIO) AudioUnit on iOS/macOS for
voice capture and playback. The audio session is configured in Swift
(`AppDelegate`) with `AVAudioSession.Category.playAndRecord` and
`AVAudioSession.Mode.default`. This document defines the integration tests
needed to verify correct behavior across session transitions, interruptions,
and route changes.
## Architecture Summary
| Layer | Responsibility |
|-------|---------------|
| `AppDelegate.swift` | Sets `AVAudioSession` category/mode, handles route-change and interruption notifications |
| `IosVoiceUnit` (Rust) | Opens VPIO AudioUnit, pins 48 kHz Int16 mono, installs render + input callbacks |
| `AudioEngine::ios_restart_voice_unit` | Restarts the VPIO unit after a route change |
| `AudioEngine::ios_pause_voice_unit` / `ios_resume_voice_unit` | Suspends audio during interruptions |
| `route_policy.rs` | Maps `AudioRoute` to recommended `AudioProcessingConfig` (AEC/NS/AGC ownership) |
## Test Scenarios
### 1. Session Activation and Deactivation
| ID | Scenario | Steps | Expected Behavior |
|----|----------|-------|-------------------|
| S-01 | Cold start session activation | Launch app → connect to server → join voice channel | VPIO unit starts, mic input flows, audio plays through default route |
| S-02 | Session deactivation on disconnect | While in voice → disconnect from server | VPIO unit stops, `AudioEngine::stop()` called, session category restored |
| S-03 | Session mode verification | After activation, query `AVAudioSession.mode` | Must be `.default` (not `.voiceChat`) to avoid ducking |
### 2. Audio Interruption Handling
| ID | Scenario | Steps | Expected Behavior |
|----|----------|-------|-------------------|
| I-01 | Phone call interruption | While in voice → receive incoming call | `AVAudioSession.interruptionNotification` fires with `.began`; VPIO paused via `ios_pause_voice_unit` |
| I-02 | Phone call ends | After I-01 → call ends | Interruption notification fires with `.ended`; VPIO resumed via `ios_resume_voice_unit` if session was active |
| I-03 | Siri activation | While in voice → invoke Siri | Interruption `.began` → pause; Siri dismisses → `.ended` → resume |
| I-04 | Alarm / timer | While in voice → alarm fires | Audio ducks (not interrupted); voice continues at reduced volume |
| I-05 | Third-party audio app | While in voice → open Spotify and play music | Other audio ducks; Chanora voice remains active |
| I-06 | Interruption during route change | While switching routes → phone call arrives | Both events handled; no crash, VPIO restarts cleanly after both resolve |
### 3. Route Change Handling
| ID | Scenario | Steps | Expected Behavior |
|----|----------|-------|-------------------|
| R-01 | Plug in wired headset | While on speaker → connect Lightning/USB-C headphones | Route changes to `.wiredHeadset`; `ios_restart_voice_unit` called; AEC disabled (no acoustic echo path); `route_policy.rs` returns `WiredHeadset` config |
| R-02 | Unplug wired headset | While on wired headset → disconnect | Route changes to `.speaker`; VPIO restarts; AEC re-enabled via platform VPIO |
| R-03 | Connect Bluetooth HFP | While on speaker → connect BT headset in HFP mode | Route changes to `.bluetoothHfp`; VPIO restarts; AEC off (headset firmware handles it) |
| R-04 | Disconnect Bluetooth HFP | While on BT HFP → turn off headset | Route falls back to speaker; VPIO restarts with platform AEC |
| R-05 | Switch to Bluetooth A2DP | While on speaker → connect A2DP-only device | Route changes to `.bluetoothA2dp`; transmit blocked (A2DP is output-only); playback continues |
| R-06 | Toggle speaker/earpiece | Use in-app audio output picker | `overrideOutputAudioPort` called; VPIO restarts; audio actually moves (not just metadata) |
| R-07 | AirPods connect/disconnect | While on speaker → AirPods connect → AirPods case closed | Route transitions handled; VPIO restarts on each change |
| R-08 | Rapid route changes | Connect/disconnect headset 5 times in 10 seconds | No crash, no audio leak, VPIO restarts cleanly each time |
| R-09 | Route change during mute | While muted → route changes | VPIO restarts; mute state preserved; no audio leak |
### 4. Audio Ducking Configuration
| ID | Scenario | Steps | Expected Behavior |
|----|----------|-------|-------------------|
| D-01 | Ducking disabled on startup | App launches and joins voice | `kAUVoiceIOProperty_OtherAudioDuckingConfiguration` set with `mEnableAdvancedDucking=0`, `mDuckingLevel=Min` |
| D-02 | Music playback while in voice | Play music via Music app → join voice channel | Music volume is NOT heavily attenuated; voice and music coexist |
| D-03 | Game audio while in voice | Play a game with audio → join voice | Game audio is NOT heavily attenuated |
### 5. VPIO Stream Format Verification
| ID | Scenario | Steps | Expected Behavior |
|----|----------|-------|-------------------|
| F-01 | Output bus format | After VPIO start, inspect bus 0 stream format | 48 kHz, Int16, mono, signed integer, packed |
| F-02 | Input bus format | After VPIO start, inspect bus 1 stream format | 48 kHz, Int16, mono, signed integer, packed |
| F-03 | Callback frame count | Log `num_frames` in render callback | iOS: 480 frames (10 ms); macOS: 512 frames (10.67 ms) |
| F-04 | Audio quality roundtrip | Speak into mic → loopback to speaker | No distortion, no resampling artifacts, correct latency |
### 6. Route Policy Correctness
| ID | Scenario | Steps | Expected Behavior |
|----|----------|-------|-------------------|
| P-01 | Speaker route policy | Route = Speaker | `ios_route_policy` returns: AEC=Platform, NS=Platform, AGC=Platform, backend=PlatformVoiceProcessing |
| P-02 | Wired headset policy | Route = WiredHeadset | AEC=Off, NS=Conservative, AGC=Conservative, backend=Noop |
| P-03 | BT HFP policy | Route = BluetoothHfp | AEC=Off, NS=Conservative, AGC=Conservative, backend=PlatformVoiceProcessing |
| P-04 | A2DP policy | Route = BluetoothA2dp | All processing off, VAD disabled, transmit blocked |
| P-05 | INV-009 invariant | Any route | Sonora AEC never enabled simultaneously with platform VPIO |
| P-06 | User VAD preserved on route change | Set VAD=WebRTC → change route | New config keeps VAD=WebRTC and hangover timing |
## Device Requirements
### Required Devices
| Device | OS | Reason |
|--------|-----|--------|
| iPhone (Lightning or USB-C) | iOS 16+ | Primary target; VPIO, route changes, interruptions |
| iPhone with Face ID | iOS 17+ | `OtherAudioDuckingConfiguration` property availability |
| AirPods (any generation) | — | Bluetooth A2DP/HFP route testing |
| Bluetooth HFP headset | — | Non-Apple BT headset route testing |
| Lightning/USB-C wired headset | — | Wired route testing |
| iPad (optional) | iPadOS 16+ | Verify identical VPIO behavior |
### Simulator Limitations
- VPIO render callback cadence differs from real hardware
- Route changes are not testable on simulator
- Interruption notifications are unreliable on simulator
- **Recommendation**: All integration tests must run on physical devices
## Automation Approach
### Phase 1: Manual Test Matrix
Execute scenarios S-01 through P-06 on physical devices using this checklist.
Record pass/fail and any audio artifacts observed.
### Phase 2: XCUITest + Rust Harness
```
┌─────────────┐ ┌──────────────┐ ┌───────────────┐
│ XCUITest │────▶│ FRB bridge │────▶│ AudioEngine │
│ (Swift) │ │ test helper │ │ (Rust) │
└─────────────┘ └──────────────┘ └───────────────┘
```
1. **FRB test helper**: Add a `#[flutter_rust_bridge::frb]` test function
that starts `AudioEngine`, runs for N seconds, and returns stats
(frames_sent, frames_received, xruns, output_underruns).
2. **XCUITest**: Launches the app, connects to a test server, triggers
the FRB helper, then uses `XCUIDevice` APIs to simulate:
- Route changes via `XCUIDevice.shared().press(.volumeUp)` + BT pairing
- Interruptions via `XCUISiriService` (Siri) or call simulation
3. **Assertions**: Verify stats counters are within expected ranges
(no xruns, no output underruns, frames_sent > 0).
### Phase 3: Continuous Monitoring
Add a telemetry event for each VPIO restart, pause, resume, and
interruption. Track:
- Restart count per session (should be ≤ number of route changes)
- Pause-to-resume latency (should be < 500 ms)
- Xrun count per session (should be 0 under normal conditions)
## References
- `crates/chanora_audio/src/ios_voice_unit.rs` — VPIO AudioUnit setup
- `crates/chanora_audio/src/engine/lifecycle.rs` — Engine start/stop/restart
- `crates/chanora_audio/src/route_policy.rs` — Route-to-config policy
- Apple: [Audio Session Programming Guide](https://developer.apple.com/library/archive/documentation/Audio/Conceptual/AudioSessionProgrammingGuide/)
- Apple: [Audio Unit Hosting Guide for iOS](https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/AudioUnitHostingGuide_iOS/)
docs/designs/protocol-probe-tool.md
+78
View File
@@ -0,0 +1,78 @@
# Protocol Probe Tool Design
**Date:** 2026-06-11
**Status:** Design proposal
**TODO:** TODO-047
**Requirements:** SysRS-128, SRS-123
**Effort:** L
## Purpose
Connect to a TeamSpeak 3-compatible server and enumerate its capabilities. Validates server compatibility, reports supported features, permissions, and protocol behavior for verification and compatibility tracking.
## Inputs
- Server address (host:port)
- Optional: nickname, identity, server password
## Outputs
- Server info: version, platform, name, welcome message
- Supported features: text messaging, voice, file transfer, channel permissions
- Client permissions: talk power, poke power, channel join capabilities
- Protocol compatibility report: pass/fail/warning per feature
- Error codes encountered during probing
## Architecture
```text
┌──────────┐ ┌───────────────┐ ┌──────────────────┐
│ CLI │────>│ Probe Session │────>│ tsclientlib │
│ (host, │ │ (chanora_ │ │ Protocol Adapter │
│ port) │ │ protocol) │ │ │
└──────────┘ └───────┬───────┘ └──────────────────┘
│
┌──────────v──────────┐
│ Probe Commands: │
│ 1. connect │
│ 2. server info │
│ 3. channel list │
│ 4. client list │
│ 5. send test msg │
│ 6. voice capability│
│ 7. permissions │
└──────────┬──────────┘
│
┌──────────v──────────┐
│ Compatibility │
│ Report (JSON/text) │
└────────────────────┘
```
1. **Connect:** Establish session via `chanora_protocol` adapter
2. **Query:** Execute probe commands sequentially with timeout
3. **Collect:** Gather responses, errors, and timing for each probe
4. **Report:** Output structured compatibility report
## Implementation Plan
- New Rust binary crate: `tools/protocol-probe/`
- Reuse `chanora_protocol::adapter` for tsclientlib connection
- Probe sequence: connect → server info → channels → clients → text test → voice check → permissions
- Each probe step has independent timeout (5s default)
- CLI interface: `protocol-probe <host[:port]> [--nick probe-bot] [--password <pw>] [--output json|text]`
- JSON output for CI integration; text output for human readability
## Dependencies
- `chanora_protocol` (adapter, tsclientlib wrapper)
- `chanora_core` (connection manager, event types)
- `serde_json` (report output)
- `clap` (CLI argument parsing)
## Verification
- Unit test: mock protocol responses, verify report generation
- Integration test: probe local test server, verify all features detected
- Demo: probe public TeamSpeak server, show compatibility report
- CI: run against test server in CI environment, assert pass on required features