- Add iOS to voiceActivityTransmitAvailable — iOS has CoreML Silero
VAD pipeline (AppleCoreMlVadWorker) but was excluded by DEC-030
gating that predated the CoreML integration
- Inline deleted VadWorkerPolicy in android_voice_unit.rs — PR #37
removed the enum from vad/mod.rs but missed updating Android
PR #37 inlined VadWorkerPolicy into the desktop capture path but missed
the iOS files. Instead of restoring the deleted types, inline the same
direct if-let-Some pattern into ios_voice_unit.rs (the only remaining
iOS backend) and permanently remove VadWorkerPolicy and
callback_vad_worker_policy from vad/mod.rs.
iOS uses Option<AppleCoreMlVadWorker> with &mut self (no Mutex), so the
inline is simpler than the desktop's try_lock pattern.
Delete ios_raw_unit.rs (538 lines) and all SonoraExperimental references
from the core audio crate. The experimental RemoteIO path that bypassed
Apple VPIO in favor of software WebRTC APM was never shipped and is no
longer needed. VPIO is the sole production iOS audio backend.
- Delete ios_raw_unit.rs entirely
- Remove Raw variant from IosVoiceBackend enum in engine.rs
- Remove SonoraExperimental from IosVoiceProcessingMode enum
- Simplify validate_for_ios() (single-variant enum, no mode check)
- Remove 2 SonoraExperimental validation tests
- Remove ios_raw_unit module declaration from lib.rs
- Remove include_str!-based debug_wav test for the deleted file
* feat(audio): add Silero ONNX VAD with WebRTC fallback
Introduce SileroOnnxVad and SileroOnnxVadWorker for desktop targets. The worker runs Silero v6 ONNX inference on a dedicated thread, accumulating 10 ms frames into the 512-sample 16 kHz input the model expects. Add VadOutput, VoiceActivityDetector trait, and WebRtcFallbackVad to provide a uniform VAD interface with graceful fallback when the ONNX model is unavailable. Wire the new VadBackend variants through AudioProcessingConfig and the snapshot stats so the bridge can report which detector is active.
* feat(audio): integrate desktop VAD worker into capture engine
Wire SileroOnnxVadWorker into the desktop capture path so voice activity can open the transmit gate before encoding. The capture callback now processes all audio through resample, downmix, and VAD unconditionally; transmit_active still gates Opus encoding.
Add new_desktop_audio_processing_state() to construct the config/stats/worker triple, and apply_desktop_vad_backend() to synchronously load or clear the worker on config changes. Override processing_backend to Noop for desktop so bridge diagnostics report the correct backend rather than the iOS-oriented PlatformVoiceProcessing default.
Includes review-driven cleanups: StreamConfig clone to deref per clippy, and a comment explaining why two try_lock calls on silero_vad_worker are structurally necessary (borrow checker requires the policy probe and the fallback path to not share a lock guard because mark_vad_fallback_active takes &mut self).
* fix(audio): modernize Windows PTT to current windows-rs API
Port the Raw Input plus low-level keyboard hook PTT backend to the newer windows-rs patterns: OptionalHandle, Result-returning CreateWindowExW, and None for CallNextHookEx. Replaces the old HHOOK(0) pointer casts. Add deterministic tests for mouse button 4 and 5 press and release driving the gate.
* build(windows): force MSVC release CRT for audiopus cmake builds
audiopus_sys calls cmake::build(opus_path), so downstream Cargo env cannot use cmake-rs Config::define() to override CMake's MSVC Debug CRT defaults. Point cmake-rs at a small wrapper that injects the policy and cache variables during configure while passing cmake --build, --version, and -E through unchanged. Keeps Opus Debug builds on Rust's release dynamic CRT (/MD) instead of CMake's default debug CRT (/MDd), which otherwise pulls in unresolved __imp__CrtDbgReportW symbols at test link.
Document that the iOS deployment target is intentionally absent from this file. It is enforced by tools/build-ios.sh and the Xcode project; setting it globally here would make native macOS cargo check runs try to link iPhone objects against the macOS SDK.
* build(flutter): update pubspec.lock after plugin additions
Regenerated lockfile reflecting the local_notifications and connectivity_plus plugin additions from the poke-notifications feature.
* fix(audio): address PR #37 review findings
Six fixes from independent PR review:
1. BLOCKER: Replace Windows-only cmake .cmd wrapper with cross-platform
CMake env vars. Setting CMAKE=tools/cmake-msvc-release-crt.cmd
globally broke non-Windows hosts because cmake-rs would try to
execute a .cmd file on macOS/Linux. Instead, set
CMAKE_POLICY_DEFAULT_CMP0091=NEW and CMAKE_MSVC_RUNTIME_LIBRARY=
MultiThreadedDLL as env vars that CMake reads natively. MSVC-
specific vars are safely ignored by GCC/Clang toolchains. Delete
the now-unnecessary wrapper script.
2. IMPORTANT: Join the Silero worker thread in Drop instead of
detaching it. The old code dropped the JoinHandle which detaches
the thread; the new code calls handle.join() after closing the
channel, ensuring the ONNX session is cleaned up before the
worker is replaced during config changes.
3. IMPORTANT: Single-try_lock refactor of the capture VAD callback.
The double try_lock (policy probe + send) is replaced by a single
scoped try_lock that both probes availability and sends the frame.
The guard is dropped before the fallback path, which needs &mut
self for mark_vad_fallback_active. This also eliminates the
VadWorkerPolicy enum and callback_vad_worker_policy function,
whose behavior is now inlined into the callback.
4. IMPORTANT: Remove tracing from the realtime capture callback.
mark_vad_fallback_active and sync_vad_backend emitted info!/warn!
from the audio thread. Replace with silent atomic state
publishing via SharedAudioProcessingStats; the bridge stats
stream already exposes vad_fallback_active for diagnostics.
5. IMPORTANT: Defer ONNX model load outside the worker mutex.
apply_desktop_vad_backend_to_worker now constructs the new worker
before taking the lock, then swaps it in under a short hold.
This prevents the realtime callback from being blocked during
model I/O + thread spawn.
6. MINOR: Remove unused VadBackend import from vad/mod.rs after
deleting the policy code.
* fix(audio): address PR #37 second-pass review findings
5-agent review found 5 blocking issues. All addressed:
1. BLOCKER: CMake env vars don't reach CMake cache. Restored .cmd wrapper
but scoped to Windows MSVC targets only via [target.x86_64-pc-windows-msvc]
and [target.aarch64-pc-windows-msvc] in .cargo/config.toml. Non-Windows
hosts are unaffected.
2. BLOCKER: processing_backend normalized in set_audio_processing_config
on desktop (cfg-gated override to Noop), mirroring startup default.
3. BLOCKER: Model-path reload was already wired via reload_audio_processing_config.
Fixed misleading doc comment in core/lib.rs.
4. BLOCKER: DEC-030 updated to reflect desktop VoiceActivity enablement.
Traceability docs (SRS, SysDes, SAD, SDD, implementation-status) updated.
5. Silero ONNX cfg narrowed to desktop-only (excludes macOS/Android).
Cargo.toml ort dependency target cfg narrowed similarly.
6. Realtime callback debt documented as TODO at CaptureState::ingest.
* fix(audio): exclude ort dep on Android target
ort does not provide first-class Android prebuilts in our pin, mirror the
iOS/macOS exclusion so cargo metadata succeeds for android targets.
* test(audio): fix stale select_ptt_backend import in ptt_privacy
The helper moved out of the ptt_backends submodule onto the crate root;
update the integration test imports so the test compiles again.
* build(windows): scope MSVC release CRT cmake wrapper via Cargo [env]
Cargo's [target.<triple>] table only forwards a fixed allowlist
(linker, runner, rustflags, rustdocflags, ar), so setting CMAKE there
was silently dropped and audiopus_sys kept linking the debug CRT,
producing LNK4098 'MSVCRTD conflicts' and __imp__CrtDbgReportW errors
on x86_64-pc-windows-msvc test builds.
Move the override to Cargo's [env] table using cc/cmake-rs's
target-suffixed CMAKE_<triple> lookup (force=true, relative=true) so it
applies to MSVC targets only and not to host tooling. Add stdout
markers to the wrapper so its invocation is provable in cargo -vv logs.
Verified: cargo test -p chanora_audio --target x86_64-pc-windows-msvc
--lib --no-run now links cleanly; CMakeCache.txt records
CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL and CMP0091=NEW.
* fix(flutter): gate VoiceActivity transmit mode by platform support
VoiceActivity relies on the native VAD worker, which is only wired up
on Windows, Linux, and Android. Showing the option on iOS, macOS, or
web let users select a mode that silently never transmitted.
Add voiceActivityTransmitAvailable + transmitModeSegmentsFor() helpers
in voice_settings_controls.dart, hide the VAD row in voice_compact.dart
and drop the VAD segment from the settings dialog when unsupported.
Keep the legacy const transmitModeSegments for the existing widget test
and add two new tests covering the gated helper.
Adopt a call-scoped VoIP audio session lifecycle so other apps' audio is
not stopped while Chanora is idle and the in-call session does not get
clobbered by media-server resets unrelated to voice.
AppDelegate.swift
- Set .ambient + .mixWithOthers as the idle baseline so the app does not
hold a VoiceChat session when no call is active.
- Switch to .playAndRecord + .voiceChat + .mixWithOthers + .duckOthers
on demand via the new chanora/ios_audio_session MethodChannel, and
revert to .ambient on deactivate.
- Gate the media-services-reset rebuild on voiceSessionActive so a
stray reset during idle no longer reactivates VoiceChat.
ios_audio_session_controller.dart (new)
- Thin Dart wrapper around chanora/ios_audio_session with activate/
deactivate; no-op on non-iOS; swallows PlatformException to keep
audio start/stop resilient to platform-side races.
main.dart
- Activate the iOS audio session on BridgeEvent_AudioStarted, deactivate
on BridgeEvent_AudioStopped, fire-and-forget via unawaited().
ios_voice_unit.rs
- Add the 10 local bindings required by the render-callback closure
preamble (wav_recorder_for_render, render_recorder_active,
render_ref_len, render_ref_accum, cb_count, last_num_frames,
num_frames_changes, callbacks_with_audio, callbacks_with_silence)
so the iOS target compiles cleanly with the new lifecycle wiring.
Tests
- 5 unit tests in test/services/ios_audio_session_controller_test.dart
cover activate/deactivate on iOS, no-op on non-iOS, and graceful
PlatformException handling.
Docs
- SRS SRS-110 expanded to cover the call-scoped lifecycle invariant.
- SysDes mobile-voice row updated to reflect the MethodChannel and
.ambient idle baseline.
- implementation-status-2026-05-28 voiceChat row flipped to done.
Verification
- flutter analyze: No issues found (2.8s)
- flutter test: 195 passed / 2 skipped / 0 failed
- cargo build -p chanora_audio --target aarch64-apple-ios: clean
- cargo build -p chanora_audio (macOS host): clean
Device QA matrix (Spotify-keeps-playing-while-idle, mix-during-call,
revert-on-call-end, media-services-reset-during-idle) remains pending
on physical hardware.
- ViewportInfo.updateShouldNotify: compare layoutClass only
(not width/height), avoiding unnecessary rebuilds on every
resize frame within the same layout class.
- ChatPanel: use BorderDirectional(start:) for RTL support.
- ChatPanel: localize 'Close chat' tooltip via AppL10n.chatCloseAction.
- Inline panel snackbar: localize via AppL10n.chatPanelCollapsedHint.
New en/zh ARB entries added.
- _saveCurrentDraft(): removed — it was a self-assignment no-op.
Draft persistence relies on ChatDetailView's didUpdateWidget
(fires onDraftChanged on target switch) and dispose (fires on
panel tear-down), both of which already populate _chatDrafts
correctly without an explicit save call.
- _handleInlineChatViewport layout snackbar: use AppL10n.
- Audio level-meter: switch from callback-count (% 3) to time-based
gating (std::time::Duration::from_millis(33)), robust to cpal
buffer-size or sample-rate changes. Remove level_decimation_counter.
- chat_panel_test.dart: add AppL10n.localizationsDelegates so the
test resolves l10n keys.
Tests: 183 passed, 2 skipped. Dart analyze clean.
cargo test -p chanora_audio --lib: 125 passed.
- ios_voice_unit.rs: add producer_shutdown AtomicBool flag (macOS only).
The macOS start path spawns a tokio producer task that holds clones
of Arc<Mutex<AudioHandler>>, Arc<ArrayQueue<f32>>, and the output
gain/muted atomics, then loops on a 20 ms tokio interval. Without
a shutdown signal the task runs forever on engine stop/restart and
leaks all four Arcs every cycle. Drop now stores 'true' on the
flag; the producer checks it at the top of each tick and exits,
releasing its clones within at most one 20 ms tick.
- macos/Runner/Release.entitlements: strengthen the existing
justification comment for com.apple.security.network.server.
Document the specific failure mode (tokio::net::UdpSocket::bind
-> sandbox 'network-outbound deny' -> EPERM) and explain why
network.client alone does not cover bind()-then-sendto. The
entitlement is required, not over-broad.
cargo check (host + aarch64-apple-darwin): clean
cargo test -p chanora_audio --lib: 133 passed
flutter test: 186 passed, 2 skipped
dart analyze: clean
The PR #27 refactor moved the iOS render callback to a direct-fill
path with its own peak_i16 == 0 check, dropping the !muted gate
that PR #28 added to the pre-refactor callback. Without this
gate, every muted callback fires a false-positive
increment_output_underrun() because the downmix helper writes
silence (peak_i16 = 0) by design when muted.
Re-apply PR #28's gate to the refactored iOS path so this PR does
not silently reintroduce the bug PR #28 was opened to fix.
Verified:
- cargo test -p chanora_audio --lib: 133 passed, 0 failed
- cargo build -p chanora_audio --target aarch64-apple-ios: clean
crates/chanora_audio/src/engine.rs: drop the macOS-specific event-queue producer/consumer path; macOS now uses the iOS-style direct AudioHandler::fill_buffer in the VPIO render callback. The shared AudioHandler is an Arc<Mutex<...>>; the realtime callback uses try_lock so it never blocks on the tokio decode task (see ios_voice_unit.rs render callback).
crates/chanora_audio/src/mobile_voice_backend.rs: update VoiceAudioParams cfg gates — handler is now the iOS/macOS/desktop shape (Arc<Mutex<AudioHandler<SessionAudioId>>>), event_producer is Android-only.
crates/chanora_audio/src/lib.rs: widen the audio_event_queue module visibility to test so the macOS-specific path can be exercised by the unit test suite.
Cargo.toml: bump cpal 0.17.3 -> 0.18.0, jni 0.21 -> 0.22.4, windows 0.54 -> 0.62, criterion 0.5 -> 0.8. Cargo.lock follows.
voice_render.rs: new limit_peak_inplace helper (single-pass, allocation-free peak scaler) with 4 unit tests. Applied in both the macOS and iOS render callbacks before the i16 conversion to prevent hard clipping on multi-client mixes that sum past 0 dBFS. Threshold 0.99 keeps the limiter transparent for normal voice levels (sub-millisecond per-frame latency at 48 kHz; pumping risk negligible for speech).
ios_voice_unit.rs: limit_peak_inplace call sites added to the macOS producer/ring render callback (per-frame, 2-element stack array) and the iOS direct-fill_buffer render callback (per-callback, on the scratch_stereo buffer); the downmix helper's hard-clamp is kept as defense-in-depth and is not expected to engage in the integrated flow.
- ios_voice_unit.rs:856-868: Revert iOS render callback from blocking
lock() back to try_lock() with silence-on-contention (WouldBlock
branch increments callback_xrun stat and returns the pre-zeroed
scratch buffer). Blocking lock() inside the CoreAudio HAL render
callback can stall the realtime IO thread when the decode task on
engine.rs:1309 holds the same AudioHandler Mutex, re-introducing
the underrun pattern this codebase already fixed elsewhere.
- ios_voice_unit.rs:782: Preallocate scratch_stereo to Apple's VPIO
MaximumFramesPerSlice (4096 frames * 2 channels = 8192 f32) at
setup time, so the realtime render callback never grows the Vec
via resize(). The defensive 'len() < needed' branch is kept for
the (impossible) case that the audio unit later raises max frames.
- main.dart:52-56: Gate _showAudioDebugOverlay behind kDebugMode &&
_isMacOS so the internal audio stats panel does not ship in
release builds. kDebugMode is a Dart compile-time const, so the
overlay subtree is tree-shaken out of release/profile binaries.
- Rebuilt macOS chanora_bridge.framework binary (universal arm64 +
x86_64) from the fixed source with the new CARGO_PROFILE_RELEASE_*
env vars (DWARF preserved for dsymutil). install_name patched back
to @rpath/chanora_bridge.framework/Versions/A/chanora_bridge.
Verified:
cargo test -p chanora_audio --lib: 129 passed, 0 failed
cargo check -p chanora_audio --target aarch64-apple-ios: clean
dart analyze lib/main.dart: no issues
xcrun lipo -archs: x86_64 arm64
xcrun otool -D: @rpath install_name preserved
macOS realtime audio was suffering buffer underruns on CoreAudio's VPIO
output callback. Root cause was twofold: AudioHandler was decoded under
a Mutex held across the realtime callback, and the render path
hard-coded mono i16 output regardless of the channel count the
callback actually exposed (CoreAudio occasionally hands the callback
stereo or quad output buffers, in which case writing only every Nth
sample produced silence + clicks).
This change brings macOS in line with the lock-free Android audio
architecture introduced for output stutter elimination:
* chanora_audio: AudioPacket / AudioCommand / AudioEventQueue
(previously gated to `target_os = "android"`) are now compiled on
macOS too. The decode loop in AudioEngine pushes inbound packets
into the queue; the VPIO render callback owns AudioHandler outright
and drains the queue, so the realtime thread never blocks on a
cross-thread mutex. set_client_volume also routes through the
command queue on macOS instead of locking the handler.
* voice_render.rs: new downmix_stereo_f32_to_interleaved_i16 helper
downmixes stereo f32 from AudioHandler to mono i16 and replicates
that mono sample across every output channel the callback exposes.
The existing downmix_stereo_f32_to_mono_i16 helper is retained for
iOS, where VPIO is reliably configured for single-channel output
via the AudioUnit stream format we pin at unit-create time.
Compile-gated to ios + test so the macos build doesn't warn on
dead code.
* ios_voice_unit.rs: render callback reads data.channels from the
args struct and forwards it to the new interleaved helper, so the
macOS path tolerates whatever channel count CoreAudio assigns. A
level decimation counter avoids running sqrt+log10 on every
callback (~93 Hz) when the Flutter consumer only reads at 30 Hz;
same regression class as the capture-side fix already in engine.rs.
* mobile_voice_backend.rs: VoiceAudioParams now carries
event_producer on macOS, and the AudioHandler is no longer wrapped
in Arc<Mutex<…>> on macOS because ownership moves into the render
callback. iOS keeps Arc<Mutex<…>> because its callback design
shares the handler with the decode task.
* lib.rs: audio_event_queue module is now compiled on macOS in
addition to android.
apps/chanora_flutter/lib/main.dart wraps the home tree in a Stack and
overlays AudioDebugStatsPanel on macOS so the live engine counters
(callback rate, drift, queue depth) used to diagnose the underrun are
visible while iterating on this code. iOS and other platforms are
unaffected.
apps/chanora_flutter/macos/Frameworks/chanora_bridge.framework binary
is rebuilt with these changes so flutter run on macOS picks up the new
realtime path without requiring developers to rebuild the Rust crate
locally. cargo check -p chanora_audio passes on macOS host.
ios_voice_unit.rs:1033: change the condition from `mix_stats.peak_i16 == 0` to `mix_stats.peak_i16 == 0 && !muted`. When the user mutes the channel via output_muted, the downmix helper fills the output buffer with silence (peak = 0), which previously falsely incremented the output_underrun counter. The mute toggle is intentional silence, not a real underrun.
Note: this does not address the separate false positive where peak_i16 == 0 with output unmuted but no audio incoming (e.g., just joined a channel with no remote speaking). A complete fix would require tracking whether the audio handler actually produced data; deferred to a follow-up.
* feat(voice): unified mobile voice bar with gesture-isolated PTT row
Replace separate VoiceStatusChip + VoicePttButton with a single
CompactVoiceBar widget that combines both into a two-row layout:
- Control row (tap): status text, mute, deafen, settings chevron
- PTT row (hold): full-width hold-to-talk, shown only in PTT mode
Gesture isolation prevents mis-touch between rows: the control row
uses tap-only InkWell/IconButton while the PTT row uses a raw
Listener for pointer-down/up events.
Key changes:
- Add CompactVoiceBar widget with state-colored container (normal,
muted, talk-power-blocked)
- Remove mute/deafen IconButtons from AppBar headerActions
- Restructure voice details sheet into primary section + collapsible
ExpansionTiles (audio processing, PTT capability, debug)
- Optimistic state updates for mute/deafen to eliminate tap delay
- Instant PTT visual feedback (no AnimatedContainer fade)
- Constant geometry across all states (no layout shift on toggle)
* fix(voice): preserve current PTT button format
* feat(voice): move mute/deafen controls into VoiceStatusChip
* fix(voice): ensure consistent chip height across mute states
Remove isSelected/selectedIcon from IconButtons inside VoiceStatusChip.
Material 3 toggle IconButtons (_SelectableIconButton) can vary in height
when the selected state changes due to tap target sizing. Use simple
conditional icons instead and set shrinkWrap tap target size with tight
constraints for stable 40x40 buttons regardless of state.
* fix(voice): remove leftover duplicate mute/deafen buttons in VoiceStatusChip
* fix(voice): replace unsafe stereo cast with bytemuck and localise talk-power tooltip
Replace the raw-pointer `&mut [(f32, f32)]` to `&mut [f32]` cast in
the oboe output callback with `bytemuck::cast_slice_mut`, eliminating
the unsafe block and relying on bytemuck compile-time NoUninit
verification instead.
Add voiceTalkPowerBlocked l10n key (en + zh) and replace the only
remaining hard-coded English tooltip in VoiceStatusChip with it.
* feat(voice): add real-time mic input level metering at 30 Hz
Expose input RMS from the audio engine through the bridge as a
dedicated Rust→Dart Stream<double>, replacing the binary on/off
indicator with a proportional dBFS level meter.
Rust side:
- chanora_audio: add set_input_dbfs/input_dbfs accessors to
SharedAudioProcessingStats; restructure CaptureState::ingest()
to compute dBFS from mono buffer before the PTT guard so the
meter shows mic activity even when not transmitting.
- chanora_core: widen audio_stats() return to include f32 input
level.
- chanora_bridge: add input_level: f32 to BridgeAudioStats and
new input_level_stream(sink: StreamSink<f32>) that pushes at
~30 Hz via tokio interval task.
- Update frb_generated.rs serialization for the new field.
Flutter side:
- VoiceLevelMeter: accept optional double level (dBFS), map
-60..0 dBFS to 0..1 fill fraction, animate with
TweenAnimationBuilder for smooth transitions.
- voice_compact.dart: subscribe to inputLevelStream in the voice
details sheet for 30 Hz meter updates, keeping 250 ms poll for
TX/RX counters.
- voice_bar.dart: accept optional inputLevel from the stream.
- main.dart: subscribe to inputLevelStream, pass to VoiceBar.
* chore: sync Flutter build config and dependency updates
- Add Flutter migrator flags to gradle.properties (builtInKotlin, newDsl)
- Add FlutterGeneratedPluginSwiftPackage to iOS/macOS Xcode projects
- Update meta 1.17→1.18, test_api 0.7.10→0.7.11
- Rebuild chanora_bridge framework for macOS
- Update Podfile.lock for iOS and macOS
* fix(voice): correct meter animation, pre-gain dBFS, stream lifecycle, and protocol warnings
B1: Convert VoiceLevelMeter to StatefulWidget tracking previous fill
as Tween begin so the meter animates smoothly instead of resetting
to zero on every frame.
B2: Compute dBFS from pre-gain mono samples in CaptureState::ingest()
so the level meter reflects raw mic input, matching mobile paths.
B4: End input_level_stream after 10 consecutive session errors instead
of emitting -120 dBFS forever when the session is gone.
Also fixes all 13 clippy warnings in chanora_protocol: collapsed
nested if-let patterns, replaced .ok() + Some matching with Ok, used
? operator, and introduced EventChannels struct to reduce the four
helper functions below the 7-argument threshold.
* fix(voice): use MissedTickBehavior::Skip for level meter stream and align dBFS doc
Set MissedTickBehavior::Skip on the input_level_stream tokio interval
so slow audio_stats() calls skip missed ticks instead of bursting,
preventing CPU spikes on the UI meter thread.
Align VoiceLevelMeter class doc: the mapping floors at -60 dBFS
(via dbfsToFraction), not the full -120 range.
* fix(audio): eliminate Android output stutter via Oboe config + lock-free callback
Phase 1 — Oboe configuration:
- Change output stream from Usage::VoiceCommunication to Usage::Game with
ContentType::Sonification to avoid forcing the Legacy (OpenSL ES) data
path on most devices (Oboe issue #2075)
- Switch output format from i16 Mono to f32 Stereo, matching Qint's proven
configuration and eliminating per-callback downmix conversion
- Set buffer size to 2x burst after stream open, reducing default buffer
from 8-20x burst to 2x burst for lower latency
- Remove scratch Mutex<Vec<f32>>; callback writes directly to Oboe buffer
Phase 2 — Lock-free output callback:
- Add audio_event_queue.rs: lock-free SPSC bridge using crossbeam ArrayQueue
with separate packet (lossy) and control (reliable) channels
- OutputCallback now owns AudioHandler directly (no Arc<Mutex<>> on Android)
- Inbound forwarder pushes packets via AudioEventProducer (no mutex)
- set_client_volume pushes control commands via event queue on Android
- iOS/desktop Arc<Mutex<AudioHandler>> path unchanged
* fix(audio): address PR #20 review findings
- Store AudioEventConsumer directly in OutputCallback to eliminate
per-callback Arc clone on the real-time audio thread
- Add SAFETY comment for the unsafe from_raw_parts_mut transmute
- Bound set_client_volume spin-loop to 64 retries with warn log
- Remove redundant crossbeam-utils direct dependency
- Regenerate license inventory for new crossbeam deps (CI fix)
* fix(audio): use ASCII TODO punctuation
* docs(rust): add doc comments to delta enums and fix dead_code warnings
Add missing documentation to ProtocolDelta, CoreDelta, and BridgeDelta
enum variants and their struct fields across the protocol, core, and
bridge crates. Document the ChannelId::ROOT constant and the
take_delta_rx adapter method.
Fix dead_code warnings:
- keyring_disabled: add #[cfg] gate matching its callers
- snapshot_signature: add #[cfg(test)] for future test use
* fix(rust): correct `order` field docs to predecessor channel ID, narrow audio engine cfg gates
- Correct `order` field documentation in ProtocolDelta, SessionEvent,
and BridgeEvent from 'sort order' to 'predecessor channel ID
(TeamSpeak linked-list ordering hint)' per Copilot review feedback.
- Narrow AudioEngine voice_out_tx, voice_activity_selector, and mic_gain
cfg gates from ios+macos+android to android-only, since these fields
are only read from self in android_restart_voice_unit. On iOS/macOS the
values are passed directly to the voice backend at construction time.
* fix(diag): record bridge events in diagnostics page
Connection lost/reconnecting/disconnected and iOS audio interruption
events now appear in the diagnostics dialog alongside existing error
snackbar entries.
* fix(diag): reduce audio callback sample verbosity to debug
The render callback diagnostic sample logged every 100 callbacks
(~2s) at INFO level, flooding the 256-entry release log buffer and
pushing out useful events. Changed to DEBUG so it only appears in
debug builds with the larger 4096-entry buffer.
* fix(diag): add timestamps to Rust diagnostic log entries
Clarify ONNX Runtime guidance with direct-open install hints, restore desktop WebRTC VAD visibility, map mouse side buttons through focused PTT capture/runtime paths, and wait for server acks before showing chat sends as successful.
Constraint: Linux release UX must stay functional when ONNX Runtime is optional and GNOME portal availability varies
Rejected: Keep desktop VAD locked to Silero only | misleads users when ONNX Runtime is skipped
Confidence: medium
Scope-risk: moderate
Directive: Preserve the protocol send-ack wait path for chat so UI success always tracks real server acceptance
Tested: flutter analyze lib/main.dart lib/widgets/chat_views.dart lib/widgets/input_dialogs.dart lib/widgets/startup_dependency_screen.dart; flutter test test/widgets/input_dialogs_test.dart test/widgets/chat_views_test.dart test/services/startup_dependency_check_test.dart test/widgets/startup_dependency_screen_test.dart test/widgets/voice_settings_controls_test.dart test/widgets/audio_processing_config_state_test.dart; cargo test -p chanora_protocol --lib; cargo test -p chanora_audio ptt_backends --lib
Not-tested: Live manual GNOME portal rebind/global PTT on a real desktop session; observer-bot chat against a live server after the sender-name fallback change
Implementation of SDD-120 §1-§8:
Bench harness (crates/chanora_audio/benches/):
- common.rs: deterministic synthetic audio (440 Hz sine, no RNG).
- realtime_capture.rs: bench_capture_alloc_count (dhat) +
bench_capture_callback_wall_clock (criterion).
- opus_codec.rs: bench_opus_encode_latency + bench_opus_decode_latency
(direct audiopus, not AudioHandler — SDD-120 §3 item 4).
- resampler.rs: bench_resampler_throughput across 44.1->48 /
16->48 / 48->48 passthrough.
CI tooling (crates/chanora_audio/examples/):
- emit_baseline.rs: aggregates criterion estimates.json outputs
into the SRS-217 baseline schema.
- compare_baseline.rs: applies SRS-219 tolerance, renders markdown
table with 🟢/🟡/🔴 markers + yellow simpler-form realization per
SDD-120 §8.
Deviation from SDD-120 §2 / §5 / §7 placement: these tools live
under examples/, not benches/ or src/bin/. Rationale: they must
consume serde_json (a dev-only dep — production builds must not
pull it). Cargo only resolves dev-dependencies for [[test]],
[[bench]], and [[example]] targets; [[bin]] targets under
src/bin/ see only regular [dependencies]. examples/ keeps the
binaries out of the production dep tree while still giving them
cargo run --example invocation. An SDD-120 amendment should
reflect this.
Workflows (.github/workflows/):
- bench-advisory.yml: PR + push triggers; runs benches; posts a
sticky PR comment via actions/github-script@v7; job status is
always success (SRS-218 clause 4 — non-blocking).
- bench-baseline-update.yml: workflow_dispatch only; runs benches;
opens PR via peter-evans/create-pull-request@v6 (sole writer of
the SAD-089 baseline JSON).
Cargo.toml additions ([dev-dependencies] only — verified excluded
from --release builds): criterion 0.5, dhat 0.3, serde_json 1.
Source-code seam: minimal pub-but-#[doc(hidden)] bench_seam module
in chanora_audio (engine.rs + lib.rs re-export) so the criterion
bench harness can construct a CaptureState and drive
CaptureState::ingest without re-implementing the engine (SDD-120
§3). Non-iOS targets only — CaptureState itself is iOS-gated.
Initial baseline seed: crates/chanora_audio/benches/baselines/
x86_64-unknown-linux-gnu.json = {}. compare_baseline handles the
missing-baseline case gracefully and emits a 'no red markers'
report; the first manual dispatch of bench-baseline-update.yml
after merge establishes the real values.
Out of scope per SDD-120 §10: production telemetry export,
build-failing hard CI gate, multi-host benchmarking, IDE
integration, Dart-side bridge round-trip bench.
Verification:
- cargo check --workspace --all-targets: PASS.
- cargo bench --bench realtime_capture --no-run: PASS.
- cargo bench --bench opus_codec --no-run: PASS.
- cargo bench --bench resampler --no-run: PASS.
- cargo build --example emit_baseline --example compare_baseline
-p chanora_audio: PASS.
- cargo test --workspace: 106 passed, 0 failed, 3 ignored — no
regression from prior count.
The capture cpal callback (CaptureState::ingest) ran two heap
allocations per callback on the realtime audio thread:
1. engine.rs:1196-1202 — fresh `mono: Vec<f32>` for the downmix
output, once per cpal callback (50–100 Hz).
2. engine.rs:1217-1218 — `pcm_accum.drain(..FRAME_SAMPLES).collect()`
building a fresh Vec<f32> of 960 samples per Opus frame.
Both sites mirror the pattern already fixed for the output side at
engine.rs:1389-1397, where allocating per callback on glibc malloc
was correlated with user-perceptible audio popping. The output-side
fix replaced the per-callback allocation with a pre-allocated
`scratch` Vec that is cleared and resized in place; this commit
applies the same template to the capture side.
Changes:
- Add `mono_scratch: Vec<f32>` and `frame_scratch: Vec<f32>` to
CaptureState. Initialised with Vec::with_capacity(4096) and
Vec::with_capacity(FRAME_SAMPLES=960) respectively in
CaptureState::new.
- Replace the downmix Vec construction with in-place push into
`self.mono_scratch`; `clear()` retains capacity across callbacks.
- Replace the drain().collect() with `self.frame_scratch.extend(
self.pcm_accum.drain(..FRAME_SAMPLES))`; same capacity-retention.
- The resampler call uses std::mem::take to swap the scratch buffer
out for the duration of the &mut self call, then moves it back —
the backing allocation is preserved across callbacks.
Algorithm semantics are unchanged: same downmix arithmetic, same
clamp loop, same Opus encode call sequence. Only the storage
strategy differs.
Out of scope (intentionally not touched):
- Android audio path (android_voice_unit.rs, mobile_voice_backend.rs):
researcher constraint C-4 — the Android cpal data path is mid-
migration and being replaced.
- Output callback at engine.rs:1409+: the only obvious per-callback
allocation there (`scratch`) was already fixed; a fuller audit
is a separate scope decision.
- The `scratch` buffer at engine.rs:1389-1397 — already correct.
Verification:
- cargo check --workspace --all-targets: passes.
- cargo test --workspace: 106 passed / 0 failed / 3 ignored.
- cargo clippy --workspace --all-targets: no new lints introduced;
the one warning inside the edited region (clamp-like pattern at
line 1266) was pre-existing on the copied clamp loop.
Per SDD-106 §6 add a permission-state clamp to TransmitModeSelector.
When RECORD_AUDIO is Denied or PermanentlyDenied the transmit gate
is forced false regardless of PTT or voice-activity state; on
Granted the clamp releases and normal transmit decisions resume.
The clamp takes precedence over PTT and hard_mute in the decision
ordering documented inline.
Three new tests cover the clamp behavior, the release-on-grant
transition, and the non-RECORD_AUDIO ignore path.
Trace: SDD-106 §6, SRS-209.
Replace the prior one-shot android_engage_voice_communication call
with a ModeStack-mediated acquire/release pair. AudioEngine snapshots
the system audio mode on first acquire via android_get_audio_mode()
and restores it on last release via android_set_audio_mode(prior).
MODE_IN_COMMUNICATION (3) is engaged across the voice-session lifetime
per SDD-108.
Includes the Android AudioManager getMode/setMode JNI helpers
(placed in chanora_audio::engine alongside the existing JNI surface)
and the small ptt.rs touch needed for the SDD-108 ID-tag on the
existing tests.
Trace: SDD-108, SDD-115.
Introduce ModeStack, a pure-Rust refcount-composable wrapper for
Android audio-mode acquire/release with prior-mode snapshot. Per
SDD-108 §1/§2 the engine snapshots the system audio mode on first
acquire and restores it on last release; composed acquires are
no-ops while the mode is held.
ModeStack is panic-free; release-on-zero returns AlreadyReleased
rather than panicking. Six SWE4-UV-045-tagged unit tests cover the
acquire/release semantics on the host target.
Trace: SDD-108, SWE4-UV-045.
Edison Jwa