Compare commits

..
Author SHA1 Message Date
Edison Jwa 609c4cee8c docs(security): regenerate license inventories
Cargo inventory: pick up chanora_resolver bump from 0.1.0 to
0.2.0-beta.1 so it matches the workspace; also adds a trailing newline
so 'cargo about generate' is idempotent in CI license-drift checks.

Flutter inventory: pick up flutter_local_notifications (+ platform
interfaces) and timezone pulled in by the prior notification
permission work.
2026-06-09 19:29:04 +09:00
Edison Jwa c7e51c2e48 fix(ios-audio): keep session active when already-in-channel
The 'already in channel' server response (code 0x0302) is treated as a
successful join by _onJoinChannel: the user stays in the channel and
local state is updated to reflect the joined target. But the underlying
voiceJoin call still raises BridgeError_ServerRejected, which the
joinVoiceChannelWithIosAudioSession helper used to interpret as a join
failure and deactivate the iOS audio session. Result: the UI shows the
user as joined while the audio session is dead and capture/playback
remain silent.

Add an isJoinSuccess predicate to the ordering helper. When the
predicate matches, the helper rethrows (so the caller can still run its
success-on-already-joined branch) without deactivating the session.
Wire _onJoinChannel to pass _isAlreadyInChannel as the predicate so the
0x0302 path keeps the session active.

Adds two regression tests covering the success-on-rethrow and the
predicate-false-still-deactivates paths.
2026-06-09 19:28:54 +09:00
Edison Jwa 703f44d731 docs(ios-audio): align activation lifecycle comments 2026-06-09 02:16:20 +09:00
Edison Jwa 7be3934c86 fix(ios-audio): activate session before voice joins 2026-06-09 02:16:08 +09:00
Edison Jwa f224347836 fix(ios-audio): add voice join session coordinator 2026-06-09 02:15:55 +09:00
33 changed files with 909 additions and 1010 deletions
+31 -46
View File
@@ -1,49 +1,34 @@
# audiopus_sys calls cmake::build(opus_path), so downstream Cargo env cannot
# call cmake-rs Config::define() to override CMake's MSVC Debug CRT defaults.
# Instead, point cmake-rs at a small wrapper that injects -D cache/policy
# variables during configure while passing cmake --build / --version / -E /
# --install / --open through unchanged. This keeps Opus Debug builds on
# Rust's release dynamic CRT (/MD) instead of CMake's default debug CRT
# (/MDd), which otherwise pulls in unresolved __imp__CrtDbgReportW symbols
# at test link.
#
# IMPORTANT: Cargo's `[target.<triple>]` config sections only forward a
# fixed allowlist of keys (linker, runner, rustflags, rustdocflags, ar)
# to build scripts. Arbitrary keys such as `CMAKE` placed under
# `[target.<triple>]` are silently ignored and never reach the
# audiopus_sys build script. cmake-rs (via cc-style env resolution)
# looks up CMAKE in this order:
# 1. CMAKE_<target-triple-with-dashes>
# 2. CMAKE_<target_triple_with_underscores>
# 3. TARGET_CMAKE (or HOST_CMAKE when host == target)
# 4. CMAKE
# We therefore scope the wrapper to Windows MSVC targets by setting the
# target-suffixed variant in the global [env] section. Non-Windows
# hosts (macOS, Linux, iOS, Android) never see CMAKE set and invoke
# `cmake` directly.
#
# NOTE: CMAKE_POLICY_DEFAULT_CMP0091 and CMAKE_MSVC_RUNTIME_LIBRARY cannot
# be set via the process environment because CMake does NOT auto-import
# them into its cache; they must be passed as `-D` definitions, which the
# wrapper does.
#
# iOS deployment target (DEC-003: iOS 13.0 minimum) is NOT set here. It is
# enforced in two places that own the iOS build:
# 1. tools/build-ios.sh — sets IPHONEOS_DEPLOYMENT_TARGET for the cargo
# invocation and bypasses audiopus_sys's CMake build via
# LIBOPUS_STATIC=1 / LIBOPUS_NO_PKG=1 / LIBOPUS_LIB_DIR.
# 2. apps/chanora_flutter/ios/Runner.xcodeproj — sets the Xcode
# IPHONEOS_DEPLOYMENT_TARGET build setting for the final link.
# Setting it globally here would make native macOS `cargo check` runs try
# to link iPhone objects against the macOS SDK.
# Environment variables set for all cargo invocations in this workspace.
# CMAKE_POLICY_VERSION_MINIMUM is required for audiopus_sys's bundled
# Opus CMake build to succeed on CMake 4.x (which removed compatibility
# with cmake_minimum_required < 3.5). audiopus_sys v0.2.2 bundles
# Opus 1.3.1 whose CMakeLists.txt uses a very old minimum version.
[env]
CMAKE_POLICY_VERSION_MINIMUM = "3.5"
# Scope the cmake wrapper to Windows MSVC targets only via the
# target-suffixed env var name that cc/cmake-rs already resolve.
# Force = true so a developer's pre-existing CMAKE_x86_64-pc-windows-msvc
# does not silently bypass the wrapper. Relative = true so the path
# resolves from the workspace root regardless of where cargo is invoked.
CMAKE_x86_64-pc-windows-msvc = { value = "tools/cmake-msvc-release-crt.cmd", force = true, relative = true }
CMAKE_aarch64-pc-windows-msvc = { value = "tools/cmake-msvc-release-crt.cmd", force = true, relative = true }
# iOS builds must set IPHONEOS_DEPLOYMENT_TARGET in the invoking script
# or Xcode build phase. Do not set it globally here: native macOS cargo
# checks also compile bundled C/C++ dependencies, and a global iOS
# deployment target makes clang try to link iPhone objects against the
# macOS SDK.
# iOS target linker flags (DEC-003: minimum deployment target iOS 13.0).
#
# These rustflags pass -miphoneos-version-min=13.0 to the linker, ensuring
# the final binary targets iOS 13.0+. This is defense-in-depth alongside
# the IPHONEOS_DEPLOYMENT_TARGET env var above — the env var affects C
# compilation (cc crate, CMake), while these rustflags affect the final
# link step.
#
# NOTE: The canonical iOS build is done via tools/build-ios.sh, which
# sets LIBOPUS_STATIC=1, LIBOPUS_NO_PKG=1, and LIBOPUS_LIB_DIR to
# bypass audiopus_sys's CMake build entirely.
[target.aarch64-apple-ios]
rustflags = ["-C", "link-arg=-miphoneos-version-min=13.0"]
[target.aarch64-apple-ios-sim]
rustflags = ["-C", "link-arg=-miphonesimulator-version-min=13.0"]
[target.x86_64-apple-ios]
rustflags = ["-C", "link-arg=-miphonesimulator-version-min=13.0"]
+1 -1
View File
@@ -23,7 +23,7 @@ EXTERNAL SOURCES:
:path: ".symlinks/plugins/haptic_kit/ios"
SPEC CHECKSUMS:
chanora_bridge: 27a03592058709f6f38701343eb51c3a55b02da0
chanora_bridge: 26252acdf9ca660ce9c132ad25cd5ad5af467b16
Flutter: cabc95a1d2626b1b06e7179b784ebcf0c0cde467
flutter_foreground_task: a159d2c2173b33699ddb3e6c2a067045d7cebb89
haptic_kit: b22c4fbb2aa7b0d66f2891f81a9e950ad2de5758
+4 -1
View File
@@ -1339,8 +1339,11 @@ sealed class BridgeEvent with _$BridgeEvent {
/// Bridge iOS voice-processing mode.
enum BridgeIosVoiceProcessingMode {
/// Apple VoiceProcessingIO path.
/// Shipping VPIO path.
platformVoiceProcessing,
/// Experimental Sonora path.
sonoraExperimental,
}
@freezed
@@ -638,19 +638,12 @@ class _VoiceSheetBodyState extends State<_VoiceSheetBody> {
selected: _mode == rust.BridgeTransmitMode.continuous,
onTap: () => _setMode(rust.BridgeTransmitMode.continuous),
),
// Voice-activity transmit is only honoured by the engine on
// hosts that ship a Chanora-owned VAD pipeline (DEC-030:
// Windows + Linux desktop and Android). iOS / macOS rely
// on Apple VoiceProcessingIO and have no VAD bridge, so
// hiding the row prevents the UI from advertising a
// transmit mode the engine cannot honour.
if (voiceActivityTransmitAvailable)
_ModeRow(
label: l10n.voiceModeVoiceActivity,
icon: Icons.graphic_eq,
selected: _mode == rust.BridgeTransmitMode.voiceActivity,
onTap: () => _setMode(rust.BridgeTransmitMode.voiceActivity),
),
_ModeRow(
label: l10n.voiceModeVoiceActivity,
icon: Icons.graphic_eq,
selected: _mode == rust.BridgeTransmitMode.voiceActivity,
onTap: () => _setMode(rust.BridgeTransmitMode.voiceActivity),
),
// 3) Release-tail slider (PTT only).
if (isPtt) ...[
@@ -128,12 +128,7 @@ class _VoiceSettingsDialogState extends State<VoiceSettingsDialog> {
VoiceSectionHeader(l10n.voiceModeLabel),
SegmentedButton<rust.BridgeTransmitMode>(
style: voiceSegmentedButtonStyle(theme),
// DEC-030: hide the voice-activity segment on hosts
// that ship no Chanora-owned VAD pipeline (iOS,
// macOS, web).
segments: transmitModeSegmentsFor(
voiceActivityAvailable: voiceActivityTransmitAvailable,
),
segments: transmitModeSegments,
selected: {_mode},
onSelectionChanged: (s) => setState(() => _mode = s.first),
),
@@ -1,6 +1,3 @@
import 'dart:io' show Platform;
import 'package:flutter/foundation.dart' show kIsWeb;
import 'package:flutter/material.dart';
import '../src/rust/api.dart' as rust;
@@ -13,11 +10,7 @@ ButtonStyle voiceSegmentedButtonStyle(ThemeData theme) {
);
}
/// Transmit mode selector segments — full set, all three modes.
///
/// This list is kept stable for legacy call sites and tests; UI
/// surfaces that must respect DEC-030 platform gating should prefer
/// [transmitModeSegmentsFor] with [voiceActivityTransmitAvailable].
/// Transmit mode selector segments.
const transmitModeSegments = [
ButtonSegment(
value: rust.BridgeTransmitMode.ptt,
@@ -36,44 +29,6 @@ const transmitModeSegments = [
),
];
/// Transmit mode selector segments, optionally dropping the
/// voice-activity entry on hosts that do not ship a VAD pipeline.
///
/// Voice activity transmit is gated by [voiceActivityTransmitAvailable]
/// because the underlying VAD pipeline ships only on Windows, Linux, and
/// Android per DEC-030. Builds for unsupported platforms (iOS, macOS,
/// web) drop the VAD segment entirely so the UI never advertises a
/// transmit mode the engine cannot honour.
List<ButtonSegment<rust.BridgeTransmitMode>> transmitModeSegmentsFor({
required bool voiceActivityAvailable,
}) {
if (voiceActivityAvailable) return transmitModeSegments;
return const [
ButtonSegment(
value: rust.BridgeTransmitMode.ptt,
label: Text('PTT'),
icon: Icon(Icons.radio_button_checked, size: 14),
),
ButtonSegment(
value: rust.BridgeTransmitMode.continuous,
label: Text('Always'),
icon: Icon(Icons.podcasts, size: 14),
),
];
}
/// True when this host advertises VAD transmit per DEC-030.
///
/// The desktop Silero ONNX + WebRTC fallback ships on Windows and
/// Linux; the Android Oboe + WebRTC path covers Android. iOS and
/// macOS still rely on the Apple VoiceProcessingIO unit and have no
/// Chanora-owned VAD pipeline, so the bridge cannot honour the
/// voice-activity transmit mode there.
bool get voiceActivityTransmitAvailable {
if (kIsWeb) return false;
return Platform.isWindows || Platform.isLinux || Platform.isAndroid;
}
/// Android hardware/WebRTC selector segments.
const androidProcessingSegments = [
ButtonSegment(
+32 -24
View File
@@ -5,18 +5,18 @@ packages:
dependency: transitive
description:
name: _fe_analyzer_shared
sha256: "3b19a47f6ea7c2632760777c78174f47f6aec1e05f0cd611380d4593b8af1dbc"
sha256: "8d7ff3948166b8ec5da0fbb5962000926b8e02f2ed9b3e51d1738905fbd4c98d"
url: "https://pub.dev"
source: hosted
version: "96.0.0"
version: "93.0.0"
analyzer:
dependency: transitive
description:
name: analyzer
sha256: "0c516bc4ad36a1a75759e54d5047cb9d15cded4459df01aa35a0b5ec7db2c2a0"
sha256: de7148ed2fcec579b19f122c1800933dfa028f6d9fd38a152b04b1516cec120b
url: "https://pub.dev"
source: hosted
version: "10.2.0"
version: "10.0.1"
args:
dependency: transitive
description:
@@ -133,10 +133,10 @@ packages:
dependency: transitive
description:
name: code_assets
sha256: bf394f466ba9205f1812a0433b392d6af280f155f56651eda7c18cc32ed493b8
sha256: "83ccdaa064c980b5596c35dd64a8d3ecc68620174ab9b90b6343b753aa721687"
url: "https://pub.dev"
source: hosted
version: "1.2.1"
version: "1.0.0"
collection:
dependency: transitive
description:
@@ -197,10 +197,10 @@ packages:
dependency: transitive
description:
name: dbus
sha256: "792974a4007974fbc5c1b5433eb2330a9db3e368c3f906253af4c007d0f49a91"
sha256: d0c98dcd4f5169878b6cf8f6e0a52403a9dff371a3e2f019697accbf6f44a270
url: "https://pub.dev"
source: hosted
version: "0.7.13"
version: "0.7.12"
fake_async:
dependency: transitive
description:
@@ -361,18 +361,18 @@ packages:
dependency: "direct main"
description:
name: haptic_kit
sha256: "457f825a3413be2651954639bed27bb2987570f75d90c4e8e1cb9be62db2e59d"
sha256: "39efffa513c9f8ce3cdded8a4423797f69d71c9281779b83727337f3ee1ed9b8"
url: "https://pub.dev"
source: hosted
version: "1.0.1"
version: "1.0.0"
hooks:
dependency: transitive
description:
name: hooks
sha256: "9a62a50b50b769a737bc0a8ff381f333529df3ab746b2f6b02e83760231455ba"
sha256: "025f060e86d2d4c3c47b56e33caf7f93bf9283340f26d23424ebcfccf34f621e"
url: "https://pub.dev"
source: hosted
version: "2.0.2"
version: "1.0.3"
http:
dependency: transitive
description:
@@ -509,6 +509,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "2.0.0"
native_toolchain_c:
dependency: transitive
description:
name: native_toolchain_c
sha256: "6ba77bb18063eebe9de401f5e6437e95e1438af0a87a3a39084fbd37c90df572"
url: "https://pub.dev"
source: hosted
version: "0.17.6"
nm:
dependency: transitive
description:
@@ -521,10 +529,10 @@ packages:
dependency: transitive
description:
name: objective_c
sha256: "6cb691c686fa2838c6deb34980d426145c2a5d537491cb83d463c33cdbc726ed"
sha256: "100a1c87616ab6ed41ec263b083c0ef3261ee6cd1dc3b0f35f8ddfa4f996fe52"
url: "https://pub.dev"
source: hosted
version: "9.4.1"
version: "9.3.0"
package_config:
dependency: transitive
description:
@@ -697,10 +705,10 @@ packages:
dependency: transitive
description:
name: shared_preferences_android
sha256: a2c49fc1fed7140cadd892d765bd47edbe4ac0b9c7e7e3c493dcb58126f99cf0
sha256: e8d4762b1e2e8578fc4d0fd548cebf24afd24f49719c08974df92834565e2c53
url: "https://pub.dev"
source: hosted
version: "2.4.25"
version: "2.4.23"
shared_preferences_foundation:
dependency: transitive
description:
@@ -854,10 +862,10 @@ packages:
dependency: transitive
description:
name: url_launcher_android
sha256: b413d49b73867ac08dd2f9890efd3cc11f2a0e577618d50843440a1fb3776c32
sha256: "17bc677f0b301615530dd1d67e0a9828cafa2d0b6b6eae4cd3679b7eac4a273c"
url: "https://pub.dev"
source: hosted
version: "6.3.32"
version: "6.3.30"
url_launcher_ios:
dependency: transitive
description:
@@ -966,10 +974,10 @@ packages:
dependency: transitive
description:
name: win32
sha256: ba6f4bba816c8d7e3c1580e170f3786d216951cc6b94babc3b814c08d2cb2738
sha256: a1fc9eb9248baa05dfc12ed5b66e377b3e23f095eec078e0371622b9033810d9
url: "https://pub.dev"
source: hosted
version: "6.3.0"
version: "6.2.0"
xdg_directories:
dependency: transitive
description:
@@ -982,10 +990,10 @@ packages:
dependency: transitive
description:
name: xml
sha256: "67f0aff7be013d107995e9b75bf4e7f2c3ef2dfdb2c8e68024bba0a7fd5756a4"
sha256: "971043b3a0d3da28727e40ed3e0b5d18b742fa5a68665cca88e74b7876d5e025"
url: "https://pub.dev"
source: hosted
version: "7.0.1"
version: "6.6.1"
yaml:
dependency: transitive
description:
@@ -995,5 +1003,5 @@ packages:
source: hosted
version: "3.1.3"
sdks:
dart: ">=3.12.0 <4.0.0"
flutter: ">=3.44.0"
dart: ">=3.11.5 <4.0.0"
flutter: ">=3.38.4"
@@ -13,24 +13,6 @@ void main() {
]);
});
test('gated transmit mode segments drop VAD when unsupported', () {
expect(
transmitModeSegmentsFor(voiceActivityAvailable: false).map((s) => s.value),
[rust.BridgeTransmitMode.ptt, rust.BridgeTransmitMode.continuous],
);
});
test('gated transmit mode segments include VAD when supported', () {
expect(
transmitModeSegmentsFor(voiceActivityAvailable: true).map((s) => s.value),
[
rust.BridgeTransmitMode.ptt,
rust.BridgeTransmitMode.continuous,
rust.BridgeTransmitMode.voiceActivity,
],
);
});
test('shared Android processing segments expose hardware and WebRTC', () {
expect(androidProcessingSegments.map((s) => s.value), [true, false]);
});
+3 -11
View File
@@ -1116,19 +1116,11 @@ impl ChanoraSession {
/// Configure the preferred Silero ONNX VAD model path on platforms
/// that ship the ONNX detector.
///
/// Set the Silero VAD model path on supported platforms.
///
/// This does not require an active connection. On desktop, the audio
/// engine immediately reloads the Silero ONNX worker if one is active
/// or if the model file is now available at the new path.
/// This does not require an active connection. Running non-iOS
/// audio backends can observe the model-path epoch and reload on
/// the next capture frame when Silero is selected.
pub async fn set_vad_model_path(&self, path: String) -> Result<(), CoreError> {
chanora_audio::vad::set_silero_model_path(&path)?;
let guard = self.inner.lock().await;
if let Some(state) = guard.as_ref() {
if let Some(audio) = state.audio.as_ref() {
audio.reload_audio_processing_config()?;
}
}
Ok(())
}
+1 -1
View File
@@ -51,7 +51,7 @@ coreaudio-rs = "0.14"
# on the main queue to avoid the VPIO RPC timeout on iOS simulator.
dispatch2 = "0.3"
[target.'cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))'.dependencies]
[target.'cfg(not(target_os = "ios"))'.dependencies]
ort = { version = "2.0.0-rc.12", default-features = false, features = ["load-dynamic", "ndarray", "api-24"] }
[target.'cfg(target_os = "android")'.dependencies]
+51 -11
View File
@@ -56,8 +56,10 @@ impl AudioRoute {
/// iOS voice-processing mode.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum IosVoiceProcessingMode {
/// Apple VoiceProcessingIO owns AEC/NS/AGC.
/// Shipping default: Apple VoiceProcessingIO owns AEC/NS/AGC.
PlatformVoiceProcessing,
/// Experimental raw capture-processing path.
SonoraExperimental,
}
/// Processing backend selected by policy/config.
@@ -193,20 +195,28 @@ impl AudioProcessingConfig {
"bluetooth_a2dp is output-only and cannot transmit duplex voice".to_string(),
));
}
if self.processing_backend == AudioBackend::Sonora
|| self.processing_backend == AudioBackend::WebrtcApm
|| self.aec == EffectOwner::Sonora
|| self.aec == EffectOwner::WebrtcApm
|| self.ns == EffectOwner::Sonora
|| self.ns == EffectOwner::WebrtcApm
|| self.agc == EffectOwner::Sonora
|| self.agc == EffectOwner::WebrtcApm
if self.ios_mode == IosVoiceProcessingMode::PlatformVoiceProcessing
&& (self.processing_backend == AudioBackend::Sonora
|| self.processing_backend == AudioBackend::WebrtcApm
|| self.aec == EffectOwner::Sonora
|| self.aec == EffectOwner::WebrtcApm
|| self.ns == EffectOwner::Sonora
|| self.ns == EffectOwner::WebrtcApm
|| self.agc == EffectOwner::Sonora
|| self.agc == EffectOwner::WebrtcApm)
{
return Err(AudioError::InvalidAudioProcessingConfig(
"software audio processing cannot be enabled with iOS VoiceProcessingIO"
.to_string(),
));
}
if self.ios_mode == IosVoiceProcessingMode::SonoraExperimental
&& self.processing_backend != AudioBackend::WebrtcApm
{
return Err(AudioError::InvalidAudioProcessingConfig(
"ios raw processing mode requires the WebRTC APM backend".to_string(),
));
}
Ok(())
}
@@ -252,6 +262,34 @@ mod tests {
assert!(config.validate_for_ios().is_err());
}
#[test]
fn raw_processing_allows_full_webrtc_apm_chain() {
let config = AudioProcessingConfig {
ios_mode: IosVoiceProcessingMode::SonoraExperimental,
processing_backend: AudioBackend::WebrtcApm,
aec: EffectOwner::WebrtcApm,
ns: EffectOwner::WebrtcApm,
agc: EffectOwner::WebrtcApm,
..AudioProcessingConfig::default()
};
assert!(config.validate_for_ios().is_ok());
}
#[test]
fn raw_processing_rejects_non_webrtc_apm_backend() {
let config = AudioProcessingConfig {
ios_mode: IosVoiceProcessingMode::SonoraExperimental,
processing_backend: AudioBackend::PlatformVoiceProcessing,
aec: EffectOwner::WebrtcApm,
ns: EffectOwner::WebrtcApm,
agc: EffectOwner::WebrtcApm,
..AudioProcessingConfig::default()
};
assert!(config.validate_for_ios().is_err());
}
#[test]
fn disable_failed_vad_backend_demotes_to_webrtc() {
let mut config = AudioProcessingConfig {
@@ -366,8 +404,10 @@ impl Default for SharedAudioProcessingStats {
}
impl SharedAudioProcessingStats {
/// Store the raw input dBFS level for capture paths that do not
/// update the full processing/VAD snapshot on this callback.
/// Store the raw input dBFS level (desktop capture path).
/// Mobile platforms use [`Self::update_capture`] instead, which
/// also records VAD state; this lighter method is for the cpal
/// capture path that has no VAD pipeline.
pub fn set_input_dbfs(&self, dbfs: f32) {
self.input_dbfs.store(dbfs.to_bits(), Ordering::Relaxed);
}
+9
View File
@@ -314,6 +314,15 @@ mod tests {
rec.stop();
}
#[test]
fn ios_raw_debug_wav_does_not_push_from_realtime_callback() {
let src = include_str!("ios_raw_unit.rs");
assert!(
!src.contains("push_raw_mic") && !src.contains("push_processed_mic"),
"ios raw callbacks must not call WavDebugRecorder push_*_mic until it has a preallocated handoff"
);
}
#[test]
fn wav_header_is_44_bytes() {
// Write to a temp file to test the header.
+50 -640
View File
@@ -307,8 +307,6 @@ pub struct AudioEngine {
output_muted: Arc<AtomicBool>,
audio_processing_config: Arc<Mutex<crate::AudioProcessingConfig>>,
audio_processing_stats: Arc<crate::SharedAudioProcessingStats>,
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
silero_vad_worker: Arc<Mutex<Option<crate::vad::silero_onnx::SileroOnnxVadWorker>>>,
#[cfg(not(target_os = "android"))]
audio_handler: Arc<Mutex<AudioHandler<SessionAudioId>>>,
#[cfg(target_os = "android")]
@@ -395,6 +393,8 @@ unsafe impl Sync for AudioEngine {}
#[cfg(any(target_os = "ios", target_os = "macos"))]
enum IosVoiceBackend {
Vpio(crate::ios_voice_unit::IosVoiceUnit),
#[cfg(target_os = "ios")]
Raw(crate::ios_raw_unit::IosRawUnit),
}
#[cfg(any(target_os = "ios", target_os = "macos"))]
@@ -404,12 +404,14 @@ impl IosVoiceBackend {
{
match self {
Self::Vpio(unit) => unit.restart(),
Self::Raw(unit) => unit.restart(),
}
}
#[cfg(target_os = "macos")]
{
let _ = self;
Ok(())
match self {
Self::Vpio(_unit) => Ok(()),
}
}
}
@@ -418,12 +420,14 @@ impl IosVoiceBackend {
{
match self {
Self::Vpio(unit) => unit.pause(),
Self::Raw(unit) => unit.pause(),
}
}
#[cfg(target_os = "macos")]
{
let _ = self;
Ok(())
match self {
Self::Vpio(_unit) => Ok(()),
}
}
}
@@ -432,12 +436,14 @@ impl IosVoiceBackend {
{
match self {
Self::Vpio(unit) => unit.resume(),
Self::Raw(unit) => unit.resume(),
}
}
#[cfg(target_os = "macos")]
{
let _ = self;
Ok(())
match self {
Self::Vpio(_unit) => Ok(()),
}
}
}
}
@@ -446,6 +452,27 @@ impl IosVoiceBackend {
fn open_ios_voice_backend(
params: crate::mobile_voice_backend::VoiceAudioParams,
) -> Result<IosVoiceBackend, AudioError> {
let _cfg = params.audio_processing_config.lock().unwrap().clone();
#[cfg(target_os = "ios")]
{
if _cfg.ios_mode == crate::IosVoiceProcessingMode::SonoraExperimental {
let raw_params = params.clone();
match crate::ios_raw_unit::IosRawUnit::start(raw_params) {
Ok(unit) => {
info!(target: "chanora_audio", "ios: RemoteIO/WebRTC APM backend selected");
return Ok(IosVoiceBackend::Raw(unit));
}
Err(e) => {
warn!(
target: "chanora_audio",
error = %e,
"ios: RemoteIO/WebRTC APM backend failed; falling back to VoiceProcessingIO"
);
}
}
}
}
let unit = crate::ios_voice_unit::IosVoiceUnit::start(params)?;
Ok(IosVoiceBackend::Vpio(unit))
}
@@ -714,7 +741,7 @@ impl AudioEngine {
) -> Option<cpal::Device>
where
DefaultFn: Fn(&cpal::Host) -> Option<cpal::Device>,
AllFn: Fn(&cpal::Host) -> Result<Devices, cpal::Error>,
AllFn: Fn(&cpal::Host) -> Result<Devices, cpal::DevicesError>,
Devices: IntoIterator<Item = cpal::Device>,
{
if let Some(id) = prefer {
@@ -796,8 +823,8 @@ impl AudioEngine {
let frames_received = Arc::new(AtomicU32::new(0));
let output_gain = Arc::new(AtomicU32::new(1.0_f32.to_bits()));
let output_muted = Arc::new(AtomicBool::new(false));
let (audio_processing_config, audio_processing_stats, silero_vad_worker) =
new_desktop_audio_processing_state();
let audio_processing_config = Arc::new(Mutex::new(crate::AudioProcessingConfig::default()));
let audio_processing_stats = Arc::new(crate::SharedAudioProcessingStats::default());
// ---------- Capture ----------
// Capture is best-effort. If the platform default input
@@ -811,9 +838,6 @@ impl AudioEngine {
transmit_flag_for_capture,
frames_sent.clone(),
cfg.mic_gain,
cfg.voice_activity_selector.clone(),
audio_processing_config.clone(),
silero_vad_worker.clone(),
audio_processing_stats.clone(),
);
let (input_stream, capture_active) = match capture_result {
@@ -957,13 +981,11 @@ impl AudioEngine {
Ok(Self {
transmit_gate,
frames_sent,
frames_received,
output_gain,
output_muted,
audio_processing_config,
audio_processing_stats,
silero_vad_worker,
audio_handler,
_input_stream: Mutex::new(input_stream),
_output_stream: Mutex::new(Some(output_stream)),
@@ -1645,40 +1667,15 @@ impl AudioEngine {
/// Apply a voice-processing config after validating iOS invariants.
pub fn set_audio_processing_config(
&self,
mut config: crate::AudioProcessingConfig,
config: crate::AudioProcessingConfig,
) -> Result<(), AudioError> {
#[cfg(target_os = "ios")]
config.validate_for_ios()?;
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
{
config.processing_backend = crate::AudioBackend::Noop;
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
self.apply_desktop_vad_backend(&config);
let mut guard = self.audio_processing_config.lock().unwrap();
*guard = config;
Ok(())
}
/// Re-apply the current audio processing config.
///
/// Used by the core layer to trigger VAD worker reload after a
/// model-path change (the epoch increments but the worker is only
/// reconstructed when `set_audio_processing_config` is called).
pub fn reload_audio_processing_config(&self) -> Result<(), AudioError> {
let config = self.audio_processing_config_snapshot();
self.set_audio_processing_config(config)
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
fn apply_desktop_vad_backend(&self, config: &crate::AudioProcessingConfig) {
apply_desktop_vad_backend_to_worker(
config,
&self.silero_vad_worker,
self.audio_processing_stats.as_ref(),
);
}
/// Current voice-processing stats snapshot.
pub fn audio_processing_stats(&self) -> crate::AudioProcessingStats {
let config = self.audio_processing_config.lock().unwrap().clone();
@@ -1780,369 +1777,10 @@ fn release_android_audio_mode_for_startup_rollback(
audio_mode_stack.release()
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
fn new_desktop_audio_processing_state() -> (
Arc<Mutex<crate::AudioProcessingConfig>>,
Arc<crate::SharedAudioProcessingStats>,
Arc<Mutex<Option<crate::vad::silero_onnx::SileroOnnxVadWorker>>>,
) {
let mut config = crate::AudioProcessingConfig::default();
// Desktop cpal capture does not use a platform voice-processing API.
// The global default (PlatformVoiceProcessing) is correct for iOS/macOS
// VPIO but would mislabel the desktop path in bridge diagnostics and
// set `platform_voice_processing_enabled = true` when no such
// processing exists. Override to Noop; the bridge/UI stats layer
// will then report the accurate backend.
config.processing_backend = crate::AudioBackend::Noop;
let audio_processing_config = Arc::new(Mutex::new(config.clone()));
let audio_processing_stats = Arc::new(crate::SharedAudioProcessingStats::default());
let silero_vad_worker = Arc::new(Mutex::new(None));
apply_desktop_vad_backend_to_worker(
&config,
&silero_vad_worker,
audio_processing_stats.as_ref(),
);
(
audio_processing_config,
audio_processing_stats,
silero_vad_worker,
)
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
fn apply_desktop_vad_backend_to_worker(
config: &crate::AudioProcessingConfig,
silero_vad_worker: &Arc<Mutex<Option<crate::vad::silero_onnx::SileroOnnxVadWorker>>>,
audio_processing_stats: &crate::SharedAudioProcessingStats,
) {
if config.vad_backend != crate::VadBackend::SileroOnnx {
let mut worker_guard = silero_vad_worker.lock().unwrap();
if worker_guard.is_some() {
info!(
target: "chanora_audio",
backend = config.vad_backend.as_str(),
"desktop: Silero ONNX VAD worker cleared because another VAD backend is selected"
);
}
*worker_guard = None;
audio_processing_stats.set_vad_fallback_active(false);
return;
}
// Load model and spawn worker BEFORE taking the lock so the
// realtime capture callback is not blocked on try_lock() during
// model I/O + thread spawn. The old worker (if any) is dropped
// after the new one is installed under the short lock hold.
let model_path = crate::vad::silero_model_bundle_path();
info!(
target: "chanora_audio",
path = %model_path,
"desktop: Silero ONNX VAD selected; loading model worker"
);
let new_worker = crate::vad::silero_onnx::SileroOnnxVadWorker::try_new(&model_path);
{
let mut worker_guard = silero_vad_worker.lock().unwrap();
*worker_guard = new_worker;
}
// Check result after releasing the lock. Re-acquire is cheap and
// ensures we log the correct state without holding the mutex.
let worker_installed = silero_vad_worker.lock().unwrap().is_some();
if worker_installed {
info!(
target: "chanora_audio",
path = %model_path,
"desktop: Silero ONNX VAD worker loaded"
);
audio_processing_stats.set_vad_fallback_active(false);
} else {
warn!(
target: "chanora_audio",
path = %model_path,
"desktop: Silero ONNX VAD worker unavailable; WebRTC fallback will be used"
);
audio_processing_stats.set_vad_fallback_active(true);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[test]
fn desktop_capture_voice_activity_opens_selector_from_speech() {
let gate = crate::ptt::AudioTransmitGate::new(false);
let selector = Arc::new(crate::TransmitModeSelector::new(gate.clone()));
selector.set_mode(crate::TransmitMode::VoiceActivity);
selector.set_in_channel(true);
let encoder = crate::opus_voice::new_voip_encoder("desktop VAD test").unwrap();
let (voice_out_tx, _voice_out_rx) = mpsc::channel::<OutPacket>(16);
let frames_sent = Arc::new(AtomicU32::new(0));
let voice_out_tx = crate::opus_voice::start_out_packet_worker(
voice_out_tx,
frames_sent,
"desktop-vad-test",
)
.unwrap();
let stats = Arc::new(crate::SharedAudioProcessingStats::default());
let mut capture = CaptureState::new(
encoder,
SAMPLE_RATE,
1,
1.0,
voice_out_tx,
gate.flag_arc(),
Some(selector.clone()),
Arc::new(Mutex::new(crate::AudioProcessingConfig {
vad_backend: crate::VadBackend::WebrtcVad,
..crate::AudioProcessingConfig::default()
})),
Arc::new(Mutex::new(None)),
stats.clone(),
);
let mut voiced = [0.0_f32; crate::frame::FRAME_10MS_SAMPLES];
for (idx, sample) in voiced.iter_mut().enumerate() {
let phase = idx as f32 * 2.0 * std::f32::consts::PI * 220.0 / SAMPLE_RATE as f32;
*sample = phase.sin() * 0.4;
}
for _ in 0..6 {
capture.ingest(&voiced);
}
assert!(
selector.voice_activity_open(),
"desktop capture must feed VAD and open VoiceActivity selector before transmit is already active"
);
assert!(
gate.load(),
"VoiceActivity selector should publish transmit gate"
);
let snapshot = stats.snapshot(&crate::AudioProcessingConfig::default());
assert!(snapshot.vad_active, "stats should expose active VAD");
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[test]
fn desktop_startup_audio_processing_state_applies_default_silero_fallback() {
let _guard = crate::vad::SILERO_MODEL_PATH_TEST_LOCK.lock().unwrap();
crate::vad::clear_silero_model_path_for_test();
let (config, stats, worker) = new_desktop_audio_processing_state();
assert_eq!(
config.lock().unwrap().vad_backend,
crate::VadBackend::SileroOnnx,
"desktop startup config should keep the default Silero backend selected"
);
assert!(
worker.lock().unwrap().is_none(),
"missing startup model should not create an ONNX worker"
);
let snapshot = stats.snapshot(&config.lock().unwrap());
assert!(
snapshot.vad_fallback_active,
"desktop startup should mark WebRTC fallback active when the default Silero worker cannot load"
);
crate::vad::clear_silero_model_path_for_test();
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[test]
fn desktop_set_audio_processing_config_normalizes_default_backend_to_noop() {
let audio_processing_config = Arc::new(Mutex::new(crate::AudioProcessingConfig::default()));
let audio_processing_stats = Arc::new(crate::SharedAudioProcessingStats::default());
let engine = AudioEngine {
transmit_gate: crate::ptt::AudioTransmitGate::new(false),
frames_sent: Arc::new(AtomicU32::new(0)),
frames_received: Arc::new(AtomicU32::new(0)),
output_gain: Arc::new(AtomicU32::new(1.0_f32.to_bits())),
output_muted: Arc::new(AtomicBool::new(false)),
audio_processing_config: audio_processing_config.clone(),
audio_processing_stats,
silero_vad_worker: Arc::new(Mutex::new(None)),
audio_handler: Arc::new(Mutex::new(AudioHandler::new())),
_input_stream: Mutex::new(None),
_output_stream: Mutex::new(None),
shutdown_tx: None,
capture_active: false,
};
engine
.set_audio_processing_config(crate::AudioProcessingConfig::default())
.unwrap();
assert_eq!(
engine.audio_processing_config_snapshot().processing_backend,
crate::AudioBackend::Noop,
"desktop setter should report the actual no-op processing backend for default configs"
);
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[test]
fn desktop_capture_silero_selection_reports_fallback_when_worker_unavailable() {
let _guard = crate::vad::SILERO_MODEL_PATH_TEST_LOCK.lock().unwrap();
crate::vad::clear_silero_model_path_for_test();
let gate = crate::ptt::AudioTransmitGate::new(false);
let selector = Arc::new(crate::TransmitModeSelector::new(gate.clone()));
selector.set_mode(crate::TransmitMode::VoiceActivity);
selector.set_in_channel(true);
let encoder = crate::opus_voice::new_voip_encoder("desktop Silero fallback test").unwrap();
let (voice_out_tx, _voice_out_rx) = mpsc::channel::<OutPacket>(16);
let frames_sent = Arc::new(AtomicU32::new(0));
let voice_out_tx = crate::opus_voice::start_out_packet_worker(
voice_out_tx,
frames_sent,
"desktop-silero-fallback-test",
)
.unwrap();
let stats = Arc::new(crate::SharedAudioProcessingStats::default());
let config = Arc::new(Mutex::new(crate::AudioProcessingConfig {
vad_backend: crate::VadBackend::SileroOnnx,
..crate::AudioProcessingConfig::default()
}));
let mut capture = CaptureState::new(
encoder,
SAMPLE_RATE,
1,
1.0,
voice_out_tx,
gate.flag_arc(),
Some(selector),
config.clone(),
Arc::new(Mutex::new(None)),
stats.clone(),
);
let mut voiced = [0.0_f32; crate::frame::FRAME_10MS_SAMPLES];
for (idx, sample) in voiced.iter_mut().enumerate() {
let phase = idx as f32 * 2.0 * std::f32::consts::PI * 220.0 / SAMPLE_RATE as f32;
*sample = phase.sin() * 0.4;
}
capture.ingest(&voiced);
let snapshot = stats.snapshot(&config.lock().unwrap());
assert_eq!(snapshot.vad_backend, crate::VadBackend::SileroOnnx);
assert!(
snapshot.vad_fallback_active,
"desktop Silero selection should make WebRTC fallback visible when ONNX worker cannot load"
);
crate::vad::clear_silero_model_path_for_test();
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[test]
fn desktop_capture_vad_consumes_only_new_pcm_while_transmitting() {
let gate = crate::ptt::AudioTransmitGate::new(true);
let selector = Arc::new(crate::TransmitModeSelector::new(gate.clone()));
selector.set_mode(crate::TransmitMode::VoiceActivity);
selector.set_in_channel(true);
let encoder = crate::opus_voice::new_voip_encoder("desktop VAD duplicate test").unwrap();
let (voice_out_tx, _voice_out_rx) = mpsc::channel::<OutPacket>(16);
let frames_sent = Arc::new(AtomicU32::new(0));
let voice_out_tx = crate::opus_voice::start_out_packet_worker(
voice_out_tx,
frames_sent,
"desktop-vad-duplicate-test",
)
.unwrap();
let stats = Arc::new(crate::SharedAudioProcessingStats::default());
let mut capture = CaptureState::new(
encoder,
SAMPLE_RATE,
1,
1.0,
voice_out_tx,
gate.flag_arc(),
Some(selector),
Arc::new(Mutex::new(crate::AudioProcessingConfig {
vad_backend: crate::VadBackend::Disabled,
..crate::AudioProcessingConfig::default()
})),
Arc::new(Mutex::new(None)),
stats,
);
capture.pcm_accum.extend(std::iter::repeat_n(
0.1_f32,
crate::frame::FRAME_10MS_SAMPLES + 240,
));
capture.pending_10ms[..240].fill(0.1);
capture.pending_10ms_len = 240;
capture.capture_frame_seq = 1;
capture.pcm_accum.extend(std::iter::repeat_n(0.2_f32, 240));
capture.process_pending_vad_frames(crate::frame::FRAME_10MS_SAMPLES + 240);
assert_eq!(
capture.capture_frame_seq, 2,
"VAD should consume only the 240 samples appended by the current ingest and complete one pending 10 ms frame"
);
}
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[test]
fn desktop_capture_silero_stale_enqueued_worker_uses_fallback() {
let gate = crate::ptt::AudioTransmitGate::new(false);
let selector = Arc::new(crate::TransmitModeSelector::new(gate.clone()));
selector.set_mode(crate::TransmitMode::VoiceActivity);
selector.set_in_channel(true);
let encoder =
crate::opus_voice::new_voip_encoder("desktop stale Silero fallback test").unwrap();
let (voice_out_tx, _voice_out_rx) = mpsc::channel::<OutPacket>(16);
let frames_sent = Arc::new(AtomicU32::new(0));
let voice_out_tx = crate::opus_voice::start_out_packet_worker(
voice_out_tx,
frames_sent,
"desktop-stale-silero-fallback-test",
)
.unwrap();
let stats = Arc::new(crate::SharedAudioProcessingStats::default());
let config = Arc::new(Mutex::new(crate::AudioProcessingConfig {
vad_backend: crate::VadBackend::SileroOnnx,
..crate::AudioProcessingConfig::default()
}));
let worker = Arc::new(Mutex::new(Some(
crate::vad::silero_onnx::SileroOnnxVadWorker::stale_test_worker(),
)));
let mut capture = CaptureState::new(
encoder,
SAMPLE_RATE,
1,
1.0,
voice_out_tx,
gate.flag_arc(),
Some(selector),
config.clone(),
worker,
stats.clone(),
);
let mut voiced = [0.0_f32; crate::frame::FRAME_10MS_SAMPLES];
for (idx, sample) in voiced.iter_mut().enumerate() {
let phase = idx as f32 * 2.0 * std::f32::consts::PI * 220.0 / SAMPLE_RATE as f32;
*sample = phase.sin() * 0.4;
}
capture.process_10ms_capture_frame(&voiced);
let snapshot = stats.snapshot(&config.lock().unwrap());
assert!(
snapshot.vad_fallback_active,
"stale Silero worker output should report active WebRTC fallback"
);
assert!(
snapshot.vad_probability > 0.5,
"stale Silero worker output should use WebRTC fallback probability instead of forced silence"
);
}
#[test]
fn android_startup_rollback_releases_acquired_mode_snapshot() {
let mut stack = crate::mode_stack::ModeStack::new();
@@ -2168,9 +1806,6 @@ fn try_open_capture(
transmit_active: Arc<AtomicBool>,
frames_sent: Arc<AtomicU32>,
mic_gain: f32,
voice_activity_selector: Option<Arc<crate::TransmitModeSelector>>,
audio_processing_config: Arc<Mutex<crate::AudioProcessingConfig>>,
silero_vad_worker: Arc<Mutex<Option<crate::vad::silero_onnx::SileroOnnxVadWorker>>>,
audio_processing_stats: Arc<crate::SharedAudioProcessingStats>,
) -> Result<cpal::Stream, AudioError> {
let in_cfg = in_dev
@@ -2210,9 +1845,6 @@ fn try_open_capture(
"cpal-capture",
)?,
transmit_active,
voice_activity_selector,
audio_processing_config,
silero_vad_worker,
audio_processing_stats,
)));
@@ -2251,17 +1883,6 @@ struct CaptureState {
/// The PTT transmission gate. Read once per outbound frame; the
/// CaptureState never mutates this flag.
transmit_active: Arc<AtomicBool>,
voice_activity_selector: Option<Arc<crate::TransmitModeSelector>>,
vad_detector: crate::vad::WebRtcFallbackVad,
silero_vad_worker: Arc<Mutex<Option<crate::vad::silero_onnx::SileroOnnxVadWorker>>>,
silero_model_epoch: u64,
current_vad_backend: crate::VadBackend,
fallback_warned_backend: Option<crate::VadBackend>,
capture_frame_seq: u64,
vad_state: crate::voice_activity::VoiceActivityStateMachine,
audio_processing_config: Arc<Mutex<crate::AudioProcessingConfig>>,
pending_10ms: [f32; crate::frame::FRAME_10MS_SAMPLES],
pending_10ms_len: usize,
/// Pre-allocated mono downmix buffer. Resized in-place each
/// callback; `clear()` retains capacity. SDD-094 realtime-thread
/// invariant: this avoids the heap allocation that the prior fix
@@ -2305,9 +1926,6 @@ impl CaptureState {
mic_gain: f32,
voice_out_tx: crate::opus_voice::EncodedVoiceFrameSender,
transmit_active: Arc<AtomicBool>,
voice_activity_selector: Option<Arc<crate::TransmitModeSelector>>,
audio_processing_config: Arc<Mutex<crate::AudioProcessingConfig>>,
silero_vad_worker: Arc<Mutex<Option<crate::vad::silero_onnx::SileroOnnxVadWorker>>>,
audio_processing_stats: Arc<crate::SharedAudioProcessingStats>,
) -> Self {
Self {
@@ -2321,17 +1939,6 @@ impl CaptureState {
opus_out: [0u8; crate::opus_voice::MAX_OPUS_FRAME],
voice_out_tx,
transmit_active,
voice_activity_selector,
vad_detector: crate::vad::WebRtcFallbackVad::default(),
silero_vad_worker,
silero_model_epoch: crate::vad::silero_model_epoch(),
current_vad_backend: crate::VadBackend::Disabled,
fallback_warned_backend: None,
capture_frame_seq: 0,
vad_state: crate::voice_activity::VoiceActivityStateMachine::default(),
audio_processing_config,
pending_10ms: [0.0; crate::frame::FRAME_10MS_SAMPLES],
pending_10ms_len: 0,
mono_scratch: Vec::with_capacity(4096),
frame_scratch: Vec::with_capacity(FRAME_SAMPLES),
audio_processing_stats,
@@ -2345,16 +1952,6 @@ impl CaptureState {
/// Consume an arbitrary-rate, multichannel cpal buffer; produce
/// 48 kHz mono frames; encode and send when `transmit_active`
/// is true (PTT engaged).
// TODO(realtime-audio): This method runs on the cpal audio callback
// thread with a ~10 ms deadline. Known pre-existing violations of the
// realtime safety constraint that should be addressed in a future
// iteration:
// 1. Blocking Mutex::lock().unwrap() on self (via cpal callback)
// 2. Potential allocation in mono_scratch.reserve() and
// pcm_accum.extend_from_slice() when buffer capacity is exceeded
// 3. Encode-path warn!/error! logging via send_voip_frame closures
// These were present before the VAD integration and are not
// introduced by this changeset.
fn ingest<T: ToF32 + Copy>(&mut self, buf: &[T]) {
// 1. Down-mix to mono (pre-gain). Always performed so the level
// meter reflects real mic input even when PTT is released.
@@ -2384,11 +1981,15 @@ impl CaptureState {
}
}
if !self.transmit_active.load(Ordering::Relaxed) {
self.pcm_accum.clear();
return;
}
// 2. Resample to 48 kHz if needed. We re-borrow
// `mono_scratch` as a shared slice per branch to satisfy
// the borrow checker against `&mut self` on the
// resample path.
let vad_start_offset = self.pcm_accum.len();
if self.in_sample_rate == SAMPLE_RATE {
// Disjoint-borrow: copy the slice into pcm_accum without
// aliasing &mut self.
@@ -2407,13 +2008,6 @@ impl CaptureState {
self.mono_scratch = mono;
}
self.process_pending_vad_frames(vad_start_offset);
if !self.transmit_active.load(Ordering::Relaxed) {
self.pcm_accum.clear();
return;
}
// 3. Encode any complete frames. Clamp each sample to
// [-1.0, 1.0] before handing to libopus's float encoder —
// out-of-range samples are hard-clipped inside libopus,
@@ -2461,179 +2055,6 @@ impl CaptureState {
}
}
fn process_pending_vad_frames(&mut self, start_offset: usize) {
let mut offset = start_offset.min(self.pcm_accum.len());
while offset < self.pcm_accum.len() {
let remaining = crate::frame::FRAME_10MS_SAMPLES - self.pending_10ms_len;
let take = remaining.min(self.pcm_accum.len() - offset);
self.pending_10ms[self.pending_10ms_len..self.pending_10ms_len + take]
.copy_from_slice(&self.pcm_accum[offset..offset + take]);
self.pending_10ms_len += take;
offset += take;
if self.pending_10ms_len == crate::frame::FRAME_10MS_SAMPLES {
let frame = self.pending_10ms;
self.process_10ms_capture_frame(&frame);
self.pending_10ms_len = 0;
}
}
}
fn mark_vad_fallback_active(&mut self, failed_backend: crate::VadBackend) {
// Realtime callback: do not log here. Publish state via atomics
// and let a non-realtime consumer translate transitions into
// info/warn events. The transmit/diagnostic stats stream
// already exposes vad_fallback_active for this purpose.
self.fallback_warned_backend = Some(failed_backend);
}
fn sync_vad_backend(&mut self, voice_activity_mode: bool, vad_backend: crate::VadBackend) {
if !voice_activity_mode {
self.current_vad_backend = crate::VadBackend::Disabled;
self.fallback_warned_backend = None;
self.audio_processing_stats.set_vad_fallback_active(false);
return;
}
let silero_epoch = crate::vad::silero_model_epoch();
let silero_changed =
vad_backend == crate::VadBackend::SileroOnnx && silero_epoch != self.silero_model_epoch;
if vad_backend == self.current_vad_backend && !silero_changed {
return;
}
self.current_vad_backend = vad_backend;
self.silero_model_epoch = silero_epoch;
self.fallback_warned_backend = None;
self.vad_state.reset();
match vad_backend {
crate::VadBackend::SileroOnnx => {
let worker_available = self
.silero_vad_worker
.try_lock()
.map(|worker| worker.is_some())
.unwrap_or(false);
if worker_available {
self.audio_processing_stats.set_vad_fallback_active(false);
} else {
self.mark_vad_fallback_active(crate::VadBackend::SileroOnnx);
self.audio_processing_stats.set_vad_fallback_active(true);
}
}
crate::VadBackend::WebrtcVad => {
self.audio_processing_stats.set_vad_fallback_active(false);
}
crate::VadBackend::EnergyDebug => {
self.audio_processing_stats.set_vad_fallback_active(true);
}
crate::VadBackend::Disabled => {
self.audio_processing_stats.set_vad_fallback_active(false);
}
}
}
fn process_10ms_capture_frame(&mut self, frame: &[f32; crate::frame::FRAME_10MS_SAMPLES]) {
let input_dbfs = crate::frame::dbfs(frame);
let (vad_backend, vad_hangover) = self
.audio_processing_config
.try_lock()
.map(|cfg| (cfg.vad_backend, cfg.vad_hangover_ms))
.unwrap_or((
crate::VadBackend::WebrtcVad,
crate::voice_activity::VAD_HANGOVER_MS,
));
let voice_activity_mode = self
.voice_activity_selector
.as_ref()
.map(|selector| selector.mode() == crate::TransmitMode::VoiceActivity)
.unwrap_or(false);
if voice_activity_mode {
self.sync_vad_backend(true, vad_backend);
self.vad_state.configure(
crate::voice_activity::VAD_OPEN_AFTER_MS,
vad_hangover,
crate::voice_activity::VAD_MIN_TX_MS,
);
} else {
self.sync_vad_backend(false, vad_backend);
}
let (vad_probability, gate_open, used_fallback_vad) = if voice_activity_mode {
self.capture_frame_seq = self.capture_frame_seq.wrapping_add(1);
let capture_seq = self.capture_frame_seq;
let mut used_fallback_vad = false;
let vad = match vad_backend {
crate::VadBackend::Disabled => crate::vad::VadOutput {
probability: 1.0,
speech: true,
},
crate::VadBackend::SileroOnnx => {
// Single `try_lock` that both probes availability and
// sends the frame. The guard is scoped to the block
// so it drops before the fallback path (which needs
// `&mut self` for `mark_vad_fallback_active`).
// Returns Some(VadOutput) on a successful, non-stale
// send; None means "fall back to WebRTC VAD".
let worker_output = {
let guard = self.silero_vad_worker.try_lock().ok();
guard.and_then(|guard| {
let worker = guard.as_ref()?;
if worker.try_send(capture_seq, frame) && !worker.is_stale(capture_seq)
{
let p = worker.latest_probability();
Some(crate::vad::VadOutput {
probability: p,
speech: p >= 0.5,
})
} else {
None
}
})
};
if let Some(output) = worker_output {
output
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
frame,
)
}
}
crate::VadBackend::WebrtcVad | crate::VadBackend::EnergyDebug => {
used_fallback_vad = vad_backend == crate::VadBackend::EnergyDebug;
crate::vad::VoiceActivityDetector::process_10ms(&mut self.vad_detector, frame)
}
};
(
vad.probability,
self.vad_state.update(vad.speech),
used_fallback_vad,
)
} else {
(0.0, false, false)
};
self.audio_processing_stats
.set_vad_fallback_active(used_fallback_vad);
let vad_active = voice_activity_mode && gate_open;
if let Some(selector) = &self.voice_activity_selector {
selector.set_voice_activity_open(vad_active);
}
self.audio_processing_stats.update_capture(
input_dbfs,
input_dbfs,
vad_probability,
vad_active,
self.transmit_active.load(Ordering::Relaxed),
);
self.audio_processing_stats
.record_capture_frame(frame.iter().all(|sample| sample.abs() <= 0.000_001));
}
/// Simple linear resampler for `in_sample_rate → 48000`.
///
/// The resampler maintains continuity across cpal buffer
@@ -2723,8 +2144,8 @@ where
{
let stream = device
.build_input_stream(
*config,
move |data: &[T], _: &cpal::InputCallbackInfo| {
config,
move |data: &[T], _| {
let mut s = state.lock().unwrap();
s.ingest(data);
},
@@ -2790,8 +2211,8 @@ where
.unwrap_or_else(std::time::Instant::now);
let stream = device
.build_output_stream(
*config,
move |out: &mut [T], _: &cpal::OutputCallbackInfo| {
config,
move |out: &mut [T], _| {
let cb_start = std::time::Instant::now();
let muted = output_muted.load(Ordering::Relaxed);
let dev_frames = out.len() / dev_channels.max(1);
@@ -3199,25 +2620,14 @@ pub mod bench_seam {
let (tx, rx) = mpsc::channel::<OutPacket>(64);
let transmit_active = Arc::new(AtomicBool::new(true));
let frames_sent = Arc::new(AtomicU32::new(0));
// Bridge the bench's private mpsc<OutPacket> to the
// realtime-thread-safe EncodedVoiceFrameSender that
// CaptureState expects. The worker task forwards
// encoded Opus frames to `tx` via `frames_sent`.
let voice_out_tx =
crate::opus_voice::start_out_packet_worker(tx, frames_sent, "cpal-bench")
.expect("start_out_packet_worker");
let state = CaptureState::new(
encoder,
in_sample_rate,
in_channels,
1.0,
voice_out_tx,
tx,
transmit_active.clone(),
None,
Arc::new(std::sync::Mutex::new(
crate::AudioProcessingConfig::default(),
)),
Arc::new(std::sync::Mutex::new(None)),
frames_sent,
Arc::new(crate::SharedAudioProcessingStats::default()),
);
Self {
+538
View File
@@ -0,0 +1,538 @@
//! Optional raw iOS RemoteIO path for the WebRTC APM experimental mode.
//!
//! Provides an alternative to `ios_voice_unit.rs` for the
//! `SonoraExperimental` processing mode. Instead of
//! `kAudioUnitSubType_VoiceProcessingIO` (which owns AEC/NS/AGC), it
//! opens `kAudioUnitSubType_RemoteIO` with voice processing explicitly
//! disabled so WebRTC APM can own the full signal path.
//!
//! ## Hard invariants enforced here
//!
//! * INV_009: Rust AEC only active when platform AEC is disabled.
//! * INV_010: VoiceProcessingIO and WebRTC APM AEC are mutually exclusive.
//! * INV_011: Software AEC backend receives both capture and render-reference.
//! * INV_012: Render reference is copied from decoded/mixed remote PCM
//! before playout.
//!
//! ## Fallback
//!
//! If RemoteIO construction fails, the caller falls back to `IosVoiceUnit`
//! (VPIO) and logs the error.
//!
//! ## Status
//!
//! Experimental / disabled by default. Only activated when the user
//! explicitly selects `SonoraExperimental` mode via the bridge API.
//!
//! ## Platform
//!
//! `kAudioUnitSubType_RemoteIO` is only available in the iOS SDK.
//! This module is gated to `target_os = "ios"`.
#[cfg(target_os = "ios")]
pub use inner::IosRawUnit;
#[cfg(target_os = "ios")]
mod inner {
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::{Arc, Mutex};
use audiopus::coder::Encoder as OpusEncoder;
use coreaudio::audio_unit::audio_format::LinearPcmFlags;
use coreaudio::audio_unit::render_callback::{self, data};
use coreaudio::audio_unit::IOType;
use coreaudio::audio_unit::{AudioUnit, Element, SampleFormat, Scope, StreamFormat};
use tracing::{info, warn};
use crate::mobile_voice_backend::VoiceAudioParams;
use crate::processor::AudioProcessor;
use crate::AudioError;
const SAMPLE_RATE_HZ: f64 = 48_000.0;
// ------------------------------------------------------------------ //
// Render-reference ring buffer //
// ------------------------------------------------------------------ //
/// 4-slot ring buffer shared between the render callback (writer) and
/// the capture callback (reader for Sonora AEC3). Capacity: 4 × 10 ms
/// = 40 ms of headroom.
///
/// If the capture callback runs before the render callback has written
/// a frame it reads zeros (silence reference), which is safe — Sonora
/// AEC3 simply skips cancellation for that frame.
type RenderReferenceBuffer = crate::render_reference::RenderReferenceBuffer<480, 4>;
type RenderReferenceFrameAccumulator =
crate::render_reference::RenderReferenceFrameAccumulator<480>;
const RAW_RENDER_SCRATCH_FRAMES: usize = 1024;
// ------------------------------------------------------------------ //
// Capture pipeline state //
// ------------------------------------------------------------------ //
struct RawCaptureState {
encoder: OpusEncoder,
pcm_accum: Vec<i16>,
opus_out: [u8; crate::opus_voice::MAX_OPUS_FRAME],
voice_out_tx: crate::opus_voice::EncodedVoiceFrameSender,
transmit_active: Arc<AtomicBool>,
output_muted: Arc<AtomicBool>,
mic_gain: f32,
voice_activity_selector: Option<Arc<crate::TransmitModeSelector>>,
vad_detector: crate::vad::WebRtcFallbackVad,
silero_coreml_worker: Option<crate::vad::apple_coreml::AppleCoreMlVadWorker>,
current_vad_backend: crate::VadBackend,
capture_frame_seq: u64,
vad_state: crate::voice_activity::VoiceActivityStateMachine,
/// Processing config — retained for route-change reloads.
audio_processing_config: Arc<Mutex<crate::AudioProcessingConfig>>,
webrtc_apm_processor: crate::processor::WebRtcApmProcessor,
audio_processing_stats: Arc<crate::SharedAudioProcessingStats>,
render_reference: Arc<RenderReferenceBuffer>,
pending_10ms: [i16; crate::frame::FRAME_10MS_SAMPLES],
pending_10ms_len: usize,
fallback_warned_backend: Option<crate::VadBackend>,
}
impl RawCaptureState {
fn new(
params: &VoiceAudioParams,
render_reference: Arc<RenderReferenceBuffer>,
) -> Result<Self, AudioError> {
let encoder = crate::opus_voice::new_voip_encoder("ios raw")?;
let webrtc_apm_config = params
.audio_processing_config
.lock()
.map(|cfg| crate::processor::webrtc_apm::WebRtcApmConfig::from_audio_config(&cfg))
.unwrap_or_default();
Ok(Self {
encoder,
pcm_accum: Vec::with_capacity(crate::frame::FRAME_20MS_SAMPLES * 2),
opus_out: [0u8; crate::opus_voice::MAX_OPUS_FRAME],
voice_out_tx: crate::opus_voice::start_out_packet_worker(
params.voice_out_tx.clone(),
params.frames_sent.clone(),
"ios-raw",
)?,
transmit_active: params.transmit_active.clone(),
output_muted: params.output_muted.clone(),
mic_gain: params.mic_gain,
voice_activity_selector: params.voice_activity_selector.clone(),
vad_detector: crate::vad::WebRtcFallbackVad::default(),
silero_coreml_worker: None,
current_vad_backend: crate::VadBackend::WebrtcVad,
capture_frame_seq: 0,
vad_state: crate::voice_activity::VoiceActivityStateMachine::default(),
audio_processing_config: params.audio_processing_config.clone(),
webrtc_apm_processor: crate::processor::WebRtcApmProcessor::with_config(
webrtc_apm_config,
)?,
audio_processing_stats: params.audio_processing_stats.clone(),
render_reference,
pending_10ms: [0_i16; crate::frame::FRAME_10MS_SAMPLES],
pending_10ms_len: 0,
fallback_warned_backend: None,
})
}
fn mark_vad_fallback_active(&mut self, failed_backend: crate::VadBackend) {
if self.fallback_warned_backend != Some(failed_backend) {
self.fallback_warned_backend = Some(failed_backend);
if self.capture_frame_seq < 128 {
tracing::info!(
target: "chanora_audio",
backend = failed_backend.as_str(),
seq = self.capture_frame_seq,
"VAD backend warming up; using WebRTC fallback"
);
} else {
tracing::warn!(
target: "chanora_audio",
backend = failed_backend.as_str(),
"VAD backend unavailable; using WebRTC fallback for runtime detection"
);
}
}
}
fn ingest_i16(&mut self, samples: &[i16]) {
// Accumulate into 10 ms frames for VAD / Sonora processing.
let mut offset = 0;
while offset < samples.len() {
let remaining = crate::frame::FRAME_10MS_SAMPLES - self.pending_10ms_len;
let take = remaining.min(samples.len() - offset);
self.pending_10ms[self.pending_10ms_len..self.pending_10ms_len + take]
.copy_from_slice(&samples[offset..offset + take]);
self.pending_10ms_len += take;
offset += take;
if self.pending_10ms_len == crate::frame::FRAME_10MS_SAMPLES {
let frame = self.pending_10ms;
self.process_10ms_capture_frame(&frame);
self.encode_complete_20ms_frames();
self.pending_10ms_len = 0;
}
}
if !self.transmit_active.load(Ordering::Relaxed) {
self.pcm_accum.clear();
return;
}
self.encode_complete_20ms_frames();
}
fn encode_complete_20ms_frames(&mut self) {
while self.pcm_accum.len() >= crate::frame::FRAME_20MS_SAMPLES {
let mut frame = [0i16; crate::frame::FRAME_20MS_SAMPLES];
frame.copy_from_slice(&self.pcm_accum[..crate::frame::FRAME_20MS_SAMPLES]);
self.pcm_accum.drain(..crate::frame::FRAME_20MS_SAMPLES);
match self.encoder.encode(&frame, &mut self.opus_out[..]) {
Ok(len) => {
crate::opus_voice::send_voip_frame(
&self.voice_out_tx,
&self.opus_out,
len,
|| {
warn!(
target: "chanora_audio",
"ios raw: voice_out queue full; dropping frame"
);
},
|| {},
);
}
Err(e) => {
tracing::error!(target: "chanora_audio",
error = %e, "ios raw opus encode failed");
}
}
}
}
fn process_10ms_capture_frame(
&mut self,
samples: &[i16; crate::frame::FRAME_10MS_SAMPLES],
) {
let mut frame = [0.0_f32; crate::frame::FRAME_10MS_SAMPLES];
for (dst, src) in frame.iter_mut().zip(samples.iter().copied()) {
*dst = crate::frame::i16_to_f32(src);
}
let input_dbfs = crate::frame::dbfs(&frame);
// Debug WAV mic taps are intentionally unavailable on iOS raw
// realtime callbacks until WavDebugRecorder supports a
// preallocated handoff path; the current recorder push path
// allocates per frame.
// Feed render reference to WebRTC APM before capture so AEC can adapt.
let render_ref = self.render_reference.read_latest();
self.webrtc_apm_processor.process_render(&render_ref);
self.webrtc_apm_processor.process_capture(&mut frame);
// Processed-mic debug WAV capture is disabled for the same
// realtime allocation reason as the raw-mic tap above.
let voice_activity_mode = self
.voice_activity_selector
.as_ref()
.map(|selector| selector.mode() == crate::TransmitMode::VoiceActivity)
.unwrap_or(false);
if !voice_activity_mode {
self.silero_coreml_worker = None;
self.current_vad_backend = crate::VadBackend::Disabled;
self.fallback_warned_backend = None;
self.audio_processing_stats.set_vad_fallback_active(false);
}
let (vad_backend, vad_hangover) = self
.audio_processing_config
.try_lock()
.map(|cfg| (cfg.vad_backend, cfg.vad_hangover_ms))
.unwrap_or((
crate::VadBackend::WebrtcVad,
crate::voice_activity::VAD_HANGOVER_MS,
));
if voice_activity_mode {
self.vad_state.configure(
crate::voice_activity::VAD_OPEN_AFTER_MS,
vad_hangover,
crate::voice_activity::VAD_MIN_TX_MS,
);
}
if voice_activity_mode && vad_backend != self.current_vad_backend {
self.current_vad_backend = vad_backend;
self.fallback_warned_backend = None;
if vad_backend == crate::VadBackend::SileroOnnx {
self.silero_coreml_worker = None;
self.mark_vad_fallback_active(crate::VadBackend::SileroOnnx);
self.audio_processing_stats.set_vad_fallback_active(true);
} else {
self.silero_coreml_worker = None;
self.audio_processing_stats.set_vad_fallback_active(false);
}
self.vad_state.reset();
}
let (vad_probability, active) = if voice_activity_mode {
self.capture_frame_seq = self.capture_frame_seq.wrapping_add(1);
let capture_seq = self.capture_frame_seq;
let mut used_fallback_vad = false;
let vad = if vad_backend == crate::VadBackend::Disabled {
crate::vad::VadOutput {
probability: 1.0,
speech: true,
}
} else if vad_backend == crate::VadBackend::SileroOnnx {
match crate::vad::callback_vad_worker_policy(
voice_activity_mode,
vad_backend,
self.silero_coreml_worker.is_some(),
) {
crate::vad::VadWorkerPolicy::UseWorker => {
let worker = self
.silero_coreml_worker
.as_ref()
.expect("policy checked worker");
let enqueued = worker.try_send(capture_seq, &frame);
if !worker.is_stale(capture_seq) {
let p = worker.latest_probability();
crate::vad::VadOutput {
probability: p,
speech: p >= 0.5,
}
} else if enqueued {
crate::vad::VadOutput {
probability: 0.0,
speech: false,
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
&frame,
)
}
}
crate::vad::VadWorkerPolicy::UseFallback => {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
&frame,
)
}
crate::vad::VadWorkerPolicy::NotModelBacked => crate::vad::VadOutput {
probability: 1.0,
speech: true,
},
}
} else {
crate::vad::VoiceActivityDetector::process_10ms(&mut self.vad_detector, &frame)
};
self.audio_processing_stats
.set_vad_fallback_active(used_fallback_vad);
(vad.probability, self.vad_state.update(vad.speech))
} else {
(0.0, false)
};
if let Some(sel) = &self.voice_activity_selector {
sel.set_voice_activity_open(voice_activity_mode && active);
}
self.audio_processing_stats.update_capture(
input_dbfs,
crate::frame::dbfs(&frame),
vad_probability,
voice_activity_mode && active,
self.transmit_active.load(Ordering::Relaxed),
);
if !self.transmit_active.load(Ordering::Relaxed) {
return;
}
if crate::capture_accumulator::append_processed_i16_bounded(
&mut self.pcm_accum,
&frame,
self.mic_gain,
) {
self.audio_processing_stats.increment_callback_xrun();
}
}
}
// ------------------------------------------------------------------ //
// IosRawUnit //
// ------------------------------------------------------------------ //
/// Raw iOS RemoteIO audio unit for the Sonora experimental path.
pub struct IosRawUnit {
unit: AudioUnit,
}
impl IosRawUnit {
/// Open a RemoteIO AudioUnit, install render + input callbacks, start.
pub(crate) fn start(params: VoiceAudioParams) -> Result<Self, AudioError> {
// INV_010: reject if config requests VPIO (that's IosVoiceUnit's job).
{
let cfg = params.audio_processing_config.lock().unwrap();
if cfg.ios_mode == crate::IosVoiceProcessingMode::PlatformVoiceProcessing {
return Err(AudioError::InvalidAudioProcessingConfig(
"IosRawUnit requires raw WebRTC APM mode".to_string(),
));
}
}
let mut unit = AudioUnit::new_uninitialized(IOType::RemoteIO)
.map_err(|e| AudioError::Backend(format!("remoteio new: {e}")))?;
// Enable input on bus 1.
const ENABLE_IO: u32 = 2003;
let enable: u32 = 1;
unit.set_property(ENABLE_IO, Scope::Input, Element::Input, Some(&enable))
.map_err(|e| AudioError::Backend(format!("remoteio enable input: {e}")))?;
// 48 kHz Int16 mono on both buses.
let fmt = StreamFormat {
sample_rate: SAMPLE_RATE_HZ,
sample_format: SampleFormat::I16,
flags: LinearPcmFlags::IS_SIGNED_INTEGER | LinearPcmFlags::IS_PACKED,
channels: 1,
};
unit.set_stream_format(fmt, Scope::Input, Element::Output)
.map_err(|e| AudioError::StreamConfig(format!("remoteio fmt output: {e}")))?;
unit.set_stream_format(fmt, Scope::Output, Element::Input)
.map_err(|e| AudioError::StreamConfig(format!("remoteio fmt input: {e}")))?;
// Shared render-reference buffer (INV_011 / INV_012).
let render_ref_buf = RenderReferenceBuffer::new();
let render_ref_for_capture = render_ref_buf.clone();
let mut capture_state = RawCaptureState::new(&params, render_ref_for_capture)?;
unit.set_input_callback(move |args: render_callback::Args<data::Interleaved<i16>>| {
capture_state.ingest_i16(args.data.buffer);
Ok(())
})
.map_err(|e| AudioError::Backend(format!("remoteio input cb: {e}")))?;
let mut scratch = [0.0_f32; RAW_RENDER_SCRATCH_FRAMES * 2];
let mut mono = [0.0_f32; RAW_RENDER_SCRATCH_FRAMES];
let mut render_ref_accum = RenderReferenceFrameAccumulator::new();
let handler = params.handler.clone();
let output_gain = params.output_gain.clone();
let output_muted = params.output_muted.clone();
let stats_render = params.audio_processing_stats.clone();
unit.set_render_callback(move |args: render_callback::Args<data::Interleaved<i16>>| {
let out = args.data.buffer;
let n = out.len();
let process_n = n.min(RAW_RENDER_SCRATCH_FRAMES);
let stereo_n = process_n * 2;
if n > RAW_RENDER_SCRATCH_FRAMES {
stats_render.increment_callback_xrun();
}
scratch[..stereo_n].fill(0.0);
match handler.try_lock() {
Ok(mut h) => {
let _ = h.fill_buffer(&mut scratch[..stereo_n]);
}
Err(std::sync::TryLockError::WouldBlock) => {
stats_render.increment_callback_xrun();
}
Err(std::sync::TryLockError::Poisoned(e)) => {
warn!(target: "chanora_audio",
"AudioHandler poisoned (raw render): {e}");
}
}
// INV_012: copy render reference BEFORE playout.
crate::voice_render::downmix_stereo_f32_to_mono_f32(
&scratch[..stereo_n],
&mut mono[..process_n],
);
render_ref_accum.push_mono_samples(&mono[..process_n], |frame| {
render_ref_buf.write(frame);
});
let gain = f32::from_bits(output_gain.load(Ordering::Relaxed));
let muted = output_muted.load(Ordering::Relaxed);
let mix_stats = crate::voice_render::downmix_stereo_f32_to_interleaved_i16(
&scratch[..stereo_n],
&mut out[..process_n],
1,
gain,
muted,
);
if process_n < n {
out[process_n..].fill(0);
}
if mix_stats.clipped_samples > 0 {
stats_render.add_clipped_samples(mix_stats.clipped_samples);
}
stats_render.update_render(crate::frame::dbfs(&scratch[..stereo_n]), n as u32);
Ok(())
})
.map_err(|e| AudioError::Backend(format!("remoteio render cb: {e}")))?;
unit.initialize()
.map_err(|e| AudioError::Backend(format!("remoteio init: {e}")))?;
unit.start()
.map_err(|e| AudioError::Backend(format!("remoteio start: {e}")))?;
info!(
target: "chanora_audio",
sample_rate_hz = SAMPLE_RATE_HZ,
"ios RemoteIO (Sonora experimental) started"
);
Ok(Self { unit })
}
/// Restart the unit after a route change (stop → uninit → init → start).
pub fn restart(&mut self) -> Result<(), AudioError> {
self.unit
.stop()
.map_err(|e| AudioError::Backend(format!("remoteio restart stop: {e}")))?;
self.unit
.uninitialize()
.map_err(|e| AudioError::Backend(format!("remoteio restart uninit: {e}")))?;
self.unit
.initialize()
.map_err(|e| AudioError::Backend(format!("remoteio restart init: {e}")))?;
self.unit
.start()
.map_err(|e| AudioError::Backend(format!("remoteio restart start: {e}")))?;
info!(target: "chanora_audio", "ios RemoteIO restarted");
Ok(())
}
/// Pause the unit during an AVAudioSession interruption.
pub fn pause(&mut self) -> Result<(), AudioError> {
self.unit
.stop()
.map_err(|e| AudioError::Backend(format!("remoteio pause: {e}")))
}
/// Resume the unit after an interruption ends.
pub fn resume(&mut self) -> Result<(), AudioError> {
self.unit
.start()
.map_err(|e| AudioError::Backend(format!("remoteio resume: {e}")))
}
}
impl Drop for IosRawUnit {
fn drop(&mut self) {
if let Err(e) = self.unit.stop() {
warn!(target: "chanora_audio", error = %e,
"ios RemoteIO stop on drop failed");
} else {
info!(target: "chanora_audio", "ios RemoteIO stopped");
}
}
}
}
+35 -20
View File
@@ -406,20 +406,38 @@ impl IosCaptureState {
speech: true,
}
} else if vad_backend == crate::VadBackend::SileroOnnx {
if let Some(worker) = self.silero_coreml_worker.as_ref() {
let enqueued = worker.try_send(capture_seq, &frame);
if !worker.is_stale(capture_seq) {
let p = worker.latest_probability();
crate::vad::VadOutput {
probability: p,
speech: p >= 0.5,
match crate::vad::callback_vad_worker_policy(
voice_activity_mode,
vad_backend,
self.silero_coreml_worker.is_some(),
) {
crate::vad::VadWorkerPolicy::UseWorker => {
let worker = self
.silero_coreml_worker
.as_ref()
.expect("policy checked worker");
let enqueued = worker.try_send(capture_seq, &frame);
if !worker.is_stale(capture_seq) {
let p = worker.latest_probability();
crate::vad::VadOutput {
probability: p,
speech: p >= 0.5,
}
} else if enqueued {
crate::vad::VadOutput {
probability: 0.0,
speech: false,
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(crate::VadBackend::SileroOnnx);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
&frame,
)
}
} else if enqueued {
crate::vad::VadOutput {
probability: 0.0,
speech: false,
}
} else {
}
crate::vad::VadWorkerPolicy::UseFallback => {
used_fallback_vad = true;
self.mark_vad_fallback_active(crate::VadBackend::SileroOnnx);
crate::vad::VoiceActivityDetector::process_10ms(
@@ -427,13 +445,10 @@ impl IosCaptureState {
&frame,
)
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(crate::VadBackend::SileroOnnx);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
&frame,
)
crate::vad::VadWorkerPolicy::NotModelBacked => crate::vad::VadOutput {
probability: 1.0,
speech: true,
},
}
} else {
crate::vad::VoiceActivityDetector::process_10ms(&mut self.vad_detector, &frame)
+4 -1
View File
@@ -65,7 +65,10 @@ pub(crate) mod voice_render;
mod sdl_output;
#[cfg(any(target_os = "ios", target_os = "macos"))]
mod ios_voice_unit;
mod ios_voice_unit;
#[cfg(target_os = "ios")]
pub mod ios_raw_unit;
#[cfg(target_os = "android")]
pub mod android_voice_unit;
@@ -26,7 +26,7 @@ use std::thread;
use tracing::{info, warn};
use windows::core::{w, PCWSTR};
use windows::Win32::Foundation::{HINSTANCE, HMODULE, HWND, LPARAM, LRESULT, WPARAM};
use windows::Win32::Foundation::{HMODULE, HWND, LPARAM, LRESULT, WPARAM};
use windows::Win32::System::LibraryLoader::GetModuleHandleW;
use windows::Win32::UI::Input::{
GetRawInputData, RegisterRawInputDevices, HRAWINPUT, RAWINPUT, RAWINPUTDEVICE, RAWINPUTHEADER,
@@ -35,7 +35,7 @@ use windows::Win32::UI::Input::{
use windows::Win32::UI::WindowsAndMessaging::{
CallNextHookEx, CreateWindowExW, DefWindowProcW, DispatchMessageW, GetMessageW,
PostThreadMessageW, RegisterClassExW, SetWindowsHookExW, TranslateMessage, UnhookWindowsHookEx,
HC_ACTION, HOOKPROC, KBDLLHOOKSTRUCT, MSG, MSLLHOOKSTRUCT, WH_KEYBOARD_LL, WH_MOUSE_LL,
HC_ACTION, HHOOK, HOOKPROC, KBDLLHOOKSTRUCT, MSG, MSLLHOOKSTRUCT, WH_KEYBOARD_LL, WH_MOUSE_LL,
WINDOW_EX_STYLE, WINDOW_STYLE, WM_INPUT, WM_KEYDOWN, WM_KEYUP, WM_QUIT, WM_SYSKEYDOWN,
WM_SYSKEYUP, WM_XBUTTONDOWN, WM_XBUTTONUP, WNDCLASSEXW, XBUTTON1, XBUTTON2,
};
@@ -423,7 +423,7 @@ unsafe fn run_raw_input_loop(
// class.
let _atom = RegisterClassExW(&wc);
let hwnd = match unsafe {
let hwnd = unsafe {
CreateWindowExW(
WINDOW_EX_STYLE(0),
class_name,
@@ -433,23 +433,13 @@ unsafe fn run_raw_input_loop(
0,
0,
0,
Some(HWND(HWND_MESSAGE_PTR as *mut core::ffi::c_void)),
HWND(HWND_MESSAGE_PTR),
None,
Some(HINSTANCE(h_instance.0)),
h_instance,
None,
)
} {
Ok(h) => h,
Err(_) => {
warn!(
target: "chanora_audio",
"windows ptt: CreateWindowExW(HWND_MESSAGE) returned null"
);
report!(false);
return false;
}
};
if hwnd.0.is_null() {
if hwnd.0 == 0 {
warn!(
target: "chanora_audio",
"windows ptt: CreateWindowExW(HWND_MESSAGE) returned null"
@@ -527,13 +517,13 @@ unsafe fn run_raw_input_loop(
usUsagePage: 0x01,
usUsage: 0x06,
dwFlags: RIDEV_REMOVE,
hwndTarget: HWND(std::ptr::null_mut()),
hwndTarget: HWND(0),
},
RAWINPUTDEVICE {
usUsagePage: 0x01,
usUsage: 0x02,
dwFlags: RIDEV_REMOVE,
hwndTarget: HWND(std::ptr::null_mut()),
hwndTarget: HWND(0),
},
];
let _ = RegisterRawInputDevices(&undo, std::mem::size_of::<RAWINPUTDEVICE>() as u32);
@@ -554,7 +544,7 @@ unsafe extern "system" fn raw_input_wnd_proc(
}
unsafe fn handle_wm_input(lparam: LPARAM) {
let h_raw = HRAWINPUT(lparam.0 as *mut core::ffi::c_void);
let h_raw = HRAWINPUT(lparam.0);
let mut size: u32 = 0;
let header_sz = std::mem::size_of::<RAWINPUTHEADER>() as u32;
// First call: query buffer size.
@@ -851,33 +841,31 @@ unsafe fn run_hook_loop(
let kbd_proc: HOOKPROC = Some(kbd_hook_proc);
let mouse_proc: HOOKPROC = Some(mouse_hook_proc);
let kbd_hook =
match SetWindowsHookExW(WH_KEYBOARD_LL, kbd_proc, Some(HINSTANCE(h_instance.0)), 0) {
Ok(h) => h,
Err(e) => {
warn!(
target: "chanora_audio",
error = %e,
"windows ptt: SetWindowsHookExW(WH_KEYBOARD_LL) failed"
);
report!(false);
return false;
}
};
let mouse_hook =
match SetWindowsHookExW(WH_MOUSE_LL, mouse_proc, Some(HINSTANCE(h_instance.0)), 0) {
Ok(h) => h,
Err(e) => {
warn!(
target: "chanora_audio",
error = %e,
"windows ptt: SetWindowsHookExW(WH_MOUSE_LL) failed"
);
let _ = UnhookWindowsHookEx(kbd_hook);
report!(false);
return false;
}
};
let kbd_hook = match SetWindowsHookExW(WH_KEYBOARD_LL, kbd_proc, h_instance, 0) {
Ok(h) => h,
Err(e) => {
warn!(
target: "chanora_audio",
error = %e,
"windows ptt: SetWindowsHookExW(WH_KEYBOARD_LL) failed"
);
report!(false);
return false;
}
};
let mouse_hook = match SetWindowsHookExW(WH_MOUSE_LL, mouse_proc, h_instance, 0) {
Ok(h) => h,
Err(e) => {
warn!(
target: "chanora_audio",
error = %e,
"windows ptt: SetWindowsHookExW(WH_MOUSE_LL) failed"
);
let _ = UnhookWindowsHookEx(kbd_hook);
report!(false);
return false;
}
};
info!(
target: "chanora_audio",
@@ -920,7 +908,7 @@ unsafe extern "system" fn kbd_hook_proc(code: i32, wparam: WPARAM, lparam: LPARA
}
});
}
CallNextHookEx(None, code, wparam, lparam)
CallNextHookEx(HHOOK(0), code, wparam, lparam)
}
/// Pure-logic dispatcher for a low-level keyboard hook event (L0
@@ -955,7 +943,7 @@ unsafe extern "system" fn mouse_hook_proc(code: i32, wparam: WPARAM, lparam: LPA
}
});
}
CallNextHookEx(None, code, wparam, lparam)
CallNextHookEx(HHOOK(0), code, wparam, lparam)
}
/// Pure-logic dispatcher for a low-level mouse hook event (L0
+61 -27
View File
@@ -8,17 +8,17 @@
#[cfg(any(target_os = "ios", target_os = "macos"))]
pub mod apple_coreml;
pub mod resampler;
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[cfg(not(target_os = "ios"))]
pub mod silero_onnx;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{OnceLock, RwLock};
use crate::frame::{f32_to_i16, i16_to_f32};
use crate::AudioError;
use crate::{AudioError, VadBackend};
use resampler::{Downsampler48to16, INPUT_FRAME_10MS};
#[cfg(not(any(target_os = "ios", target_os = "macos", target_os = "android")))]
#[cfg(not(target_os = "ios"))]
pub use silero_onnx::SileroOnnxVad;
/// Voice activity detector output for one 10 ms frame.
@@ -108,12 +108,39 @@ pub fn process_i16_10ms(detector: &mut dyn VoiceActivityDetector, samples: &[i16
detector.process_10ms(&frame)
}
/// Callback-side policy for optional model-backed VAD workers.
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
pub(crate) enum VadWorkerPolicy {
/// Keep using the already-available model worker.
UseWorker,
/// No worker may be constructed on the callback thread; use WebRTC fallback.
UseFallback,
/// This backend does not need a model worker.
NotModelBacked,
}
/// Decide whether a realtime callback may use a model-backed VAD worker.
///
/// Model/worker construction is intentionally absent from this policy: if a
/// worker is not already present, callbacks must stay nonblocking and fall back.
pub(crate) fn callback_vad_worker_policy(
voice_activity_mode: bool,
backend: VadBackend,
worker_available: bool,
) -> VadWorkerPolicy {
if !voice_activity_mode || backend != VadBackend::SileroOnnx {
return VadWorkerPolicy::NotModelBacked;
}
if worker_available {
VadWorkerPolicy::UseWorker
} else {
VadWorkerPolicy::UseFallback
}
}
static SILERO_MODEL_PATH_OVERRIDE: OnceLock<RwLock<Option<String>>> = OnceLock::new();
static SILERO_MODEL_EPOCH: AtomicU64 = AtomicU64::new(0);
#[cfg(test)]
pub(crate) static SILERO_MODEL_PATH_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
fn silero_model_path_override() -> &'static RwLock<Option<String>> {
SILERO_MODEL_PATH_OVERRIDE.get_or_init(|| RwLock::new(None))
}
@@ -148,19 +175,6 @@ pub fn silero_model_epoch() -> u64 {
SILERO_MODEL_EPOCH.load(Ordering::Relaxed)
}
#[cfg(test)]
pub(crate) fn clear_silero_model_path_for_test() {
set_silero_model_path_for_test(None);
}
#[cfg(test)]
pub(crate) fn set_silero_model_path_for_test(path: Option<String>) {
if let Ok(mut guard) = silero_model_path_override().write() {
*guard = path;
SILERO_MODEL_EPOCH.fetch_add(1, Ordering::Relaxed);
}
}
/// Return the expected path of the Silero VAD v6 ONNX model on
/// supported platforms.
/// The model is shipped as a Flutter asset and copied to the app's
@@ -250,20 +264,13 @@ mod tests {
#[test]
fn set_silero_model_path_rejects_missing_file() {
let _guard = SILERO_MODEL_PATH_TEST_LOCK.lock().unwrap();
clear_silero_model_path_for_test();
let result = set_silero_model_path("/definitely/not/a/silero_vad.onnx");
assert!(result.is_err());
clear_silero_model_path_for_test();
}
#[test]
fn set_silero_model_path_updates_override_and_epoch() {
let _guard = SILERO_MODEL_PATH_TEST_LOCK.lock().unwrap();
clear_silero_model_path_for_test();
let path =
std::env::temp_dir().join(format!("chanora_test_silero_{}.onnx", std::process::id()));
std::fs::write(&path, b"test").unwrap();
@@ -273,7 +280,34 @@ mod tests {
assert!(silero_model_epoch() > before);
assert_eq!(silero_model_bundle_path(), path.to_string_lossy());
clear_silero_model_path_for_test();
let _ = std::fs::remove_file(path);
}
#[test]
fn callback_policy_uses_existing_model_worker_only() {
assert_eq!(
callback_vad_worker_policy(true, VadBackend::SileroOnnx, true),
VadWorkerPolicy::UseWorker
);
assert_eq!(
callback_vad_worker_policy(true, VadBackend::SileroOnnx, false),
VadWorkerPolicy::UseFallback
);
}
#[test]
fn callback_policy_keeps_disabled_and_webrtc_paths_worker_free() {
assert_eq!(
callback_vad_worker_policy(false, VadBackend::SileroOnnx, false),
VadWorkerPolicy::NotModelBacked
);
assert_eq!(
callback_vad_worker_policy(true, VadBackend::Disabled, false),
VadWorkerPolicy::NotModelBacked
);
assert_eq!(
callback_vad_worker_policy(true, VadBackend::WebrtcVad, false),
VadWorkerPolicy::NotModelBacked
);
}
}
+1 -33
View File
@@ -361,31 +361,6 @@ impl SileroOnnxVadWorker {
})
}
#[cfg(test)]
pub(crate) fn stale_test_worker() -> Self {
let (tx, rx) = std::sync::mpsc::sync_channel::<SileroFrameMessage>(64);
let alive = Arc::new(AtomicBool::new(true));
let alive_for_thread = alive.clone();
let handle = std::thread::Builder::new()
.name("chanora-silero-vad-stale-test".to_string())
.spawn(move || {
while alive_for_thread.load(Ordering::Relaxed) {
match rx.recv_timeout(std::time::Duration::from_millis(10)) {
Ok(_) | Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {}
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => break,
}
}
})
.ok();
Self {
tx: Some(tx),
latest_probability: Arc::new(AtomicU32::new(0.0_f32.to_bits())),
latest_processed_seq: Arc::new(AtomicU64::new(u64::MAX)),
alive,
handle,
}
}
/// Best-effort enqueue of a 10 ms frame for background inference.
pub fn try_send(&self, seq: u64, frame: &[f32; super::resampler::INPUT_FRAME_10MS]) -> bool {
let Some(tx) = &self.tx else {
@@ -418,15 +393,8 @@ impl SileroOnnxVadWorker {
impl Drop for SileroOnnxVadWorker {
fn drop(&mut self) {
self.alive.store(false, Ordering::Relaxed);
// Drop the sender first so the worker thread's rx.recv() returns
// Err and the loop exits promptly.
let _ = self.tx.take();
// Join the thread instead of detaching. The channel close
// unblocks rx.recv() so the join is bounded; it waits at most
// until the current in-flight inference completes.
if let Some(handle) = self.handle.take() {
let _ = handle.join();
}
let _ = self.handle.take();
}
}
+2 -2
View File
@@ -196,8 +196,8 @@ fn windows_exercise() {
// start/stop lifecycle through the public trait surface so
// any info!/warn! the factory or the backend's `start` path
// emits is captured by the layer.
use chanora_audio::ptt_backends::{PttBinding, PttInputClass};
use chanora_audio::{select_ptt_backend, AudioTransmitGate};
use chanora_audio::ptt_backends::{select_ptt_backend, PttBinding, PttInputClass};
use chanora_audio::AudioTransmitGate;
let mut backend = select_ptt_backend();
let gate = AudioTransmitGate::new(false);
+23 -5
View File
@@ -1062,8 +1062,10 @@ pub enum BridgeAudioRoute {
/// Bridge iOS voice-processing mode.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BridgeIosVoiceProcessingMode {
/// Apple VoiceProcessingIO path.
/// Shipping VPIO path.
PlatformVoiceProcessing,
/// Experimental Sonora path.
SonoraExperimental,
}
/// Bridge processing backend.
@@ -1221,6 +1223,7 @@ impl From<BridgeIosVoiceProcessingMode> for chanora_core::IosVoiceProcessingMode
fn from(mode: BridgeIosVoiceProcessingMode) -> Self {
match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => Self::PlatformVoiceProcessing,
BridgeIosVoiceProcessingMode::SonoraExperimental => Self::SonoraExperimental,
}
}
}
@@ -1231,6 +1234,7 @@ impl From<chanora_core::IosVoiceProcessingMode> for BridgeIosVoiceProcessingMode
chanora_core::IosVoiceProcessingMode::PlatformVoiceProcessing => {
Self::PlatformVoiceProcessing
}
chanora_core::IosVoiceProcessingMode::SonoraExperimental => Self::SonoraExperimental,
}
}
}
@@ -2357,11 +2361,25 @@ pub async fn set_ios_voice_processing_mode(
let config = BridgeAudioProcessingConfig {
route: BridgeAudioRoute::Speaker,
ios_mode: mode,
processing_backend: BridgeAudioBackend::PlatformVoiceProcessing,
processing_backend: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => {
BridgeAudioBackend::PlatformVoiceProcessing
}
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeAudioBackend::WebrtcApm,
},
vad_backend: BridgeVadBackend::SileroOnnx,
aec: BridgeEffectOwner::Platform,
ns: BridgeEffectOwner::Platform,
agc: BridgeEffectOwner::Platform,
aec: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => BridgeEffectOwner::Platform,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::WebrtcApm,
},
ns: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => BridgeEffectOwner::Platform,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::WebrtcApm,
},
agc: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => BridgeEffectOwner::Platform,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::WebrtcApm,
},
hpf_enabled: true,
limiter_enabled: true,
vad_hangover_ms: 500,
@@ -2409,6 +2409,7 @@ impl SseDecode for crate::api::BridgeIosVoiceProcessingMode {
let mut inner = <i32>::sse_decode(deserializer);
return match inner {
0 => crate::api::BridgeIosVoiceProcessingMode::PlatformVoiceProcessing,
1 => crate::api::BridgeIosVoiceProcessingMode::SonoraExperimental,
_ => unreachable!(
"Invalid variant for BridgeIosVoiceProcessingMode: {}",
inner
@@ -3444,6 +3445,7 @@ impl flutter_rust_bridge::IntoDart for crate::api::BridgeIosVoiceProcessingMode
fn into_dart(self) -> flutter_rust_bridge::for_generated::DartAbi {
match self {
Self::PlatformVoiceProcessing => 0.into_dart(),
Self::SonoraExperimental => 1.into_dart(),
_ => unreachable!(),
}
}
@@ -4213,6 +4215,7 @@ impl SseEncode for crate::api::BridgeIosVoiceProcessingMode {
<i32>::sse_encode(
match self {
crate::api::BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => 0,
crate::api::BridgeIosVoiceProcessingMode::SonoraExperimental => 1,
_ => {
unimplemented!("");
}
+2 -2
View File
@@ -123,7 +123,7 @@ Runtime event or error
| Bridge command DTOs | Flutter generated API | `chanora_bridge`, Rust core | Stable typed DTOs; no raw protocol-library types cross to Flutter |
| Bridge event DTOs | Rust core / bridge | Flutter services/widgets | User-safe errors and capability fields are explicit |
| Protocol DTOs | `chanora_protocol` | Rust core, state sync | Protocol adapter isolates `tsclientlib` |
| Audio configuration | Flutter settings / Rust core | `chanora_audio` | Voice modes and processing flags are explicit; Windows/Linux desktop VAD-backed `VoiceActivity` is enabled only where runtime evidence exists, with unsupported platforms disabled/deferred |
| Audio configuration | Flutter settings / Rust core | `chanora_audio` | Voice modes and processing flags are explicit; VAD remains disabled/deferred |
| Storage records | Storage crate | Rust core / Flutter UI via bridge | Secrets stay behind secure-storage abstraction |
| Diagnostic bundles | Diagnostics crate | Flutter diagnostics UI | Redaction runs before export or display |
| Platform capability records | Platform adapters/audio/PTT backends | UI and release record | UI/release wording must not over-claim capability |
@@ -159,7 +159,7 @@ Runtime event or error
| Secure storage abstraction | Platform storage details do not leak into UI or unrelated crates |
| Advisory audio benchmarks | Performance regressions are surfaced without making CI a hard release gate at this stage |
| PTT capability levels | Platform PTT support is represented as capability data and must match release wording |
| VoiceActivity platform scope | Windows/Linux desktop `VoiceActivity` is implemented through the capture VAD path; unsupported platforms remain disabled/deferred until backend allocation and runtime verification exist |
| VoiceActivity deferral | `VoiceActivity` remains reserved/disabled until a later baseline allocates implementation |
| No automatic diagnostic upload in MVP | Diagnostics are local and user-initiated unless future approved requirements change policy |
## 11. Verification Handoff
+1 -1
View File
@@ -63,7 +63,7 @@ Design rules:
| Capture/playback | Platform-specific units handle Android, iOS, desktop/fallback paths behind Rust audio abstractions |
| Codec | Opus encode/decode lives in `opus_voice.rs` and associated audio modules |
| DSP chain | High-pass filter, noise suppression, echo cancellation, and AGC are represented by audio processing modules/backends |
| Transmit control | `TransmitMode` supports `Ptt`, `Continuous`, and `VoiceActivity`; `VoiceActivity` is active for Windows/Linux desktop capture when VAD is configured, while mobile, macOS, and unverified-platform enablement remain deferred |
| Transmit control | `TransmitMode` supports `Ptt`, `Continuous`, and reserved `VoiceActivity`; `VoiceActivity` has no active MVP implementation |
| VoiceActivity gate (capture-side) | `voice_activity::VoiceActivityStateMachine` is the 10 ms-cadence gate for `TransmitMode::VoiceActivity`; open-after 40 ms (debounce), hangover 500 ms (anti-chatter), min-tx 200 ms (anti-flicker), weak-hold 30-100 frames (anti-stale-VAD); live `configure()` re-clamps existing timers on settings change without resetting state; 9 unit tests cover the main paths |
| PTT | Desktop/mobile backends expose capability level and active backend; missed-key-up watchdog prevents stuck transmit |
| Release tail | Tail handling prevents abrupt cutoffs after PTT release where configured |
@@ -11,7 +11,7 @@
| Android Keystore-backed DEK deferred | Secure-storage claim limited | Storage design carries fallback limitation | Platform audit required | Waiver required |
| Full reducer tests incomplete | State verification incomplete | State design remains valid but evidence partial | SWE.4/SWE.5 partial | Blocks full state-sync claim |
| Desktop/iOS artifacts not release-ready | Platform packaging requirements partial | Release design remains source-build/unsigned | SYS.4 evidence partial | Public binary release No-Go |
| VAD platform-scoped | VoiceActivity active only for verified Windows/Linux desktop paths | UI must show disabled/unavailable on unsupported platforms | VAD pass claim must name verified platform/runtime evidence | No broad VAD marketing claim without platform scope |
| VAD deferred | VoiceActivity not active | UI must show disabled/coming-soon | No VAD pass claim | No VAD marketing claim |
## 2. Conclusion
@@ -43,7 +43,7 @@ This record captures the current maintainability review so implementation, verif
| Android Keystore-backed DEK remains deferred | Android identity/bookmark encryption has weaker fail-safe properties than final target secure-storage design. | Android secure-storage audit or waiver; explicit release-readiness limitation. |
| Android permission/audio lifecycle needs deeper route exercise | Build/install/launch smoke now passes on the emulator, but full permission-flow and audio-route lifecycle behavior still need an interactive scenario or device test before release. | Device/emulator scenario covering permission request/denial/grant, voice controls, foreground service, audio focus, and route/SCO transitions. |
| iOS device runtime verification not executed in this review | The iOS audio-session error path is hardened, but VoiceProcessingIO/session ordering and runtime audio behavior still need device evidence. | iOS device or simulator build/run plus audio-session smoke evidence before iOS runtime success is claimed. |
| VAD / VoiceActivity wording drift | VAD assets, tests, and Windows/Linux desktop runtime wiring exist, but product-enabled `VoiceActivity` must remain platform-scoped per DEC-030. | Release, README, and verification wording must distinguish verified desktop behavior from unsupported mobile/macOS/unverified-platform behavior. |
| VAD / VoiceActivity wording drift | VAD assets, tests, and scaffolding exist, but product-enabled `VoiceActivity` remains reserved/disabled per DEC-030. | Release, README, and verification wording must distinguish scaffolding/assets/tests from shipped product behavior. |
| Protocol voice packet re-export is an intentional exception | Future maintainers may assume complete protocol isolation and accidentally widen the Seam. | Architecture note in SAD/SDD or a decision-register entry. |
| Bridge DTO mirror drift | Field additions can be missed across Core, Bridge, and Dart generated DTOs. | Bridge generation check plus Flutter analyze/test after bridge DTO changes. |
| Full live reducer integration remains separate from reducer unit coverage | State reducer tests are strong, but runtime UI still has snapshot/probe paths. | SWE.5 integration run proving live protocol events fold through the intended state path, or explicit P1 deferral. |
@@ -91,7 +91,7 @@ adb -s emulator-5554 shell pidof app.chanora.chanora_flutter
- Android minimum runtime baseline is API 28 (Android 9.0) per SysRS-288, SRS-187, DEC-004, and the Gradle `minSdk = 28` configuration. Documents must not revive the older API 24 baseline.
- Flutter app version/build is `0.3.0+100` in `apps/chanora_flutter/pubspec.yaml`. Rust workspace package version remains `0.2.0-beta.1`. Release documents must distinguish these values instead of treating them as one candidate version.
- The v0.3.0 changelog entry may mention Windows/Linux desktop VAD-backed `VoiceActivity` only with matching runtime evidence; mobile, macOS, and unverified-platform `VoiceActivity` remain disabled/unavailable until DEC-030 is superseded and runtime verification exists.
- The v0.3.0 changelog entry may mention VAD assets/backends only as implementation scaffolding; product-enabled `VoiceActivity` remains disabled/coming-soon until DEC-030 is superseded and runtime verification exists.
- README wording must describe the existing Flutter/Rust workspace and app scaffold, not a future scaffold that has not been created.
## 8. Git Policy
+1 -1
View File
@@ -15,7 +15,7 @@ This register records product and engineering decisions referenced by the DV doc
| DEC-012 legal/trademark/OSS review | Open | Blocks public/store release |
| DEC-020 dual license MIT OR Apache-2.0 | Accepted per README | Supports license posture; dependency notices still require review |
| DEC-027 desktop mouse side-button PTT | Accepted by requirements baseline | Verification must not over-claim unsupported platform input classes |
| DEC-030 VAD platform scope | Partially superseded by desktop enablement | Windows/Linux desktop `VoiceActivity` is enabled through the audio capture VAD path and must be claimed only with matching runtime evidence; mobile, macOS, and unverified-platform `VoiceActivity` remain disabled/deferred until a later baseline enables and verifies them |
| DEC-030 VAD deferral | Accepted as deferral | VAD scaffolding, assets, and tests may exist, but product `VoiceActivity` remains disabled/coming-soon until a later baseline enables and verifies it |
| DEC-032 Android CMake patch exit path | Active tracking | Patched dependency requires reevaluation |
| DEC-033 macOS VPIO ducking configuration | Accepted | Write `kAUVoiceIOProperty_OtherAudioDuckingConfiguration` with `mEnableAdvancedDucking=0` (disables dynamic voice-activity-driven ducking) and `mDuckingLevel=Min` (= 10) to minimise the ducking of other apps' audio during a voice session; property is macOS 14+ only, the macOS 13 set fails silently (debug log) and VPIO uses its default behaviour; matches the iOS `.voiceChat` baseline on macOS 14+ |
| DEC-034 Android runtime verification gate | Active tracking | Android target compilation, install, and runtime smoke are blocked locally until `aarch64-linux-android-clang` is available and `adb devices -l` shows an authorized target; release docs must not claim Android runtime success |
+2 -2
View File
@@ -21,7 +21,7 @@
| Protocol adapter | `chanora_protocol``tsclientlib` isolated behind `ProtocolClient`, typed DTOs, `ProtocolError` catalogue |
| Connection lifecycle | `chanora_core` — supervisor task, exponential backoff reconnect (1s→60s), user-disconnect suppresses reconnect; branch `simplify-project-review` has started splitting the previous large `lib.rs` into focused internal Modules (`events.rs`, `network_diagnostics.rs`) while preserving public re-exports |
| State sync reducer unit | `chanora_state``ConnectionState`, `channel_join`, snapshot/delta reducers, reconnect handling, deterministic ordering, malformed duplicate normalization, channel-delete/client cleanup, and reducer unit tests. Runtime core integration still uses snapshot/probe refresh paths and remains separate validation work. |
| Audio subsystem | `chanora_audio` — Opus encode/decode, HPF/NS/AEC3/AGC2 DSP, PTT backends (Windows/macOS/Linux/focused), iOS VoiceProcessingIO, Android Oboe, jitter buffer via `tsclientlib::audio::AudioHandler`, mixer, mute/deaf gates, release-tail timer, and Windows/Linux desktop `VoiceActivity` through the capture VAD path. Mobile, macOS, and unverified-platform `VoiceActivity` remain deferred per DEC-030. |
| Audio subsystem | `chanora_audio` — Opus encode/decode, HPF/NS/AEC3/AGC2 DSP, PTT backends (Windows/macOS/Linux/focused), iOS VoiceProcessingIO, Android Oboe, jitter buffer via `tsclientlib::audio::AudioHandler`, mixer, mute/deaf gates, release-tail timer, and VAD scaffolding/assets. Product `VoiceActivity` remains disabled per DEC-030. |
| Push-to-talk | Per-platform backends: Windows Raw Input + hook fallback, macOS Event Tap, Linux freedesktop portal, focused fallback; `PttCapabilityLevel` (L0L3); missed-key-up watchdog |
| Voice controls UI | `voice_bar`, `voice_compact`, `voice_haptics`, `voice_level_meter`, `voice_platform`, `ptt_capability_badge`, `talk_power_warning` |
| Storage (non-secret) | `chanora_storage``BookmarkRepository` (SQLite/rusqlite bundled, schema v2), ChaCha20-Poly1305 encrypted passwords |
@@ -54,7 +54,7 @@
|---|---|
| Event replay tooling | Reducer tests cover the state-sync contract, but standalone replay-file tooling remains a P1 verification gap. |
| Reducer runtime integration evidence | The standalone reducer is unit-tested, but `chanora_core` still refreshes UI state through snapshot/probe paths rather than folding all live protocol events through `chanora_state::reduce`. |
| Mobile/macOS VoiceActivity | `assets/models/silero_vad.onnx` is bundled and used by the desktop VAD path where runtime evidence supports it; mobile, macOS, and unverified-platform `TransmitMode::VoiceActivity` remain disabled/deferred until a later baseline supplies backend enablement and verification evidence. |
| Silero VAD | `assets/models/silero_vad.onnx` bundled but DEC-030 defers VAD to P1; `TransmitMode::VoiceActivity` is reserved and disabled in this baseline. |
| macOS build | Source-buildable only; no public release artifact is approved. |
| Windows build | Source-buildable only; no public release artifact is approved. |
| iOS build | Source-buildable/unsigned validation only; no TestFlight/App Store release artifact is approved. |
+1 -1
View File
@@ -18,7 +18,7 @@ A waiver records a known gap that reviewers may accept for a limited decision sc
| DV-WVR-004 | Standalone event replay and runtime reducer-integration evidence not yet complete | `docs/implementation-status-2026-05-28.md`, local `cargo test -p chanora_state --locked` evidence | Reducer unit coverage supports state synchronization, but file-based replay evidence and live-event integration evidence remain open | DV documentation pass and internal validation | Event replay tooling implemented or requirement reprioritized; runtime reducer integration evidence attached |
| DV-WVR-005 | Event replay tool not found | `docs/implementation-status-2026-05-28.md` | Limits P1 state verification hooks SRS-061/SRS-098 | Accepted as P1 deferral | Event replay tooling implemented or requirement reprioritized |
| DV-WVR-006 | Desktop and iOS artifacts are source-buildable or unsigned only | `docs/implementation-status-2026-05-28.md`, `docs/release/ios-build.md` | Blocks packaged public release claims | Internal validation from source/unsigned builds only | Signed/notarized/package artifacts exist and hashes are recorded |
| DV-WVR-007 | Silero VAD asset bundled while `VoiceActivity` is platform-scoped | `docs/implementation-status-2026-05-28.md` | Risk that UI/release wording overstates VAD availability beyond verified Windows/Linux desktop paths | DV may pass if VoiceActivity claims are limited to verified desktop evidence and unsupported platforms remain disabled/unavailable | Mobile/macOS implementation allocated in a later baseline or asset/wording reconciled |
| DV-WVR-007 | Silero VAD asset bundled while `VoiceActivity` is deferred | `docs/implementation-status-2026-05-28.md` | Risk that UI/release wording overstates VAD availability | DV may pass if VoiceActivity remains disabled/coming-soon | VAD implementation allocated in a later baseline or asset/wording reconciled |
| DV-WVR-008 | Artifact hashes, tag, and candidate run IDs are not recorded in release record | `docs/release/release-readiness-go-nogo-record.md` | Blocks final release approval and reproducibility | DV documentation review only | Candidate build run records, tag, commit SHA, and artifact hashes are recorded |
| DV-WVR-009 | Android target compile and runtime smoke blocked locally | `docs/governance/maintainability-review-2026-06-08.md`, `docs/release/release-readiness-go-nogo-record.md` | Blocks Android runtime, permission-flow, and audio-lifecycle success claims | Documentation review only; internal validation must keep Android limitation stated | Android NDK compiler `aarch64-linux-android-clang` is available, `adb devices -l` shows an authorized target, and Android build/install/smoke evidence is attached |
+2 -2
View File
@@ -2597,7 +2597,7 @@ This section extends the ASPICE SWE.1 Software Requirements Specification. The s
- Source SysDes: SysDes-150
- Verification method: Integration Test, UI Review
**SRS-205**: The software shall represent the user's voice transmit mode as a `TransmitMode` enum with variants `Ptt`, `Continuous`, and `VoiceActivity`. `VoiceActivity` shall be selectable only on platforms with an implemented and verified VAD capture path in this baseline (currently Windows/Linux desktop); mobile, macOS, and unverified-platform enablement remain deferred per DEC-030. The setting shall be persisted per identity via the identity store where the selected platform supports it. The default value for a fresh install shall be `Ptt`. The UI shall render `VoiceActivity` as disabled/unavailable on unsupported platforms rather than claiming runtime support.
**SRS-205**: The software shall represent the user's voice transmit mode as a `TransmitMode` enum with variants `Ptt`, `Continuous`, and `VoiceActivity` (the last reserved with no v1 implementation per DEC-030). The setting shall be persisted per identity via the identity store. The default value for a fresh install shall be `Ptt`. The UI shall render `VoiceActivity` as a disabled "coming soon" option until an implementation is allocated in a later baseline.
- Status: Baseline Candidate
- Type: Software Interface Requirement
@@ -2837,7 +2837,7 @@ Consistent with the SysRS-307 / SysRS-308 / SysRS-309 deferrals propagated throu
| Version | Date | Description |
|---|---|---|
| 0.9.5 | 2026-05-15 | Added v1 audio + PTT lifecycle software requirements SRS-204 through SRS-207 sourced from SysDes-149..151: bridge surface drops `start_audio` / `stop_audio` and adds `voice_join(channel_id)` / `voice_leave()`, audio engine opens streams on first voice-channel join and closes on last leave with output independent of mic-permission state, `TransmitMode` enum (`Ptt` default, `Continuous`, platform-scoped `VoiceActivity` per DEC-030) persisted per identity where supported, `release_tail_ms` (default 200, range 0500) gate on the `true → false` transition of `transmit_active` with re-press cancellation, Voice Bar hard-mute override of `transmit_active`. Strict layered sourcing preserved (`SRS -> SysDes` only). |
| 0.9.5 | 2026-05-15 | Added v1 audio + PTT lifecycle software requirements SRS-204 through SRS-207 sourced from SysDes-149..151: bridge surface drops `start_audio` / `stop_audio` and adds `voice_join(channel_id)` / `voice_leave()`, audio engine opens streams on first voice-channel join and closes on last leave with output independent of mic-permission state, `TransmitMode` enum (`Ptt` default, `Continuous`, reserved `VoiceActivity` per DEC-030) persisted per identity, `release_tail_ms` (default 200, range 0500) gate on the `true → false` transition of `transmit_active` with re-press cancellation, Voice Bar hard-mute override of `transmit_active`. Strict layered sourcing preserved (`SRS -> SysDes` only). |
## Baseline Candidate 0.9.6 Update
+2 -2
View File
@@ -2852,7 +2852,7 @@ This SysDes version covers all known SysRS requirements from `SysRS-001` through
- ASPICE SYS.3 alignment: Architecture constraints
- Allocated SysRS: SysRS-298
**SysDes-149**: The system architecture shall define a `TransmitMode` element carried as an enum at the audio + bridge + UI boundary with variants `Ptt`, `Continuous`, and `VoiceActivity`. `VoiceActivity` shall be surfaced only when the current platform has an allocated, implemented, and verified VAD capture path in this baseline (currently Windows/Linux desktop); mobile, macOS, and unverified-platform enablement remain deferred per DEC-030. The active mode shall be persisted in the identity store where the selected platform supports it; unsupported platforms shall surface `VoiceActivity` as disabled/unavailable rather than claiming runtime support.
**SysDes-149**: The system architecture shall define a `TransmitMode` element carried as an enum at the audio + bridge + UI boundary with variants `Ptt`, `Continuous`, and a reserved `VoiceActivity` placeholder that has no allocated implementation in this baseline (deferred per DEC-030). The active mode shall be persisted in the identity store; the UI shall surface `VoiceActivity` as a disabled "coming soon" option until an implementation is allocated in a later baseline.
- Status: Baseline Candidate
- Type: Subsystem Interface
@@ -3029,7 +3029,7 @@ This SysDes version covers all known SysRS requirements from `SysRS-001` through
| Version | Date | Description |
|---|---|---|
| 0.9.5 | 2026-05-15 | Added v1 audio + PTT lifecycle allocation SysDes-149 through SysDes-151 sourced from SysRS-303 / SysRS-304: `TransmitMode` enum element (`Ptt` / `Continuous` / platform-scoped `VoiceActivity` per DEC-030) at the audio + bridge + UI boundary, audio engine lifecycle bound to voice-channel membership with no manual start affordance and a listen-only path independent of mic permission, hard-mute override element, and the release-tail timer adapter (default 200 ms, range 0500 ms) on `transmit_active`. Strict layered sourcing preserved (`SysDes -> SysRS` only). |
| 0.9.5 | 2026-05-15 | Added v1 audio + PTT lifecycle allocation SysDes-149 through SysDes-151 sourced from SysRS-303 / SysRS-304: `TransmitMode` enum element (`Ptt` / `Continuous` / reserved `VoiceActivity` per DEC-030) at the audio + bridge + UI boundary, audio engine lifecycle bound to voice-channel membership with no manual start affordance and a listen-only path independent of mic permission, hard-mute override element, and the release-tail timer adapter (default 200 ms, range 0500 ms) on `transmit_active`. Strict layered sourcing preserved (`SysDes -> SysRS` only). |
## Baseline Candidate 0.9.6 Update
+2 -2
View File
@@ -1924,7 +1924,7 @@ This section converts the baseline product decisions into auditable system-level
- Priority: P0
- Verification: Privacy Review, Security Audit, Diagnostic Inspection
**SysRS-303**: The Chanora application system shall define the v1 voice transmit mode set as `Ptt`, `Continuous`, and `VoiceActivity`. `VoiceActivity` shall be enabled only on platforms with an implemented and verified VAD capture path in this baseline (currently Windows/Linux desktop); mobile, macOS, and unverified-platform enablement remain deferred per DEC-030. The default mode on a fresh install shall be `Ptt`; the user-selected mode shall be persisted per identity where the selected platform supports it. The audio engine lifecycle shall be bound to voice-channel membership: input and output streams shall open on the user's first voice-channel join of the session and shall close on the last voice-channel leave, with no manual start affordance exposed at any system interface. The output stream shall open regardless of microphone-permission state so listen-only is a first-class flow. A user-facing hard-mute toggle shall force the transmit gate closed and shall override the active transmit mode, the PTT key state, and every other internal signal.
**SysRS-303**: The Chanora application system shall define the v1 voice transmit mode set as `Ptt` and `Continuous`, with `VoiceActivity` reserved on the enum surface but unimplemented in this baseline (deferred per DEC-030). The default mode on a fresh install shall be `Ptt`; the user-selected mode shall be persisted per identity. The audio engine lifecycle shall be bound to voice-channel membership: input and output streams shall open on the user's first voice-channel join of the session and shall close on the last voice-channel leave, with no manual start affordance exposed at any system interface. The output stream shall open regardless of microphone-permission state so listen-only is a first-class flow. A user-facing hard-mute toggle shall force the transmit gate closed and shall override the active transmit mode, the PTT key state, and every other internal signal.
- Priority: P0
- Verification: Functional Test, UX Review
@@ -1989,7 +1989,7 @@ This section converts the baseline product decisions into auditable system-level
| Version | Date | Description |
|---|---|---|
| 0.9.5 | 2026-05-15 | Added v1 audio + PTT lifecycle requirements SysRS-303 and SysRS-304 capturing the no-manual-start audio engine bound to voice-channel membership, the v1 transmit-mode set (`Ptt` default, `Continuous`, and platform-scoped `VoiceActivity` per DEC-030), the listen-only flow (output independent of mic permission), the hard-mute override, and the 200 ms (0500 ms) PTT release tail for word-boundary anti-clipping. |
| 0.9.5 | 2026-05-15 | Added v1 audio + PTT lifecycle requirements SysRS-303 and SysRS-304 capturing the no-manual-start audio engine bound to voice-channel membership, the v1 transmit-mode set (`Ptt` default + `Continuous`, with `VoiceActivity` reserved per DEC-030), the listen-only flow (output independent of mic permission), the hard-mute override, and the 200 ms (0500 ms) PTT release tail for word-boundary anti-clipping. |
## Baseline Candidate 0.9.9 Update
-40
View File
@@ -1,40 +0,0 @@
@echo off
rem Wrapper around cmake.exe that injects policy/cache variables to force
rem the release dynamic CRT (/MD) for Debug configurations. audiopus_sys
rem calls cmake::build(opus_path), which invokes CMake configure; the
rem upstream Opus CMakeLists.txt does not set CMP0091 or
rem CMAKE_MSVC_RUNTIME_LIBRARY, so CMake defaults to /MDd in Debug
rem builds. That pulls in __imp__CrtDbgReportW which is unresolved at
rem test link time because the rest of the Rust/Cargo graph uses /MD.
rem
rem This wrapper is only invoked on Windows MSVC targets (see
rem .cargo/config.toml [env] CMAKE_<target-triple> entries). Non-Windows
rem hosts are unaffected.
rem
rem cmake-rs invokes this as: cmake-msvc-release-crt.cmd <configure-args>
rem and also: cmake-msvc-release-crt.cmd --build ...
rem and also: cmake-msvc-release-crt.cmd --version
rem and also: cmake-msvc-release-crt.cmd -E ...
rem
rem Echoes a marker to stdout so the cargo build log proves the wrapper
rem was actually invoked. cmake-rs forwards cmake stdout to the build
rem script log, so this surfaces in `cargo build -vv` output.
setlocal
echo cmake-msvc-release-crt.cmd: invoked with %*
rem Pass through non-configure invocations unchanged.
if "%~1"=="--build" goto :passthrough
if "%~1"=="--version" goto :passthrough
if "%~1"=="-E" goto :passthrough
if "%~1"=="--install" goto :passthrough
if "%~1"=="--open" goto :passthrough
rem Configure invocation: inject -D flags before user args.
echo cmake-msvc-release-crt.cmd: injecting CMP0091=NEW and CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL
cmake.exe -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL %*
exit /b %ERRORLEVEL%
:passthrough
cmake.exe %*
exit /b %ERRORLEVEL%