edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Android Oboe voice backend — WebRTC APM, VAD, HW/SW toggle, BBCode welcome, link trust, foreground task

Browse Source 
Audio engine (Rust):
- Android Oboe: WebRTC APM (AEC/NS/AGC/HPF) + TEN/Silero ONNX VAD
- Hardware effects (JNI) with software fallback per-effect
- Render reference buffer for AEC between output/capture callbacks
- Voice activity gate: suppress transmission when speaker muted (all platforms)
- Audio focus (SDD-109) + Bluetooth SCO (SDD-110) via JNI
- ONNX Runtime 1.26 via ort 2.0.0-rc.12 (down from rc.10, ndarray 0.17)
- VAD worker channel capacity 8→32, initial seq u64::MAX (warm-up fix)
- TEN VAD default backend (was Silero)
- Platform→WebrtcApm resolution after hardware binding
- oboe-rs edisonjwa fork with get_raw_session_id()

Android Kotlin:
- AndroidAudioFocusController + AndroidBluetoothScoController
- AndroidAudioLifecycleController (route changes to Flutter)
- ProGuard rules for new controllers

Flutter UI:
- VoiceSettings: Android HW/SW toggle (Platform auto / WebRTC APM)
- VoiceStatusChip: mute warning border + Speaker muted label
- BBCode welcome message parser (BbCodeText, case-insensitive)
- Welcome message foldable (expanded by default)
- Link trust dialog (domain wildcards, SharedPreferences)
- HapticFeedback on voice sheet opener
- Server name in AppBar, version v0.1.0
- Default channel (id=1) visible, serverquery clients hidden
- flutter_foreground_task integration

Config:
- ort load-dynamic on all non-iOS (Android/Linux/Windows)
- ONNX Runtime AAR 1.26.0
- ndarray moved to common deps (was Apple-only)
This commit is contained in:
Edison Jwa
2026-05-22 09:29:57 +09:00
parent 6af4ecab0f
commit bf284018e6
37 changed files with 3676 additions and 703 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/chanora_bridge/src/api.rs
+19 -4
View File
@@ -1725,6 +1725,21 @@ pub async fn set_vad_model_path(path: String) -> Result<(), BridgeError> {
Ok(())
}
/// Configure the TEN VAD ONNX model path.
pub async fn set_ten_vad_model_path(path: String) -> Result<(), BridgeError> {
if path.trim().is_empty() {
return Err(BridgeError::InvalidCommand(
"ten vad model path must not be empty".to_string(),
));
}
runtime()
.spawn(async move { chanora_audio::vad::set_ten_model_path(&path).map_err(|e| e) })
.await
.map_err(|e| task_join_error("set_ten_vad_model_path", e))?
.map_err(|e| BridgeError::Unmapped(format!("set_ten_vad_model_path: {e}")))?;
Ok(())
}
/// Enable or disable audio debug WAV dumping.
pub async fn enable_audio_debug_wav_dump(enabled: bool) -> Result<(), BridgeError> {
runtime()
@@ -1746,20 +1761,20 @@ pub async fn set_ios_voice_processing_mode(
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => {
BridgeAudioBackend::PlatformVoiceProcessing
}
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeAudioBackend::Sonora,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeAudioBackend::WebrtcApm,
},
vad_backend: BridgeVadBackend::SileroOnnx,
aec: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => BridgeEffectOwner::Platform,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::Sonora,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::WebrtcApm,
},
ns: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => BridgeEffectOwner::Platform,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::Sonora,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::WebrtcApm,
},
agc: match mode {
BridgeIosVoiceProcessingMode::PlatformVoiceProcessing => BridgeEffectOwner::Platform,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::Sonora,
BridgeIosVoiceProcessingMode::SonoraExperimental => BridgeEffectOwner::WebrtcApm,
},
hpf_enabled: true,
limiter_enabled: true,
crates/chanora_bridge/src/frb_generated.rs
+140 -30
View File
@@ -38,7 +38,7 @@ flutter_rust_bridge::frb_generated_boilerplate!(
default_rust_auto_opaque = RustAutoOpaqueMoi,
);
pub(crate) const FLUTTER_RUST_BRIDGE_CODEGEN_VERSION: &str = "2.12.0";
pub(crate) const FLUTTER_RUST_BRIDGE_CODEGEN_CONTENT_HASH: i32 = -1835973251;
pub(crate) const FLUTTER_RUST_BRIDGE_CODEGEN_CONTENT_HASH: i32 = -436507436;
// Section: executor
@@ -398,6 +398,41 @@ fn wire__crate__api__export_diagnostics_impl(
},
)
}
fn wire__crate__api__get_audio_processing_config_impl(
port_: flutter_rust_bridge::for_generated::MessagePort,
ptr_: flutter_rust_bridge::for_generated::PlatformGeneralizedUint8ListPtr,
rust_vec_len_: i32,
data_len_: i32,
) {
FLUTTER_RUST_BRIDGE_HANDLER.wrap_async::<flutter_rust_bridge::for_generated::SseCodec, _, _, _>(
flutter_rust_bridge::for_generated::TaskInfo {
debug_name: "get_audio_processing_config",
port: Some(port_),
mode: flutter_rust_bridge::for_generated::FfiCallMode::Normal,
},
move || {
let message = unsafe {
flutter_rust_bridge::for_generated::Dart2RustMessageSse::from_wire(
ptr_,
rust_vec_len_,
data_len_,
)
};
let mut deserializer =
flutter_rust_bridge::for_generated::SseDeserializer::new(message);
deserializer.end();
move |context| async move {
transform_result_sse::<_, crate::BridgeError>(
(move || async move {
let output_ok = crate::api::get_audio_processing_config().await?;
Ok(output_ok)
})()
.await,
)
}
},
)
}
fn wire__crate__api__get_ptt_binding_impl(
port_: flutter_rust_bridge::for_generated::MessagePort,
ptr_: flutter_rust_bridge::for_generated::PlatformGeneralizedUint8ListPtr,
@@ -598,6 +633,38 @@ fn wire__crate__api__handle_media_services_reset_impl(
},
)
}
fn wire__crate__api__handle_media_services_reset_with_route_impl(
ptr_: flutter_rust_bridge::for_generated::PlatformGeneralizedUint8ListPtr,
rust_vec_len_: i32,
data_len_: i32,
) -> flutter_rust_bridge::for_generated::WireSyncRust2DartSse {
FLUTTER_RUST_BRIDGE_HANDLER.wrap_sync::<flutter_rust_bridge::for_generated::SseCodec, _>(
flutter_rust_bridge::for_generated::TaskInfo {
debug_name: "handle_media_services_reset_with_route",
port: None,
mode: flutter_rust_bridge::for_generated::FfiCallMode::Sync,
},
move || {
let message = unsafe {
flutter_rust_bridge::for_generated::Dart2RustMessageSse::from_wire(
ptr_,
rust_vec_len_,
data_len_,
)
};
let mut deserializer =
flutter_rust_bridge::for_generated::SseDeserializer::new(message);
let api_route_class = <String>::sse_decode(&mut deserializer);
deserializer.end();
transform_result_sse::<_, ()>((move || {
let output_ok = Result::<_, ()>::Ok({
crate::api::handle_media_services_reset_with_route(api_route_class);
})?;
Ok(output_ok)
})())
},
)
}
fn wire__crate__api__handle_route_change_impl(
ptr_: flutter_rust_bridge::for_generated::PlatformGeneralizedUint8ListPtr,
rust_vec_len_: i32,
@@ -1198,6 +1265,42 @@ fn wire__crate__api__set_release_tail_ms_impl(
},
)
}
fn wire__crate__api__set_ten_vad_model_path_impl(
port_: flutter_rust_bridge::for_generated::MessagePort,
ptr_: flutter_rust_bridge::for_generated::PlatformGeneralizedUint8ListPtr,
rust_vec_len_: i32,
data_len_: i32,
) {
FLUTTER_RUST_BRIDGE_HANDLER.wrap_async::<flutter_rust_bridge::for_generated::SseCodec, _, _, _>(
flutter_rust_bridge::for_generated::TaskInfo {
debug_name: "set_ten_vad_model_path",
port: Some(port_),
mode: flutter_rust_bridge::for_generated::FfiCallMode::Normal,
},
move || {
let message = unsafe {
flutter_rust_bridge::for_generated::Dart2RustMessageSse::from_wire(
ptr_,
rust_vec_len_,
data_len_,
)
};
let mut deserializer =
flutter_rust_bridge::for_generated::SseDeserializer::new(message);
let api_path = <String>::sse_decode(&mut deserializer);
deserializer.end();
move |context| async move {
transform_result_sse::<_, crate::BridgeError>(
(move || async move {
let output_ok = crate::api::set_ten_vad_model_path(api_path).await?;
Ok(output_ok)
})()
.await,
)
}
},
)
}
fn wire__crate__api__set_transmit_mode_impl(
port_: flutter_rust_bridge::for_generated::MessagePort,
ptr_: flutter_rust_bridge::for_generated::PlatformGeneralizedUint8ListPtr,
@@ -2090,31 +2193,33 @@ fn pde_ffi_dispatcher_primary_impl(
7 => wire__crate__api__disconnect_impl(port, ptr, rust_vec_len, data_len),
8 => wire__crate__api__enable_audio_debug_wav_dump_impl(port, ptr, rust_vec_len, data_len),
9 => wire__crate__api__events_stream_impl(port, ptr, rust_vec_len, data_len),
11 => wire__crate__api__get_ptt_binding_impl(port, ptr, rust_vec_len, data_len),
12 => wire__crate__api__get_release_tail_ms_impl(port, ptr, rust_vec_len, data_len),
13 => wire__crate__api__get_transmit_mode_impl(port, ptr, rust_vec_len, data_len),
18 => wire__crate__api__init_storage_impl(port, ptr, rust_vec_len, data_len),
19 => wire__crate__api__is_connected_impl(port, ptr, rust_vec_len, data_len),
20 => wire__crate__api__list_bookmarks_impl(port, ptr, rust_vec_len, data_len),
22 => wire__crate__api__move_to_channel_impl(port, ptr, rust_vec_len, data_len),
23 => wire__crate__api__ptt_descriptor_impl(port, ptr, rust_vec_len, data_len),
24 => wire__crate__api__set_audio_processing_config_impl(port, ptr, rust_vec_len, data_len),
25 => wire__crate__api__set_hard_mute_impl(port, ptr, rust_vec_len, data_len),
26 => wire__crate__api__set_input_muted_impl(port, ptr, rust_vec_len, data_len),
27 => {
11 => wire__crate__api__get_audio_processing_config_impl(port, ptr, rust_vec_len, data_len),
12 => wire__crate__api__get_ptt_binding_impl(port, ptr, rust_vec_len, data_len),
13 => wire__crate__api__get_release_tail_ms_impl(port, ptr, rust_vec_len, data_len),
14 => wire__crate__api__get_transmit_mode_impl(port, ptr, rust_vec_len, data_len),
20 => wire__crate__api__init_storage_impl(port, ptr, rust_vec_len, data_len),
21 => wire__crate__api__is_connected_impl(port, ptr, rust_vec_len, data_len),
22 => wire__crate__api__list_bookmarks_impl(port, ptr, rust_vec_len, data_len),
24 => wire__crate__api__move_to_channel_impl(port, ptr, rust_vec_len, data_len),
25 => wire__crate__api__ptt_descriptor_impl(port, ptr, rust_vec_len, data_len),
26 => wire__crate__api__set_audio_processing_config_impl(port, ptr, rust_vec_len, data_len),
27 => wire__crate__api__set_hard_mute_impl(port, ptr, rust_vec_len, data_len),
28 => wire__crate__api__set_input_muted_impl(port, ptr, rust_vec_len, data_len),
29 => {
wire__crate__api__set_ios_voice_processing_mode_impl(port, ptr, rust_vec_len, data_len)
}
29 => wire__crate__api__set_output_gain_impl(port, ptr, rust_vec_len, data_len),
30 => wire__crate__api__set_output_muted_impl(port, ptr, rust_vec_len, data_len),
31 => wire__crate__api__set_ptt_impl(port, ptr, rust_vec_len, data_len),
32 => wire__crate__api__set_ptt_binding_impl(port, ptr, rust_vec_len, data_len),
33 => wire__crate__api__set_release_tail_ms_impl(port, ptr, rust_vec_len, data_len),
34 => wire__crate__api__set_transmit_mode_impl(port, ptr, rust_vec_len, data_len),
35 => wire__crate__api__set_vad_model_path_impl(port, ptr, rust_vec_len, data_len),
36 => wire__crate__api__snapshot_impl(port, ptr, rust_vec_len, data_len),
37 => wire__crate__api__update_bookmark_impl(port, ptr, rust_vec_len, data_len),
38 => wire__crate__api__voice_join_impl(port, ptr, rust_vec_len, data_len),
39 => wire__crate__api__voice_leave_impl(port, ptr, rust_vec_len, data_len),
31 => wire__crate__api__set_output_gain_impl(port, ptr, rust_vec_len, data_len),
32 => wire__crate__api__set_output_muted_impl(port, ptr, rust_vec_len, data_len),
33 => wire__crate__api__set_ptt_impl(port, ptr, rust_vec_len, data_len),
34 => wire__crate__api__set_ptt_binding_impl(port, ptr, rust_vec_len, data_len),
35 => wire__crate__api__set_release_tail_ms_impl(port, ptr, rust_vec_len, data_len),
36 => wire__crate__api__set_ten_vad_model_path_impl(port, ptr, rust_vec_len, data_len),
37 => wire__crate__api__set_transmit_mode_impl(port, ptr, rust_vec_len, data_len),
38 => wire__crate__api__set_vad_model_path_impl(port, ptr, rust_vec_len, data_len),
39 => wire__crate__api__snapshot_impl(port, ptr, rust_vec_len, data_len),
40 => wire__crate__api__update_bookmark_impl(port, ptr, rust_vec_len, data_len),
41 => wire__crate__api__voice_join_impl(port, ptr, rust_vec_len, data_len),
42 => wire__crate__api__voice_leave_impl(port, ptr, rust_vec_len, data_len),
_ => unreachable!(),
}
}
@@ -2128,12 +2233,17 @@ fn pde_ffi_dispatcher_sync_impl(
// Codec=Pde (Serialization + dispatch), see doc to use other codecs
match func_id {
10 => wire__crate__api__export_diagnostics_impl(ptr, rust_vec_len, data_len),
14 => wire__crate__api__handle_interruption_began_impl(ptr, rust_vec_len, data_len),
15 => wire__crate__api__handle_interruption_ended_impl(ptr, rust_vec_len, data_len),
16 => wire__crate__api__handle_media_services_reset_impl(ptr, rust_vec_len, data_len),
17 => wire__crate__api__handle_route_change_impl(ptr, rust_vec_len, data_len),
21 => wire__crate__api__log_file_path_str_impl(ptr, rust_vec_len, data_len),
28 => wire__crate__api__set_network_state_impl(ptr, rust_vec_len, data_len),
15 => wire__crate__api__handle_interruption_began_impl(ptr, rust_vec_len, data_len),
16 => wire__crate__api__handle_interruption_ended_impl(ptr, rust_vec_len, data_len),
17 => wire__crate__api__handle_media_services_reset_impl(ptr, rust_vec_len, data_len),
18 => wire__crate__api__handle_media_services_reset_with_route_impl(
ptr,
rust_vec_len,
data_len,
),
19 => wire__crate__api__handle_route_change_impl(ptr, rust_vec_len, data_len),
23 => wire__crate__api__log_file_path_str_impl(ptr, rust_vec_len, data_len),
30 => wire__crate__api__set_network_state_impl(ptr, rust_vec_len, data_len),
_ => unreachable!(),
}
}