edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Android Oboe voice backend — WebRTC APM, VAD, HW/SW toggle, BBCode welcome, link trust, foreground task

Browse Source 
Audio engine (Rust):
- Android Oboe: WebRTC APM (AEC/NS/AGC/HPF) + TEN/Silero ONNX VAD
- Hardware effects (JNI) with software fallback per-effect
- Render reference buffer for AEC between output/capture callbacks
- Voice activity gate: suppress transmission when speaker muted (all platforms)
- Audio focus (SDD-109) + Bluetooth SCO (SDD-110) via JNI
- ONNX Runtime 1.26 via ort 2.0.0-rc.12 (down from rc.10, ndarray 0.17)
- VAD worker channel capacity 8→32, initial seq u64::MAX (warm-up fix)
- TEN VAD default backend (was Silero)
- Platform→WebrtcApm resolution after hardware binding
- oboe-rs edisonjwa fork with get_raw_session_id()

Android Kotlin:
- AndroidAudioFocusController + AndroidBluetoothScoController
- AndroidAudioLifecycleController (route changes to Flutter)
- ProGuard rules for new controllers

Flutter UI:
- VoiceSettings: Android HW/SW toggle (Platform auto / WebRTC APM)
- VoiceStatusChip: mute warning border + Speaker muted label
- BBCode welcome message parser (BbCodeText, case-insensitive)
- Welcome message foldable (expanded by default)
- Link trust dialog (domain wildcards, SharedPreferences)
- HapticFeedback on voice sheet opener
- Server name in AppBar, version v0.1.0
- Default channel (id=1) visible, serverquery clients hidden
- flutter_foreground_task integration

Config:
- ort load-dynamic on all non-iOS (Android/Linux/Windows)
- ONNX Runtime AAR 1.26.0
- ndarray moved to common deps (was Apple-only)
This commit is contained in:
Edison Jwa
2026-05-22 09:29:57 +09:00
parent 6af4ecab0f
commit bf284018e6
37 changed files with 3676 additions and 703 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/chanora_audio/src/ios_raw_unit.rs
+132 -27
View File
@@ -1,15 +1,15 @@
//! Optional raw iOS RemoteIO path for the Sonora experimental mode.
//! Optional raw iOS RemoteIO path for the WebRTC APM experimental mode.
//!
//! Provides an alternative to `ios_voice_unit.rs` for the
//! `SonoraExperimental` processing mode. Instead of
//! `kAudioUnitSubType_VoiceProcessingIO` (which owns AEC/NS/AGC), it
//! opens `kAudioUnitSubType_RemoteIO` with voice processing explicitly
//! disabled so Rust's Sonora DSP chain can own the full signal path.
//! disabled so WebRTC APM can own the full signal path.
//!
//! ## Hard invariants enforced here
//!
//! * INV_009: Rust AEC only active when platform AEC is disabled.
//! * INV_010: VoiceProcessingIO and Sonora AEC3 are mutually exclusive.
//! * INV_010: VoiceProcessingIO and WebRTC APM AEC are mutually exclusive.
//! * INV_011: Software AEC backend receives both capture and render-reference.
//! * INV_012: Render reference is copied from decoded/mixed remote PCM
//! before playout.
@@ -112,19 +112,25 @@ mod inner {
opus_out: [u8; crate::opus_voice::MAX_OPUS_FRAME],
voice_out_tx: mpsc::Sender<OutPacket>,
transmit_active: Arc<AtomicBool>,
output_muted: Arc<AtomicBool>,
frames_sent: Arc<AtomicU32>,
mic_gain: f32,
voice_activity_selector: Option<Arc<crate::TransmitModeSelector>>,
vad_detector: crate::vad::WebRtcFallbackVad,
silero_vad_worker: Option<crate::vad::silero_onnx::SileroOnnxVadWorker>,
ten_vad_worker: Option<crate::vad::TenOnnxVadWorker>,
current_vad_backend: crate::VadBackend,
silero_model_epoch: u64,
capture_frame_seq: u64,
vad_state: crate::voice_activity::VoiceActivityStateMachine,
/// Processing config — retained for route-change reloads; not read in the hot path.
#[allow(dead_code)]
/// Processing config — retained for route-change reloads.
audio_processing_config: Arc<Mutex<crate::AudioProcessingConfig>>,
sonora_processor: crate::processor::SonoraProcessor,
webrtc_apm_processor: crate::processor::WebRtcApmProcessor,
audio_processing_stats: Arc<crate::SharedAudioProcessingStats>,
render_reference: Arc<RenderReferenceBuffer>,
pending_10ms: [i16; crate::frame::FRAME_10MS_SAMPLES],
pending_10ms_len: usize,
fallback_warned_backend: Option<crate::VadBackend>,
wav_recorder: Option<Arc<crate::debug_wav::WavDebugRecorder>>,
}
@@ -132,6 +138,7 @@ mod inner {
fn new(
voice_out_tx: mpsc::Sender<OutPacket>,
transmit_active: Arc<AtomicBool>,
output_muted: Arc<AtomicBool>,
frames_sent: Arc<AtomicU32>,
mic_gain: f32,
voice_activity_selector: Option<Arc<crate::TransmitModeSelector>>,
@@ -140,32 +147,48 @@ mod inner {
render_reference: Arc<RenderReferenceBuffer>,
) -> Result<Self, AudioError> {
let encoder = crate::opus_voice::new_voip_encoder("ios raw")?;
let webrtc_apm_config = audio_processing_config
.lock()
.map(|cfg| crate::processor::webrtc_apm::WebRtcApmConfig::from_audio_config(&cfg))
.unwrap_or_default();
Ok(Self {
encoder,
pcm_accum: Vec::with_capacity(crate::frame::FRAME_20MS_SAMPLES * 2),
opus_out: [0u8; crate::opus_voice::MAX_OPUS_FRAME],
voice_out_tx,
transmit_active,
output_muted,
frames_sent,
mic_gain,
voice_activity_selector,
vad_detector: crate::vad::WebRtcFallbackVad::default(),
silero_vad_worker: None,
ten_vad_worker: None,
current_vad_backend: crate::VadBackend::WebrtcVad,
silero_model_epoch: crate::vad::silero_model_epoch(),
capture_frame_seq: 0,
vad_state: crate::voice_activity::VoiceActivityStateMachine::default(),
audio_processing_config,
sonora_processor: crate::processor::SonoraProcessor::with_config(
crate::processor::sonora::SonoraConfig::with_aec3(),
),
webrtc_apm_processor: crate::processor::WebRtcApmProcessor::with_config(
webrtc_apm_config,
)?,
audio_processing_stats,
render_reference,
pending_10ms: [0_i16; crate::frame::FRAME_10MS_SAMPLES],
pending_10ms_len: 0,
fallback_warned_backend: None,
wav_recorder: None,
})
}
fn disable_failed_vad_backend(&mut self, failed_backend: crate::VadBackend) {
if let Ok(mut cfg) = self.audio_processing_config.try_lock() {
let _ = cfg.disable_failed_vad_backend(failed_backend);
fn mark_vad_fallback_active(&mut self, failed_backend: crate::VadBackend) {
if self.fallback_warned_backend != Some(failed_backend) {
self.fallback_warned_backend = Some(failed_backend);
tracing::warn!(
target: "chanora_audio",
backend = failed_backend.as_str(),
"VAD backend unavailable; using WebRTC fallback for runtime detection"
);
}
}
@@ -237,12 +260,12 @@ mod inner {
rec.push_raw_mic(&frame);
}
// INV_012: feed render reference to Sonora AEC3 before capture.
// Feed render reference to WebRTC APM before capture so AEC can adapt.
let render_ref = self.render_reference.read_latest();
self.sonora_processor.process_render(&render_ref);
self.sonora_processor.process_capture(&mut frame);
self.webrtc_apm_processor.process_render(&render_ref);
self.webrtc_apm_processor.process_capture(&mut frame);
// WAV tap: processed mic (after Sonora).
// WAV tap: processed mic (after WebRTC APM).
if let Some(ref rec) = self.wav_recorder {
rec.push_processed_mic(&frame);
}
@@ -260,37 +283,118 @@ mod inner {
vad_hangover,
crate::voice_activity::VAD_MIN_TX_MS,
);
// Switch VAD backend when config changes.
let silero_epoch = crate::vad::silero_model_epoch();
let silero_changed = vad_backend == crate::VadBackend::SileroOnnx
&& silero_epoch != self.silero_model_epoch;
if vad_backend != self.current_vad_backend || silero_changed {
self.current_vad_backend = vad_backend;
self.silero_model_epoch = silero_epoch;
self.fallback_warned_backend = None;
match vad_backend {
crate::VadBackend::SileroOnnx => {
let path = crate::vad::silero_model_bundle_path();
self.silero_vad_worker =
crate::vad::silero_onnx::SileroOnnxVadWorker::try_new(&path);
self.ten_vad_worker = None;
if self.silero_vad_worker.is_none() {
tracing::warn!(
target: "chanora_audio",
"Silero VAD model not found at {path}; falling back to WebRTC VAD"
);
}
}
crate::VadBackend::TenVad => {
let path = crate::vad::ten_model_bundle_path();
self.ten_vad_worker = crate::vad::TenOnnxVadWorker::try_new(&path);
self.silero_vad_worker = None;
if self.ten_vad_worker.is_none() {
tracing::warn!(
target: "chanora_audio",
"TEN VAD ONNX model not found at {path}; falling back to WebRTC VAD"
);
}
}
_ => {
self.silero_vad_worker = None;
self.ten_vad_worker = None;
}
}
self.vad_state.reset();
}
self.capture_frame_seq = self.capture_frame_seq.wrapping_add(1);
let capture_seq = self.capture_frame_seq;
let mut used_fallback_vad = false;
let vad = if vad_backend == crate::VadBackend::Disabled {
crate::vad::VadOutput {
probability: 1.0,
speech: true,
}
} else {
used_fallback_vad = matches!(
vad_backend,
crate::VadBackend::SileroOnnx | crate::VadBackend::TenVad
);
if used_fallback_vad {
self.disable_failed_vad_backend(vad_backend);
} else if vad_backend == crate::VadBackend::SileroOnnx {
if let Some(worker) = self.silero_vad_worker.as_ref() {
let enqueued = worker.try_send(capture_seq, &frame);
if enqueued && !worker.is_stale(capture_seq) {
let p = worker.latest_probability();
crate::vad::VadOutput {
probability: p,
speech: p >= 0.5,
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
&frame,
)
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(&mut self.vad_detector, &frame)
}
} else if vad_backend == crate::VadBackend::TenVad {
if let Some(worker) = self.ten_vad_worker.as_ref() {
let enqueued = worker.try_send(capture_seq, &frame);
if enqueued && !worker.is_stale(capture_seq) {
let p = worker.latest_probability();
crate::vad::VadOutput {
probability: p,
speech: p >= 0.5,
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(
&mut self.vad_detector,
&frame,
)
}
} else {
used_fallback_vad = true;
self.mark_vad_fallback_active(vad_backend);
crate::vad::VoiceActivityDetector::process_10ms(&mut self.vad_detector, &frame)
}
} else {
crate::vad::VoiceActivityDetector::process_10ms(&mut self.vad_detector, &frame)
};
self.audio_processing_stats
.set_vad_fallback_active(used_fallback_vad);
let active = self.vad_state.update(vad.speech);
let output_muted = self.output_muted.load(Ordering::Relaxed);
if let Some(sel) = &self.voice_activity_selector {
sel.set_voice_activity_open(active);
sel.set_voice_activity_open(active && !output_muted);
}
self.audio_processing_stats.update_capture(
input_dbfs,
crate::frame::dbfs(&frame),
vad.probability,
active,
active && !output_muted,
self.transmit_active.load(Ordering::Relaxed),
);
if !self.transmit_active.load(Ordering::Relaxed) {
if !self.transmit_active.load(Ordering::Relaxed) || output_muted {
return;
}
@@ -336,7 +440,7 @@ mod inner {
let cfg = audio_processing_config.lock().unwrap();
if cfg.ios_mode == crate::IosVoiceProcessingMode::PlatformVoiceProcessing {
return Err(AudioError::InvalidAudioProcessingConfig(
"IosRawUnit requires SonoraExperimental mode".to_string(),
"IosRawUnit requires raw WebRTC APM mode".to_string(),
));
}
}
@@ -369,6 +473,7 @@ mod inner {
let mut capture_state = RawCaptureState::new(
voice_out_tx,
transmit_active,
output_muted.clone(),
frames_sent,
mic_gain,
voice_activity_selector,