710 lines
32 KiB
Rust
710 lines
32 KiB
Rust
//! iOS output + capture stream via the **VoiceProcessingIO**
|
|
//! AudioUnit (`kAudioUnitSubType_VoiceProcessingIO`, a.k.a. VPIO).
|
|
//!
|
|
//! ## Why not cpal on iOS
|
|
//!
|
|
//! cpal's iOS backend opens `kAudioUnitSubType_RemoteIO` with no
|
|
//! control over the stream format, buffer size, or channel count;
|
|
//! on iPhone 16 Pro running iOS 18 it reports the output element as
|
|
//! **mono 48 kHz** even when the session category is `.playAndRecord`
|
|
//! with mode `.default`. More importantly, RemoteIO opened by cpal
|
|
//! stays bound to the route that was active at construction time:
|
|
//! a later `AVAudioSession.overrideOutputAudioPort(.speaker)` flips
|
|
//! the session route metadata (visible in
|
|
//! `AVAudioSession.currentRoute`) but the underlying AudioUnit
|
|
//! keeps writing to the original transducer. End-user symptom: the
|
|
//! Speaker / Receiver toggle in our picker shows the route change
|
|
//! in logs but produces no audible difference — the audio is still
|
|
//! coming out the earpiece.
|
|
//!
|
|
//! Every production iOS VoIP client (Mumble iOS, Linphone /
|
|
//! mediastreamer2, Signal-iOS, Jitsi, the WebRTC reference impl)
|
|
//! avoids RemoteIO and drives VPIO directly instead. VPIO is
|
|
//! Apple's recommended voice unit: it ships hardware AEC + AGC +
|
|
//! NS, it accepts arbitrary stream-format requests on bus 0
|
|
//! (output) and bus 1 (input), and it re-binds its underlying HAL
|
|
//! transducer correctly when the AVAudioSession route changes,
|
|
//! because it IS the canonical voice unit on iOS — Apple's own
|
|
//! FaceTime audio path runs through it.
|
|
//!
|
|
//! This file replaces the cpal capture + playback streams on iOS
|
|
//! only. macOS continues to use cpal's CoreAudio HAL backend (which
|
|
//! works correctly for desktop audio). Linux uses SDL2 (see
|
|
//! `sdl_output.rs`). Windows uses cpal's WASAPI backend.
|
|
//!
|
|
//! ## What VPIO gives us
|
|
//!
|
|
//! * Pinned **48 kHz Int16 mono** stream format on both bus 0
|
|
//! (output to hardware) and bus 1 (input from hardware). 48 kHz
|
|
//! matches the Opus encoder + `tsclientlib::AudioHandler` mix
|
|
//! rate exactly, so no resampling is needed inside the audio
|
|
//! callback. Int16 is Apple's documented canonical iOS sample
|
|
//! format for AudioUnits (see Audio Unit Hosting Guide for iOS
|
|
//! §"Canonical formats").
|
|
//! * Hardware **AEC** (acoustic echo cancellation), **AGC**
|
|
//! (automatic gain control), and **NS** (noise suppression) ran
|
|
//! in the secure-enclave-adjacent voice processor. Free DSP that
|
|
//! we'd otherwise need to ship as software (DEC-007/008/009).
|
|
//! * **Route-change-correct** physical binding: tapping Speaker or
|
|
//! Receiver in our picker now actually moves the audio.
|
|
//!
|
|
//! ## Threading & lifecycle
|
|
//!
|
|
//! `coreaudio::audio_unit::AudioUnit` is `Send` but `!Sync` — the
|
|
//! AudioUnit internally holds the C `AudioUnit` opaque pointer and
|
|
//! the wrapper's destructor calls `AudioComponentInstanceDispose`,
|
|
//! which (per Apple's threading rules) must be called from the
|
|
//! thread that owns the unit. We open the unit on the same thread
|
|
//! that calls `Self::start` (the tokio worker that runs
|
|
//! `chanora_core::ChanoraSession::start_audio`, the same pattern
|
|
//! cpal + SDL use) and never move it. The outer `AudioEngine`
|
|
//! already carries an `unsafe impl Send` to satisfy the same
|
|
//! constraint for cpal's `!Send` Stream type; that impl covers
|
|
//! VPIO too.
|
|
//!
|
|
//! Dropping `IosVoiceUnit` calls `audio_unit.stop()` via the
|
|
//! wrapper's `Drop`, which detaches the render + input callbacks
|
|
//! and stops the unit. The AudioHandler + CaptureState `Arc`s the
|
|
//! callbacks held are then released.
|
|
//!
|
|
//! ## What this file does NOT do
|
|
//!
|
|
//! * Route-change observation — that lives in Swift
|
|
//! (`AppDelegate.handleRouteChange`) and bounces the unit via a
|
|
//! future FRB call. Tracked as Commit 5 of the VPIO rollout.
|
|
//! * AVAudioSession category / mode configuration — Swift owns the
|
|
//! session (it must be set up before Flutter loads).
|
|
|
|
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
|
|
use std::sync::{Arc, Mutex};
|
|
|
|
use audiopus::coder::Encoder as OpusEncoder;
|
|
use audiopus::{
|
|
Application as OpusApp, Bitrate as OpusBitrate, Channels as OpusChannels,
|
|
SampleRate as OpusSampleRate,
|
|
};
|
|
use coreaudio::audio_unit::audio_format::LinearPcmFlags;
|
|
use coreaudio::audio_unit::render_callback::{self, data};
|
|
use coreaudio::audio_unit::IOType;
|
|
use coreaudio::audio_unit::{AudioUnit, Element, SampleFormat, Scope, StreamFormat};
|
|
use tokio::sync::mpsc;
|
|
use tracing::{debug, error, info, warn};
|
|
use tsclientlib::audio::AudioHandler;
|
|
|
|
use crate::engine::SessionAudioId;
|
|
use crate::AudioError;
|
|
use chanora_protocol::{AudioData, CodecType, OutAudio, OutPacket};
|
|
|
|
/// 20 ms at 48 kHz mono — one Opus frame's worth of samples.
|
|
/// Aligning the AudioUnit IO buffer to this frame size keeps the
|
|
/// jitter-buffer / encoder handshake tight (no fractional-frame
|
|
/// reads inside fill_buffer or accumulator drift inside the
|
|
/// capture pipeline).
|
|
const FRAME_SAMPLES_MONO: usize = 960;
|
|
|
|
/// Maximum size of an encoded Opus frame in bytes (per RFC 6716
|
|
/// §3.2.1). Same constant the cpal-side `CaptureState` uses; we
|
|
/// duplicate it here instead of cross-importing from engine.rs
|
|
/// because engine.rs's copy is cfg-gated to non-iOS for cpal-only
|
|
/// reasons. Post-step-5 review may dedupe by promoting both to a
|
|
/// shared `crate::framing` module.
|
|
const MAX_OPUS_FRAME: usize = 1275;
|
|
|
|
/// Sample rate every layer above us assumes. Matches the Opus
|
|
/// encoder rate, the `tsclientlib::AudioHandler` mix rate, and the
|
|
/// sample rate we ask iOS to give us via VPIO's StreamFormat.
|
|
const SAMPLE_RATE_HZ: f64 = 48_000.0;
|
|
|
|
/// Output element (bus 0) of an `IOType::VoiceProcessingIO` unit
|
|
/// drives the hardware speaker / receiver / AirPods / BT. The
|
|
/// stream format we set on `Scope::Input` of this element is the
|
|
/// format **we** push samples in; VPIO converts internally to
|
|
/// whatever the hardware needs.
|
|
const OUTPUT_BUS: Element = Element::Output;
|
|
|
|
/// Input element (bus 1) of an `IOType::VoiceProcessingIO` unit
|
|
/// pulls from the hardware microphone. The stream format we set on
|
|
/// `Scope::Output` of this element is the format **we** receive
|
|
/// samples in.
|
|
const INPUT_BUS: Element = Element::Input;
|
|
|
|
/// Capture pipeline state owned by the VPIO input callback. The
|
|
/// AudioUnit hands us 48 kHz signed-int16 mono PCM directly (no
|
|
/// downmix or resample needed — VPIO's hardware-side mix-down
|
|
/// from whatever the route's native format is happens before we
|
|
/// see the samples). All this struct does is gate on PTT, scale
|
|
/// by mic_gain, accumulate to a 20 ms / 960-sample frame, encode
|
|
/// to Opus, and try-send the resulting packet on the protocol
|
|
/// queue.
|
|
///
|
|
/// Mirrors the cpal-side `CaptureState` in engine.rs but is
|
|
/// type-specialised to i16 (the cpal version is generic over
|
|
/// `T: ToF32` to handle arbitrary HAL formats). Same Opus VoIP
|
|
/// tuning (32 kbps, complexity 10, inband FEC, 5% packet-loss
|
|
/// budget) lifted verbatim from `try_open_capture` so iOS audio
|
|
/// quality matches every other platform.
|
|
///
|
|
/// This struct is moved into the VPIO `set_input_callback` closure
|
|
/// and is therefore `'static + Send`. The OpusEncoder + Vec + arrays
|
|
/// are all owned; the two atomics + sender are `Arc<...>` clones
|
|
/// shared with `AudioEngine`.
|
|
struct IosCaptureState {
|
|
encoder: OpusEncoder,
|
|
/// 48 kHz mono PCM scratch accumulating to FRAME_SAMPLES_MONO
|
|
/// per encode. Capacity 2x to absorb cpal-style buffer-size
|
|
/// jitter without reallocating.
|
|
pcm_accum: Vec<i16>,
|
|
opus_out: [u8; MAX_OPUS_FRAME],
|
|
voice_out_tx: mpsc::Sender<OutPacket>,
|
|
/// PTT transmission gate. Read once per outbound frame; this
|
|
/// struct never mutates the flag (SAD-075 / SDD-089).
|
|
transmit_active: Arc<AtomicBool>,
|
|
frames_sent: Arc<AtomicU32>,
|
|
mic_gain: f32,
|
|
}
|
|
|
|
impl IosCaptureState {
|
|
/// Build a VoIP-tuned Opus encoder + the capture-state wrapper.
|
|
/// Encoder configuration is the same as cpal-side
|
|
/// `try_open_capture` (engine.rs) so audio quality is platform-
|
|
/// neutral.
|
|
fn new(
|
|
voice_out_tx: mpsc::Sender<OutPacket>,
|
|
transmit_active: Arc<AtomicBool>,
|
|
frames_sent: Arc<AtomicU32>,
|
|
mic_gain: f32,
|
|
) -> Result<Self, AudioError> {
|
|
let mut encoder =
|
|
OpusEncoder::new(OpusSampleRate::Hz48000, OpusChannels::Mono, OpusApp::Voip)
|
|
.map_err(|e| AudioError::Opus(format!("encoder new (ios): {e}")))?;
|
|
|
|
// VoIP-tuned settings — bitrate 32 kbps, complexity 10,
|
|
// inband FEC on, packet-loss-perc 5. Soft-fail each setter
|
|
// with a warn log to match the cpal-side behaviour: an
|
|
// unusual libopus build that rejects one setter shouldn't
|
|
// tank the whole pipeline. Full rationale + RFC citations
|
|
// are in engine.rs::try_open_capture line ~640.
|
|
if let Err(e) = encoder.set_bitrate(OpusBitrate::BitsPerSecond(32_000)) {
|
|
warn!(target: "chanora_audio", error = %e, "opus(ios): set_bitrate(32000) failed");
|
|
}
|
|
if let Err(e) = encoder.set_complexity(10) {
|
|
warn!(target: "chanora_audio", error = %e, "opus(ios): set_complexity(10) failed");
|
|
}
|
|
if let Err(e) = encoder.set_inband_fec(true) {
|
|
warn!(target: "chanora_audio", error = %e, "opus(ios): set_inband_fec(true) failed");
|
|
}
|
|
if let Err(e) = encoder.set_packet_loss_perc(5) {
|
|
warn!(target: "chanora_audio", error = %e, "opus(ios): set_packet_loss_perc(5) failed");
|
|
}
|
|
info!(
|
|
target: "chanora_audio",
|
|
bitrate_bps = 32_000,
|
|
complexity = 10,
|
|
inband_fec = true,
|
|
packet_loss_perc = 5,
|
|
"ios VPIO opus encoder tuned for VoIP"
|
|
);
|
|
|
|
Ok(Self {
|
|
encoder,
|
|
pcm_accum: Vec::with_capacity(FRAME_SAMPLES_MONO * 2),
|
|
opus_out: [0u8; MAX_OPUS_FRAME],
|
|
voice_out_tx,
|
|
transmit_active,
|
|
frames_sent,
|
|
mic_gain,
|
|
})
|
|
}
|
|
|
|
/// Consume the i16 mono buffer delivered by VPIO, accumulate
|
|
/// to a 20 ms frame boundary, encode + send when PTT is held.
|
|
///
|
|
/// VPIO's input element delivers samples already at the
|
|
/// stream format we pinned (48 kHz Int16 mono interleaved).
|
|
/// In practice "interleaved mono" is the same byte layout as
|
|
/// "planar mono" so we just take the buffer as-is.
|
|
fn ingest_i16(&mut self, samples: &[i16]) {
|
|
if !self.transmit_active.load(Ordering::Relaxed) {
|
|
// Drain accumulator while muted so we don't pop on the
|
|
// PTT release edge. Matches cpal-side behaviour.
|
|
self.pcm_accum.clear();
|
|
return;
|
|
}
|
|
|
|
// Mic-gain application. When gain==1.0 we skip the
|
|
// multiply + saturate loop entirely — that's the common
|
|
// case and the loop is the inner-most hot path of the
|
|
// realtime audio thread.
|
|
if (self.mic_gain - 1.0).abs() < f32::EPSILON {
|
|
self.pcm_accum.extend_from_slice(samples);
|
|
} else {
|
|
let gain = self.mic_gain;
|
|
self.pcm_accum.extend(samples.iter().map(|&s| {
|
|
// Saturating mul-then-cast keeps the signal in
|
|
// the i16 envelope. Clipping in this branch is
|
|
// expected — if the user pushed mic_gain past 1.0
|
|
// and is shouting, the alternative is wrap-around
|
|
// distortion which sounds far worse.
|
|
let scaled = (s as f32) * gain;
|
|
scaled.clamp(i16::MIN as f32, i16::MAX as f32) as i16
|
|
}));
|
|
}
|
|
|
|
// Drain complete 20 ms frames out of the accumulator, encode
|
|
// each, send the resulting Opus packet on the protocol
|
|
// queue. The `while` covers the case where a single VPIO
|
|
// callback delivers more than one frame's worth (rare on
|
|
// iOS where the HW IO buffer duration aligns with the
|
|
// Opus frame, but always possible during route changes).
|
|
while self.pcm_accum.len() >= FRAME_SAMPLES_MONO {
|
|
// Use a stack-allocated frame buffer to avoid the
|
|
// per-callback allocation a `drain(..N).collect()`
|
|
// would incur. The encoder doesn't need ownership.
|
|
let mut frame = [0i16; FRAME_SAMPLES_MONO];
|
|
frame.copy_from_slice(&self.pcm_accum[..FRAME_SAMPLES_MONO]);
|
|
self.pcm_accum.drain(..FRAME_SAMPLES_MONO);
|
|
|
|
match self.encoder.encode(&frame, &mut self.opus_out[..]) {
|
|
Ok(len) => {
|
|
let packet = OutAudio::new(&AudioData::C2S {
|
|
id: 0,
|
|
codec: CodecType::OpusVoice,
|
|
data: &self.opus_out[..len],
|
|
});
|
|
match self.voice_out_tx.try_send(packet) {
|
|
Ok(()) => {
|
|
self.frames_sent.fetch_add(1, Ordering::Relaxed);
|
|
}
|
|
Err(mpsc::error::TrySendError::Full(_)) => {
|
|
warn!(
|
|
target: "chanora_audio",
|
|
"ios VPIO: voice_out queue full; dropping frame"
|
|
);
|
|
}
|
|
Err(mpsc::error::TrySendError::Closed(_)) => {
|
|
debug!(
|
|
target: "chanora_audio",
|
|
"ios VPIO: voice_out closed; capture pipeline stopping"
|
|
);
|
|
}
|
|
}
|
|
}
|
|
Err(e) => {
|
|
error!(target: "chanora_audio", error = %e, "ios VPIO opus encode failed");
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Live iOS audio unit wrapper. Construct + start = audio
|
|
/// flowing; drop = audio stopped.
|
|
pub struct IosVoiceUnit {
|
|
// Drop = stop the audio unit (severs render callback). The
|
|
// wrapper's own Drop calls AudioComponentInstanceDispose
|
|
// after stop returns.
|
|
unit: AudioUnit,
|
|
}
|
|
|
|
impl IosVoiceUnit {
|
|
/// Open a VoiceProcessingIO AudioUnit, pin its stream format
|
|
/// to 48 kHz Int16 mono on both buses, install render + input
|
|
/// callbacks, and start it. The unit begins pumping audio
|
|
/// immediately on return — input callback fires when the mic
|
|
/// captures samples, render callback fires when the hardware
|
|
/// needs samples to play.
|
|
///
|
|
/// Parameters mirror the capture + playback inputs the cpal
|
|
/// and SDL backends accept so engine.rs can swap backends with
|
|
/// a `cfg`.
|
|
///
|
|
/// * `handler` — shared AudioHandler the inbound forwarder
|
|
/// feeds Opus packets into. The output render callback pulls
|
|
/// decoded f32 frames from it (48 kHz stereo) and downmixes
|
|
/// to the i16 mono buffer VPIO expects.
|
|
/// * `output_gain` / `output_muted` — same atomics the cpal
|
|
/// and SDL output paths read on every callback so the master
|
|
/// volume + local-mute UI works identically across backends.
|
|
/// * `voice_out_tx` — channel the capture pipeline sends
|
|
/// encoded `OutPacket`s on.
|
|
/// * `transmit_active` — PTT gate flag the capture pipeline
|
|
/// consults before encoding.
|
|
/// * `frames_sent` — counter the bridge stats surface reads.
|
|
/// * `mic_gain` — pre-encode amplitude scale.
|
|
///
|
|
/// Capture wiring landed in commit 3; playback wiring landed
|
|
/// in commit 4. Route-change observation is commit 5.
|
|
#[allow(clippy::too_many_arguments)]
|
|
pub fn start(
|
|
handler: Arc<Mutex<AudioHandler<SessionAudioId>>>,
|
|
output_gain: Arc<AtomicU32>,
|
|
output_muted: Arc<AtomicBool>,
|
|
voice_out_tx: mpsc::Sender<OutPacket>,
|
|
transmit_active: Arc<AtomicBool>,
|
|
frames_sent: Arc<AtomicU32>,
|
|
mic_gain: f32,
|
|
) -> Result<Self, AudioError> {
|
|
// Construct the VoiceProcessingIO AudioUnit. cpal exposes
|
|
// `Default::default()` which on iOS picks the inferior
|
|
// RemoteIO unit; we explicitly pick VPIO. `coreaudio-rs`
|
|
// returns an already-initialized unit from `new`, but we
|
|
// need to set properties before init so use
|
|
// `new_uninitialized` and call `initialize` ourselves
|
|
// after the property set is complete.
|
|
let mut unit = AudioUnit::new_uninitialized(IOType::VoiceProcessingIO)
|
|
.map_err(|e| AudioError::Backend(format!("vpio audio unit new: {e}")))?;
|
|
|
|
// Enable input I/O on element 1. VPIO's input element is
|
|
// OFF by default — without this `kAudioOutputUnitProperty_EnableIO`
|
|
// toggle no audio flows in and the input callback never
|
|
// fires. The constant is 2003 per Apple's headers
|
|
// (`AudioUnitProperties.h`). The value is a u32 with
|
|
// 1 = enabled. Element::Input on `Scope::Input` is the
|
|
// mic side of the unit (element 1 of bus 1; despite the
|
|
// confusing nomenclature, `Scope::Input` here means
|
|
// "input to the unit" i.e. mic samples coming IN).
|
|
//
|
|
// Output element 0 is enabled by default for any
|
|
// `kAudioUnitType_Output` subtype (which VPIO is), so we
|
|
// don't need to toggle anything for playback.
|
|
//
|
|
// Apple's documented sequence for VPIO setup:
|
|
// 1. AudioComponentInstanceNew (= AudioUnit::new_uninitialized)
|
|
// 2. EnableIO on element 1 (= this set_property call)
|
|
// 3. Set stream format on both elements
|
|
// 4. Install callbacks
|
|
// 5. AudioUnitInitialize (= unit.initialize)
|
|
// 6. AudioOutputUnitStart (= unit.start)
|
|
const K_AUDIO_OUTPUT_UNIT_PROPERTY_ENABLE_IO: u32 = 2003;
|
|
let enable: u32 = 1;
|
|
unit.set_property(
|
|
K_AUDIO_OUTPUT_UNIT_PROPERTY_ENABLE_IO,
|
|
Scope::Input,
|
|
Element::Input,
|
|
Some(&enable),
|
|
)
|
|
.map_err(|e| AudioError::Backend(format!("vpio enable input I/O: {e}")))?;
|
|
|
|
// Note: we keep VPIO's voice processing chain ENABLED
|
|
// (AEC + AGC + NS on the mic path) because it gives us
|
|
// clean capture for free. The historical playback
|
|
// breakage we attributed to this chain (commit c16318c
|
|
// tried to bypass it) was actually caused by the
|
|
// AVAudioSession mode .voiceChat ducking output to the
|
|
// earpiece \u2014 fixed in AppDelegate.swift by switching
|
|
// to .default + .defaultToSpeaker. With the session mode
|
|
// correct, voice processing can stay on.
|
|
|
|
// Stream format. Apple's iOS canonical format for
|
|
// AudioUnits is Linear PCM, 16-bit signed integer samples
|
|
// (see "Canonical formats" in the Audio Unit Hosting Guide
|
|
// for iOS). Mono channel matches the Opus encoder + the
|
|
// mic input pipe. 48 kHz matches every layer above us.
|
|
//
|
|
// The StreamFormat is set on:
|
|
// * OUTPUT_BUS (bus 0), Scope::Input — the format WE
|
|
// push to the unit, i.e. what fill_buffer writes.
|
|
// * INPUT_BUS (bus 1), Scope::Output — the format we
|
|
// RECEIVE from the unit, i.e. what the mic input
|
|
// callback hands us.
|
|
// This pairing is documented in Audio Unit Hosting Guide
|
|
// for iOS §"Specifying the Audio Stream Format".
|
|
let stream_format = StreamFormat {
|
|
sample_rate: SAMPLE_RATE_HZ,
|
|
sample_format: SampleFormat::I16,
|
|
flags: LinearPcmFlags::IS_SIGNED_INTEGER | LinearPcmFlags::IS_PACKED,
|
|
channels: 1,
|
|
};
|
|
|
|
unit.set_stream_format(stream_format, Scope::Input, OUTPUT_BUS)
|
|
.map_err(|e| {
|
|
AudioError::StreamConfig(format!(
|
|
"vpio set output stream format (bus 0 input scope): {e}"
|
|
))
|
|
})?;
|
|
unit.set_stream_format(stream_format, Scope::Output, INPUT_BUS)
|
|
.map_err(|e| {
|
|
AudioError::StreamConfig(format!(
|
|
"vpio set input stream format (bus 1 output scope): {e}"
|
|
))
|
|
})?;
|
|
|
|
// Build the capture pipeline and move it into the input
|
|
// callback. The Opus encoder + accumulator + opus_out
|
|
// scratch are owned by the closure — no Mutex needed
|
|
// because the input callback is the sole writer/reader on
|
|
// the audio thread.
|
|
let mut capture_state =
|
|
IosCaptureState::new(voice_out_tx, transmit_active, frames_sent, mic_gain)?;
|
|
|
|
unit.set_input_callback(move |args: render_callback::Args<data::Interleaved<i16>>| {
|
|
// VPIO with our pinned stream format delivers
|
|
// interleaved Int16 mono. `args.data.buffer` is a
|
|
// `&mut [i16]` of length num_frames * channels = N * 1.
|
|
// coreaudio-rs handles the AudioBufferList plumbing +
|
|
// the AudioUnitRender call internally before invoking
|
|
// this closure.
|
|
capture_state.ingest_i16(args.data.buffer);
|
|
Ok(())
|
|
})
|
|
.map_err(|e| AudioError::Backend(format!("vpio set input callback: {e}")))?;
|
|
|
|
// Install the render callback that drives playback. The
|
|
// buffer iOS hands us is uninitialised — we MUST fill it
|
|
// (writing silence if we have nothing, never leaving stale
|
|
// frames).
|
|
//
|
|
// Pipeline per callback:
|
|
// 1. Lock the AudioHandler, ask it to fill a scratch
|
|
// f32 stereo buffer (length = 2 * num_frames). The
|
|
// handler runs Opus decode + per-client jitter
|
|
// buffer + mix. Same primitive cpal + SDL output
|
|
// paths use; this is the platform-neutral playback
|
|
// contract from `tsclientlib::audio::AudioHandler`.
|
|
// 2. Downmix to i16 mono with master gain. VPIO expects
|
|
// mono int16 (the stream format we pinned above);
|
|
// the handler produces stereo f32. We average L+R
|
|
// to a single mono channel rather than dropping R —
|
|
// the cpal-side mono-output path made the same
|
|
// mistake briefly (commit 6a4dbad / fix) and lost
|
|
// half the spatial mix.
|
|
// 3. Local-mute zeroes the output but STILL drains
|
|
// AudioHandler in step 1 so its jitter buffer
|
|
// doesn't grow unbounded while muted. This is the
|
|
// contract every other backend follows (matches
|
|
// SdlOutput::callback and the cpal output stream).
|
|
//
|
|
// Build the playback pipeline (direct fill_buffer in
|
|
// render callback; matches tsclientlib's reference SDL
|
|
// example at
|
|
// tsclientlib/examples/audio_utils/ts_to_audio.rs).
|
|
//
|
|
// The earlier ring-buffer attempt (rc.8+73..+74) decoupled
|
|
// AudioHandler from the render callback via a 50 Hz
|
|
// producer task + SPSC ring buffer. The +74 diagnostics
|
|
// showed that approach was making things worse: the
|
|
// producer drained AudioHandler at 50 Hz, but iOS VPIO
|
|
// calls our render callback at ~43.5 Hz (consuming 1440
|
|
// mono samples per 23 ms call). With consumer slightly
|
|
// slower than producer in chunks-per-second but each
|
|
// consumer pull being larger, the ring averaged out empty
|
|
// — fill_buffer was returning silence 65-84% of ticks
|
|
// because we drained it too aggressively before packets
|
|
// arrived. Linux/SDL's same pattern works fine because
|
|
// SDL calls fill_buffer at exactly the device callback
|
|
// rate.
|
|
//
|
|
// Revert to direct call: render callback locks
|
|
// AudioHandler, asks for `num_frames` stereo frames, and
|
|
// immediately downmixes to i16 mono into the output
|
|
// buffer. Same as Linux/SDL, just stereo-f32 -> mono-i16
|
|
// converted at the boundary.
|
|
let mut scratch_stereo: Vec<f32> = Vec::with_capacity(2048);
|
|
let handler_for_render = handler.clone();
|
|
let output_gain_for_render = output_gain.clone();
|
|
let output_muted_for_render = output_muted.clone();
|
|
// Diagnostic counters (sampled every 100 callbacks ~= 2 s).
|
|
let mut cb_count: u64 = 0;
|
|
let mut last_num_frames: usize = 0;
|
|
let mut num_frames_changes: u32 = 0;
|
|
let mut callbacks_with_audio: u64 = 0;
|
|
let mut callbacks_with_silence: u64 = 0;
|
|
unit.set_render_callback(move |args: render_callback::Args<data::Interleaved<i16>>| {
|
|
let out: &mut [i16] = args.data.buffer;
|
|
let num_frames = out.len();
|
|
// AudioHandler produces 48 kHz stereo f32 (= num_frames * 2 floats).
|
|
let needed = num_frames * 2;
|
|
if scratch_stereo.len() < needed {
|
|
scratch_stereo.resize(needed, 0.0);
|
|
}
|
|
// Zero the live slice. AudioHandler::fill_buffer is
|
|
// additive (does NOT clear); residual values from
|
|
// earlier callbacks (when scratch was bigger) would
|
|
// leak through otherwise.
|
|
scratch_stereo[..needed].fill(0.0);
|
|
// Non-blocking fill on the realtime callback thread.
|
|
// If the inbound forwarder currently owns this mutex,
|
|
// emit this period as silence instead of blocking and
|
|
// risking an AudioUnit underrun pop/click.
|
|
match handler_for_render.try_lock() {
|
|
Ok(mut h) => {
|
|
let _removed = h.fill_buffer(&mut scratch_stereo[..needed]);
|
|
}
|
|
Err(std::sync::TryLockError::WouldBlock) => {
|
|
// scratch_stereo is already zeroed above.
|
|
}
|
|
Err(std::sync::TryLockError::Poisoned(e)) => {
|
|
// Never panic on the realtime IO thread.
|
|
warn!(target: "chanora_audio", "AudioHandler mutex poisoned: {e}");
|
|
}
|
|
}
|
|
|
|
// Downmix stereo f32 -> mono i16 with master gain.
|
|
// (l + r) * 0.5 preserves total signal energy with
|
|
// 3 dB headroom against sum-of-correlated-peaks
|
|
// clipping. Hard-clip i16 cast at the boundary.
|
|
let gain = f32::from_bits(output_gain_for_render.load(Ordering::Relaxed));
|
|
let muted = output_muted_for_render.load(Ordering::Relaxed);
|
|
let mut peak_out: i16 = 0;
|
|
for (i, dst) in out.iter_mut().enumerate() {
|
|
if muted {
|
|
*dst = 0;
|
|
continue;
|
|
}
|
|
let l = scratch_stereo[i * 2];
|
|
let r = scratch_stereo[i * 2 + 1];
|
|
let mono_f32 = (l + r) * 0.5 * gain;
|
|
let clamped = mono_f32.clamp(-1.0, 1.0);
|
|
let sample = (clamped * i16::MAX as f32) as i16;
|
|
*dst = sample;
|
|
let a = sample.unsigned_abs() as i16;
|
|
if a > peak_out {
|
|
peak_out = a;
|
|
}
|
|
}
|
|
|
|
// Track audio-vs-silence for the diagnostic.
|
|
if peak_out > 0 {
|
|
callbacks_with_audio = callbacks_with_audio.wrapping_add(1);
|
|
} else {
|
|
callbacks_with_silence = callbacks_with_silence.wrapping_add(1);
|
|
}
|
|
|
|
// Diagnostic sampling.
|
|
if last_num_frames != 0 && last_num_frames != num_frames {
|
|
num_frames_changes = num_frames_changes.wrapping_add(1);
|
|
}
|
|
last_num_frames = num_frames;
|
|
cb_count = cb_count.wrapping_add(1);
|
|
if cb_count.is_multiple_of(100) {
|
|
info!(
|
|
target: "chanora_audio",
|
|
cb = cb_count,
|
|
num_frames,
|
|
frames_changes = num_frames_changes,
|
|
callbacks_with_audio,
|
|
callbacks_with_silence,
|
|
peak_out_i16 = peak_out,
|
|
gain,
|
|
"ios audio unit render callback diagnostic sample (direct fill_buffer)"
|
|
);
|
|
}
|
|
Ok(())
|
|
})
|
|
.map_err(|e| AudioError::Backend(format!("audio unit set render callback: {e}")))?;
|
|
|
|
// Finalise the unit — allocates internal buffers per the
|
|
// stream formats we set above. After initialize() most
|
|
// property changes are rejected (you have to uninitialize +
|
|
// re-initialize), which is why the property set must come
|
|
// first. Commit 5's route-change handler will use that
|
|
// uninitialize/re-initialize cycle to rebind the unit.
|
|
unit.initialize()
|
|
.map_err(|e| AudioError::Backend(format!("vpio initialize: {e}")))?;
|
|
|
|
unit.start()
|
|
.map_err(|e| AudioError::Backend(format!("vpio start: {e}")))?;
|
|
|
|
info!(
|
|
target: "chanora_audio",
|
|
sample_rate_hz = SAMPLE_RATE_HZ,
|
|
channels = stream_format.channels,
|
|
sample_format = ?stream_format.sample_format,
|
|
"ios VPIO audio unit started"
|
|
);
|
|
|
|
// Read back the ACTUAL stream format VPIO accepted on each
|
|
// bus (iOS sometimes substitutes its own format if the
|
|
// hardware can't satisfy our preference) and the actual
|
|
// AVAudioSession sample rate + IO buffer duration. Without
|
|
// these we can't tell whether our 48 kHz Int16 mono format
|
|
// was honoured or silently downgraded to e.g. 44.1 kHz
|
|
// Float32 (which would cause our render callback to write
|
|
// i16 values into a buffer iOS interprets as f32 = severe
|
|
// distortion). Diagnostic prompted by external review
|
|
// pointing out that 'preferredSampleRate' is a hint, not
|
|
// a guarantee \u2014 must verify post-init.
|
|
match unit.output_stream_format() {
|
|
Ok(fmt) => info!(
|
|
target: "chanora_audio",
|
|
sample_rate = fmt.sample_rate,
|
|
channels = fmt.channels,
|
|
sample_format = ?fmt.sample_format,
|
|
flags = ?fmt.flags,
|
|
"ios VPIO actual OUTPUT stream format (post-init)"
|
|
),
|
|
Err(e) => warn!(
|
|
target: "chanora_audio",
|
|
error = %e,
|
|
"ios VPIO output_stream_format read failed"
|
|
),
|
|
}
|
|
match unit.input_stream_format() {
|
|
Ok(fmt) => info!(
|
|
target: "chanora_audio",
|
|
sample_rate = fmt.sample_rate,
|
|
channels = fmt.channels,
|
|
sample_format = ?fmt.sample_format,
|
|
flags = ?fmt.flags,
|
|
"ios VPIO actual INPUT stream format (post-init)"
|
|
),
|
|
Err(e) => warn!(
|
|
target: "chanora_audio",
|
|
error = %e,
|
|
"ios VPIO input_stream_format read failed"
|
|
),
|
|
}
|
|
|
|
Ok(Self { unit })
|
|
}
|
|
|
|
/// Restart the audio unit after route change handling.
|
|
///
|
|
/// Route rebinding on iOS is most reliable when we bounce the
|
|
/// VoiceProcessingIO unit through an uninitialize/reinitialize
|
|
/// cycle, then start again.
|
|
pub fn restart(&mut self) -> Result<(), AudioError> {
|
|
self.unit
|
|
.stop()
|
|
.map_err(|e| AudioError::Backend(format!("vpio restart stop: {e}")))?;
|
|
self.unit
|
|
.uninitialize()
|
|
.map_err(|e| AudioError::Backend(format!("vpio restart uninit: {e}")))?;
|
|
self.unit
|
|
.initialize()
|
|
.map_err(|e| AudioError::Backend(format!("vpio restart init: {e}")))?;
|
|
self.unit
|
|
.start()
|
|
.map_err(|e| AudioError::Backend(format!("vpio restart start: {e}")))?;
|
|
info!(target: "chanora_audio", "ios VPIO audio unit restarted");
|
|
Ok(())
|
|
}
|
|
|
|
/// Pause the audio unit during an interruption.
|
|
pub fn pause(&mut self) -> Result<(), AudioError> {
|
|
self.unit
|
|
.stop()
|
|
.map_err(|e| AudioError::Backend(format!("vpio pause stop: {e}")))
|
|
}
|
|
|
|
/// Resume the audio unit after an interruption.
|
|
pub fn resume(&mut self) -> Result<(), AudioError> {
|
|
self.unit
|
|
.start()
|
|
.map_err(|e| AudioError::Backend(format!("vpio resume start: {e}")))
|
|
}
|
|
}
|
|
|
|
impl Drop for IosVoiceUnit {
|
|
fn drop(&mut self) {
|
|
// Stop the audio unit so the render callback no longer
|
|
// fires. The coreaudio-rs wrapper's own Drop calls
|
|
// AudioComponentInstanceDispose afterwards.
|
|
if let Err(e) = self.unit.stop() {
|
|
warn!(target: "chanora_audio", error = %e, "ios audio unit stop on drop failed");
|
|
} else {
|
|
info!(target: "chanora_audio", "ios audio unit stopped");
|
|
}
|
|
}
|
|
}
|