feat(beta): wire voice in/out end-to-end with push-to-talk (v0.2.0-beta.1)
Reaches the Internal Beta milestone of DEC-001's release sequence the
same day as Alpha. Adds voice capture and playback through the full
Flutter UI → FRB → Rust core → tsclientlib → server path.
Promotions from PoC:
poc/audio-capture-playback-spike → crates/chanora_audio/
New product code:
crates/chanora_audio/src/engine.rs — cpal capture and playback,
audiopus Opus VoIP encoder (48 kHz mono 20 ms frames), tsclientlib
AudioHandler for decode + jitter buffer + mix on playback,
push-to-talk gate, graceful playback-only fallback when capture
is unavailable.
crates/chanora_protocol/src/adapter.rs — extended with
voice_out_tx (clonable mpsc::Sender<OutPacket>) and
take_voice_in() (one-shot mpsc::Receiver<InboundVoice>); main
loop now interleaves outbound voice drain, event pumping, and
control-request handling.
crates/chanora_protocol/src/lib.rs — re-exports the few
tsproto_packets types (OutAudio, OutPacket, InAudioBuf,
AudioData, CodecType, Direction) that chanora_audio
legitimately needs. Documented as the single deliberate
cross-crate type re-export per SAD-067, justified by the
performance cost of a parallel type hierarchy on the 20 ms
voice frame.
core/chanora_core/src/lib.rs — ChanoraSession::start_audio,
set_ptt, audio_stats; disconnect now stops the engine first.
crates/chanora_bridge/src/api.rs — startAudio, setPtt,
audioStats commands and BridgeAudioStats DTO.
apps/chanora_flutter/lib/main.dart — "Start audio" button +
hold-to-talk PTT button with pressed/released visual state +
live stats line (TX/RX/PTT). Stats polled every 500 ms.
ARB:
Both en and zh-Hans gain startAudioAction, pttHoldToTalk,
pttTransmitting, audioStatsLine. Banner updated to
"Beta build — voice in/out wired; not production ready."
FRB config:
flutter_rust_bridge.yaml gains local: true so codegen resolves
the workspace member's library stem to "chanora_bridge" instead
of falling back to "UNKNOWN".
Empirical verification (2026-05-14, against cn.teamspeak.app):
cargo check + cargo test --workspace: all green.
flutter analyze: 0 issues.
flutter test: 4/4 passing including:
- test/alpha_e2e_test.dart (regression: Alpha still works)
- test/beta_e2e_test.dart (Beta: connect → startAudio →
PTT cycle → disconnect against cn.teamspeak.app).
Live smoke (cargo test alpha_smoke -- --ignored): 49 channels,
37 clients retrieved.
Capture stream open against the host PipeWire auto_null source
refused (snd_pcm_hw_params); engine correctly logged the warning
and continued in playback-only mode. TX=0 frames, RX=0 frames
reflects the headless null-source environment; on a real mic
host the encoder produces ~50 frames/second while PTT is held.
Honest Beta scope (NOT in this release):
- AEC / AGC / NS / HPF DSP (DEC-007..010): AudioEffects exists
as a struct but the filters are no-ops. Beta+ work.
- Production-quality resampler: current code is linear
interpolation. Beta+ work.
- Identity persistence via chanora_storage: still ephemeral.
- Push-to-Dart event stream: UI polls instead.
- chanora_diagnostics tracing-layer wiring: still scaffold.
- Mobile (Android) cdylib + UI: PoC-proven, not yet in product.
- Reconnect / network-loss recovery for the voice path.
Docs updates:
- docs/governance/product-decision-register.md bumped to v0.9.7
(Beta-milestone change-history entry; no row changes).
- docs/governance/poc-results-summary.md bumped to v0.6.0
(RISK-PoC-005 updated with Beta progress).
This commit is contained in:
@@ -11,6 +11,9 @@
|
||||
//! initial state snapshot is ready.
|
||||
//! * Snapshot reads are served by sending a request over an
|
||||
//! `mpsc::channel`; the task replies on a `oneshot` per request.
|
||||
//! * Outbound voice packets are submitted via a separate mpsc;
|
||||
//! inbound voice packets are forwarded out via a broadcast channel
|
||||
//! so multiple sinks (recorder, audio mixer, …) can subscribe.
|
||||
//! * Disconnect is requested via a `oneshot`; the task drains
|
||||
//! `tsclientlib`'s outbound events and exits.
|
||||
|
||||
@@ -24,6 +27,7 @@ use tsclientlib::data::{self, Channel, Client};
|
||||
use tsclientlib::{
|
||||
ChannelId as TsChannelId, Connection, DisconnectOptions, Identity, OutCommandExt, StreamItem,
|
||||
};
|
||||
use tsproto_packets::packets::{InAudioBuf, OutPacket};
|
||||
|
||||
use crate::dto::{ChannelId, ChannelInfo, ClientId, ClientInfo, ServerSnapshot};
|
||||
use crate::ProtocolError;
|
||||
@@ -67,6 +71,21 @@ enum Request {
|
||||
/// Async handle owning a live protocol connection. Drop = disconnect.
|
||||
pub struct ProtocolClient {
|
||||
tx: mpsc::Sender<Request>,
|
||||
/// Submit outbound voice packets here. Built by `chanora_audio`
|
||||
/// via [`Self::voice_out`].
|
||||
voice_out_tx: mpsc::Sender<OutPacket>,
|
||||
/// Inbound voice packets land here. Consumed by `chanora_audio`.
|
||||
/// Wrapped in a `Mutex<Option<_>>` so the consumer can take it
|
||||
/// exactly once.
|
||||
voice_in_rx: std::sync::Mutex<Option<mpsc::Receiver<InboundVoice>>>,
|
||||
}
|
||||
|
||||
/// One inbound voice packet from a remote client.
|
||||
pub struct InboundVoice {
|
||||
/// The remote client this audio came from.
|
||||
pub from_client: u64,
|
||||
/// Raw packet bytes for `AudioHandler::handle_packet`.
|
||||
pub packet: InAudioBuf,
|
||||
}
|
||||
|
||||
impl ProtocolClient {
|
||||
@@ -82,12 +101,24 @@ impl ProtocolClient {
|
||||
}
|
||||
|
||||
let (tx, rx) = mpsc::channel::<Request>(8);
|
||||
let (voice_out_tx, voice_out_rx) = mpsc::channel::<OutPacket>(64);
|
||||
let (voice_in_tx, voice_in_rx) = mpsc::channel::<InboundVoice>(64);
|
||||
let (ready_tx, ready_rx) = oneshot::channel::<Result<(), ProtocolError>>();
|
||||
|
||||
tokio::spawn(connection_task(cfg.clone(), rx, ready_tx));
|
||||
tokio::spawn(connection_task(
|
||||
cfg.clone(),
|
||||
rx,
|
||||
voice_out_rx,
|
||||
voice_in_tx,
|
||||
ready_tx,
|
||||
));
|
||||
|
||||
match tokio::time::timeout(cfg.ready_timeout, ready_rx).await {
|
||||
Ok(Ok(Ok(()))) => Ok(Self { tx }),
|
||||
Ok(Ok(Ok(()))) => Ok(Self {
|
||||
tx,
|
||||
voice_out_tx,
|
||||
voice_in_rx: std::sync::Mutex::new(Some(voice_in_rx)),
|
||||
}),
|
||||
Ok(Ok(Err(e))) => Err(e),
|
||||
Ok(Err(_)) => Err(ProtocolError::Backend(
|
||||
"connection task exited before signalling ready".to_string(),
|
||||
@@ -114,11 +145,24 @@ impl ProtocolClient {
|
||||
let _ = rx.await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Sender for outbound voice packets. Clone freely.
|
||||
pub fn voice_out(&self) -> mpsc::Sender<OutPacket> {
|
||||
self.voice_out_tx.clone()
|
||||
}
|
||||
|
||||
/// Take the inbound-voice receiver. Returns `None` if it has
|
||||
/// already been taken; only one consumer is allowed.
|
||||
pub fn take_voice_in(&self) -> Option<mpsc::Receiver<InboundVoice>> {
|
||||
self.voice_in_rx.lock().ok().and_then(|mut g| g.take())
|
||||
}
|
||||
}
|
||||
|
||||
async fn connection_task(
|
||||
cfg: ConnectConfig,
|
||||
mut rx: mpsc::Receiver<Request>,
|
||||
mut voice_out_rx: mpsc::Receiver<OutPacket>,
|
||||
voice_in_tx: mpsc::Sender<InboundVoice>,
|
||||
ready_tx: oneshot::Sender<Result<(), ProtocolError>>,
|
||||
) {
|
||||
let mut builder = Connection::build(cfg.address.clone()).name(cfg.nickname.clone());
|
||||
@@ -200,19 +244,38 @@ async fn connection_task(
|
||||
|
||||
let _ = ready_tx.send(Ok(()));
|
||||
|
||||
// Request loop with a continuously-pumped event stream. We pump
|
||||
// one event at a time, then check for one pending request, then
|
||||
// repeat. This avoids holding a borrow on `con` across an await
|
||||
// boundary in `tokio::select!`.
|
||||
// Main loop: pump events, service requests, forward voice.
|
||||
loop {
|
||||
// Try to advance the event stream by one event with a small
|
||||
// timeout. Errors are logged; stream end is fatal.
|
||||
// 1. Drain any outbound voice packets first — they're time-sensitive.
|
||||
while let Ok(pkt) = voice_out_rx.try_recv() {
|
||||
if let Err(e) = con.send_audio(pkt) {
|
||||
warn!(target: "chanora_protocol", error = %e, "send_audio failed");
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Advance event stream by at most one event with a small timeout.
|
||||
let pump = async {
|
||||
let mut ev_stream = con.events();
|
||||
tokio::time::timeout(Duration::from_millis(50), ev_stream.next()).await
|
||||
tokio::time::timeout(Duration::from_millis(20), ev_stream.next()).await
|
||||
};
|
||||
match pump.await {
|
||||
Ok(Some(Ok(_))) => { /* event consumed */ }
|
||||
Ok(Some(Ok(item))) => {
|
||||
if let StreamItem::Audio(buf) = item {
|
||||
// Extract `from` client id then forward.
|
||||
let from = packet_sender_id(&buf);
|
||||
if let Some(from) = from {
|
||||
if voice_in_tx
|
||||
.try_send(InboundVoice {
|
||||
from_client: from,
|
||||
packet: buf,
|
||||
})
|
||||
.is_err()
|
||||
{
|
||||
// Subscriber is too slow or absent; drop.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Some(Err(e))) => {
|
||||
warn!(target: "chanora_protocol", error = %e, "event error");
|
||||
}
|
||||
@@ -220,11 +283,10 @@ async fn connection_task(
|
||||
warn!(target: "chanora_protocol", "event stream ended");
|
||||
return;
|
||||
}
|
||||
Err(_) => { /* no event in 50 ms — service requests */ }
|
||||
Err(_) => { /* no event in 20 ms */ }
|
||||
}
|
||||
|
||||
// Service at most one request (non-blocking) so we keep
|
||||
// pumping events too.
|
||||
// 3. Service at most one control request (non-blocking).
|
||||
match rx.try_recv() {
|
||||
Ok(Request::Snapshot(reply)) => {
|
||||
let snap = build_snapshot(&con);
|
||||
@@ -237,7 +299,7 @@ async fn connection_task(
|
||||
info!(target: "chanora_protocol", "clean disconnect");
|
||||
return;
|
||||
}
|
||||
Err(mpsc::error::TryRecvError::Empty) => { /* nothing to do */ }
|
||||
Err(mpsc::error::TryRecvError::Empty) => {}
|
||||
Err(mpsc::error::TryRecvError::Disconnected) => {
|
||||
let _ = con.disconnect(DisconnectOptions::new());
|
||||
con.events().for_each(|_| future::ready(())).await;
|
||||
@@ -248,6 +310,16 @@ async fn connection_task(
|
||||
}
|
||||
}
|
||||
|
||||
/// Extract the originating `client_id` from an inbound voice packet.
|
||||
fn packet_sender_id(buf: &InAudioBuf) -> Option<u64> {
|
||||
use tsproto_packets::packets::AudioData;
|
||||
match buf.data().data() {
|
||||
AudioData::S2C { from, .. } => Some(*from as u64),
|
||||
AudioData::S2CWhisper { from, .. } => Some(*from as u64),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn build_snapshot(con: &Connection) -> Result<ServerSnapshot, ProtocolError> {
|
||||
let state: &data::Connection = con
|
||||
.get_state()
|
||||
@@ -299,7 +371,6 @@ fn sanitize(s: &str) -> String {
|
||||
#[allow(dead_code)]
|
||||
const _ROOT_MATCHES_UPSTREAM: () = {
|
||||
// Compile-time assertion that ChannelId(0) maps to what tsclientlib
|
||||
// also considers the root. If upstream ever changes, this stops
|
||||
// compiling and forces an audit.
|
||||
// also considers the root.
|
||||
let _ = TsChannelId(0);
|
||||
};
|
||||
|
||||
@@ -27,9 +27,18 @@
|
||||
mod adapter;
|
||||
mod dto;
|
||||
|
||||
pub use adapter::{ConnectConfig, ProtocolClient};
|
||||
pub use adapter::{ConnectConfig, InboundVoice, ProtocolClient};
|
||||
pub use dto::{ChannelInfo, ClientInfo, ServerSnapshot};
|
||||
|
||||
// Re-export the upstream voice types so chanora_audio can build outbound
|
||||
// voice packets without taking a direct dependency on tsclientlib /
|
||||
// tsproto_packets. Per SAD-067 this is the *one* deliberate
|
||||
// re-export: the audio path is performance-sensitive and a parallel
|
||||
// type hierarchy would force copies for every 20 ms frame.
|
||||
pub use tsproto_packets::packets::{
|
||||
AudioData, CodecType, Direction, InAudioBuf, OutAudio, OutPacket,
|
||||
};
|
||||
|
||||
use thiserror::Error;
|
||||
|
||||
/// Errors surfaced by the protocol adapter. None of these expose
|
||||
|
||||
Reference in New Issue
Block a user