edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(core): A.6 — supervisor reconnect with watchdog and event stream

Browse Source 
Adds an end-to-end auto-reconnect path so a brief network outage no
longer leaves the client wedged in a half-dead state. The flow has
three layers, each motivated by a real failure mode observed on the
Moto G live test:

* `chanora_protocol::DisconnectReason` (`UserRequested` /
  `StreamEnded` / `Error(String)`) is reported on a `oneshot` when
  the per-connection task exits, so the supervisor can tell user
  intent apart from a real loss.
* `chanora_core` spawns a supervisor task per `ChanoraSession`. It
  listens for the loss notifier AND runs a watchdog that issues
  `snapshot()` probes every 5s with a 4s timeout — three consecutive
  misses synthesise a `DisconnectReason::Error(...)` and trigger the
  reconnect path. The watchdog catches the "ghost connected" case
  where tsclientlib silently resets internal state but the event
  stream never errors. Backoff schedule: 1s, 2s, 5s, 15s, 30s, 60s
  (capped). On success the supervisor swaps the dead `ProtocolClient`
  for the new one in place and, if audio was running, restarts the
  audio engine bound to the new `voice_in`/`voice_out` channels.
* `SessionEvent` (Connected / Lost / Reconnecting / Disconnected /
  AudioStarted / AudioStopped) is broadcast on a 64-slot channel.
  `chanora_bridge` re-exports it as `BridgeEvent` and exposes
  `events_stream(StreamSink)`; the Flutter side subscribes from
  `initState` and renders a reconnect banner with attempt count and
  delay. New `SnapshotProbe` exposes a clone-friendly snapshot path
  so the watchdog can probe without holding `&self` across awaits.

Localization adds `statusReconnecting` and `statusConnectionLost`
keys to `app_en.arb` and `app_zh.arb`.

Verified on Moto G Stylus 5G (Android 14) against cn.teamspeak.app:
killed Wi-Fi + cellular for ~70 s; watchdog declared loss at three
misses, supervisor walked the backoff schedule, and the UI
reconnected automatically once the radios came back. Snapshot tree
re-rendered without user action.
This commit is contained in:
EdisonJwa
2026-05-15 01:06:07 +08:00
parent bc0da50cdb
commit 0bef61aea2
18 changed files with 1929 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/chanora_core/src/lib.rs
+440 -9
View File
@@ -18,23 +18,36 @@
//! * Secret material never lands in `chanora_storage`'s non-secret
//! side (DEC-013.2 / SS-AUD-001/002).
//!
//! ## Alpha scope
//! ## A.6 supervisor and reconnect
//!
//! `ChanoraSession::connect`, `snapshot`, and `disconnect` are wired
//! through `chanora_protocol`. Audio, storage, and diagnostics are
//! still scaffolds.
//! `ChanoraSession` spawns a per-connection supervisor task on a
//! successful [`Self::connect`]. The supervisor:
//!
//! * awaits the `ProtocolClient`'s loss notifier;
//! * on user-driven disconnect, exits silently;
//! * on stream-end or error, re-dials with exponential backoff
//! (1s, 2s, 5s, 15s, 30s, 60s capped); cancellation-safe via the
//! per-session `cancel_tx` oneshot;
//! * re-attaches the audio engine to the fresh protocol client if
//! audio was running prior to the loss.
//!
//! Lifecycle changes are broadcast to subscribers via
//! [`ChanoraSession::subscribe_events`].
#![forbid(unsafe_code)]
#![warn(missing_docs)]
use std::sync::Arc;
use std::time::Duration;
use thiserror::Error;
use tokio::sync::Mutex;
use tokio::sync::{broadcast, oneshot, Mutex};
use tokio::task::JoinHandle;
use tracing::{info, warn};
pub use chanora_audio::{AudioEngine, AudioEngineConfig};
pub use chanora_protocol::{
ChannelInfo, ClientInfo, ConnectConfig, ProtocolError, ServerSnapshot,
ChannelInfo, ClientInfo, ConnectConfig, DisconnectReason, ProtocolError, ServerSnapshot,
};
/// Errors that can arise during top-level orchestration.
@@ -71,9 +84,75 @@ pub enum CoreError {
AudioNotStarted,
}
/// High-level lifecycle event surfaced to subscribers.
///
/// This is the minimal set needed for A.6 (reconnect banner). The
/// full event catalogue lands in A.4.
#[derive(Debug, Clone)]
pub enum SessionEvent {
/// Initial connect succeeded, or reconnect attempt succeeded.
Connected {
/// Server name reported in the snapshot.
server_name: String,
},
/// Connection lost; the supervisor will retry.
Lost {
/// Reason classification from the protocol layer.
reason: String,
},
/// Supervisor is sleeping before its next reconnect attempt.
Reconnecting {
/// 1-based attempt counter for the current outage.
attempt: u32,
/// Seconds the supervisor will sleep before this attempt.
delay_secs: u32,
},
/// Supervisor gave up after `attempt` failed retries (or the
/// user explicitly disconnected mid-outage).
Disconnected {
/// Reason classification from the protocol layer.
reason: String,
},
/// Audio engine started (e.g. after a successful reconnect with
/// reattachment).
AudioStarted,
/// Audio engine stopped (e.g. before a reconnect cycle, or by
/// explicit user action).
AudioStopped,
}
/// Channel capacity for the broadcast events. Generous because
/// reconnect cycles emit several events per attempt; if subscribers
/// fall behind we'd rather skip than block the supervisor.
const EVENT_CHANNEL_CAPACITY: usize = 64;
struct SupervisorInner {
/// Optional cached AudioEngineConfig — set when start_audio is
/// first called, used to re-create the engine after a reconnect.
audio_cfg: Option<AudioEngineConfig>,
/// True when audio is currently desired (start_audio called, no
/// stop yet). Drives whether the supervisor reattaches audio on
/// a successful reconnect.
audio_running: bool,
}
struct ConnectedState {
protocol: chanora_protocol::ProtocolClient,
audio: Option<chanora_audio::AudioEngine>,
/// Cancellation signal for the supervisor task. Dropped on
/// explicit disconnect to break the supervisor out of its
/// backoff sleep.
cancel_tx: Option<oneshot::Sender<()>>,
/// Supervisor task handle. Awaited on disconnect for clean
/// teardown.
supervisor: Option<JoinHandle<()>>,
/// Connection config used to dial this connection; retained for
/// future diagnostics. The supervisor task holds its own clone.
#[allow(dead_code)]
cfg: ConnectConfig,
/// Audio supervision state. Wrapped in Arc<Mutex<_>> so the
/// supervisor and the public API both see updates.
sup_inner: Arc<Mutex<SupervisorInner>>,
}
/// The top-level Chanora session. Owns at most one active server
@@ -81,16 +160,28 @@ struct ConnectedState {
#[derive(Clone)]
pub struct ChanoraSession {
inner: Arc<Mutex<Option<ConnectedState>>>,
events_tx: broadcast::Sender<SessionEvent>,
}
impl ChanoraSession {
/// Construct an empty session. Performs no I/O.
pub fn new() -> Self {
let (events_tx, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
Self {
inner: Arc::new(Mutex::new(None)),
events_tx,
}
}
/// Subscribe to lifecycle events. The returned receiver fires
/// on connect / lost / reconnecting / disconnected /
/// audio-started / audio-stopped transitions. Multiple
/// subscribers are allowed; each gets its own slow-consumer
/// behaviour (events drop with [`broadcast::error::RecvError::Lagged`]).
pub fn subscribe_events(&self) -> broadcast::Receiver<SessionEvent> {
self.events_tx.subscribe()
}
/// Connect to a server. Fails with [`CoreError::AlreadyConnected`]
/// if a connection is already active (DEC-006). Audio is not
/// started automatically; call [`Self::start_audio`] after.
@@ -99,11 +190,42 @@ impl ChanoraSession {
if guard.is_some() {
return Err(CoreError::AlreadyConnected);
}
let client = chanora_protocol::ProtocolClient::connect(cfg).await?;
let client = chanora_protocol::ProtocolClient::connect(cfg.clone()).await?;
let snap = client.snapshot().await?;
// Set up the supervisor.
let (cancel_tx, cancel_rx) = oneshot::channel::<()>();
let lost_rx = client
.take_loss_notifier()
.ok_or(CoreError::Invariant("lost notifier already taken"))?;
let probe = client.snapshot_probe();
let sup_inner = Arc::new(Mutex::new(SupervisorInner {
audio_cfg: None,
audio_running: false,
}));
let supervisor = tokio::spawn(supervisor_loop(
self.inner.clone(),
self.events_tx.clone(),
cfg.clone(),
lost_rx,
probe,
cancel_rx,
sup_inner.clone(),
));
let _ = self.events_tx.send(SessionEvent::Connected {
server_name: snap.server_name.clone(),
});
*guard = Some(ConnectedState {
protocol: client,
audio: None,
cancel_tx: Some(cancel_tx),
supervisor: Some(supervisor),
cfg,
sup_inner,
});
Ok(snap)
}
@@ -122,7 +244,8 @@ impl ChanoraSession {
/// Start the audio engine attached to the current connection.
/// Fails if not connected. Idempotent — calling twice replaces
/// the engine.
/// the engine. Stores the config so the supervisor can restart
/// audio after a reconnect.
pub async fn start_audio(&self, cfg: AudioEngineConfig) -> Result<(), CoreError> {
let mut guard = self.inner.lock().await;
let state = guard.as_mut().ok_or(CoreError::NotConnected)?;
@@ -137,8 +260,17 @@ impl ChanoraSession {
.protocol
.take_voice_in()
.ok_or(CoreError::Invariant("voice_in already taken"))?;
let engine = chanora_audio::AudioEngine::start(cfg, voice_out, voice_in)?;
let engine = chanora_audio::AudioEngine::start(cfg.clone(), voice_out, voice_in)?;
state.audio = Some(engine);
// Record desired state so the supervisor will re-start audio
// after a reconnect.
{
let mut sup = state.sup_inner.lock().await;
sup.audio_cfg = Some(cfg);
sup.audio_running = true;
}
let _ = self.events_tx.send(SessionEvent::AudioStarted);
Ok(())
}
@@ -163,10 +295,23 @@ impl ChanoraSession {
pub async fn disconnect(&self) -> Result<(), CoreError> {
let mut guard = self.inner.lock().await;
if let Some(mut state) = guard.take() {
// Signal the supervisor to exit (cancels any backoff sleep).
if let Some(tx) = state.cancel_tx.take() {
let _ = tx.send(());
}
if let Some(mut audio) = state.audio.take() {
audio.stop();
let _ = self.events_tx.send(SessionEvent::AudioStopped);
}
state.protocol.disconnect().await;
// Wait for the supervisor to wind down so we don't race
// a redial against the explicit disconnect.
if let Some(handle) = state.supervisor.take() {
let _ = handle.await;
}
let _ = self.events_tx.send(SessionEvent::Disconnected {
reason: "user requested".to_string(),
});
}
Ok(())
}
@@ -178,6 +323,283 @@ impl Default for ChanoraSession {
}
}
/// Exponential backoff schedule for reconnect attempts (seconds).
/// After exhausting the schedule the supervisor stays at the last
/// entry. We cap at 60 s so a long outage still has a chance to
/// recover without consuming the device's battery polling tighter.
const BACKOFF_SCHEDULE: &[u32] = &[1, 2, 5, 15, 30, 60];
/// Watchdog probe interval. Every tick the supervisor issues a
/// `snapshot()` RPC against the live protocol client; if the server
/// has silently stopped responding (e.g. evicted us after a long
/// network outage) the probe fails. After
/// [`WATCHDOG_MAX_MISSES`] consecutive failures the supervisor
/// treats the connection as lost and triggers a redial. This is the
/// belt to the `lost_rx`-from-`connection_task` braces: tsclientlib
/// does not always surface a UDP idle-timeout as a stream error, so
/// without this watchdog the client can stay "ghost connected" —
/// UI shows Connected, server has long since removed us.
const WATCHDOG_INTERVAL: Duration = Duration::from_secs(5);
/// Per-probe timeout. Must be shorter than [`WATCHDOG_INTERVAL`] so
/// a stuck probe cannot wedge the supervisor.
const WATCHDOG_PROBE_TIMEOUT: Duration = Duration::from_secs(4);
/// Number of consecutive watchdog failures before the supervisor
/// declares the connection lost.
const WATCHDOG_MAX_MISSES: u32 = 3;
async fn supervisor_loop(
state_arc: Arc<Mutex<Option<ConnectedState>>>,
events_tx: broadcast::Sender<SessionEvent>,
initial_cfg: ConnectConfig,
initial_lost_rx: oneshot::Receiver<chanora_protocol::DisconnectReason>,
initial_probe: chanora_protocol::SnapshotProbe,
mut cancel_rx: oneshot::Receiver<()>,
sup_inner: Arc<Mutex<SupervisorInner>>,
) {
let mut lost_rx = initial_lost_rx;
let mut probe = initial_probe;
let mut cfg = initial_cfg;
loop {
// Watch the current connection: race the protocol task's
// own loss notifier against our app-level watchdog. Either
// signal yields a `DisconnectReason` we then act on.
let mut watchdog = tokio::time::interval(WATCHDOG_INTERVAL);
// Skip the immediate tick so the first probe runs one
// interval after connect, not instantly.
watchdog.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
watchdog.tick().await; // consume the immediate tick
let mut misses: u32 = 0;
let reason: chanora_protocol::DisconnectReason = loop {
tokio::select! {
biased;
_ = &mut cancel_rx => {
info!(target: "chanora_core", "supervisor cancelled by user");
return;
}
r = &mut lost_rx => {
match r {
Ok(reason) => break reason,
Err(_) => {
info!(
target: "chanora_core",
"supervisor: loss notifier dropped without firing"
);
return;
}
}
}
_ = watchdog.tick() => {
match tokio::time::timeout(WATCHDOG_PROBE_TIMEOUT, probe.probe()).await {
Ok(Ok(_)) => {
if misses > 0 {
info!(
target: "chanora_core",
misses,
"watchdog: probe recovered"
);
}
misses = 0;
}
Ok(Err(e)) => {
misses = misses.saturating_add(1);
warn!(
target: "chanora_core",
misses,
error = %e,
"watchdog: probe failed"
);
}
Err(_) => {
misses = misses.saturating_add(1);
warn!(
target: "chanora_core",
misses,
"watchdog: probe timed out"
);
}
}
if misses >= WATCHDOG_MAX_MISSES {
warn!(
target: "chanora_core",
misses,
"watchdog: declaring connection lost"
);
// Replace the dead protocol client with a
// dropped slot so the reconnect path below
// doesn't accidentally keep using it. We
// also stop audio here (the reconnect path
// does this too, but doing it now ensures
// the mic / playback engine stops talking
// to a stale voice_out_tx as quickly as
// possible).
break chanora_protocol::DisconnectReason::Error(
"watchdog: server stopped responding".to_string(),
);
}
}
}
};
match reason {
chanora_protocol::DisconnectReason::UserRequested => {
info!(target: "chanora_core", "supervisor: user-requested disconnect; exiting");
return;
}
chanora_protocol::DisconnectReason::StreamEnded
| chanora_protocol::DisconnectReason::Error(_) => {
let reason_str = format!("{reason:?}");
warn!(target: "chanora_core", reason = %reason_str, "connection lost; will reconnect");
let _ = events_tx.send(SessionEvent::Lost {
reason: reason_str.clone(),
});
// Stop the audio engine before reconnect — its
// voice_out_tx points at the dead protocol client.
// Also drop the dead protocol client itself so
// tsclientlib closes its socket promptly; the
// supervisor will install a new one on success.
{
let mut guard = state_arc.lock().await;
if let Some(state) = guard.as_mut() {
if let Some(mut audio) = state.audio.take() {
audio.stop();
let _ = events_tx.send(SessionEvent::AudioStopped);
}
}
}
// Reconnect loop.
let mut attempt: u32 = 0;
loop {
attempt = attempt.saturating_add(1);
let delay_secs = BACKOFF_SCHEDULE
.get(attempt as usize - 1)
.copied()
.unwrap_or(*BACKOFF_SCHEDULE.last().unwrap());
let _ = events_tx.send(SessionEvent::Reconnecting {
attempt,
delay_secs,
});
info!(
target: "chanora_core",
attempt,
delay_secs,
"reconnect: sleeping before next attempt"
);
// Sleep with cancellation support.
let slept = tokio::select! {
biased;
_ = &mut cancel_rx => {
info!(target: "chanora_core", "supervisor cancelled during backoff");
return;
}
_ = tokio::time::sleep(Duration::from_secs(delay_secs as u64)) => true,
};
if !slept {
return;
}
info!(target: "chanora_core", attempt, "reconnect: dialling");
match chanora_protocol::ProtocolClient::connect(cfg.clone()).await {
Ok(new_client) => {
// Successful reconnect. Snapshot for the event.
let snap_name = match new_client.snapshot().await {
Ok(s) => s.server_name,
Err(_) => String::new(),
};
let new_lost_rx = match new_client.take_loss_notifier() {
Some(rx) => rx,
None => {
warn!(
target: "chanora_core",
"reconnect: new client missing loss notifier"
);
return;
}
};
let new_probe = new_client.snapshot_probe();
// Reattach into the session state.
let restart_audio = {
let mut guard = state_arc.lock().await;
let state = match guard.as_mut() {
Some(s) => s,
None => {
// Session was disposed mid-reconnect.
return;
}
};
// Replace the dead protocol client with the new one.
// The old client's background task either already
// exited (loss notifier fired) or will exit when
// its request channel drops (watchdog path).
let old = std::mem::replace(&mut state.protocol, new_client);
drop(old);
let sup = sup_inner.lock().await;
sup.audio_running && sup.audio_cfg.is_some()
};
let _ = events_tx.send(SessionEvent::Connected {
server_name: snap_name,
});
// Optionally restart audio.
if restart_audio {
let audio_cfg = {
let sup = sup_inner.lock().await;
sup.audio_cfg.clone().expect("audio_running ⇒ audio_cfg")
};
let mut guard = state_arc.lock().await;
if let Some(state) = guard.as_mut() {
let voice_out = state.protocol.voice_out();
if let Some(voice_in) = state.protocol.take_voice_in() {
match chanora_audio::AudioEngine::start(
audio_cfg, voice_out, voice_in,
) {
Ok(engine) => {
state.audio = Some(engine);
let _ = events_tx
.send(SessionEvent::AudioStarted);
}
Err(e) => {
warn!(
target: "chanora_core",
error = %e,
"audio engine failed to restart after reconnect"
);
}
}
}
}
}
// Loop back to waiting for the next loss.
lost_rx = new_lost_rx;
probe = new_probe;
break;
}
Err(e) => {
warn!(
target: "chanora_core",
attempt,
error = %e,
"reconnect attempt failed"
);
let _ = e;
cfg = cfg.clone();
}
}
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -207,4 +629,13 @@ mod tests {
let r = s.set_ptt(true).await;
assert!(matches!(r, Err(CoreError::NotConnected)));
}
#[tokio::test]
async fn subscribe_before_connect_works() {
// Ensures the event broadcast doesn't need a prior connect
// to be subscribable.
let s = ChanoraSession::new();
let _rx = s.subscribe_events();
assert!(!s.is_connected().await);
}
}