Edison Jwa
2f6d45fb04
* feat(audio): add Silero ONNX VAD with WebRTC fallback Introduce SileroOnnxVad and SileroOnnxVadWorker for desktop targets. The worker runs Silero v6 ONNX inference on a dedicated thread, accumulating 10 ms frames into the 512-sample 16 kHz input the model expects. Add VadOutput, VoiceActivityDetector trait, and WebRtcFallbackVad to provide a uniform VAD interface with graceful fallback when the ONNX model is unavailable. Wire the new VadBackend variants through AudioProcessingConfig and the snapshot stats so the bridge can report which detector is active. * feat(audio): integrate desktop VAD worker into capture engine Wire SileroOnnxVadWorker into the desktop capture path so voice activity can open the transmit gate before encoding. The capture callback now processes all audio through resample, downmix, and VAD unconditionally; transmit_active still gates Opus encoding. Add new_desktop_audio_processing_state() to construct the config/stats/worker triple, and apply_desktop_vad_backend() to synchronously load or clear the worker on config changes. Override processing_backend to Noop for desktop so bridge diagnostics report the correct backend rather than the iOS-oriented PlatformVoiceProcessing default. Includes review-driven cleanups: StreamConfig clone to deref per clippy, and a comment explaining why two try_lock calls on silero_vad_worker are structurally necessary (borrow checker requires the policy probe and the fallback path to not share a lock guard because mark_vad_fallback_active takes &mut self). * fix(audio): modernize Windows PTT to current windows-rs API Port the Raw Input plus low-level keyboard hook PTT backend to the newer windows-rs patterns: OptionalHandle, Result-returning CreateWindowExW, and None for CallNextHookEx. Replaces the old HHOOK(0) pointer casts. Add deterministic tests for mouse button 4 and 5 press and release driving the gate. * build(windows): force MSVC release CRT for audiopus cmake builds audiopus_sys calls cmake::build(opus_path), so downstream Cargo env cannot use cmake-rs Config::define() to override CMake's MSVC Debug CRT defaults. Point cmake-rs at a small wrapper that injects the policy and cache variables during configure while passing cmake --build, --version, and -E through unchanged. Keeps Opus Debug builds on Rust's release dynamic CRT (/MD) instead of CMake's default debug CRT (/MDd), which otherwise pulls in unresolved __imp__CrtDbgReportW symbols at test link. Document that the iOS deployment target is intentionally absent from this file. It is enforced by tools/build-ios.sh and the Xcode project; setting it globally here would make native macOS cargo check runs try to link iPhone objects against the macOS SDK. * build(flutter): update pubspec.lock after plugin additions Regenerated lockfile reflecting the local_notifications and connectivity_plus plugin additions from the poke-notifications feature. * fix(audio): address PR #37 review findings Six fixes from independent PR review: 1. BLOCKER: Replace Windows-only cmake .cmd wrapper with cross-platform CMake env vars. Setting CMAKE=tools/cmake-msvc-release-crt.cmd globally broke non-Windows hosts because cmake-rs would try to execute a .cmd file on macOS/Linux. Instead, set CMAKE_POLICY_DEFAULT_CMP0091=NEW and CMAKE_MSVC_RUNTIME_LIBRARY= MultiThreadedDLL as env vars that CMake reads natively. MSVC- specific vars are safely ignored by GCC/Clang toolchains. Delete the now-unnecessary wrapper script. 2. IMPORTANT: Join the Silero worker thread in Drop instead of detaching it. The old code dropped the JoinHandle which detaches the thread; the new code calls handle.join() after closing the channel, ensuring the ONNX session is cleaned up before the worker is replaced during config changes. 3. IMPORTANT: Single-try_lock refactor of the capture VAD callback. The double try_lock (policy probe + send) is replaced by a single scoped try_lock that both probes availability and sends the frame. The guard is dropped before the fallback path, which needs &mut self for mark_vad_fallback_active. This also eliminates the VadWorkerPolicy enum and callback_vad_worker_policy function, whose behavior is now inlined into the callback. 4. IMPORTANT: Remove tracing from the realtime capture callback. mark_vad_fallback_active and sync_vad_backend emitted info!/warn! from the audio thread. Replace with silent atomic state publishing via SharedAudioProcessingStats; the bridge stats stream already exposes vad_fallback_active for diagnostics. 5. IMPORTANT: Defer ONNX model load outside the worker mutex. apply_desktop_vad_backend_to_worker now constructs the new worker before taking the lock, then swaps it in under a short hold. This prevents the realtime callback from being blocked during model I/O + thread spawn. 6. MINOR: Remove unused VadBackend import from vad/mod.rs after deleting the policy code. * fix(audio): address PR #37 second-pass review findings 5-agent review found 5 blocking issues. All addressed: 1. BLOCKER: CMake env vars don't reach CMake cache. Restored .cmd wrapper but scoped to Windows MSVC targets only via [target.x86_64-pc-windows-msvc] and [target.aarch64-pc-windows-msvc] in .cargo/config.toml. Non-Windows hosts are unaffected. 2. BLOCKER: processing_backend normalized in set_audio_processing_config on desktop (cfg-gated override to Noop), mirroring startup default. 3. BLOCKER: Model-path reload was already wired via reload_audio_processing_config. Fixed misleading doc comment in core/lib.rs. 4. BLOCKER: DEC-030 updated to reflect desktop VoiceActivity enablement. Traceability docs (SRS, SysDes, SAD, SDD, implementation-status) updated. 5. Silero ONNX cfg narrowed to desktop-only (excludes macOS/Android). Cargo.toml ort dependency target cfg narrowed similarly. 6. Realtime callback debt documented as TODO at CaptureState::ingest. * fix(audio): exclude ort dep on Android target ort does not provide first-class Android prebuilts in our pin, mirror the iOS/macOS exclusion so cargo metadata succeeds for android targets. * test(audio): fix stale select_ptt_backend import in ptt_privacy The helper moved out of the ptt_backends submodule onto the crate root; update the integration test imports so the test compiles again. * build(windows): scope MSVC release CRT cmake wrapper via Cargo [env] Cargo's [target.<triple>] table only forwards a fixed allowlist (linker, runner, rustflags, rustdocflags, ar), so setting CMAKE there was silently dropped and audiopus_sys kept linking the debug CRT, producing LNK4098 'MSVCRTD conflicts' and __imp__CrtDbgReportW errors on x86_64-pc-windows-msvc test builds. Move the override to Cargo's [env] table using cc/cmake-rs's target-suffixed CMAKE_<triple> lookup (force=true, relative=true) so it applies to MSVC targets only and not to host tooling. Add stdout markers to the wrapper so its invocation is provable in cargo -vv logs. Verified: cargo test -p chanora_audio --target x86_64-pc-windows-msvc --lib --no-run now links cleanly; CMakeCache.txt records CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL and CMP0091=NEW. * fix(flutter): gate VoiceActivity transmit mode by platform support VoiceActivity relies on the native VAD worker, which is only wired up on Windows, Linux, and Android. Showing the option on iOS, macOS, or web let users select a mode that silently never transmitted. Add voiceActivityTransmitAvailable + transmitModeSegmentsFor() helpers in voice_settings_controls.dart, hide the VAD row in voice_compact.dart and drop the VAD segment from the settings dialog when unsupported. Keep the legacy const transmitModeSegments for the existing widget test and add two new tests covering the gated helper.
11 KiB
11 KiB
Chanora Implementation Status — 2026-05-28
Workspace version: v0.2.0-beta.1
Flutter app version/build: 0.3.0+100
CHANGELOG latest: v0.3.0
Build status: Host Rust workspace evidence shows all 9 crates compile cleanly. This does not claim Android target success; Android target compile/install/smoke evidence remains blocked locally as noted below.
P0 / MVP
Done
| Area | Evidence |
|---|---|
| App shell / startup | main.dart (2313 lines), app_bootstrap.dart, RustLib.init() wired |
| Flutter UI | Full widget set: connect_widgets, snapshot_view, chat_views, voice_bar, voice_compact, voice_settings, voice_settings_controls, client_info_sheet, input_dialogs, bbcode_text |
| Material 3 + design tokens | chanora_tokens.dart, platform_capabilities.dart |
| Localization (en + zh-Hans) | l10n/generated/app_localizations_en.dart + app_localizations_zh.dart, l10n.yaml |
| Flutter/Rust bridge | chanora_bridge crate (2152-line api.rs), generated frb_generated.rs, Dart side generated |
| Protocol adapter | chanora_protocol — tsclientlib isolated behind ProtocolClient, typed DTOs, ProtocolError catalogue |
| Connection lifecycle | chanora_core — supervisor task, exponential backoff reconnect (1s→60s), user-disconnect suppresses reconnect; branch simplify-project-review has started splitting the previous large lib.rs into focused internal Modules (events.rs, network_diagnostics.rs) while preserving public re-exports |
| State sync reducer unit | chanora_state — ConnectionState, channel_join, snapshot/delta reducers, reconnect handling, deterministic ordering, malformed duplicate normalization, channel-delete/client cleanup, and reducer unit tests. Runtime core integration still uses snapshot/probe refresh paths and remains separate validation work. |
| Audio subsystem | chanora_audio — Opus encode/decode, HPF/NS/AEC3/AGC2 DSP, PTT backends (Windows/macOS/Linux/focused), iOS VoiceProcessingIO, Android Oboe, jitter buffer via tsclientlib::audio::AudioHandler, mixer, mute/deaf gates, release-tail timer, and Windows/Linux desktop VoiceActivity through the capture VAD path. Mobile, macOS, and unverified-platform VoiceActivity remain deferred per DEC-030. |
| Push-to-talk | Per-platform backends: Windows Raw Input + hook fallback, macOS Event Tap, Linux freedesktop portal, focused fallback; PttCapabilityLevel (L0–L3); missed-key-up watchdog |
| Voice controls UI | voice_bar, voice_compact, voice_haptics, voice_level_meter, voice_platform, ptt_capability_badge, talk_power_warning |
| Storage (non-secret) | chanora_storage — BookmarkRepository (SQLite/rusqlite bundled, schema v2), ChaCha20-Poly1305 encrypted passwords |
| Storage (secrets) | IdentityFileStore with platform keyring (Linux Secret Service, macOS Keychain, Windows Credential Manager, iOS Keychain); file fallback with 0600 perms |
| Diagnostics | chanora_diagnostics — Redactor (IP/host/email/token/path/secret scrubbing), InMemoryLogSink, DiagnosticExport JSON bundle, KnownSecretRegistry, panic hook |
| Server address resolution | chanora_resolver — SRV/TSDNS/DNS fallback |
| Server prefetch | chanora_prefetch crate + prefetch_debouncer.dart — invisible host-field prefetch, TTL cache, generation-safe |
| Android platform | android_voice_unit.rs, android_permissions_service.dart, MODE_IN_COMMUNICATION routing, foreground service (flutter_foreground_task) |
| iOS platform | ios_voice_unit.rs, ios_raw_unit.rs, ios_permissions_service.dart, AVAudioSession integration, audio_session package |
| Permission UX | permission_state_banner.dart, pre-request explainers |
| Bookmark UI | Save/connect/delete in connect_widgets.dart |
| Channel join | Tap-to-join with optional password, channel_join_error_mapper.dart, channel_spacer.dart |
| Chat | chat_views.dart, BBCode rendering (bbcode_text.dart) |
| Audio settings UI | voice_settings.dart, voice_settings_controls.dart, audio_processing_config_state.dart, audio_device_list_tile.dart, audio_output_tile.dart |
| Audio debug stats | audio_debug_stats_panel.dart |
| TS3 server link | ts3_server_link.dart — ts3server:// URI parsing |
| UI preferences | ui_preferences_service.dart, shared_preferences |
| Connection phase state | connection_phase_state.dart |
| Snapshot state mapper | snapshot_state_mapper.dart |
| Back intent (Android) | back_intent_policy.dart, back_intent_service.dart |
| Audio lifecycle | audio_lifecycle_service.dart |
| Link trust | link_trust_service.dart |
| About dialog | Non-affiliation statement, dual-license declaration, NOTICE pointer |
| CI | GitHub Actions on every push |
| Workspace compiles | Host Rust workspace evidence shows all 9 crates build cleanly; Android target compilation remains blocked locally as noted below. |
Partial / Scaffold Only
| Area | Gap |
|---|---|
| Event replay tooling | Reducer tests cover the state-sync contract, but standalone replay-file tooling remains a P1 verification gap. |
| Reducer runtime integration evidence | The standalone reducer is unit-tested, but chanora_core still refreshes UI state through snapshot/probe paths rather than folding all live protocol events through chanora_state::reduce. |
| Mobile/macOS VoiceActivity | assets/models/silero_vad.onnx is bundled and used by the desktop VAD path where runtime evidence supports it; mobile, macOS, and unverified-platform TransmitMode::VoiceActivity remain disabled/deferred until a later baseline supplies backend enablement and verification evidence. |
| macOS build | Source-buildable only; no public release artifact is approved. |
| Windows build | Source-buildable only; no public release artifact is approved. |
| iOS build | Source-buildable/unsigned validation only; no TestFlight/App Store release artifact is approved. |
Not Done (P0 blockers remaining)
| Item | Status |
|---|---|
| DEC-012 legal/trademark/OSS review | Explicitly open. Public release is blocked. |
| Android Keystore-backed DEK | Deferred to v1.1. Android still uses file-fallback for the Data Encryption Key. |
| Android target compile/install/smoke evidence | Blocked locally until the Android NDK compiler aarch64-linux-android-clang is available and adb devices -l shows an authorized device or emulator. |
iOS AVAudioSession.Mode.voiceChat |
Implemented in apps/chanora_flutter/ios/Runner/AppDelegate.swift with call-scoped activation (idle .ambient baseline; VoIP .playAndRecord + .voiceChat + .mixWithOthers engaged only on BridgeEvent::AudioStarted via chanora/ios_audio_session MethodChannel). Release readiness still requires device audio validation and candidate evidence attachment. |
| Candidate state-sync evidence attachment | Reducer tests exist and pass locally; release readiness still needs candidate CI/run IDs and runtime integration evidence attached before public release approval. |
P1 / Beta
Done (promoted from Beta work)
| Area | Evidence |
|---|---|
| Diagnostics export UI | Diagnostics dialog in main.dart, share_plus for export |
| Reconnect banner | Referenced in CHANGELOG v0.4 |
| Identity persistence | IdentityFileStore shipped |
| Audio loopback/processing test hooks | compare_baseline.rs, emit_baseline.rs examples; ptt_privacy.rs, linux_portal_smoke.rs tests |
| Opus codec benchmarks | benches/opus_codec.rs, benches/resampler.rs, benches/realtime_capture.rs |
| Audio processing backend abstraction | processor/mod.rs with sonora, webrtc_apm, noop backends |
| Per-user mute | SRS-074 implemented in audio gate |
| Protocol adapter isolation | SRS-053 trait boundary in place |
Not Done (P1 backlog)
| Item | Notes |
|---|---|
| Per-user volume (SRS-075) | Not yet wired to UI/storage |
| Recent servers persistence (SRS-085) | Not confirmed in storage crate |
| UI settings persistence (SRS-087) | Implemented for current P1 scope using shared_preferences: host, nickname, permission explanation flag, and theme mode (system / light / dark). SQLite-backed UI settings remain a future hardening option if multi-profile or transactional settings are introduced. |
| Event replay tool (SRS-061, SRS-098) | No replay infrastructure found |
| Network diagnostics (SRS-100) | Core tracks connect/disconnect counts and last-loss reasons in network_diagnostics.rs; export/integration evidence still needs release-candidate attachment |
| Side navigation rail for medium layout (SRS-153) | Not confirmed |
| Keyboard focus traversal (SRS-160) | Not confirmed |
| Android audio focus / BT route changes (SRS-112) | Partial — MODE_IN_COMMUNICATION done; full focus/BT handling not confirmed |
| Windows installer packaging (SRS-116) | Not in rc.1 artifacts |
| Linux packaging (AppImage/Flatpak/deb/rpm) (SRS-118) | Not confirmed |
| Android AAB release build pipeline (SDD-109) | Referenced but not confirmed as CI-automated |
| iOS TestFlight/App Store build pipeline (SRS-120) | Deferred |
| Light/dark theme toggle (SAD-043) | Not confirmed in UI |
| Audio device hot-plug recovery (SRS-082) | Noted as follow-up work in chanora_audio/src/lib.rs |
P2 / Production
Not Done (all P2 items pending)
| Item | Notes |
|---|---|
| DEC-012 legal sign-off | Hard blocker for public release |
| macOS signed + notarized builds (SRS-117) | Requires macOS build host |
| C4 architecture views in SAD (SAD-046) | Documentation artifact |
| ADRs for significant decisions (SAD-047) | Documentation artifact |
| Architecture glossary (SAD-055, SDD-063) | Documentation artifact |
| Staged rollout plan | staged-release-plan.md referenced but not confirmed complete |
| Bidirectional text (SRS-175) | Deferred |
| Expanded layout persistent side panes (SRS-154) | Deferred |
SOP (Standing Operating Procedures)
In Place
- Agent router + phase spec docs (in git history, currently untracked/renamed)
phase_index.jsonmachine-readable requirement index- Conventional Commits style enforced
- Privacy/security gates documented (
SOP_AGENT_OPERATIONS.md) - Traceability chain: SysRS → SysDes → SRS → SAD → SDD
SECURITY.md,CONTRIBUTING.md,NOTICE, dual-license files present
Needs Attention
- The agent spec docs (
P0_MVP_AGENT_SPEC.md,P1_BETA_AGENT_SPEC.md, etc.) are deleted from the working tree but still in git HEAD. The newdocs/srs.md,docs/sysdes.md,docs/sysrs.mdare untracked. Docs reorganization in progress — files need to be committed or deletions reverted.
Summary
P0 / MVP: ~85% done. Core product works end-to-end (connect, voice, chat,
bookmarks, storage, diagnostics). Main blockers: DEC-012 legal
review (hard gate), Android Keystore DEK, iOS voiceChat evidence,
and candidate evidence attachment.
P1 / Beta: ~40% done. Audio processing backend, diagnostics export, and
loopback tests are in. Per-user volume, event replay, network
diagnostics, packaging pipelines, and several UI hardening items
remain.
P2 / Prod: ~5% done. Blocked on P0 legal gate. Documentation artifacts
(C4 views, ADRs, glossary) and production signing pipelines
not started.
SOP: Docs in place but a working-tree reorganization is uncommitted.