edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
329 Commits 17 Branches 3 Tags
5ef0eee3f19ef0baabacb8317e9f9c7d276e2a79
Commit Graph
6 Commits
Author SHA1 Message Date
Edison Jwa 2f6d45fb04 feat(audio): desktop Silero ONNX VAD + Windows PTT modernization + MSVC CRT build fix (#37) 
* feat(audio): add Silero ONNX VAD with WebRTC fallback

Introduce SileroOnnxVad and SileroOnnxVadWorker for desktop targets. The worker runs Silero v6 ONNX inference on a dedicated thread, accumulating 10 ms frames into the 512-sample 16 kHz input the model expects. Add VadOutput, VoiceActivityDetector trait, and WebRtcFallbackVad to provide a uniform VAD interface with graceful fallback when the ONNX model is unavailable. Wire the new VadBackend variants through AudioProcessingConfig and the snapshot stats so the bridge can report which detector is active.

* feat(audio): integrate desktop VAD worker into capture engine

Wire SileroOnnxVadWorker into the desktop capture path so voice activity can open the transmit gate before encoding. The capture callback now processes all audio through resample, downmix, and VAD unconditionally; transmit_active still gates Opus encoding.

Add new_desktop_audio_processing_state() to construct the config/stats/worker triple, and apply_desktop_vad_backend() to synchronously load or clear the worker on config changes. Override processing_backend to Noop for desktop so bridge diagnostics report the correct backend rather than the iOS-oriented PlatformVoiceProcessing default.

Includes review-driven cleanups: StreamConfig clone to deref per clippy, and a comment explaining why two try_lock calls on silero_vad_worker are structurally necessary (borrow checker requires the policy probe and the fallback path to not share a lock guard because mark_vad_fallback_active takes &mut self).

* fix(audio): modernize Windows PTT to current windows-rs API

Port the Raw Input plus low-level keyboard hook PTT backend to the newer windows-rs patterns: OptionalHandle, Result-returning CreateWindowExW, and None for CallNextHookEx. Replaces the old HHOOK(0) pointer casts. Add deterministic tests for mouse button 4 and 5 press and release driving the gate.

* build(windows): force MSVC release CRT for audiopus cmake builds

audiopus_sys calls cmake::build(opus_path), so downstream Cargo env cannot use cmake-rs Config::define() to override CMake's MSVC Debug CRT defaults. Point cmake-rs at a small wrapper that injects the policy and cache variables during configure while passing cmake --build, --version, and -E through unchanged. Keeps Opus Debug builds on Rust's release dynamic CRT (/MD) instead of CMake's default debug CRT (/MDd), which otherwise pulls in unresolved __imp__CrtDbgReportW symbols at test link.

Document that the iOS deployment target is intentionally absent from this file. It is enforced by tools/build-ios.sh and the Xcode project; setting it globally here would make native macOS cargo check runs try to link iPhone objects against the macOS SDK.

* build(flutter): update pubspec.lock after plugin additions

Regenerated lockfile reflecting the local_notifications and connectivity_plus plugin additions from the poke-notifications feature.

* fix(audio): address PR #37 review findings

Six fixes from independent PR review:

1. BLOCKER: Replace Windows-only cmake .cmd wrapper with cross-platform
   CMake env vars. Setting CMAKE=tools/cmake-msvc-release-crt.cmd
   globally broke non-Windows hosts because cmake-rs would try to
   execute a .cmd file on macOS/Linux. Instead, set
   CMAKE_POLICY_DEFAULT_CMP0091=NEW and CMAKE_MSVC_RUNTIME_LIBRARY=
   MultiThreadedDLL as env vars that CMake reads natively. MSVC-
   specific vars are safely ignored by GCC/Clang toolchains. Delete
   the now-unnecessary wrapper script.

2. IMPORTANT: Join the Silero worker thread in Drop instead of
   detaching it. The old code dropped the JoinHandle which detaches
   the thread; the new code calls handle.join() after closing the
   channel, ensuring the ONNX session is cleaned up before the
   worker is replaced during config changes.

3. IMPORTANT: Single-try_lock refactor of the capture VAD callback.
   The double try_lock (policy probe + send) is replaced by a single
   scoped try_lock that both probes availability and sends the frame.
   The guard is dropped before the fallback path, which needs &mut
   self for mark_vad_fallback_active. This also eliminates the
   VadWorkerPolicy enum and callback_vad_worker_policy function,
   whose behavior is now inlined into the callback.

4. IMPORTANT: Remove tracing from the realtime capture callback.
   mark_vad_fallback_active and sync_vad_backend emitted info!/warn!
   from the audio thread. Replace with silent atomic state
   publishing via SharedAudioProcessingStats; the bridge stats
   stream already exposes vad_fallback_active for diagnostics.

5. IMPORTANT: Defer ONNX model load outside the worker mutex.
   apply_desktop_vad_backend_to_worker now constructs the new worker
   before taking the lock, then swaps it in under a short hold.
   This prevents the realtime callback from being blocked during
   model I/O + thread spawn.

6. MINOR: Remove unused VadBackend import from vad/mod.rs after
   deleting the policy code.

* fix(audio): address PR #37 second-pass review findings

5-agent review found 5 blocking issues. All addressed:

1. BLOCKER: CMake env vars don't reach CMake cache. Restored .cmd wrapper
   but scoped to Windows MSVC targets only via [target.x86_64-pc-windows-msvc]
   and [target.aarch64-pc-windows-msvc] in .cargo/config.toml. Non-Windows
   hosts are unaffected.

2. BLOCKER: processing_backend normalized in set_audio_processing_config
   on desktop (cfg-gated override to Noop), mirroring startup default.

3. BLOCKER: Model-path reload was already wired via reload_audio_processing_config.
   Fixed misleading doc comment in core/lib.rs.

4. BLOCKER: DEC-030 updated to reflect desktop VoiceActivity enablement.
   Traceability docs (SRS, SysDes, SAD, SDD, implementation-status) updated.

5. Silero ONNX cfg narrowed to desktop-only (excludes macOS/Android).
   Cargo.toml ort dependency target cfg narrowed similarly.

6. Realtime callback debt documented as TODO at CaptureState::ingest.

* fix(audio): exclude ort dep on Android target

ort does not provide first-class Android prebuilts in our pin, mirror the
iOS/macOS exclusion so cargo metadata succeeds for android targets.

* test(audio): fix stale select_ptt_backend import in ptt_privacy

The helper moved out of the ptt_backends submodule onto the crate root;
update the integration test imports so the test compiles again.

* build(windows): scope MSVC release CRT cmake wrapper via Cargo [env]

Cargo's [target.<triple>] table only forwards a fixed allowlist
(linker, runner, rustflags, rustdocflags, ar), so setting CMAKE there
was silently dropped and audiopus_sys kept linking the debug CRT,
producing LNK4098 'MSVCRTD conflicts' and __imp__CrtDbgReportW errors
on x86_64-pc-windows-msvc test builds.

Move the override to Cargo's [env] table using cc/cmake-rs's
target-suffixed CMAKE_<triple> lookup (force=true, relative=true) so it
applies to MSVC targets only and not to host tooling. Add stdout
markers to the wrapper so its invocation is provable in cargo -vv logs.

Verified: cargo test -p chanora_audio --target x86_64-pc-windows-msvc
--lib --no-run now links cleanly; CMakeCache.txt records
CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL and CMP0091=NEW.

* fix(flutter): gate VoiceActivity transmit mode by platform support

VoiceActivity relies on the native VAD worker, which is only wired up
on Windows, Linux, and Android. Showing the option on iOS, macOS, or
web let users select a mode that silently never transmitted.

Add voiceActivityTransmitAvailable + transmitModeSegmentsFor() helpers
in voice_settings_controls.dart, hide the VAD row in voice_compact.dart
and drop the VAD segment from the settings dialog when unsupported.
Keep the legacy const transmitModeSegments for the existing widget test
and add two new tests covering the gated helper.
 2026-06-09 20:47:16 +09:00
Edison Jwa fe6e07353e chore: restore product scaffold to rollback baseline 2026-05-29 14:02:04 +09:00
Edison Jwa 7d6d56e330 docs(p0): compact MVP spec for Android Oboe focus 2026-05-20 14:52:33 +09:00
EdisonJwa 5aa51c310f docs(p0): SysRS/SysDes/SRS/SAD/SDD/Verification + traceability for Android P0 reconciliation 
Full P0 Android documentation chain:

- SysRS: API 24 → API 28 reconciliation per DEC-004 (SysRS-288);
  add SysRS-305 (Android in-call audio mode), SysRS-306
  (RECORD_AUDIO runtime timing).
- SysDes: SysDes-152 (in-call audio mode subsystem), SysDes-153
  (RECORD_AUDIO permission flow), SysDes-154 (Android voice audio
  backend), SysDes-155 (macOS runtime baseline).
- SRS: SRS-187 → API 28; add SRS-208 (in-call audio mode), SRS-209
  (RECORD_AUDIO + listen-only fallback), SRS-210..215 (Android voice
  audio backend latency/preset/AEC/usage/sharing/foreground service);
  retarget SysDes anchors from generic SysDes-135 to SysDes-152/153/154.
- SAD: SAD-063 refreshed (API 28); add SAD-084 (audio mode
  controller), SAD-085 (permission adapter with listen-only),
  SAD-086 (foreground service), SAD-087 (macOS runtime baseline);
  formalize cross-cutting + platform-specific allocation pattern
  in §24.1.
- SDD: expand SDD-028 (BackIntentService); refresh SDD-073 (build
  config); add SDD-105 (JNI bootstrap), SDD-106 (permission
  requester), SDD-107 (foreground service), SDD-108 (audio mode
  controller), SDD-109 (AAB pipeline), SDD-110 (PTT capability),
  SDD-111..116 (Android voice audio backend), SDD-118 (Android
  bridge build automation), SDD-119 (iOS/macOS bridge build
  automation back-fill).
- Verification: create android-p0-acceptance.md TC-1..TC-18; add
  SWE4-UV-040..052, SWE5-IV-016..026, SWE6-SV-018..030,
  SYS4-SIV-015 strengthened + SYS4-SIV-017/018.
- Governance: traceability matrix v0.9.9 with end-to-end chain
  closure; DEC-032 documents the temporary abiFilters reduction to
  arm64-v8a only and its restore-by gate.

Trace: full chain SysRS → SysDes → SRS → SAD → SDD → Code → Verification.
 2026-05-18 12:48:28 +08:00
EdisonJwa 02ffadfa52 docs(ptt): land Baseline Candidate v0.9.3 — capability-based desktop PTT 
Applies the gen2 desktop-PTT review summary
(`gen2/chanora-desktop-ptt-review-summary-v0.9.2.md`) to our doc set
with the owner rulings PTT-OPEN-001 through PTT-OPEN-006 resolved as
accepted decisions DEC-023 through DEC-028:

  * DEC-023 Windows Global PTT P0 / MVP
  * DEC-024 macOS Global PTT P0 / MVP with permission UX
  * DEC-025 Linux officially-tested env: GNOME on Wayland only
  * DEC-026 Mouse side buttons supported (Win + macOS; Linux portal)
  * DEC-027 PTT diagnostics: capability + availability only, no
            raw key codes ever
  * DEC-028 Missed-key-up watchdog: P0

Requirements (SysRS / SRS) and architecture (SysDes / SAD / SDD)
gain the desktop-PTT ID set the gen2 summary describes:

  SysRS-296..302  -> SysDes-142..148
                  -> SRS-195..203
                  -> SAD-071..079
                  -> SDD-081..092

ID totals advance from 295 / 141 / 194 / 70 / 80 to 302 / 148 / 203
/ 79 / 92. The strict layered sourcing rule (`SRS -> SysDes` only,
`SAD -> SRS` only, `SDD -> SAD` only) is preserved; the
`tools/validate_docs.py` validator reports zero undefined refs and
zero direct-layer-rule violations.

New document:

  * `docs/architecture/desktop-ptt-architecture.md` — capability
    ladder (L0Focused, L1GlobalShortcut, L2GlobalHoldToTalk,
    L3GlobalWithMouseButtons, L4DeviceAware reserved), Windows /
    macOS / Linux strategies, privacy rule, audio-gate rule,
    missed-key-up watchdog, release-readiness evidence requirement,
    traceability summary.

Doc addenda (Baseline Candidate 0.9.3):

  * `privacy/privacy-policy.md` — no raw key history, capability-
    dependent Global PTT, UI reflects actual runtime capability
  * `security/threat-model.md` — THREAT-PTT-001..006
  * `security/diagnostic-redaction-audit-report.md` —
    REDACT-PTT-001..006 banned field list enforced by `PttSanitizer`
  * `release/platform-release-policy.md` — per-platform evidence
    fields, no over-claim on untested Linux compositors
  * `release/release-readiness-go-nogo-record.md` — RR-PTT-001..008
    release-readiness items
  * `verification/swe4-unit-verification-plan.md` —
    SWE4-UV-035..039
  * `verification/swe5-software-integration-verification-plan.md` —
    SWE5-IV-015
  * `verification/swe6-software-verification-plan.md` — SWE6-SV-017
  * `verification/sys4-system-integration-verification-plan.md` —
    SYS4-SIV-016
  * `governance/traceability-matrix.md` — full PTT trace rows +
    verification map
  * `governance/decision-impact-assessment.md` — DEC-023..028
    impact matrix
  * `governance/product-decision-register.md` v0.9.9 entry
    recording DEC-023..028 in the decision table and the status
    table at §7
  * `governance/document-index.md` — adds
    `desktop-ptt-architecture.md` to the controlled set
  * `architecture/proof-of-concept-plan.md` —
    PoC-PTT-001..005 platform items
  * `references/external-references.md` — Windows Raw Input,
    macOS event-tap, Linux GlobalShortcuts portal references
  * Both validation reports
    (`baseline-candidate-validation-report.md`,
    `repo-format-validation-report.md`) bumped to v0.9.3 with the
    new ID totals (302 / 148 / 203 / 79 / 92).

README §"Desktop Push-to-Talk" added between Architecture Overview
and Repository Layout: capability levels, per-platform strategy,
privacy posture, missed-key-up watchdog.

Tooling:

  * `tools/validate_docs.py` copied from the gen2 zip into the
    repo tree (was previously available only inside the zip).
    Reports zero undefined refs, zero direct-layer-rule violations,
    English-only CJK check passes. The 35 "old package-style
    filename" hits are pre-existing and identical to the gen2
    baseline (they live in `path-migration-map.md` and config-ID
    headers of governance docs and are intentional per the path
    migration policy).
  * `.gitignore` adds `/gen2/` so the externally-provided review
    package does not enter the repo.

No code changes in this commit; B (the implementation split into
`transmit_active` / `capture_active`, `PttCapabilityLevel`
reporting, `PttSanitizer` diagnostics rule, and the UI capability
badge) follows in a separate commit.
 2026-05-15 14:51:22 +08:00
EdisonJwa f1bc9a6c85 chore(repo): initial baseline import (docs v0.9.2 + bootstrap) 
Imports the v0.9.2 documentation baseline and the bootstrap files
required by docs/governance/repository-bootstrap-plan.md v0.1.0 §3,
minus the justfile (added in the next commit).

This commit establishes the git history for the project. All previous
work lived only as filesystem state with no version control.
 2026-05-14 12:25:33 +08:00