edison/chanora
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ecdd68eff2273ae94c87bf189112d6ebd0877987
chanora/docs/verification/test-environment-requirements.md
T
Edison Jwa 3b13a7edb4 docs: comprehensive codebase analysis and reference documentation 
Add full codebase analysis report, issue history, test environment
requirements, external research (TeaSpeak/ReSpeak/YaTQA), and offline
protocol references (ReSpeak ts3protocol, TeaSpeak, YaTQA). Add master
TODO list with 88 items across 11 categories. Add missing LICENSE files.

Branch: docs/codebase-analysis-v2
2026-06-11 09:23:10 +09:00

8.2 KiB
Raw Blame History

Automated Test Environment Requirements

Date: 2026-06-11
Purpose: Define what's needed for a fully automated test system

1. Current Test Infrastructure

What's Automated

Area Status
Rust unit/integration tests ✅ CI on Ubuntu
Rust static analysis (clippy) ✅ Advisory
Flutter analyze + test ✅ CI on Ubuntu
Supply-chain checks ✅ CI
License inventories ✅ CI
Audio benchmarks ⚠️ Advisory only
iOS unsigned build ✅ CI on macOS

What's Missing

Gap Impact
Android CI build or test Android verification blocked
iOS device/simulator test Audio session untested
Windows CI PTT and audio untested
macOS CI Native audio untested
Linux CI (non-Ubuntu) Portal behavior untested
Audio quality metrics No POLQA/PESQ/ViSQOL
Network condition simulation No latency/packet loss testing
Integration test with real server E2E excluded from CI
Performance regression gates Benchmarks advisory only

2. Platform Requirements

iOS / iPadOS

Requirement Details
Minimum OS iOS 16+ (CoreML Silero VAD)
Hardware Physical iPhone (A12+), physical iPad
Simulator UI/audio session mock testing
Platform features AVAudioSession, CoreML VAD, haptics, push-to-talk

Android

Requirement Details
Minimum SDK API 28 (Android 9)
Hardware Physical device (arm64), emulator for CI
Platform features Oboe audio, AudioManager, permissions, foreground service

macOS

Requirement Details
Minimum OS macOS 13+ (CoreML Silero VAD)
Hardware Apple Silicon Mac (primary), Intel Mac (compatibility)
Platform features CoreAudio VoiceProcessingIO, Event Tap PTT, permissions

Windows

Requirement Details
Minimum OS Windows 10+
Hardware Windows PC with audio hardware
Platform features cpal audio, Raw Input PTT, Credential Manager

Linux

Requirement Details
Environment GNOME on Wayland
Hardware Linux PC with audio hardware
Platform features cpal capture, SDL2 playback, D-Bus GlobalShortcuts

3. Physical Devices Needed

Device Purpose Est. Cost
iPhone 14+ (A15+) iOS audio, CoreML VAD, PTT $600-800
iPad (A12+) iPadOS layout, audio session $350-500
Android phone (arm64) Android audio, Oboe $200-400
Android tablet Layout testing $200-300
Apple Silicon Mac macOS build/test, iOS Simulator (dev machine)
Intel Mac macOS x86_64 compatibility $500-800 (used)
Windows PC Windows PTT, audio $500-800
Linux PC GNOME/Wayland, PipeWire $500-800
USB headset Audio device switching $30-60
Bluetooth earbuds Bluetooth audio route $50-150
Total $2,400-4,200

4. Audio Testing Tools

Tool Purpose Cost
ViSQOL (Google, open-source) Speech quality metric Free
PESQ / POLQA ITU-T speech quality $500-2000 (license)
Virtual audio devices Software loopback for CI Free (BlackHole, VB-Audio, snd-aloop)
Opus test vectors Codec conformance Free (IETF)
Custom DSP analysis THD, SNR, frequency response Build in-house

5. Network Simulation Tools

Tool Purpose Cost
tc (traffic control) Linux latency/jitter/packet loss Free
Network Link Conditioner Apple's built-in network impairment Free
clumsy / NetLimiter Windows network impairment Free / $30
comcast Cross-platform network impairment Free
Toxiproxy TCP proxy with configurable latency Free

6. Test Automation Architecture

Test Layers

┌─────────────────────────────────────────────────────────────┐
│                    SYS.4 System Integration                  │
│  Real devices, real servers, real networks, store builds     │
├─────────────────────────────────────────────────────────────┤
│                  SWE.6 Software Verification                 │
│  MVP acceptance matrix, E2E flows, audio quality            │
├─────────────────────────────────────────────────────────────┤
│               SWE.5 Software Integration                     │
│  Flutter ↔ Bridge ↔ Rust ↔ Protocol, platform channels      │
├─────────────────────────────────────────────────────────────┤
│                    SWE.4 Unit Tests                          │
│  Rust crate tests, Flutter widget/service tests, mocks      │
└─────────────────────────────────────────────────────────────┘

Recommended Test Matrix

Test Type Runner Frequency Blocking?
Rust cargo test Ubuntu CI Every PR Yes
Rust cargo clippy Ubuntu CI Every PR Advisory
Flutter analyze + test Ubuntu CI Every PR Yes
Supply-chain Ubuntu CI Every PR Yes
iOS unsigned build macOS CI Every PR Yes
Android build (multi-ABI) Ubuntu CI + NDK Every PR Yes
Windows build + smoke Windows runner Every PR Yes
macOS build + smoke macOS runner Every PR Yes
Audio benchmarks Ubuntu CI Every PR Advisory
E2E (live server) Dedicated runner Nightly No
Audio quality (loopback) Device runners Weekly Advisory
Network condition tests Linux + tc Weekly Advisory

7. Cloud Testing Services

Service Use Case Est. Monthly Cost
Firebase Test Lab Android device matrix $0-150 (free tier)
BrowserStack App Live Real iOS + Android devices $200-400
AWS Device Farm Automated device testing $0.17/device-minute
GitHub Actions Current CI baseline $0-100
MacStadium macOS CI for signed builds $80-150

8. Cost Estimates

Hardware (One-Time)

Item Cost
iPhone 14+ $700
iPad $400
Android phone $300
Windows PC $600
Linux PC $600
Audio peripherals $100
Total hardware $2,700

Cloud Services (Monthly)

Service Cost
GitHub Actions (public) $0
GitHub Actions (private) $0-40
MacStadium macOS runner $80-150
Firebase Test Lab $0-150
Total monthly $80-740

Maintenance Effort

Area Effort
CI workflow maintenance 2-4 hrs/week
Test fixture updates 1-2 hrs/week
Device OS updates 2-4 hrs/month
Benchmark reviews 1-2 hrs/month
Total maintenance ~15-25 hrs/month

9. Implementation Roadmap

Phase 1: Foundation (Weeks 1-4)

  • Add Android build job to CI (NDK + cargo-ndk + multi-ABI)
  • Add Windows build job to CI (windows-latest runner)
  • Add macOS build + smoke job to CI
  • Add Linux multi-distro build matrix (Docker)
  • Enable cargo clippy as blocking check
  • Add cargo llvm-cov coverage reporting

Phase 2: Device Integration (Weeks 5-8)

  • Acquire physical test devices
  • Set up self-hosted runners for device-connected tests
  • Implement audio loopback test harness
  • Add Flutter integration tests with platform channels
  • Set up Firebase Test Lab for Android device matrix
  • Implement network condition test suite (tc-based)

Phase 3: Full Automation (Weeks 9-12)

  • Nightly E2E test against controlled TeamSpeak server
  • Weekly audio quality regression suite on physical devices
  • Automated Android multi-device testing (Firebase)
  • Benchmark regression gates
  • iOS TestFlight build + device smoke automation
  • Windows/macOS signed build + smoke automation

Generated by test environment research agents on 2026-06-11

Reference in New Issue View Git Blame Copy Permalink