Edison Jwa
3b13a7edb4
Add full codebase analysis report, issue history, test environment requirements, external research (TeaSpeak/ReSpeak/YaTQA), and offline protocol references (ReSpeak ts3protocol, TeaSpeak, YaTQA). Add master TODO list with 88 items across 11 categories. Add missing LICENSE files. Branch: docs/codebase-analysis-v2
244 lines
8.2 KiB
Markdown
244 lines
8.2 KiB
Markdown
# Automated Test Environment Requirements
|
|
|
|
**Date:** 2026-06-11
|
|
**Purpose:** Define what's needed for a fully automated test system
|
|
|
|
---
|
|
|
|
## 1. Current Test Infrastructure
|
|
|
|
### What's Automated
|
|
|
|
| Area | Status |
|
|
|---|---|
|
|
| Rust unit/integration tests | ✅ CI on Ubuntu |
|
|
| Rust static analysis (clippy) | ✅ Advisory |
|
|
| Flutter analyze + test | ✅ CI on Ubuntu |
|
|
| Supply-chain checks | ✅ CI |
|
|
| License inventories | ✅ CI |
|
|
| Audio benchmarks | ⚠️ Advisory only |
|
|
| iOS unsigned build | ✅ CI on macOS |
|
|
|
|
### What's Missing
|
|
|
|
| Gap | Impact |
|
|
|---|---|
|
|
| Android CI build or test | Android verification blocked |
|
|
| iOS device/simulator test | Audio session untested |
|
|
| Windows CI | PTT and audio untested |
|
|
| macOS CI | Native audio untested |
|
|
| Linux CI (non-Ubuntu) | Portal behavior untested |
|
|
| Audio quality metrics | No POLQA/PESQ/ViSQOL |
|
|
| Network condition simulation | No latency/packet loss testing |
|
|
| Integration test with real server | E2E excluded from CI |
|
|
| Performance regression gates | Benchmarks advisory only |
|
|
|
|
---
|
|
|
|
## 2. Platform Requirements
|
|
|
|
### iOS / iPadOS
|
|
|
|
| Requirement | Details |
|
|
|---|---|
|
|
| Minimum OS | iOS 16+ (CoreML Silero VAD) |
|
|
| Hardware | Physical iPhone (A12+), physical iPad |
|
|
| Simulator | UI/audio session mock testing |
|
|
| Platform features | AVAudioSession, CoreML VAD, haptics, push-to-talk |
|
|
|
|
### Android
|
|
|
|
| Requirement | Details |
|
|
|---|---|
|
|
| Minimum SDK | API 28 (Android 9) |
|
|
| Hardware | Physical device (arm64), emulator for CI |
|
|
| Platform features | Oboe audio, AudioManager, permissions, foreground service |
|
|
|
|
### macOS
|
|
|
|
| Requirement | Details |
|
|
|---|---|
|
|
| Minimum OS | macOS 13+ (CoreML Silero VAD) |
|
|
| Hardware | Apple Silicon Mac (primary), Intel Mac (compatibility) |
|
|
| Platform features | CoreAudio VoiceProcessingIO, Event Tap PTT, permissions |
|
|
|
|
### Windows
|
|
|
|
| Requirement | Details |
|
|
|---|---|
|
|
| Minimum OS | Windows 10+ |
|
|
| Hardware | Windows PC with audio hardware |
|
|
| Platform features | cpal audio, Raw Input PTT, Credential Manager |
|
|
|
|
### Linux
|
|
|
|
| Requirement | Details |
|
|
|---|---|
|
|
| Environment | GNOME on Wayland |
|
|
| Hardware | Linux PC with audio hardware |
|
|
| Platform features | cpal capture, SDL2 playback, D-Bus GlobalShortcuts |
|
|
|
|
---
|
|
|
|
## 3. Physical Devices Needed
|
|
|
|
| Device | Purpose | Est. Cost |
|
|
|---|---|---|
|
|
| iPhone 14+ (A15+) | iOS audio, CoreML VAD, PTT | $600-800 |
|
|
| iPad (A12+) | iPadOS layout, audio session | $350-500 |
|
|
| Android phone (arm64) | Android audio, Oboe | $200-400 |
|
|
| Android tablet | Layout testing | $200-300 |
|
|
| Apple Silicon Mac | macOS build/test, iOS Simulator | (dev machine) |
|
|
| Intel Mac | macOS x86_64 compatibility | $500-800 (used) |
|
|
| Windows PC | Windows PTT, audio | $500-800 |
|
|
| Linux PC | GNOME/Wayland, PipeWire | $500-800 |
|
|
| USB headset | Audio device switching | $30-60 |
|
|
| Bluetooth earbuds | Bluetooth audio route | $50-150 |
|
|
| **Total** | | **$2,400-4,200** |
|
|
|
|
---
|
|
|
|
## 4. Audio Testing Tools
|
|
|
|
| Tool | Purpose | Cost |
|
|
|---|---|---|
|
|
| **ViSQOL** (Google, open-source) | Speech quality metric | Free |
|
|
| **PESQ / POLQA** | ITU-T speech quality | $500-2000 (license) |
|
|
| **Virtual audio devices** | Software loopback for CI | Free (BlackHole, VB-Audio, snd-aloop) |
|
|
| **Opus test vectors** | Codec conformance | Free (IETF) |
|
|
| **Custom DSP analysis** | THD, SNR, frequency response | Build in-house |
|
|
|
|
---
|
|
|
|
## 5. Network Simulation Tools
|
|
|
|
| Tool | Purpose | Cost |
|
|
|---|---|---|
|
|
| **tc (traffic control)** | Linux latency/jitter/packet loss | Free |
|
|
| **Network Link Conditioner** | Apple's built-in network impairment | Free |
|
|
| **clumsy / NetLimiter** | Windows network impairment | Free / $30 |
|
|
| **comcast** | Cross-platform network impairment | Free |
|
|
| **Toxiproxy** | TCP proxy with configurable latency | Free |
|
|
|
|
---
|
|
|
|
## 6. Test Automation Architecture
|
|
|
|
### Test Layers
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ SYS.4 System Integration │
|
|
│ Real devices, real servers, real networks, store builds │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ SWE.6 Software Verification │
|
|
│ MVP acceptance matrix, E2E flows, audio quality │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ SWE.5 Software Integration │
|
|
│ Flutter ↔ Bridge ↔ Rust ↔ Protocol, platform channels │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ SWE.4 Unit Tests │
|
|
│ Rust crate tests, Flutter widget/service tests, mocks │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Recommended Test Matrix
|
|
|
|
| Test Type | Runner | Frequency | Blocking? |
|
|
|---|---|---|---|
|
|
| Rust `cargo test` | Ubuntu CI | Every PR | Yes |
|
|
| Rust `cargo clippy` | Ubuntu CI | Every PR | Advisory |
|
|
| Flutter `analyze` + `test` | Ubuntu CI | Every PR | Yes |
|
|
| Supply-chain | Ubuntu CI | Every PR | Yes |
|
|
| iOS unsigned build | macOS CI | Every PR | Yes |
|
|
| Android build (multi-ABI) | Ubuntu CI + NDK | Every PR | Yes |
|
|
| Windows build + smoke | Windows runner | Every PR | Yes |
|
|
| macOS build + smoke | macOS runner | Every PR | Yes |
|
|
| Audio benchmarks | Ubuntu CI | Every PR | Advisory |
|
|
| E2E (live server) | Dedicated runner | Nightly | No |
|
|
| Audio quality (loopback) | Device runners | Weekly | Advisory |
|
|
| Network condition tests | Linux + tc | Weekly | Advisory |
|
|
|
|
---
|
|
|
|
## 7. Cloud Testing Services
|
|
|
|
| Service | Use Case | Est. Monthly Cost |
|
|
|---|---|---|
|
|
| Firebase Test Lab | Android device matrix | $0-150 (free tier) |
|
|
| BrowserStack App Live | Real iOS + Android devices | $200-400 |
|
|
| AWS Device Farm | Automated device testing | $0.17/device-minute |
|
|
| GitHub Actions | Current CI baseline | $0-100 |
|
|
| MacStadium | macOS CI for signed builds | $80-150 |
|
|
|
|
---
|
|
|
|
## 8. Cost Estimates
|
|
|
|
### Hardware (One-Time)
|
|
|
|
| Item | Cost |
|
|
|---|---|
|
|
| iPhone 14+ | $700 |
|
|
| iPad | $400 |
|
|
| Android phone | $300 |
|
|
| Windows PC | $600 |
|
|
| Linux PC | $600 |
|
|
| Audio peripherals | $100 |
|
|
| **Total hardware** | **$2,700** |
|
|
|
|
### Cloud Services (Monthly)
|
|
|
|
| Service | Cost |
|
|
|---|---|
|
|
| GitHub Actions (public) | $0 |
|
|
| GitHub Actions (private) | $0-40 |
|
|
| MacStadium macOS runner | $80-150 |
|
|
| Firebase Test Lab | $0-150 |
|
|
| **Total monthly** | **$80-740** |
|
|
|
|
### Maintenance Effort
|
|
|
|
| Area | Effort |
|
|
|---|---|
|
|
| CI workflow maintenance | 2-4 hrs/week |
|
|
| Test fixture updates | 1-2 hrs/week |
|
|
| Device OS updates | 2-4 hrs/month |
|
|
| Benchmark reviews | 1-2 hrs/month |
|
|
| **Total maintenance** | **~15-25 hrs/month** |
|
|
|
|
---
|
|
|
|
## 9. Implementation Roadmap
|
|
|
|
### Phase 1: Foundation (Weeks 1-4)
|
|
|
|
- [ ] Add Android build job to CI (NDK + cargo-ndk + multi-ABI)
|
|
- [ ] Add Windows build job to CI (windows-latest runner)
|
|
- [ ] Add macOS build + smoke job to CI
|
|
- [ ] Add Linux multi-distro build matrix (Docker)
|
|
- [ ] Enable `cargo clippy` as blocking check
|
|
- [ ] Add `cargo llvm-cov` coverage reporting
|
|
|
|
### Phase 2: Device Integration (Weeks 5-8)
|
|
|
|
- [ ] Acquire physical test devices
|
|
- [ ] Set up self-hosted runners for device-connected tests
|
|
- [ ] Implement audio loopback test harness
|
|
- [ ] Add Flutter integration tests with platform channels
|
|
- [ ] Set up Firebase Test Lab for Android device matrix
|
|
- [ ] Implement network condition test suite (tc-based)
|
|
|
|
### Phase 3: Full Automation (Weeks 9-12)
|
|
|
|
- [ ] Nightly E2E test against controlled TeamSpeak server
|
|
- [ ] Weekly audio quality regression suite on physical devices
|
|
- [ ] Automated Android multi-device testing (Firebase)
|
|
- [ ] Benchmark regression gates
|
|
- [ ] iOS TestFlight build + device smoke automation
|
|
- [ ] Windows/macOS signed build + smoke automation
|
|
|
|
---
|
|
|
|
*Generated by test environment research agents on 2026-06-11*
|