# Automated Test Environment Requirements

**Date:** 2026-06-11  
**Purpose:** Define what's needed for a fully automated test system

---

## 1. Current Test Infrastructure

### What's Automated

| Area | Status |
|---|---|
| Rust unit/integration tests | ✅ CI on Ubuntu |
| Rust static analysis (clippy) | ✅ Advisory |
| Flutter analyze + test | ✅ CI on Ubuntu |
| Supply-chain checks | ✅ CI |
| License inventories | ✅ CI |
| Audio benchmarks | ⚠️ Advisory only |
| iOS unsigned build | ✅ CI on macOS |

### What's Missing

| Gap | Impact |
|---|---|
| Android CI build or test | Android verification blocked |
| iOS device/simulator test | Audio session untested |
| Windows CI | PTT and audio untested |
| macOS CI | Native audio untested |
| Linux CI (non-Ubuntu) | Portal behavior untested |
| Audio quality metrics | No POLQA/PESQ/ViSQOL |
| Network condition simulation | No latency/packet loss testing |
| Integration test with real server | E2E excluded from CI |
| Performance regression gates | Benchmarks advisory only |

---

## 2. Platform Requirements

### iOS / iPadOS

| Requirement | Details |
|---|---|
| Minimum OS | iOS 16+ (CoreML Silero VAD) |
| Hardware | Physical iPhone (A12+), physical iPad |
| Simulator | UI/audio session mock testing |
| Platform features | AVAudioSession, CoreML VAD, haptics, push-to-talk |

### Android

| Requirement | Details |
|---|---|
| Minimum SDK | API 28 (Android 9) |
| Hardware | Physical device (arm64), emulator for CI |
| Platform features | Oboe audio, AudioManager, permissions, foreground service |

### macOS

| Requirement | Details |
|---|---|
| Minimum OS | macOS 13+ (CoreML Silero VAD) |
| Hardware | Apple Silicon Mac (primary), Intel Mac (compatibility) |
| Platform features | CoreAudio VoiceProcessingIO, Event Tap PTT, permissions |

### Windows

| Requirement | Details |
|---|---|
| Minimum OS | Windows 10+ |
| Hardware | Windows PC with audio hardware |
| Platform features | cpal audio, Raw Input PTT, Credential Manager |

### Linux

| Requirement | Details |
|---|---|
| Environment | GNOME on Wayland |
| Hardware | Linux PC with audio hardware |
| Platform features | cpal capture, SDL2 playback, D-Bus GlobalShortcuts |

---

## 3. Physical Devices Needed

| Device | Purpose | Est. Cost |
|---|---|---|
| iPhone 14+ (A15+) | iOS audio, CoreML VAD, PTT | $600-800 |
| iPad (A12+) | iPadOS layout, audio session | $350-500 |
| Android phone (arm64) | Android audio, Oboe | $200-400 |
| Android tablet | Layout testing | $200-300 |
| Apple Silicon Mac | macOS build/test, iOS Simulator | (dev machine) |
| Intel Mac | macOS x86_64 compatibility | $500-800 (used) |
| Windows PC | Windows PTT, audio | $500-800 |
| Linux PC | GNOME/Wayland, PipeWire | $500-800 |
| USB headset | Audio device switching | $30-60 |
| Bluetooth earbuds | Bluetooth audio route | $50-150 |
| **Total** | | **$2,400-4,200** |

---

## 4. Audio Testing Tools

| Tool | Purpose | Cost |
|---|---|---|
| **ViSQOL** (Google, open-source) | Speech quality metric | Free |
| **PESQ / POLQA** | ITU-T speech quality | $500-2000 (license) |
| **Virtual audio devices** | Software loopback for CI | Free (BlackHole, VB-Audio, snd-aloop) |
| **Opus test vectors** | Codec conformance | Free (IETF) |
| **Custom DSP analysis** | THD, SNR, frequency response | Build in-house |

---

## 5. Network Simulation Tools

| Tool | Purpose | Cost |
|---|---|---|
| **tc (traffic control)** | Linux latency/jitter/packet loss | Free |
| **Network Link Conditioner** | Apple's built-in network impairment | Free |
| **clumsy / NetLimiter** | Windows network impairment | Free / $30 |
| **comcast** | Cross-platform network impairment | Free |
| **Toxiproxy** | TCP proxy with configurable latency | Free |

---

## 6. Test Automation Architecture

### Test Layers

```
┌─────────────────────────────────────────────────────────────┐
│                    SYS.4 System Integration                  │
│  Real devices, real servers, real networks, store builds     │
├─────────────────────────────────────────────────────────────┤
│                  SWE.6 Software Verification                 │
│  MVP acceptance matrix, E2E flows, audio quality            │
├─────────────────────────────────────────────────────────────┤
│               SWE.5 Software Integration                     │
│  Flutter ↔ Bridge ↔ Rust ↔ Protocol, platform channels      │
├─────────────────────────────────────────────────────────────┤
│                    SWE.4 Unit Tests                          │
│  Rust crate tests, Flutter widget/service tests, mocks      │
└─────────────────────────────────────────────────────────────┘
```

### Recommended Test Matrix

| Test Type | Runner | Frequency | Blocking? |
|---|---|---|---|
| Rust `cargo test` | Ubuntu CI | Every PR | Yes |
| Rust `cargo clippy` | Ubuntu CI | Every PR | Advisory |
| Flutter `analyze` + `test` | Ubuntu CI | Every PR | Yes |
| Supply-chain | Ubuntu CI | Every PR | Yes |
| iOS unsigned build | macOS CI | Every PR | Yes |
| Android build (multi-ABI) | Ubuntu CI + NDK | Every PR | Yes |
| Windows build + smoke | Windows runner | Every PR | Yes |
| macOS build + smoke | macOS runner | Every PR | Yes |
| Audio benchmarks | Ubuntu CI | Every PR | Advisory |
| E2E (live server) | Dedicated runner | Nightly | No |
| Audio quality (loopback) | Device runners | Weekly | Advisory |
| Network condition tests | Linux + tc | Weekly | Advisory |

---

## 7. Cloud Testing Services

| Service | Use Case | Est. Monthly Cost |
|---|---|---|
| Firebase Test Lab | Android device matrix | $0-150 (free tier) |
| BrowserStack App Live | Real iOS + Android devices | $200-400 |
| AWS Device Farm | Automated device testing | $0.17/device-minute |
| GitHub Actions | Current CI baseline | $0-100 |
| MacStadium | macOS CI for signed builds | $80-150 |

---

## 8. Cost Estimates

### Hardware (One-Time)

| Item | Cost |
|---|---|
| iPhone 14+ | $700 |
| iPad | $400 |
| Android phone | $300 |
| Windows PC | $600 |
| Linux PC | $600 |
| Audio peripherals | $100 |
| **Total hardware** | **$2,700** |

### Cloud Services (Monthly)

| Service | Cost |
|---|---|
| GitHub Actions (public) | $0 |
| GitHub Actions (private) | $0-40 |
| MacStadium macOS runner | $80-150 |
| Firebase Test Lab | $0-150 |
| **Total monthly** | **$80-740** |

### Maintenance Effort

| Area | Effort |
|---|---|
| CI workflow maintenance | 2-4 hrs/week |
| Test fixture updates | 1-2 hrs/week |
| Device OS updates | 2-4 hrs/month |
| Benchmark reviews | 1-2 hrs/month |
| **Total maintenance** | **~15-25 hrs/month** |

---

## 9. Implementation Roadmap

### Phase 1: Foundation (Weeks 1-4)

- [ ] Add Android build job to CI (NDK + cargo-ndk + multi-ABI)
- [ ] Add Windows build job to CI (windows-latest runner)
- [ ] Add macOS build + smoke job to CI
- [ ] Add Linux multi-distro build matrix (Docker)
- [ ] Enable `cargo clippy` as blocking check
- [ ] Add `cargo llvm-cov` coverage reporting

### Phase 2: Device Integration (Weeks 5-8)

- [ ] Acquire physical test devices
- [ ] Set up self-hosted runners for device-connected tests
- [ ] Implement audio loopback test harness
- [ ] Add Flutter integration tests with platform channels
- [ ] Set up Firebase Test Lab for Android device matrix
- [ ] Implement network condition test suite (tc-based)

### Phase 3: Full Automation (Weeks 9-12)

- [ ] Nightly E2E test against controlled TeamSpeak server
- [ ] Weekly audio quality regression suite on physical devices
- [ ] Automated Android multi-device testing (Firebase)
- [ ] Benchmark regression gates
- [ ] iOS TestFlight build + device smoke automation
- [ ] Windows/macOS signed build + smoke automation

---

*Generated by test environment research agents on 2026-06-11*