Files
chanora/docs/architecture/file-transfer-design.md
T
Edison Jwa aa796d7395 feat: file transfer system (avatar/icon download with cacache) (#40)
* docs(architecture): add file transfer design, research, and implementation plan

* feat(cache): add chanora_cache crate with cacache-backed blob cache

- New chanora_cache crate: content-addressed blob store wrapping cacache
- BlobCache API: async put/get/remove/clear/total_size/evict
- Key validation: av_ prefix (32 hex chars), ic_ prefix (decimal digits)
- Cacache provides crash safety, SSRI integrity, content dedup
- Mtime-based eviction via cacache::list_sync + sort by timestamp
- 7 unit tests all passing
- Added to workspace members

* feat(protocol): add file download support for avatars and icons

- Add Request::DownloadFile variant with oneshot reply
- Add ProtocolClient::download_avatar(client_uid) and download_icon(icon_id)
- Track pending file downloads by FiletransferHandle
- Handle StreamItem::FileDownload: read bytes from TCP stream
- Handle StreamItem::FiletransferFailed: map to ProtocolError
- Add ProtocolError::FileTransfer(String) variant
- Add path helper tests for avatar/icon download paths
- No tsclientlib types leak across the adapter boundary

* feat(core): add blob cache wiring and avatar download orchestration

- Add chanora_cache dependency to Cargo.toml
- Add blob_cache field to ChanoraSession (Arc<Mutex<Option<BlobCache>>>)
- Add init_cache() method: creates BlobCache, runs eviction
- Add get_avatar() method: cache-first, download on miss, store in cache
- Add clear_cache() and cache_size() methods for cache management
- Add CoreError::Cache variant for BlobCacheError conversion
- Add avatar_cache integration test

* feat(bridge): add init_cache, download_avatar, and cache management functions

- Add init_cache(dir) bridge function
- Add download_avatar(avatar_hash, client_uid) bridge function
- Add clear_file_cache() and file_cache_size() bridge functions
- Map CoreError::Cache and ProtocolError::FileTransfer in BridgeError

* feat(flutter): add cache initialization wiring and avatar download shims

- Add wireCache() to app_bootstrap using getApplicationCacheDirectory()
- Call wireCache() after wireStorage() in main bootstrap flow
- Add Dart-side initCache and downloadAvatar wrapper shims in api.dart
- Update Cargo.lock for new chanora_cache dependency

* feat(core): FileTransferService with coalescing, throttling, negative cache

- New file_transfer module with FileTransferService struct
- Semaphore(2) throttles concurrent downloads
- In-flight HashMap coalesces duplicate avatar requests
- 5-min negative cache short-circuits ServerRejected misses
- ChanoraSession delegates get_avatar through the service
- connect/disconnect update shared protocol handle
- clear_cache/cache_size delegate to service
- 2 new unit tests (cached hit, negative cache)

* feat(core,bridge): add get_icon with coalescing and negative cache

- FileTransferService::get_icon() mirrors get_avatar pattern
- ChanoraSession::get_icon() delegates through FileTransferService
- Bridge download_icon() exposed for Flutter
- Dart downloadIcon() shim added
- Uses PREFIX_ICON (ic_<crc32u>) cache key format
- 1 new unit test (cached icon hit)

* fix(core,protocol): simplify store_protocol and add download size cap

- store_protocol: always write to shared Arc<Mutex<Option<ProtocolClient>>>;
  the FileTransferService holds the same Arc so it sees updates automatically
- read_download_bytes: reject downloads exceeding 10 MB to prevent
  malicious servers from causing OOM
2026-06-10 11:45:14 +09:00

738 lines
31 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# File Transfer Design
**Date:** 2026-06-10
**Status:** Draft for review
**Scope:** Download files from TeamSpeak-compatible servers via the native client protocol, starting with avatars and icons.
**Direct upstream source:** `docs/architecture/sad.md` (SAD-067, SDD-MOD-009)
## 1. Goal
Chanora needs to download files stored on TeamSpeak-compatible servers. The most visible use cases are client avatars and server/channel/client icons. The file transfer mechanism is also used for channel file browser features, but this document scopes the initial design to avatar and icon retrieval only.
This document describes:
- How the TeamSpeak file transfer protocol works.
- How `tsclientlib` exposes it.
- How Chanora should integrate it following the existing protocol adapter pattern.
- How the result flows through the bridge to the Flutter UI layer.
Upload, channel file browsing, and file deletion are explicitly out of scope for the initial implementation.
## 2. Protocol Background
### 2.1 Two-Phase Transfer
TeamSpeak file transfer is a two-phase process:
1. **Command phase** — The client sends a command over the main encrypted UDP connection to request a transfer token (`ftkey`).
2. **Transfer phase** — The client opens a separate TCP connection to the server's file transfer port (default `30033`) and sends the `ftkey` to authenticate the transfer. Raw bytes flow over this TCP stream.
### 2.2 Relevant ServerQuery Commands
| Command | Direction | Purpose |
|---|---|---|
| `ftinitdownload` | Client → Server | Initialize a download. Returns `ftkey`, `port`, `size`. |
| `ftgetfileinfo` | Client → Server | Get metadata for one or more files. |
| `ftgetfilelist` | Client → Server | List files in a channel's file repository. |
| `ftinitupload` | Client → Server | Initialize an upload. |
| `ftlist` | Client → Server | List active file transfers. |
| `ftstop` | Client → Server | Stop a running transfer. |
| `ftdeletefile` | Client → Server | Delete a file. |
| `ftcreatedir` | Client → Server | Create a directory. |
| `ftrenamefile` | Client → Server | Rename or move a file. |
Initial scope uses only `ftinitdownload` and `ftgetfileinfo`.
### 2.3 File Paths
Files are addressed by a path scoped to a channel ID (`cid`):
- `cid=0` — Server-level file repository. Avatars and icons live here.
- `cid=N` (non-zero) — Channel-specific file repository.
Avatar path: `/avatar_<hex>` where `<hex>` is derived from the client's unique identifier (UID). Each byte of the base64-decoded UID is split into two nibbles, and each nibble maps to a letter `a` through `p` (0→a, 1→b, ..., 15→p).
Icon path: `/icon_<id>` where `<id>` is the icon's signed 64-bit integer ID. If negative, treat as unsigned for the path.
### 2.4 `ftinitdownload` Command
```
ftinitdownload clientftfid={id} name={path} cid={channelId} cpw={password} seekpos={seek} proto=0
```
Parameters:
| Parameter | Type | Description |
|---|---|---|
| `clientftfid` | `u16` | Arbitrary client-side transfer ID. |
| `name` | `string` | File path, e.g. `/avatar_abcdef`. |
| `cid` | `ChannelId` | Channel scope (0 = server). |
| `cpw` | `string` | Channel password. Empty for server-level. |
| `seekpos` | `u64` | Resume offset. 0 for a fresh download. |
| `proto` | `u8` | Protocol version. Always 0. |
Server response:
| Field | Type | Description |
|---|---|---|
| `clientftfid` | `u16` | Echo of the client transfer ID. |
| `serverftfid` | `u16` | Server-side transfer ID. |
| `ftkey` | `string` | One-time transfer key (hex). |
| `port` | `u16` | File transfer TCP port (usually 30033). |
| `size` | `u64` | File size in bytes. |
| `proto` | `u8` | Protocol version echo. |
| `ip` | `string` (optional) | Override IP for the TCP connection. |
### 2.5 TCP Transfer
After receiving the `ftkey`, the client:
1. Opens a TCP connection to `server_ip:port`.
2. Sends `ftkey` followed by a newline.
3. Reads exactly `size` bytes of raw file data.
4. Closes the TCP connection.
### 2.6 Permissions
File transfer requires the following permissions on the server:
| Permission | Needed for |
|---|---|
| `i_ft_file_download_power` | Downloading files. |
| `i_ft_needed_file_download_power` | Required download power on the channel/server. |
| `b_ft_ignore_password` | Bypassing channel passwords (not needed for avatars). |
Avatar downloads typically require only basic download power because avatars are in the server-level repository (`cid=0`), which is generally accessible.
### 2.7 Avatar Detection
When a client connects or updates, the server sends `client_flag_avatar` as a string (the avatar hash). If non-empty, the client has an avatar. The avatar is downloaded from `/avatar_<hex>` where `<hex>` is computed from the client's UID (not from the hash string itself — the hash is just a presence indicator).
## 3. tsclientlib Support
`tsclientlib` implements file transfer natively. The library handles the entire command + TCP flow internally:
### 3.1 Public API
```rust
// tsclientlib/src/lib.rs (relevant signatures)
impl Connection {
pub fn download_file(
&mut self,
channel_id: ChannelId,
path: &str,
channel_password: Option<&str>,
seek_position: Option<u64>,
) -> Result<FiletransferHandle>;
pub fn upload_file(
&mut self,
channel_id: ChannelId,
path: &str,
channel_password: Option<&str>,
size: u64,
overwrite: bool,
resume: bool,
) -> Result<FiletransferHandle>;
}
```
`download_file` sends the `ftinitdownload` command and returns a `FiletransferHandle(u16)` immediately. The actual transfer completes asynchronously.
### 3.2 Stream Items
The connection's event stream emits:
| StreamItem | When | Data |
|---|---|---|
| `StreamItem::FileDownload(FileDownloadResult)` | Server responds with `ftkey`; TCP connected and `ftkey` written | `{ size: u64, stream: TcpStream }` |
| `StreamItem::FileUpload(FileUploadResult)` | Upload ready | `{ seek_position: u64, stream: TcpStream }` |
| `StreamItem::FiletransferFailed(FiletransferHandle, Error)` | Transfer failed | Handle + error |
When `FileDownload` fires, tsclientlib has already:
1. Sent `ftinitdownload` over the encrypted UDP command channel.
2. Received the `ftkey`, `port`, and `size` from the server.
3. Opened a TCP connection to `server:port`.
4. Written the `ftkey` to the TCP socket.
The `TcpStream` in `FileDownloadResult` is ready to read; Chanora only needs to read exactly `size` bytes.
### 3.3 Avatar Helper
`tsproto-types` provides `Uid::as_avatar()` which computes the avatar filename from a UID. Chanora's existing `uid_to_avatar_path()` in `adapter.rs` does the same thing independently.
### 3.4 Doc-Comment Examples
tsclientlib's source contains usage examples in doc comments:
```rust
/// Download an icon:
/// con.download_file(ChannelId(0), &format!("/icon_{}", icon_id), None, None)
/// Upload an avatar:
/// con.upload_file(ChannelId(0), "/avatar", None, data.len() as u64, true, false)
```
## 4. Architecture Integration
### 4.1 Existing Pattern
The protocol adapter (`crates/chanora_protocol`) uses a single tokio task that owns the `tsclientlib::Connection`. All operations follow this pattern:
1. Define a `Request` enum variant with parameters and a `oneshot::Sender` for the reply.
2. Send the request through the `mpsc` channel to the connection task.
3. The connection task calls tsclientlib and resolves the oneshot.
File transfer fits this pattern exactly. The only difference is that the result arrives asynchronously via `StreamItem::FileDownload` rather than immediately from the command call.
### 4.2 Design
The file transfer integration adds:
1. **`Request` variants** for file download.
2. **A pending-downloads map** (`HashMap<FiletransferHandle, DownloadContext>`) in the connection task, mirroring the existing `pending_moves` pattern.
3. **`StreamItem::FileDownload` and `StreamItem::FiletransferFailed`** handling in the event loop.
4. **New DTOs** for file transfer results.
5. **Convenience methods** on `ProtocolClient` for avatar and icon downloads.
### 4.3 Layer Responsibilities
| Layer | Responsibility |
|---|---|
| `chanora_protocol` | Call `tsclientlib::download_file`, track pending transfers, read `TcpStream`, return bytes. No tsclientlib types leak. |
| `chanora_core` | Orchestrate when to download (e.g., on profile fetch or on avatar cache miss). |
| `chanora_bridge` | Expose typed `download_avatar` / `download_icon` commands to Flutter. |
| Flutter UI | Call bridge, display with `Image.memory()`. Cache in memory/image cache. |
### 4.4 Error Mapping
File transfer errors map to the existing `ProtocolError` variants:
| tsclientlib error | ProtocolError |
|---|---|
| Permission denied (TS3 error code) | `ServerRejected { code, message }` |
| File not found | `ServerRejected { code, message }` |
| Network/TCP failure | `Backend(String)` |
| Timeout | `Timeout` |
| Connection lost mid-transfer | `Lost(String)` |
## 5. Detailed Design
### 5.1 New Types in `dto.rs`
```rust
/// A downloaded file's raw content and metadata.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct DownloadedFile {
/// Raw file bytes.
pub data: Vec<u8>,
/// The server path that was requested.
pub path: String,
/// Channel ID the file was downloaded from.
pub channel_id: u64,
}
```
### 5.2 New Request Variants in `adapter.rs`
```rust
enum Request {
// ... existing variants ...
/// Download a file from the server's file repository.
DownloadFile {
/// Channel ID. 0 for server-level (avatars, icons).
channel_id: u64,
/// File path, e.g. "/avatar_abcdef" or "/icon_12345".
path: String,
/// Channel password. None for server-level files.
channel_password: Option<String>,
/// Reply channel for the result.
reply: oneshot::Sender<Result<DownloadedFile, ProtocolError>>,
},
}
```
### 5.3 Pending Downloads Map
```rust
type PendingDownloads = HashMap<tsclientlib::FiletransferHandle, PendingDownload>;
struct PendingDownload {
path: String,
channel_id: u64,
reply: oneshot::Sender<Result<DownloadedFile, ProtocolError>>,
}
```
### 5.4 Event Loop Handling
In the connection task's main loop, add handling for file transfer stream items:
```rust
// In handle_non_audio_stream_item or in the main loop:
StreamItem::FileDownload(result) => {
// result: FileDownloadResult { size, stream }
// Look up the handle in pending_downloads
// Use tokio::io::AsyncReadExt::read_exact to read 'size' bytes
// Resolve the oneshot with DownloadedFile
}
StreamItem::FiletransferFailed(handle, error) => {
// Look up the handle in pending_downloads
// Resolve the oneshot with ProtocolError::Backend
}
```
The TCP read from the `TcpStream` is an async operation. Since the connection task already runs in a tokio context, the read can be done inline. However, for large files this would block the main event loop. Two approaches:
**Option A: Read inline (simple, good for small files like avatars)**
Avatars are typically under 100 KB. Reading them inline in the event loop is acceptable and avoids complexity.
**Option B: Spawn a reader task**
For future channel-file-browser support with potentially large files, spawn a separate tokio task that reads the stream and sends the result back.
**Recommendation:** Start with Option A. The initial scope is avatars and icons (small files). Refactor to Option B when channel file browsing is implemented.
### 5.5 Request Handling
When the connection task receives `Request::DownloadFile`:
```rust
Ok(Request::DownloadFile { channel_id, path, channel_password, reply }) => {
let ts_channel_id = TsChannelId(channel_id);
match con.download_file(ts_channel_id, &path, channel_password.as_deref(), None) {
Ok(handle) => {
pending_downloads.insert(handle, PendingDownload {
path,
channel_id,
reply,
});
}
Err(e) => {
let _ = reply.send(Err(ProtocolError::Backend(
format!("download_file init: {e}")
)));
}
}
}
```
### 5.6 Public API on `ProtocolClient`
```rust
impl ProtocolClient {
/// Download a file from the server's file repository.
/// `channel_id` 0 means server-level (avatars, icons).
pub async fn download_file(
&self,
channel_id: u64,
path: String,
channel_password: Option<String>,
) -> Result<DownloadedFile, ProtocolError> {
let (tx, rx) = oneshot::channel();
self.tx
.send(Request::DownloadFile { channel_id, path, channel_password, reply: tx })
.await
.map_err(|_| ProtocolError::Lost("connection task is gone".to_string()))?;
rx.await
.map_err(|_| ProtocolError::Lost("download_file reply dropped".to_string()))?
}
/// Download a client's avatar image. Returns raw image bytes.
/// Pass the `avatar_path` from `ClientProfile`.
pub async fn download_avatar(
&self,
avatar_path: String,
) -> Result<DownloadedFile, ProtocolError> {
self.download_file(0, avatar_path, None).await
}
/// Download a server, channel, or client icon by its icon ID.
pub async fn download_icon(
&self,
icon_id: i64,
) -> Result<DownloadedFile, ProtocolError> {
let unsigned_id = icon_id as u64;
let path = format!("/icon_{}", unsigned_id);
self.download_file(0, path, None).await
}
}
```
### 5.7 Exports in `lib.rs`
```rust
pub use dto::DownloadedFile;
```
### 5.8 Bridge Layer
In `crates/chanora_bridge/src/api.rs`, add:
```rust
pub async fn download_avatar(&self, avatar_path: String) -> Result<Vec<u8>, BridgeError> {
self.protocol
.download_avatar(avatar_path)
.await
.map(|file| file.data)
.map_err(BridgeError::Protocol)
}
```
### 5.9 Flutter Integration
Flutter side:
1. Call `clientProfile()` to get `ClientProfile` (already exists).
2. Check if `avatarPath` is non-empty.
3. Call bridge `downloadAvatar(avatarPath)` to get `Uint8List`.
4. Display with `Image.memory(bytes)`.
Caching strategy:
- In-memory: Use Flutter's standard `ImageCache` or a simple `Map<String, Uint8List>` keyed by avatar path.
- Disk: Consider caching to local storage for offline display. This is a follow-up decision, not MVP scope.
- The avatar path already encodes the UID, so it can serve as a cache key.
## 6. Avatar Path Computation
Chanora already has this implemented in `adapter.rs`:
```rust
fn uid_to_avatar_path(uid_b64: &str) -> String {
let decoded = BASE64_STANDARD.decode(uid_b64).unwrap_or_default();
let mut rendered = String::with_capacity(decoded.len() * 2);
for byte in decoded {
rendered.push((b'a' + (byte >> 4)) as char);
rendered.push((b'a' + (byte & 0x0f)) as char);
}
rendered
}
```
This maps each nibble to `a` through `p` (0→a, 1→b, ..., 15→p), matching the canonical TeamSpeak implementation.
The full avatar path is constructed as:
```rust
let avatar_path = if client.avatar_hash.is_empty() || unique_id.is_empty() {
String::new()
} else {
format!("/avatar_{}", uid_to_avatar_path(&unique_id))
};
```
This is already correct and used in `ClientProfile.avatar_path`. No changes needed.
## 7. Threading and Concurrency
| Concern | Design |
|---|---|
| TCP read blocking the event loop | For avatar/icon sizes (< 100 KB typically), inline async read is acceptable. Spawn a reader task for larger files when channel file browsing is added. |
| Multiple concurrent downloads | `pending_downloads` is a HashMap keyed by `FiletransferHandle`. Multiple downloads can be in flight simultaneously. tsclientlib assigns unique handles. |
| Download timeout | Add a deadline to pending downloads (e.g., 30 seconds). Sweep expired entries similar to the existing `pending_moves` sweep. |
| Cancellation on disconnect | When the connection task exits, all pending oneshot senders are dropped, which resolves the caller's await with a `RecvError`. The caller maps this to `ProtocolError::Lost`. |
## 8. Diagnostic and Security Considerations
### 8.1 Diagnostic Redaction
- File transfer paths may contain user-identifying information (UID-derived avatar names). These should be registered for diagnostic redaction if they appear in log output.
- File contents (avatar images) must not appear in log output or diagnostic exports.
### 8.2 Security
- The `ftkey` is a one-time token and must not be logged.
- TCP file transfer connections are not encrypted. This is a TeamSpeak protocol limitation, not a Chanora design choice. Avatar data is public (visible to anyone on the server), so the risk is acceptable.
- File download does not require secrets beyond the existing authenticated connection.
### 8.3 Privacy
- Avatar downloads reveal to the server that the user is viewing a specific client's avatar. This is inherent in the protocol.
- Chanora should not download avatars proactively for all clients. Download only when the UI needs to display a specific avatar (lazy/on-demand).
## 9. Out of Scope
The following are explicitly deferred:
- File upload (avatar upload, channel file upload).
- Channel file browser (listing, creating directories, deleting, renaming).
- Resumable downloads (seek position > 0).
- File transfer progress reporting.
- myTeamSpeak avatar resolution (the `client_myteamspeak_avatar` field).
- In-memory hot cache in Rust (Flutter's `ImageCache` handles decoded image caching; add Rust-side layer only if profiling shows need).
- Upload, file browser, and channel file management.
## 10. Cache Architecture
### 10.1 Layer Ownership
| Layer | Responsibility | Storage |
|---|---|---|
| `chanora_protocol` | Download raw bytes from server. No caching logic. | None |
| `chanora_cache` | Content-addressed blob store backed by `cacache`: crash-safe writes, SSRI integrity verification, key validation, eviction, clear. Separate crate from `chanora_storage`. | Platform cache directory |
| `chanora_core` | Session-aware cache orchestration: check freshness, coalesce requests, rate-limit downloads, persist to disk via `chanora_cache`. | Delegates to `chanora_cache` |
| `chanora_bridge` | Expose typed `download_avatar` / `clear_file_cache` / `file_cache_size` to Flutter. | None |
| Flutter | Display via `Image.memory`. Standard `ImageCache` for hot memory caching. Evict from `ImageCache` when hash changes. | In-memory only |
### 10.2 Why Separate `chanora_cache` Crate
`chanora_cache` is a separate crate from `chanora_storage` for three reasons:
1. **Different durability semantics.** `chanora_storage` holds identity, bookmarks, and connection profiles — data the user explicitly created. `chanora_cache` holds downloaded blobs that are fully reconstructible from the server. Losing the cache is an inconvenience, not data loss.
2. **Different backup semantics.** Cache should be excluded from backups; persistent storage should be included. Platform conventions (iOS `Library/Caches/` vs `Library/Application Support/`) reflect this distinction.
3. **Different directory placement.** Cache lives in the platform's cache directory (OS may evict under storage pressure on mobile). Persistent storage lives in the support directory.
The cache wraps the `cacache` crate for production-tested crash safety and integrity verification. It does not share `chanora_storage`'s crate or directory, and does not reimplement cacache's atomic write or content-addressing logic.
### 10.3 Why Hybrid (Rust Disk + Flutter Memory)
- Flutter's built-in `ImageCache` is an LRU in-memory cache (default 1000 images / 100 MiB). It handles hot display caching automatically when you use `MemoryImage`.
- Flutter has no built-in disk cache. `cached_network_image` / `flutter_cache_manager` are designed for HTTP URLs, not custom binary protocol data.
- Rust already owns the protocol, the connection state, and the anti-flood budget. Putting disk cache here avoids a feedback loop across the bridge.
### 10.4 Cache Storage
`chanora_cache` wraps the `cacache` crate for its on-disk storage. The physical layout is managed by `cacache`:
```
<app_cache_dir>/chanora/
blobs/ ← cacache content store root
content-v2/ ← content-addressed by SHA-512
<sha512-hex>/
data ← raw blob bytes
tmp/ ← temp files (in-flight writes)
index-v2/ ← entry index (key → content mapping)
```
Chanora's `BlobCache` maps protocol keys to `cacache` string keys:
| Protocol key | cacache key | Example |
|---|---|---|
| Avatar MD5 | `"av_<md5hex>"` | `"av_a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6"` |
| Icon CRC32 | `"ic_<crc32u>"` | `"ic_123456789"` |
**Why `cacache`:**
1. **Crash safety.** Production-tested atomic writes (temp → rename). Handles partial writes, power loss, crash mid-write. No custom crash safety code to maintain.
2. **Integrity verification.** SSRI integrity check on every `read()`. Detects corruption, bit rot, partial writes automatically. Better than custom "delete on read failure".
3. **Content dedup.** Same bytes stored once regardless of key. Same avatar on two servers = stored once automatically.
4. **Less code to maintain.** ~120 LOC wrapper vs ~200 LOC custom implementation. Crash safety and integrity are the hard parts — `cacache` owns them.
**Why no `<server_uid>` subdirectory:** The protocol uses content-addressed identifiers. `client_flag_avatar` is the MD5 of the avatar bytes — a given avatar hash always maps to the same bytes regardless of which server the user is on. Same avatar on two servers = same content = stored once by `cacache`. This is a deliberate dedup advantage over per-server namespacing.
**Why no metadata sidecars:** Content is immutable (a given hash always maps to the same bytes). `cacache` manages its own entry index with timestamps. No custom metadata files needed.
**Key validation rules:**
| Prefix | Key format | Validation |
|---|---|---|
| `av_` | `av_<32 hex chars>` | MD5 is exactly 32 hex characters |
| `ic_` | `ic_<1-10 digit number>` | CRC32 unsigned, 04294967295 |
Keys failing validation are rejected at the `BlobCache` API boundary. This prevents path traversal or malformed filenames on disk.
**Platform paths** (Flutter passes the base directory into Rust at startup, matching the existing `initStorage` pattern):
| Platform | Cache directory |
|---|---|
| Android | `context.cacheDir/chanora/` (via `getCacheDir()`) |
| iOS | `Library/Caches/chanora/` (via `getApplicationCacheDirectory()`) |
| macOS | `~/Library/Caches/chanora/` |
| Windows | `%LOCALAPPDATA%/chanora/cache/` |
| Linux | `$XDG_CACHE_HOME/chanora/` or `~/.cache/chanora/` |
Flutter already resolves platform-specific paths. The same `getApplicationCacheDirectory()` call that is available in `path_provider` across all Chanora target platforms should be used. This follows the existing pattern where `app_bootstrap.dart` calls `getApplicationSupportDirectory()` for persistent storage; avatar/icon cache uses the cache-equivalent directory instead.
The cache init call is separate from storage init:
```rust
// Bridge init (Flutter calls these at startup)
pub fn init_storage(support_dir: String) -> Result<(), BridgeError>; // existing
pub fn init_cache(cache_dir: String) -> Result<(), BridgeError>; // new
```
### 10.5 Cache Freshness Strategy
The `client_flag_avatar` field on each client is the authoritative freshness signal:
```
On connect / on client list update:
For each visible client with non-empty avatar_hash:
cache_key = "av_<avatar_hash>"
if cacache entry exists for "av_<avatar_hash>":
use cached file (zero downloads)
else:
enqueue download for avatar_path with expected hash = avatar_hash
On avatar_hash change for a client:
The new hash produces a different cacache key.
The old entry remains until eviction or manual clear.
The new file is downloaded on demand.
```
This means:
- **First connect:** No cache hits. Downloads happen lazily as the UI requests avatars.
- **Reconnect to same server:** All avatars hit cache instantly (hashes match keys). Zero downloads.
- **User changes avatar:** New hash = new cache key. Old entry becomes orphan. New file downloads on next UI request.
- **Same user on different server:** Same avatar hash = same cached file. Cross-server dedup for free.
### 10.6 Anti-Flood and Download Timing
TeamSpeak servers enforce anti-flood rate limiting. Downloading all avatars eagerly on connect would trigger it on servers with many users.
**Strategy: lazy + throttled prefetch**
| Phase | What | Rate |
|---|---|---|
| Connect settle (first 2-5 s) | Do nothing. Let the initial state snapshot and channel tree arrive. | — |
| After settle | UI requests avatars for visible clients in the current channel. These trigger downloads one at a time. | Max 1-2 concurrent downloads per server |
| Channel switch | UI requests avatars for newly visible clients. | Same throttle |
| Background prefetch (optional, future) | Low-priority downloads for clients in adjacent channels. | 1 request per 500 ms |
**Anti-flood handling:**
- If the server responds with an anti-flood error (TS3 error code `0x0701` = `client_could_not_be_banned` / flood-related), back off the download queue.
- Implement a simple semaphore in `chanora_core`: max 1-2 concurrent downloads.
- If a download gets a flood error, pause the queue for 5 seconds, then resume at reduced rate.
### 10.7 Retry on Failure
| Failure type | Strategy |
|---|---|
| Transient (network timeout, TCP reset) | Retry with exponential backoff: 5 s, 30 s, 2 min, 10 min. Cap at 10 min. |
| Server flood limit hit | Pause queue 5 s, then resume at reduced rate. Do not count as a per-file retry. |
| Permission denied (no download power) | Do not retry. Record negative cache entry. Only retry if hash changes. |
| File not found (avatar removed) | Do not retry. Record negative cache entry. Clear when hash changes or becomes empty. |
| Connection lost | All pending downloads fail. On reconnect, cache check runs fresh with current hashes. |
**Negative cache:** In-memory `HashMap<String, Instant>` with 5-minute TTL. Keys like `"av_<hash>"` or `"ic_<id>"` that received permanent errors are stored with an expiry. On lookup, expired entries are treated as absent. Cleared entirely on reconnect.
### 10.8 Request Coalescing
Multiple UI widgets may request the same avatar simultaneously (e.g., channel list + chat view + client info sheet).
**Pattern:** In `chanora_core`'s `FileTransferService`, maintain an in-flight map:
```rust
HashMap<String, tokio::task::JoinHandle<Result<Vec<u8>, FileTransferError>>>
```
- First request: start download, store handle.
- Subsequent requests for same key: await the same handle.
- When handle completes: write to cache, wake all waiters, remove from map.
### 10.9 Cache Eviction and Size Limits
**MVP approach:**
- No automatic size-based eviction in MVP. Avatars are small (typically 10-100 KB). Even 1000 avatars = ~50-100 MB.
- Rely on platform cache directory semantics (OS may evict under storage pressure on mobile).
- Old hash files accumulate but are harmless.
**Post-MVP:**
- `BlobCache::evict(max_bytes)` — walk `cacache::ls()` entries, sort by timestamp (oldest first), delete until total size < `max_bytes`. `cacache` manages timestamps internally. No metadata sidecars needed.
- Or simpler: `BlobCache::evict_older_than(duration)` — delete entries with timestamp older than N days.
- Call on startup and periodically (e.g., every 24 hours or on app resume).
### 10.10 User-Initiated Cache Clear
Add a bridge method:
```rust
pub fn clear_file_cache(&self) -> Result<(), BridgeError> {
// Delete the entire blobs/ directory contents
// Flutter evicts all avatar/icon-related entries from ImageCache
}
pub fn file_cache_size(&self) -> Result<u64, BridgeError> {
// Walk blobs/ and sum file sizes
}
```
Flutter side:
```dart
// In settings or storage management UI:
onPressed: () async {
await api.clearFileCache();
PaintingBinding.instance.imageCache.clear();
}
```
This should be exposed in the app's settings UI under a "Clear cache" or "Storage management" section.
### 10.11 Storage Clear Across Servers
Since the cache is flat with content-addressed keys (no server namespacing):
- Connecting to a different server does not conflict — same avatar hash = same file.
- Avatars unique to the old server remain cached. If a user on the new server has the same avatar (same hash), it hits cache instantly (cross-server dedup).
- Cache clear removes all cached data regardless of which server it came from.
### 10.12 Flutter Display Strategy
**Option A: Bytes across bridge (simpler, recommended for MVP)**
Rust returns `Vec<u8>` across the bridge. Flutter uses `Image.memory(bytes)`.
```dart
final bytes = await api.downloadAvatar(clientUid: uid);
if (bytes != null && bytes.isNotEmpty) {
return Image.memory(Uint8List.fromList(bytes));
} else {
return CircleAvatar(child: Text(initials)); // fallback
}
```
Flutter's `ImageCache` caches the decoded image in memory automatically. Same avatar bytes = cache hit in memory.
**Option B: File path across bridge (better for large images, future)**
Rust writes to disk and returns the file path. Flutter uses `FileImage`.
```dart
final path = await api.getAvatarPath(avatarHash: hash);
if (path != null) {
return Image.file(File(path));
} else {
return CircleAvatar(child: Text(initials));
}
```
`FileImage` does not watch for file changes. When the hash changes, the UI must evict the old entry from `ImageCache` using `PaintingBinding.instance.imageCache.evict(key)`.
**Recommendation:** Start with Option A for MVP. It avoids file-path cross-platform complications and works well for small avatar files. The bridge already returns `Vec<u8>` for the download result.
## 11. Implementation Sequence
| Phase | Scope | What |
|---|---|---|
| Phase 1 | Protocol download | `Request::DownloadFile`, `StreamItem::FileDownload` handling, `ProtocolClient::download_avatar()` / `download_icon()`. No caching. |
| Phase 2 | Bridge + Flutter display | Bridge `downloadAvatar()`, Flutter `Image.memory()`, initials fallback. Still no caching — every view re-downloads. |
| Phase 3 | Rust disk cache | New `chanora_cache` crate: `cacache`-backed content-addressed blob store (`BlobCache`), key validation, mtime-based eviction, `init_cache` bridge call. |
| Phase 4 | Session orchestration | `chanora_core` `FileTransferService`: request coalescing, rate limiter (semaphore), negative cache (5 min TTL), retry backoff. |
| Phase 5 | Cache management | Bridge `clearFileCache()` + `fileCacheSize()`, Flutter settings UI, eviction on startup. |
Phase 1 and 2 deliver visible value (avatars in the UI). Phase 3-5 add robustness.
## 12. References
| Reference | Use |
|---|---|
| `ReSpeak/tsdeclarations` `Messages.toml` lines 828-830 | `ftinitdownload` command declaration |
| `ReSpeak/tsdeclarations` `Messages.toml` lines 590 | `FileDownload` response structure |
| `ReSpeak/tsdeclarations` `ts3protocol.md` | Low-level TeamSpeak protocol specification |
| `ReSpeak/tsclientlib` `src/lib.rs` lines 956-1005 | `download_file` / `upload_file` public API |
| `ReSpeak/tsclientlib` `src/lib.rs` lines 1371-1427 | `StreamItem::FileDownload` handling |
| `ReSpeak/tsclientlib` `src/lib.rs` lines 1630-1672 | Outgoing init commands |
| `Multivit4min/TS3-NodeJS-Library` `src/transport/FileTransfer.ts` | Reference TCP transfer implementation |
| `Speckmops/ts3admin.class` `lib/ts3admin.class.php` lines 1352-1370 | Reference avatar download flow |
| `docs/architecture/sad.md` SAD-067, SDD-MOD-009 | Protocol adapter boundary rules |
| `crates/chanora_protocol/src/adapter.rs` lines 1561-1568 | Existing `uid_to_avatar_path` implementation |