* docs(architecture): add file transfer design, research, and implementation plan * feat(cache): add chanora_cache crate with cacache-backed blob cache - New chanora_cache crate: content-addressed blob store wrapping cacache - BlobCache API: async put/get/remove/clear/total_size/evict - Key validation: av_ prefix (32 hex chars), ic_ prefix (decimal digits) - Cacache provides crash safety, SSRI integrity, content dedup - Mtime-based eviction via cacache::list_sync + sort by timestamp - 7 unit tests all passing - Added to workspace members * feat(protocol): add file download support for avatars and icons - Add Request::DownloadFile variant with oneshot reply - Add ProtocolClient::download_avatar(client_uid) and download_icon(icon_id) - Track pending file downloads by FiletransferHandle - Handle StreamItem::FileDownload: read bytes from TCP stream - Handle StreamItem::FiletransferFailed: map to ProtocolError - Add ProtocolError::FileTransfer(String) variant - Add path helper tests for avatar/icon download paths - No tsclientlib types leak across the adapter boundary * feat(core): add blob cache wiring and avatar download orchestration - Add chanora_cache dependency to Cargo.toml - Add blob_cache field to ChanoraSession (Arc<Mutex<Option<BlobCache>>>) - Add init_cache() method: creates BlobCache, runs eviction - Add get_avatar() method: cache-first, download on miss, store in cache - Add clear_cache() and cache_size() methods for cache management - Add CoreError::Cache variant for BlobCacheError conversion - Add avatar_cache integration test * feat(bridge): add init_cache, download_avatar, and cache management functions - Add init_cache(dir) bridge function - Add download_avatar(avatar_hash, client_uid) bridge function - Add clear_file_cache() and file_cache_size() bridge functions - Map CoreError::Cache and ProtocolError::FileTransfer in BridgeError * feat(flutter): add cache initialization wiring and avatar download shims - Add wireCache() to app_bootstrap using getApplicationCacheDirectory() - Call wireCache() after wireStorage() in main bootstrap flow - Add Dart-side initCache and downloadAvatar wrapper shims in api.dart - Update Cargo.lock for new chanora_cache dependency * feat(core): FileTransferService with coalescing, throttling, negative cache - New file_transfer module with FileTransferService struct - Semaphore(2) throttles concurrent downloads - In-flight HashMap coalesces duplicate avatar requests - 5-min negative cache short-circuits ServerRejected misses - ChanoraSession delegates get_avatar through the service - connect/disconnect update shared protocol handle - clear_cache/cache_size delegate to service - 2 new unit tests (cached hit, negative cache) * feat(core,bridge): add get_icon with coalescing and negative cache - FileTransferService::get_icon() mirrors get_avatar pattern - ChanoraSession::get_icon() delegates through FileTransferService - Bridge download_icon() exposed for Flutter - Dart downloadIcon() shim added - Uses PREFIX_ICON (ic_<crc32u>) cache key format - 1 new unit test (cached icon hit) * fix(core,protocol): simplify store_protocol and add download size cap - store_protocol: always write to shared Arc<Mutex<Option<ProtocolClient>>>; the FileTransferService holds the same Arc so it sees updates automatically - read_download_bytes: reject downloads exceeding 10 MB to prevent malicious servers from causing OOM
738 lines
31 KiB
Markdown
738 lines
31 KiB
Markdown
# File Transfer Design
|
||
|
||
**Date:** 2026-06-10
|
||
**Status:** Draft for review
|
||
**Scope:** Download files from TeamSpeak-compatible servers via the native client protocol, starting with avatars and icons.
|
||
**Direct upstream source:** `docs/architecture/sad.md` (SAD-067, SDD-MOD-009)
|
||
|
||
## 1. Goal
|
||
|
||
Chanora needs to download files stored on TeamSpeak-compatible servers. The most visible use cases are client avatars and server/channel/client icons. The file transfer mechanism is also used for channel file browser features, but this document scopes the initial design to avatar and icon retrieval only.
|
||
|
||
This document describes:
|
||
|
||
- How the TeamSpeak file transfer protocol works.
|
||
- How `tsclientlib` exposes it.
|
||
- How Chanora should integrate it following the existing protocol adapter pattern.
|
||
- How the result flows through the bridge to the Flutter UI layer.
|
||
|
||
Upload, channel file browsing, and file deletion are explicitly out of scope for the initial implementation.
|
||
|
||
## 2. Protocol Background
|
||
|
||
### 2.1 Two-Phase Transfer
|
||
|
||
TeamSpeak file transfer is a two-phase process:
|
||
|
||
1. **Command phase** — The client sends a command over the main encrypted UDP connection to request a transfer token (`ftkey`).
|
||
2. **Transfer phase** — The client opens a separate TCP connection to the server's file transfer port (default `30033`) and sends the `ftkey` to authenticate the transfer. Raw bytes flow over this TCP stream.
|
||
|
||
### 2.2 Relevant ServerQuery Commands
|
||
|
||
| Command | Direction | Purpose |
|
||
|---|---|---|
|
||
| `ftinitdownload` | Client → Server | Initialize a download. Returns `ftkey`, `port`, `size`. |
|
||
| `ftgetfileinfo` | Client → Server | Get metadata for one or more files. |
|
||
| `ftgetfilelist` | Client → Server | List files in a channel's file repository. |
|
||
| `ftinitupload` | Client → Server | Initialize an upload. |
|
||
| `ftlist` | Client → Server | List active file transfers. |
|
||
| `ftstop` | Client → Server | Stop a running transfer. |
|
||
| `ftdeletefile` | Client → Server | Delete a file. |
|
||
| `ftcreatedir` | Client → Server | Create a directory. |
|
||
| `ftrenamefile` | Client → Server | Rename or move a file. |
|
||
|
||
Initial scope uses only `ftinitdownload` and `ftgetfileinfo`.
|
||
|
||
### 2.3 File Paths
|
||
|
||
Files are addressed by a path scoped to a channel ID (`cid`):
|
||
|
||
- `cid=0` — Server-level file repository. Avatars and icons live here.
|
||
- `cid=N` (non-zero) — Channel-specific file repository.
|
||
|
||
Avatar path: `/avatar_<hex>` where `<hex>` is derived from the client's unique identifier (UID). Each byte of the base64-decoded UID is split into two nibbles, and each nibble maps to a letter `a` through `p` (0→a, 1→b, ..., 15→p).
|
||
|
||
Icon path: `/icon_<id>` where `<id>` is the icon's signed 64-bit integer ID. If negative, treat as unsigned for the path.
|
||
|
||
### 2.4 `ftinitdownload` Command
|
||
|
||
```
|
||
ftinitdownload clientftfid={id} name={path} cid={channelId} cpw={password} seekpos={seek} proto=0
|
||
```
|
||
|
||
Parameters:
|
||
|
||
| Parameter | Type | Description |
|
||
|---|---|---|
|
||
| `clientftfid` | `u16` | Arbitrary client-side transfer ID. |
|
||
| `name` | `string` | File path, e.g. `/avatar_abcdef`. |
|
||
| `cid` | `ChannelId` | Channel scope (0 = server). |
|
||
| `cpw` | `string` | Channel password. Empty for server-level. |
|
||
| `seekpos` | `u64` | Resume offset. 0 for a fresh download. |
|
||
| `proto` | `u8` | Protocol version. Always 0. |
|
||
|
||
Server response:
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `clientftfid` | `u16` | Echo of the client transfer ID. |
|
||
| `serverftfid` | `u16` | Server-side transfer ID. |
|
||
| `ftkey` | `string` | One-time transfer key (hex). |
|
||
| `port` | `u16` | File transfer TCP port (usually 30033). |
|
||
| `size` | `u64` | File size in bytes. |
|
||
| `proto` | `u8` | Protocol version echo. |
|
||
| `ip` | `string` (optional) | Override IP for the TCP connection. |
|
||
|
||
### 2.5 TCP Transfer
|
||
|
||
After receiving the `ftkey`, the client:
|
||
|
||
1. Opens a TCP connection to `server_ip:port`.
|
||
2. Sends `ftkey` followed by a newline.
|
||
3. Reads exactly `size` bytes of raw file data.
|
||
4. Closes the TCP connection.
|
||
|
||
### 2.6 Permissions
|
||
|
||
File transfer requires the following permissions on the server:
|
||
|
||
| Permission | Needed for |
|
||
|---|---|
|
||
| `i_ft_file_download_power` | Downloading files. |
|
||
| `i_ft_needed_file_download_power` | Required download power on the channel/server. |
|
||
| `b_ft_ignore_password` | Bypassing channel passwords (not needed for avatars). |
|
||
|
||
Avatar downloads typically require only basic download power because avatars are in the server-level repository (`cid=0`), which is generally accessible.
|
||
|
||
### 2.7 Avatar Detection
|
||
|
||
When a client connects or updates, the server sends `client_flag_avatar` as a string (the avatar hash). If non-empty, the client has an avatar. The avatar is downloaded from `/avatar_<hex>` where `<hex>` is computed from the client's UID (not from the hash string itself — the hash is just a presence indicator).
|
||
|
||
## 3. tsclientlib Support
|
||
|
||
`tsclientlib` implements file transfer natively. The library handles the entire command + TCP flow internally:
|
||
|
||
### 3.1 Public API
|
||
|
||
```rust
|
||
// tsclientlib/src/lib.rs (relevant signatures)
|
||
impl Connection {
|
||
pub fn download_file(
|
||
&mut self,
|
||
channel_id: ChannelId,
|
||
path: &str,
|
||
channel_password: Option<&str>,
|
||
seek_position: Option<u64>,
|
||
) -> Result<FiletransferHandle>;
|
||
|
||
pub fn upload_file(
|
||
&mut self,
|
||
channel_id: ChannelId,
|
||
path: &str,
|
||
channel_password: Option<&str>,
|
||
size: u64,
|
||
overwrite: bool,
|
||
resume: bool,
|
||
) -> Result<FiletransferHandle>;
|
||
}
|
||
```
|
||
|
||
`download_file` sends the `ftinitdownload` command and returns a `FiletransferHandle(u16)` immediately. The actual transfer completes asynchronously.
|
||
|
||
### 3.2 Stream Items
|
||
|
||
The connection's event stream emits:
|
||
|
||
| StreamItem | When | Data |
|
||
|---|---|---|
|
||
| `StreamItem::FileDownload(FileDownloadResult)` | Server responds with `ftkey`; TCP connected and `ftkey` written | `{ size: u64, stream: TcpStream }` |
|
||
| `StreamItem::FileUpload(FileUploadResult)` | Upload ready | `{ seek_position: u64, stream: TcpStream }` |
|
||
| `StreamItem::FiletransferFailed(FiletransferHandle, Error)` | Transfer failed | Handle + error |
|
||
|
||
When `FileDownload` fires, tsclientlib has already:
|
||
|
||
1. Sent `ftinitdownload` over the encrypted UDP command channel.
|
||
2. Received the `ftkey`, `port`, and `size` from the server.
|
||
3. Opened a TCP connection to `server:port`.
|
||
4. Written the `ftkey` to the TCP socket.
|
||
|
||
The `TcpStream` in `FileDownloadResult` is ready to read; Chanora only needs to read exactly `size` bytes.
|
||
|
||
### 3.3 Avatar Helper
|
||
|
||
`tsproto-types` provides `Uid::as_avatar()` which computes the avatar filename from a UID. Chanora's existing `uid_to_avatar_path()` in `adapter.rs` does the same thing independently.
|
||
|
||
### 3.4 Doc-Comment Examples
|
||
|
||
tsclientlib's source contains usage examples in doc comments:
|
||
|
||
```rust
|
||
/// Download an icon:
|
||
/// con.download_file(ChannelId(0), &format!("/icon_{}", icon_id), None, None)
|
||
|
||
/// Upload an avatar:
|
||
/// con.upload_file(ChannelId(0), "/avatar", None, data.len() as u64, true, false)
|
||
```
|
||
|
||
## 4. Architecture Integration
|
||
|
||
### 4.1 Existing Pattern
|
||
|
||
The protocol adapter (`crates/chanora_protocol`) uses a single tokio task that owns the `tsclientlib::Connection`. All operations follow this pattern:
|
||
|
||
1. Define a `Request` enum variant with parameters and a `oneshot::Sender` for the reply.
|
||
2. Send the request through the `mpsc` channel to the connection task.
|
||
3. The connection task calls tsclientlib and resolves the oneshot.
|
||
|
||
File transfer fits this pattern exactly. The only difference is that the result arrives asynchronously via `StreamItem::FileDownload` rather than immediately from the command call.
|
||
|
||
### 4.2 Design
|
||
|
||
The file transfer integration adds:
|
||
|
||
1. **`Request` variants** for file download.
|
||
2. **A pending-downloads map** (`HashMap<FiletransferHandle, DownloadContext>`) in the connection task, mirroring the existing `pending_moves` pattern.
|
||
3. **`StreamItem::FileDownload` and `StreamItem::FiletransferFailed`** handling in the event loop.
|
||
4. **New DTOs** for file transfer results.
|
||
5. **Convenience methods** on `ProtocolClient` for avatar and icon downloads.
|
||
|
||
### 4.3 Layer Responsibilities
|
||
|
||
| Layer | Responsibility |
|
||
|---|---|
|
||
| `chanora_protocol` | Call `tsclientlib::download_file`, track pending transfers, read `TcpStream`, return bytes. No tsclientlib types leak. |
|
||
| `chanora_core` | Orchestrate when to download (e.g., on profile fetch or on avatar cache miss). |
|
||
| `chanora_bridge` | Expose typed `download_avatar` / `download_icon` commands to Flutter. |
|
||
| Flutter UI | Call bridge, display with `Image.memory()`. Cache in memory/image cache. |
|
||
|
||
### 4.4 Error Mapping
|
||
|
||
File transfer errors map to the existing `ProtocolError` variants:
|
||
|
||
| tsclientlib error | ProtocolError |
|
||
|---|---|
|
||
| Permission denied (TS3 error code) | `ServerRejected { code, message }` |
|
||
| File not found | `ServerRejected { code, message }` |
|
||
| Network/TCP failure | `Backend(String)` |
|
||
| Timeout | `Timeout` |
|
||
| Connection lost mid-transfer | `Lost(String)` |
|
||
|
||
## 5. Detailed Design
|
||
|
||
### 5.1 New Types in `dto.rs`
|
||
|
||
```rust
|
||
/// A downloaded file's raw content and metadata.
|
||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||
pub struct DownloadedFile {
|
||
/// Raw file bytes.
|
||
pub data: Vec<u8>,
|
||
/// The server path that was requested.
|
||
pub path: String,
|
||
/// Channel ID the file was downloaded from.
|
||
pub channel_id: u64,
|
||
}
|
||
```
|
||
|
||
### 5.2 New Request Variants in `adapter.rs`
|
||
|
||
```rust
|
||
enum Request {
|
||
// ... existing variants ...
|
||
|
||
/// Download a file from the server's file repository.
|
||
DownloadFile {
|
||
/// Channel ID. 0 for server-level (avatars, icons).
|
||
channel_id: u64,
|
||
/// File path, e.g. "/avatar_abcdef" or "/icon_12345".
|
||
path: String,
|
||
/// Channel password. None for server-level files.
|
||
channel_password: Option<String>,
|
||
/// Reply channel for the result.
|
||
reply: oneshot::Sender<Result<DownloadedFile, ProtocolError>>,
|
||
},
|
||
}
|
||
```
|
||
|
||
### 5.3 Pending Downloads Map
|
||
|
||
```rust
|
||
type PendingDownloads = HashMap<tsclientlib::FiletransferHandle, PendingDownload>;
|
||
|
||
struct PendingDownload {
|
||
path: String,
|
||
channel_id: u64,
|
||
reply: oneshot::Sender<Result<DownloadedFile, ProtocolError>>,
|
||
}
|
||
```
|
||
|
||
### 5.4 Event Loop Handling
|
||
|
||
In the connection task's main loop, add handling for file transfer stream items:
|
||
|
||
```rust
|
||
// In handle_non_audio_stream_item or in the main loop:
|
||
StreamItem::FileDownload(result) => {
|
||
// result: FileDownloadResult { size, stream }
|
||
// Look up the handle in pending_downloads
|
||
// Use tokio::io::AsyncReadExt::read_exact to read 'size' bytes
|
||
// Resolve the oneshot with DownloadedFile
|
||
}
|
||
StreamItem::FiletransferFailed(handle, error) => {
|
||
// Look up the handle in pending_downloads
|
||
// Resolve the oneshot with ProtocolError::Backend
|
||
}
|
||
```
|
||
|
||
The TCP read from the `TcpStream` is an async operation. Since the connection task already runs in a tokio context, the read can be done inline. However, for large files this would block the main event loop. Two approaches:
|
||
|
||
**Option A: Read inline (simple, good for small files like avatars)**
|
||
|
||
Avatars are typically under 100 KB. Reading them inline in the event loop is acceptable and avoids complexity.
|
||
|
||
**Option B: Spawn a reader task**
|
||
|
||
For future channel-file-browser support with potentially large files, spawn a separate tokio task that reads the stream and sends the result back.
|
||
|
||
**Recommendation:** Start with Option A. The initial scope is avatars and icons (small files). Refactor to Option B when channel file browsing is implemented.
|
||
|
||
### 5.5 Request Handling
|
||
|
||
When the connection task receives `Request::DownloadFile`:
|
||
|
||
```rust
|
||
Ok(Request::DownloadFile { channel_id, path, channel_password, reply }) => {
|
||
let ts_channel_id = TsChannelId(channel_id);
|
||
match con.download_file(ts_channel_id, &path, channel_password.as_deref(), None) {
|
||
Ok(handle) => {
|
||
pending_downloads.insert(handle, PendingDownload {
|
||
path,
|
||
channel_id,
|
||
reply,
|
||
});
|
||
}
|
||
Err(e) => {
|
||
let _ = reply.send(Err(ProtocolError::Backend(
|
||
format!("download_file init: {e}")
|
||
)));
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.6 Public API on `ProtocolClient`
|
||
|
||
```rust
|
||
impl ProtocolClient {
|
||
/// Download a file from the server's file repository.
|
||
/// `channel_id` 0 means server-level (avatars, icons).
|
||
pub async fn download_file(
|
||
&self,
|
||
channel_id: u64,
|
||
path: String,
|
||
channel_password: Option<String>,
|
||
) -> Result<DownloadedFile, ProtocolError> {
|
||
let (tx, rx) = oneshot::channel();
|
||
self.tx
|
||
.send(Request::DownloadFile { channel_id, path, channel_password, reply: tx })
|
||
.await
|
||
.map_err(|_| ProtocolError::Lost("connection task is gone".to_string()))?;
|
||
rx.await
|
||
.map_err(|_| ProtocolError::Lost("download_file reply dropped".to_string()))?
|
||
}
|
||
|
||
/// Download a client's avatar image. Returns raw image bytes.
|
||
/// Pass the `avatar_path` from `ClientProfile`.
|
||
pub async fn download_avatar(
|
||
&self,
|
||
avatar_path: String,
|
||
) -> Result<DownloadedFile, ProtocolError> {
|
||
self.download_file(0, avatar_path, None).await
|
||
}
|
||
|
||
/// Download a server, channel, or client icon by its icon ID.
|
||
pub async fn download_icon(
|
||
&self,
|
||
icon_id: i64,
|
||
) -> Result<DownloadedFile, ProtocolError> {
|
||
let unsigned_id = icon_id as u64;
|
||
let path = format!("/icon_{}", unsigned_id);
|
||
self.download_file(0, path, None).await
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.7 Exports in `lib.rs`
|
||
|
||
```rust
|
||
pub use dto::DownloadedFile;
|
||
```
|
||
|
||
### 5.8 Bridge Layer
|
||
|
||
In `crates/chanora_bridge/src/api.rs`, add:
|
||
|
||
```rust
|
||
pub async fn download_avatar(&self, avatar_path: String) -> Result<Vec<u8>, BridgeError> {
|
||
self.protocol
|
||
.download_avatar(avatar_path)
|
||
.await
|
||
.map(|file| file.data)
|
||
.map_err(BridgeError::Protocol)
|
||
}
|
||
```
|
||
|
||
### 5.9 Flutter Integration
|
||
|
||
Flutter side:
|
||
|
||
1. Call `clientProfile()` to get `ClientProfile` (already exists).
|
||
2. Check if `avatarPath` is non-empty.
|
||
3. Call bridge `downloadAvatar(avatarPath)` to get `Uint8List`.
|
||
4. Display with `Image.memory(bytes)`.
|
||
|
||
Caching strategy:
|
||
|
||
- In-memory: Use Flutter's standard `ImageCache` or a simple `Map<String, Uint8List>` keyed by avatar path.
|
||
- Disk: Consider caching to local storage for offline display. This is a follow-up decision, not MVP scope.
|
||
- The avatar path already encodes the UID, so it can serve as a cache key.
|
||
|
||
## 6. Avatar Path Computation
|
||
|
||
Chanora already has this implemented in `adapter.rs`:
|
||
|
||
```rust
|
||
fn uid_to_avatar_path(uid_b64: &str) -> String {
|
||
let decoded = BASE64_STANDARD.decode(uid_b64).unwrap_or_default();
|
||
let mut rendered = String::with_capacity(decoded.len() * 2);
|
||
for byte in decoded {
|
||
rendered.push((b'a' + (byte >> 4)) as char);
|
||
rendered.push((b'a' + (byte & 0x0f)) as char);
|
||
}
|
||
rendered
|
||
}
|
||
```
|
||
|
||
This maps each nibble to `a` through `p` (0→a, 1→b, ..., 15→p), matching the canonical TeamSpeak implementation.
|
||
|
||
The full avatar path is constructed as:
|
||
|
||
```rust
|
||
let avatar_path = if client.avatar_hash.is_empty() || unique_id.is_empty() {
|
||
String::new()
|
||
} else {
|
||
format!("/avatar_{}", uid_to_avatar_path(&unique_id))
|
||
};
|
||
```
|
||
|
||
This is already correct and used in `ClientProfile.avatar_path`. No changes needed.
|
||
|
||
## 7. Threading and Concurrency
|
||
|
||
| Concern | Design |
|
||
|---|---|
|
||
| TCP read blocking the event loop | For avatar/icon sizes (< 100 KB typically), inline async read is acceptable. Spawn a reader task for larger files when channel file browsing is added. |
|
||
| Multiple concurrent downloads | `pending_downloads` is a HashMap keyed by `FiletransferHandle`. Multiple downloads can be in flight simultaneously. tsclientlib assigns unique handles. |
|
||
| Download timeout | Add a deadline to pending downloads (e.g., 30 seconds). Sweep expired entries similar to the existing `pending_moves` sweep. |
|
||
| Cancellation on disconnect | When the connection task exits, all pending oneshot senders are dropped, which resolves the caller's await with a `RecvError`. The caller maps this to `ProtocolError::Lost`. |
|
||
|
||
## 8. Diagnostic and Security Considerations
|
||
|
||
### 8.1 Diagnostic Redaction
|
||
|
||
- File transfer paths may contain user-identifying information (UID-derived avatar names). These should be registered for diagnostic redaction if they appear in log output.
|
||
- File contents (avatar images) must not appear in log output or diagnostic exports.
|
||
|
||
### 8.2 Security
|
||
|
||
- The `ftkey` is a one-time token and must not be logged.
|
||
- TCP file transfer connections are not encrypted. This is a TeamSpeak protocol limitation, not a Chanora design choice. Avatar data is public (visible to anyone on the server), so the risk is acceptable.
|
||
- File download does not require secrets beyond the existing authenticated connection.
|
||
|
||
### 8.3 Privacy
|
||
|
||
- Avatar downloads reveal to the server that the user is viewing a specific client's avatar. This is inherent in the protocol.
|
||
- Chanora should not download avatars proactively for all clients. Download only when the UI needs to display a specific avatar (lazy/on-demand).
|
||
|
||
## 9. Out of Scope
|
||
|
||
The following are explicitly deferred:
|
||
|
||
- File upload (avatar upload, channel file upload).
|
||
- Channel file browser (listing, creating directories, deleting, renaming).
|
||
- Resumable downloads (seek position > 0).
|
||
- File transfer progress reporting.
|
||
- myTeamSpeak avatar resolution (the `client_myteamspeak_avatar` field).
|
||
- In-memory hot cache in Rust (Flutter's `ImageCache` handles decoded image caching; add Rust-side layer only if profiling shows need).
|
||
- Upload, file browser, and channel file management.
|
||
|
||
## 10. Cache Architecture
|
||
|
||
### 10.1 Layer Ownership
|
||
|
||
| Layer | Responsibility | Storage |
|
||
|---|---|---|
|
||
| `chanora_protocol` | Download raw bytes from server. No caching logic. | None |
|
||
| `chanora_cache` | Content-addressed blob store backed by `cacache`: crash-safe writes, SSRI integrity verification, key validation, eviction, clear. Separate crate from `chanora_storage`. | Platform cache directory |
|
||
| `chanora_core` | Session-aware cache orchestration: check freshness, coalesce requests, rate-limit downloads, persist to disk via `chanora_cache`. | Delegates to `chanora_cache` |
|
||
| `chanora_bridge` | Expose typed `download_avatar` / `clear_file_cache` / `file_cache_size` to Flutter. | None |
|
||
| Flutter | Display via `Image.memory`. Standard `ImageCache` for hot memory caching. Evict from `ImageCache` when hash changes. | In-memory only |
|
||
|
||
### 10.2 Why Separate `chanora_cache` Crate
|
||
|
||
`chanora_cache` is a separate crate from `chanora_storage` for three reasons:
|
||
|
||
1. **Different durability semantics.** `chanora_storage` holds identity, bookmarks, and connection profiles — data the user explicitly created. `chanora_cache` holds downloaded blobs that are fully reconstructible from the server. Losing the cache is an inconvenience, not data loss.
|
||
2. **Different backup semantics.** Cache should be excluded from backups; persistent storage should be included. Platform conventions (iOS `Library/Caches/` vs `Library/Application Support/`) reflect this distinction.
|
||
3. **Different directory placement.** Cache lives in the platform's cache directory (OS may evict under storage pressure on mobile). Persistent storage lives in the support directory.
|
||
|
||
The cache wraps the `cacache` crate for production-tested crash safety and integrity verification. It does not share `chanora_storage`'s crate or directory, and does not reimplement cacache's atomic write or content-addressing logic.
|
||
|
||
### 10.3 Why Hybrid (Rust Disk + Flutter Memory)
|
||
|
||
- Flutter's built-in `ImageCache` is an LRU in-memory cache (default 1000 images / 100 MiB). It handles hot display caching automatically when you use `MemoryImage`.
|
||
- Flutter has no built-in disk cache. `cached_network_image` / `flutter_cache_manager` are designed for HTTP URLs, not custom binary protocol data.
|
||
- Rust already owns the protocol, the connection state, and the anti-flood budget. Putting disk cache here avoids a feedback loop across the bridge.
|
||
|
||
### 10.4 Cache Storage
|
||
|
||
`chanora_cache` wraps the `cacache` crate for its on-disk storage. The physical layout is managed by `cacache`:
|
||
|
||
```
|
||
<app_cache_dir>/chanora/
|
||
blobs/ ← cacache content store root
|
||
content-v2/ ← content-addressed by SHA-512
|
||
<sha512-hex>/
|
||
data ← raw blob bytes
|
||
tmp/ ← temp files (in-flight writes)
|
||
index-v2/ ← entry index (key → content mapping)
|
||
```
|
||
|
||
Chanora's `BlobCache` maps protocol keys to `cacache` string keys:
|
||
|
||
| Protocol key | cacache key | Example |
|
||
|---|---|---|
|
||
| Avatar MD5 | `"av_<md5hex>"` | `"av_a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6"` |
|
||
| Icon CRC32 | `"ic_<crc32u>"` | `"ic_123456789"` |
|
||
|
||
**Why `cacache`:**
|
||
|
||
1. **Crash safety.** Production-tested atomic writes (temp → rename). Handles partial writes, power loss, crash mid-write. No custom crash safety code to maintain.
|
||
2. **Integrity verification.** SSRI integrity check on every `read()`. Detects corruption, bit rot, partial writes automatically. Better than custom "delete on read failure".
|
||
3. **Content dedup.** Same bytes stored once regardless of key. Same avatar on two servers = stored once automatically.
|
||
4. **Less code to maintain.** ~120 LOC wrapper vs ~200 LOC custom implementation. Crash safety and integrity are the hard parts — `cacache` owns them.
|
||
|
||
**Why no `<server_uid>` subdirectory:** The protocol uses content-addressed identifiers. `client_flag_avatar` is the MD5 of the avatar bytes — a given avatar hash always maps to the same bytes regardless of which server the user is on. Same avatar on two servers = same content = stored once by `cacache`. This is a deliberate dedup advantage over per-server namespacing.
|
||
|
||
**Why no metadata sidecars:** Content is immutable (a given hash always maps to the same bytes). `cacache` manages its own entry index with timestamps. No custom metadata files needed.
|
||
|
||
**Key validation rules:**
|
||
|
||
| Prefix | Key format | Validation |
|
||
|---|---|---|
|
||
| `av_` | `av_<32 hex chars>` | MD5 is exactly 32 hex characters |
|
||
| `ic_` | `ic_<1-10 digit number>` | CRC32 unsigned, 0–4294967295 |
|
||
|
||
Keys failing validation are rejected at the `BlobCache` API boundary. This prevents path traversal or malformed filenames on disk.
|
||
|
||
**Platform paths** (Flutter passes the base directory into Rust at startup, matching the existing `initStorage` pattern):
|
||
|
||
| Platform | Cache directory |
|
||
|---|---|
|
||
| Android | `context.cacheDir/chanora/` (via `getCacheDir()`) |
|
||
| iOS | `Library/Caches/chanora/` (via `getApplicationCacheDirectory()`) |
|
||
| macOS | `~/Library/Caches/chanora/` |
|
||
| Windows | `%LOCALAPPDATA%/chanora/cache/` |
|
||
| Linux | `$XDG_CACHE_HOME/chanora/` or `~/.cache/chanora/` |
|
||
|
||
Flutter already resolves platform-specific paths. The same `getApplicationCacheDirectory()` call that is available in `path_provider` across all Chanora target platforms should be used. This follows the existing pattern where `app_bootstrap.dart` calls `getApplicationSupportDirectory()` for persistent storage; avatar/icon cache uses the cache-equivalent directory instead.
|
||
|
||
The cache init call is separate from storage init:
|
||
|
||
```rust
|
||
// Bridge init (Flutter calls these at startup)
|
||
pub fn init_storage(support_dir: String) -> Result<(), BridgeError>; // existing
|
||
pub fn init_cache(cache_dir: String) -> Result<(), BridgeError>; // new
|
||
```
|
||
|
||
### 10.5 Cache Freshness Strategy
|
||
|
||
The `client_flag_avatar` field on each client is the authoritative freshness signal:
|
||
|
||
```
|
||
On connect / on client list update:
|
||
For each visible client with non-empty avatar_hash:
|
||
cache_key = "av_<avatar_hash>"
|
||
if cacache entry exists for "av_<avatar_hash>":
|
||
use cached file (zero downloads)
|
||
else:
|
||
enqueue download for avatar_path with expected hash = avatar_hash
|
||
|
||
On avatar_hash change for a client:
|
||
The new hash produces a different cacache key.
|
||
The old entry remains until eviction or manual clear.
|
||
The new file is downloaded on demand.
|
||
```
|
||
|
||
This means:
|
||
|
||
- **First connect:** No cache hits. Downloads happen lazily as the UI requests avatars.
|
||
- **Reconnect to same server:** All avatars hit cache instantly (hashes match keys). Zero downloads.
|
||
- **User changes avatar:** New hash = new cache key. Old entry becomes orphan. New file downloads on next UI request.
|
||
- **Same user on different server:** Same avatar hash = same cached file. Cross-server dedup for free.
|
||
|
||
### 10.6 Anti-Flood and Download Timing
|
||
|
||
TeamSpeak servers enforce anti-flood rate limiting. Downloading all avatars eagerly on connect would trigger it on servers with many users.
|
||
|
||
**Strategy: lazy + throttled prefetch**
|
||
|
||
| Phase | What | Rate |
|
||
|---|---|---|
|
||
| Connect settle (first 2-5 s) | Do nothing. Let the initial state snapshot and channel tree arrive. | — |
|
||
| After settle | UI requests avatars for visible clients in the current channel. These trigger downloads one at a time. | Max 1-2 concurrent downloads per server |
|
||
| Channel switch | UI requests avatars for newly visible clients. | Same throttle |
|
||
| Background prefetch (optional, future) | Low-priority downloads for clients in adjacent channels. | 1 request per 500 ms |
|
||
|
||
**Anti-flood handling:**
|
||
|
||
- If the server responds with an anti-flood error (TS3 error code `0x0701` = `client_could_not_be_banned` / flood-related), back off the download queue.
|
||
- Implement a simple semaphore in `chanora_core`: max 1-2 concurrent downloads.
|
||
- If a download gets a flood error, pause the queue for 5 seconds, then resume at reduced rate.
|
||
|
||
### 10.7 Retry on Failure
|
||
|
||
| Failure type | Strategy |
|
||
|---|---|
|
||
| Transient (network timeout, TCP reset) | Retry with exponential backoff: 5 s, 30 s, 2 min, 10 min. Cap at 10 min. |
|
||
| Server flood limit hit | Pause queue 5 s, then resume at reduced rate. Do not count as a per-file retry. |
|
||
| Permission denied (no download power) | Do not retry. Record negative cache entry. Only retry if hash changes. |
|
||
| File not found (avatar removed) | Do not retry. Record negative cache entry. Clear when hash changes or becomes empty. |
|
||
| Connection lost | All pending downloads fail. On reconnect, cache check runs fresh with current hashes. |
|
||
|
||
**Negative cache:** In-memory `HashMap<String, Instant>` with 5-minute TTL. Keys like `"av_<hash>"` or `"ic_<id>"` that received permanent errors are stored with an expiry. On lookup, expired entries are treated as absent. Cleared entirely on reconnect.
|
||
|
||
### 10.8 Request Coalescing
|
||
|
||
Multiple UI widgets may request the same avatar simultaneously (e.g., channel list + chat view + client info sheet).
|
||
|
||
**Pattern:** In `chanora_core`'s `FileTransferService`, maintain an in-flight map:
|
||
|
||
```rust
|
||
HashMap<String, tokio::task::JoinHandle<Result<Vec<u8>, FileTransferError>>>
|
||
```
|
||
|
||
- First request: start download, store handle.
|
||
- Subsequent requests for same key: await the same handle.
|
||
- When handle completes: write to cache, wake all waiters, remove from map.
|
||
|
||
### 10.9 Cache Eviction and Size Limits
|
||
|
||
**MVP approach:**
|
||
|
||
- No automatic size-based eviction in MVP. Avatars are small (typically 10-100 KB). Even 1000 avatars = ~50-100 MB.
|
||
- Rely on platform cache directory semantics (OS may evict under storage pressure on mobile).
|
||
- Old hash files accumulate but are harmless.
|
||
|
||
**Post-MVP:**
|
||
|
||
- `BlobCache::evict(max_bytes)` — walk `cacache::ls()` entries, sort by timestamp (oldest first), delete until total size < `max_bytes`. `cacache` manages timestamps internally. No metadata sidecars needed.
|
||
- Or simpler: `BlobCache::evict_older_than(duration)` — delete entries with timestamp older than N days.
|
||
- Call on startup and periodically (e.g., every 24 hours or on app resume).
|
||
|
||
### 10.10 User-Initiated Cache Clear
|
||
|
||
Add a bridge method:
|
||
|
||
```rust
|
||
pub fn clear_file_cache(&self) -> Result<(), BridgeError> {
|
||
// Delete the entire blobs/ directory contents
|
||
// Flutter evicts all avatar/icon-related entries from ImageCache
|
||
}
|
||
|
||
pub fn file_cache_size(&self) -> Result<u64, BridgeError> {
|
||
// Walk blobs/ and sum file sizes
|
||
}
|
||
```
|
||
|
||
Flutter side:
|
||
|
||
```dart
|
||
// In settings or storage management UI:
|
||
onPressed: () async {
|
||
await api.clearFileCache();
|
||
PaintingBinding.instance.imageCache.clear();
|
||
}
|
||
```
|
||
|
||
This should be exposed in the app's settings UI under a "Clear cache" or "Storage management" section.
|
||
|
||
### 10.11 Storage Clear Across Servers
|
||
|
||
Since the cache is flat with content-addressed keys (no server namespacing):
|
||
|
||
- Connecting to a different server does not conflict — same avatar hash = same file.
|
||
- Avatars unique to the old server remain cached. If a user on the new server has the same avatar (same hash), it hits cache instantly (cross-server dedup).
|
||
- Cache clear removes all cached data regardless of which server it came from.
|
||
|
||
### 10.12 Flutter Display Strategy
|
||
|
||
**Option A: Bytes across bridge (simpler, recommended for MVP)**
|
||
|
||
Rust returns `Vec<u8>` across the bridge. Flutter uses `Image.memory(bytes)`.
|
||
|
||
```dart
|
||
final bytes = await api.downloadAvatar(clientUid: uid);
|
||
if (bytes != null && bytes.isNotEmpty) {
|
||
return Image.memory(Uint8List.fromList(bytes));
|
||
} else {
|
||
return CircleAvatar(child: Text(initials)); // fallback
|
||
}
|
||
```
|
||
|
||
Flutter's `ImageCache` caches the decoded image in memory automatically. Same avatar bytes = cache hit in memory.
|
||
|
||
**Option B: File path across bridge (better for large images, future)**
|
||
|
||
Rust writes to disk and returns the file path. Flutter uses `FileImage`.
|
||
|
||
```dart
|
||
final path = await api.getAvatarPath(avatarHash: hash);
|
||
if (path != null) {
|
||
return Image.file(File(path));
|
||
} else {
|
||
return CircleAvatar(child: Text(initials));
|
||
}
|
||
```
|
||
|
||
`FileImage` does not watch for file changes. When the hash changes, the UI must evict the old entry from `ImageCache` using `PaintingBinding.instance.imageCache.evict(key)`.
|
||
|
||
**Recommendation:** Start with Option A for MVP. It avoids file-path cross-platform complications and works well for small avatar files. The bridge already returns `Vec<u8>` for the download result.
|
||
|
||
## 11. Implementation Sequence
|
||
|
||
| Phase | Scope | What |
|
||
|---|---|---|
|
||
| Phase 1 | Protocol download | `Request::DownloadFile`, `StreamItem::FileDownload` handling, `ProtocolClient::download_avatar()` / `download_icon()`. No caching. |
|
||
| Phase 2 | Bridge + Flutter display | Bridge `downloadAvatar()`, Flutter `Image.memory()`, initials fallback. Still no caching — every view re-downloads. |
|
||
| Phase 3 | Rust disk cache | New `chanora_cache` crate: `cacache`-backed content-addressed blob store (`BlobCache`), key validation, mtime-based eviction, `init_cache` bridge call. |
|
||
| Phase 4 | Session orchestration | `chanora_core` `FileTransferService`: request coalescing, rate limiter (semaphore), negative cache (5 min TTL), retry backoff. |
|
||
| Phase 5 | Cache management | Bridge `clearFileCache()` + `fileCacheSize()`, Flutter settings UI, eviction on startup. |
|
||
|
||
Phase 1 and 2 deliver visible value (avatars in the UI). Phase 3-5 add robustness.
|
||
|
||
## 12. References
|
||
|
||
| Reference | Use |
|
||
|---|---|
|
||
| `ReSpeak/tsdeclarations` `Messages.toml` lines 828-830 | `ftinitdownload` command declaration |
|
||
| `ReSpeak/tsdeclarations` `Messages.toml` lines 590 | `FileDownload` response structure |
|
||
| `ReSpeak/tsdeclarations` `ts3protocol.md` | Low-level TeamSpeak protocol specification |
|
||
| `ReSpeak/tsclientlib` `src/lib.rs` lines 956-1005 | `download_file` / `upload_file` public API |
|
||
| `ReSpeak/tsclientlib` `src/lib.rs` lines 1371-1427 | `StreamItem::FileDownload` handling |
|
||
| `ReSpeak/tsclientlib` `src/lib.rs` lines 1630-1672 | Outgoing init commands |
|
||
| `Multivit4min/TS3-NodeJS-Library` `src/transport/FileTransfer.ts` | Reference TCP transfer implementation |
|
||
| `Speckmops/ts3admin.class` `lib/ts3admin.class.php` lines 1352-1370 | Reference avatar download flow |
|
||
| `docs/architecture/sad.md` SAD-067, SDD-MOD-009 | Protocol adapter boundary rules |
|
||
| `crates/chanora_protocol/src/adapter.rs` lines 1561-1568 | Existing `uid_to_avatar_path` implementation |
|