docs(architecture): add file transfer design, research, and implementation plan
This commit is contained in:
Edison Jwa
@@ -0,0 +1,737 @@
|
||||
# File Transfer Design
|
||||
|
||||
**Date:** 2026-06-10
|
||||
**Status:** Draft for review
|
||||
**Scope:** Download files from TeamSpeak-compatible servers via the native client protocol, starting with avatars and icons.
|
||||
**Direct upstream source:** `docs/architecture/sad.md` (SAD-067, SDD-MOD-009)
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Chanora needs to download files stored on TeamSpeak-compatible servers. The most visible use cases are client avatars and server/channel/client icons. The file transfer mechanism is also used for channel file browser features, but this document scopes the initial design to avatar and icon retrieval only.
|
||||
|
||||
This document describes:
|
||||
|
||||
- How the TeamSpeak file transfer protocol works.
|
||||
- How `tsclientlib` exposes it.
|
||||
- How Chanora should integrate it following the existing protocol adapter pattern.
|
||||
- How the result flows through the bridge to the Flutter UI layer.
|
||||
|
||||
Upload, channel file browsing, and file deletion are explicitly out of scope for the initial implementation.
|
||||
|
||||
## 2. Protocol Background
|
||||
|
||||
### 2.1 Two-Phase Transfer
|
||||
|
||||
TeamSpeak file transfer is a two-phase process:
|
||||
|
||||
1. **Command phase** — The client sends a command over the main encrypted UDP connection to request a transfer token (`ftkey`).
|
||||
2. **Transfer phase** — The client opens a separate TCP connection to the server's file transfer port (default `30033`) and sends the `ftkey` to authenticate the transfer. Raw bytes flow over this TCP stream.
|
||||
|
||||
### 2.2 Relevant ServerQuery Commands
|
||||
|
||||
| Command | Direction | Purpose |
|
||||
|---|---|---|
|
||||
| `ftinitdownload` | Client → Server | Initialize a download. Returns `ftkey`, `port`, `size`. |
|
||||
| `ftgetfileinfo` | Client → Server | Get metadata for one or more files. |
|
||||
| `ftgetfilelist` | Client → Server | List files in a channel's file repository. |
|
||||
| `ftinitupload` | Client → Server | Initialize an upload. |
|
||||
| `ftlist` | Client → Server | List active file transfers. |
|
||||
| `ftstop` | Client → Server | Stop a running transfer. |
|
||||
| `ftdeletefile` | Client → Server | Delete a file. |
|
||||
| `ftcreatedir` | Client → Server | Create a directory. |
|
||||
| `ftrenamefile` | Client → Server | Rename or move a file. |
|
||||
|
||||
Initial scope uses only `ftinitdownload` and `ftgetfileinfo`.
|
||||
|
||||
### 2.3 File Paths
|
||||
|
||||
Files are addressed by a path scoped to a channel ID (`cid`):
|
||||
|
||||
- `cid=0` — Server-level file repository. Avatars and icons live here.
|
||||
- `cid=N` (non-zero) — Channel-specific file repository.
|
||||
|
||||
Avatar path: `/avatar_<hex>` where `<hex>` is derived from the client's unique identifier (UID). Each byte of the base64-decoded UID is split into two nibbles, and each nibble maps to a letter `a` through `p` (0→a, 1→b, ..., 15→p).
|
||||
|
||||
Icon path: `/icon_<id>` where `<id>` is the icon's signed 64-bit integer ID. If negative, treat as unsigned for the path.
|
||||
|
||||
### 2.4 `ftinitdownload` Command
|
||||
|
||||
```
|
||||
ftinitdownload clientftfid={id} name={path} cid={channelId} cpw={password} seekpos={seek} proto=0
|
||||
```
|
||||
|
||||
Parameters:
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `clientftfid` | `u16` | Arbitrary client-side transfer ID. |
|
||||
| `name` | `string` | File path, e.g. `/avatar_abcdef`. |
|
||||
| `cid` | `ChannelId` | Channel scope (0 = server). |
|
||||
| `cpw` | `string` | Channel password. Empty for server-level. |
|
||||
| `seekpos` | `u64` | Resume offset. 0 for a fresh download. |
|
||||
| `proto` | `u8` | Protocol version. Always 0. |
|
||||
|
||||
Server response:
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `clientftfid` | `u16` | Echo of the client transfer ID. |
|
||||
| `serverftfid` | `u16` | Server-side transfer ID. |
|
||||
| `ftkey` | `string` | One-time transfer key (hex). |
|
||||
| `port` | `u16` | File transfer TCP port (usually 30033). |
|
||||
| `size` | `u64` | File size in bytes. |
|
||||
| `proto` | `u8` | Protocol version echo. |
|
||||
| `ip` | `string` (optional) | Override IP for the TCP connection. |
|
||||
|
||||
### 2.5 TCP Transfer
|
||||
|
||||
After receiving the `ftkey`, the client:
|
||||
|
||||
1. Opens a TCP connection to `server_ip:port`.
|
||||
2. Sends `ftkey` followed by a newline.
|
||||
3. Reads exactly `size` bytes of raw file data.
|
||||
4. Closes the TCP connection.
|
||||
|
||||
### 2.6 Permissions
|
||||
|
||||
File transfer requires the following permissions on the server:
|
||||
|
||||
| Permission | Needed for |
|
||||
|---|---|
|
||||
| `i_ft_file_download_power` | Downloading files. |
|
||||
| `i_ft_needed_file_download_power` | Required download power on the channel/server. |
|
||||
| `b_ft_ignore_password` | Bypassing channel passwords (not needed for avatars). |
|
||||
|
||||
Avatar downloads typically require only basic download power because avatars are in the server-level repository (`cid=0`), which is generally accessible.
|
||||
|
||||
### 2.7 Avatar Detection
|
||||
|
||||
When a client connects or updates, the server sends `client_flag_avatar` as a string (the avatar hash). If non-empty, the client has an avatar. The avatar is downloaded from `/avatar_<hex>` where `<hex>` is computed from the client's UID (not from the hash string itself — the hash is just a presence indicator).
|
||||
|
||||
## 3. tsclientlib Support
|
||||
|
||||
`tsclientlib` implements file transfer natively. The library handles the entire command + TCP flow internally:
|
||||
|
||||
### 3.1 Public API
|
||||
|
||||
```rust
|
||||
// tsclientlib/src/lib.rs (relevant signatures)
|
||||
impl Connection {
|
||||
pub fn download_file(
|
||||
&mut self,
|
||||
channel_id: ChannelId,
|
||||
path: &str,
|
||||
channel_password: Option<&str>,
|
||||
seek_position: Option<u64>,
|
||||
) -> Result<FiletransferHandle>;
|
||||
|
||||
pub fn upload_file(
|
||||
&mut self,
|
||||
channel_id: ChannelId,
|
||||
path: &str,
|
||||
channel_password: Option<&str>,
|
||||
size: u64,
|
||||
overwrite: bool,
|
||||
resume: bool,
|
||||
) -> Result<FiletransferHandle>;
|
||||
}
|
||||
```
|
||||
|
||||
`download_file` sends the `ftinitdownload` command and returns a `FiletransferHandle(u16)` immediately. The actual transfer completes asynchronously.
|
||||
|
||||
### 3.2 Stream Items
|
||||
|
||||
The connection's event stream emits:
|
||||
|
||||
| StreamItem | When | Data |
|
||||
|---|---|---|
|
||||
| `StreamItem::FileDownload(FileDownloadResult)` | Server responds with `ftkey`; TCP connected and `ftkey` written | `{ size: u64, stream: TcpStream }` |
|
||||
| `StreamItem::FileUpload(FileUploadResult)` | Upload ready | `{ seek_position: u64, stream: TcpStream }` |
|
||||
| `StreamItem::FiletransferFailed(FiletransferHandle, Error)` | Transfer failed | Handle + error |
|
||||
|
||||
When `FileDownload` fires, tsclientlib has already:
|
||||
|
||||
1. Sent `ftinitdownload` over the encrypted UDP command channel.
|
||||
2. Received the `ftkey`, `port`, and `size` from the server.
|
||||
3. Opened a TCP connection to `server:port`.
|
||||
4. Written the `ftkey` to the TCP socket.
|
||||
|
||||
The `TcpStream` in `FileDownloadResult` is ready to read; Chanora only needs to read exactly `size` bytes.
|
||||
|
||||
### 3.3 Avatar Helper
|
||||
|
||||
`tsproto-types` provides `Uid::as_avatar()` which computes the avatar filename from a UID. Chanora's existing `uid_to_avatar_path()` in `adapter.rs` does the same thing independently.
|
||||
|
||||
### 3.4 Doc-Comment Examples
|
||||
|
||||
tsclientlib's source contains usage examples in doc comments:
|
||||
|
||||
```rust
|
||||
/// Download an icon:
|
||||
/// con.download_file(ChannelId(0), &format!("/icon_{}", icon_id), None, None)
|
||||
|
||||
/// Upload an avatar:
|
||||
/// con.upload_file(ChannelId(0), "/avatar", None, data.len() as u64, true, false)
|
||||
```
|
||||
|
||||
## 4. Architecture Integration
|
||||
|
||||
### 4.1 Existing Pattern
|
||||
|
||||
The protocol adapter (`crates/chanora_protocol`) uses a single tokio task that owns the `tsclientlib::Connection`. All operations follow this pattern:
|
||||
|
||||
1. Define a `Request` enum variant with parameters and a `oneshot::Sender` for the reply.
|
||||
2. Send the request through the `mpsc` channel to the connection task.
|
||||
3. The connection task calls tsclientlib and resolves the oneshot.
|
||||
|
||||
File transfer fits this pattern exactly. The only difference is that the result arrives asynchronously via `StreamItem::FileDownload` rather than immediately from the command call.
|
||||
|
||||
### 4.2 Design
|
||||
|
||||
The file transfer integration adds:
|
||||
|
||||
1. **`Request` variants** for file download.
|
||||
2. **A pending-downloads map** (`HashMap<FiletransferHandle, DownloadContext>`) in the connection task, mirroring the existing `pending_moves` pattern.
|
||||
3. **`StreamItem::FileDownload` and `StreamItem::FiletransferFailed`** handling in the event loop.
|
||||
4. **New DTOs** for file transfer results.
|
||||
5. **Convenience methods** on `ProtocolClient` for avatar and icon downloads.
|
||||
|
||||
### 4.3 Layer Responsibilities
|
||||
|
||||
| Layer | Responsibility |
|
||||
|---|---|
|
||||
| `chanora_protocol` | Call `tsclientlib::download_file`, track pending transfers, read `TcpStream`, return bytes. No tsclientlib types leak. |
|
||||
| `chanora_core` | Orchestrate when to download (e.g., on profile fetch or on avatar cache miss). |
|
||||
| `chanora_bridge` | Expose typed `download_avatar` / `download_icon` commands to Flutter. |
|
||||
| Flutter UI | Call bridge, display with `Image.memory()`. Cache in memory/image cache. |
|
||||
|
||||
### 4.4 Error Mapping
|
||||
|
||||
File transfer errors map to the existing `ProtocolError` variants:
|
||||
|
||||
| tsclientlib error | ProtocolError |
|
||||
|---|---|
|
||||
| Permission denied (TS3 error code) | `ServerRejected { code, message }` |
|
||||
| File not found | `ServerRejected { code, message }` |
|
||||
| Network/TCP failure | `Backend(String)` |
|
||||
| Timeout | `Timeout` |
|
||||
| Connection lost mid-transfer | `Lost(String)` |
|
||||
|
||||
## 5. Detailed Design
|
||||
|
||||
### 5.1 New Types in `dto.rs`
|
||||
|
||||
```rust
|
||||
/// A downloaded file's raw content and metadata.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct DownloadedFile {
|
||||
/// Raw file bytes.
|
||||
pub data: Vec<u8>,
|
||||
/// The server path that was requested.
|
||||
pub path: String,
|
||||
/// Channel ID the file was downloaded from.
|
||||
pub channel_id: u64,
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 New Request Variants in `adapter.rs`
|
||||
|
||||
```rust
|
||||
enum Request {
|
||||
// ... existing variants ...
|
||||
|
||||
/// Download a file from the server's file repository.
|
||||
DownloadFile {
|
||||
/// Channel ID. 0 for server-level (avatars, icons).
|
||||
channel_id: u64,
|
||||
/// File path, e.g. "/avatar_abcdef" or "/icon_12345".
|
||||
path: String,
|
||||
/// Channel password. None for server-level files.
|
||||
channel_password: Option<String>,
|
||||
/// Reply channel for the result.
|
||||
reply: oneshot::Sender<Result<DownloadedFile, ProtocolError>>,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 Pending Downloads Map
|
||||
|
||||
```rust
|
||||
type PendingDownloads = HashMap<tsclientlib::FiletransferHandle, PendingDownload>;
|
||||
|
||||
struct PendingDownload {
|
||||
path: String,
|
||||
channel_id: u64,
|
||||
reply: oneshot::Sender<Result<DownloadedFile, ProtocolError>>,
|
||||
}
|
||||
```
|
||||
|
||||
### 5.4 Event Loop Handling
|
||||
|
||||
In the connection task's main loop, add handling for file transfer stream items:
|
||||
|
||||
```rust
|
||||
// In handle_non_audio_stream_item or in the main loop:
|
||||
StreamItem::FileDownload(result) => {
|
||||
// result: FileDownloadResult { size, stream }
|
||||
// Look up the handle in pending_downloads
|
||||
// Use tokio::io::AsyncReadExt::read_exact to read 'size' bytes
|
||||
// Resolve the oneshot with DownloadedFile
|
||||
}
|
||||
StreamItem::FiletransferFailed(handle, error) => {
|
||||
// Look up the handle in pending_downloads
|
||||
// Resolve the oneshot with ProtocolError::Backend
|
||||
}
|
||||
```
|
||||
|
||||
The TCP read from the `TcpStream` is an async operation. Since the connection task already runs in a tokio context, the read can be done inline. However, for large files this would block the main event loop. Two approaches:
|
||||
|
||||
**Option A: Read inline (simple, good for small files like avatars)**
|
||||
|
||||
Avatars are typically under 100 KB. Reading them inline in the event loop is acceptable and avoids complexity.
|
||||
|
||||
**Option B: Spawn a reader task**
|
||||
|
||||
For future channel-file-browser support with potentially large files, spawn a separate tokio task that reads the stream and sends the result back.
|
||||
|
||||
**Recommendation:** Start with Option A. The initial scope is avatars and icons (small files). Refactor to Option B when channel file browsing is implemented.
|
||||
|
||||
### 5.5 Request Handling
|
||||
|
||||
When the connection task receives `Request::DownloadFile`:
|
||||
|
||||
```rust
|
||||
Ok(Request::DownloadFile { channel_id, path, channel_password, reply }) => {
|
||||
let ts_channel_id = TsChannelId(channel_id);
|
||||
match con.download_file(ts_channel_id, &path, channel_password.as_deref(), None) {
|
||||
Ok(handle) => {
|
||||
pending_downloads.insert(handle, PendingDownload {
|
||||
path,
|
||||
channel_id,
|
||||
reply,
|
||||
});
|
||||
}
|
||||
Err(e) => {
|
||||
let _ = reply.send(Err(ProtocolError::Backend(
|
||||
format!("download_file init: {e}")
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.6 Public API on `ProtocolClient`
|
||||
|
||||
```rust
|
||||
impl ProtocolClient {
|
||||
/// Download a file from the server's file repository.
|
||||
/// `channel_id` 0 means server-level (avatars, icons).
|
||||
pub async fn download_file(
|
||||
&self,
|
||||
channel_id: u64,
|
||||
path: String,
|
||||
channel_password: Option<String>,
|
||||
) -> Result<DownloadedFile, ProtocolError> {
|
||||
let (tx, rx) = oneshot::channel();
|
||||
self.tx
|
||||
.send(Request::DownloadFile { channel_id, path, channel_password, reply: tx })
|
||||
.await
|
||||
.map_err(|_| ProtocolError::Lost("connection task is gone".to_string()))?;
|
||||
rx.await
|
||||
.map_err(|_| ProtocolError::Lost("download_file reply dropped".to_string()))?
|
||||
}
|
||||
|
||||
/// Download a client's avatar image. Returns raw image bytes.
|
||||
/// Pass the `avatar_path` from `ClientProfile`.
|
||||
pub async fn download_avatar(
|
||||
&self,
|
||||
avatar_path: String,
|
||||
) -> Result<DownloadedFile, ProtocolError> {
|
||||
self.download_file(0, avatar_path, None).await
|
||||
}
|
||||
|
||||
/// Download a server, channel, or client icon by its icon ID.
|
||||
pub async fn download_icon(
|
||||
&self,
|
||||
icon_id: i64,
|
||||
) -> Result<DownloadedFile, ProtocolError> {
|
||||
let unsigned_id = icon_id as u64;
|
||||
let path = format!("/icon_{}", unsigned_id);
|
||||
self.download_file(0, path, None).await
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.7 Exports in `lib.rs`
|
||||
|
||||
```rust
|
||||
pub use dto::DownloadedFile;
|
||||
```
|
||||
|
||||
### 5.8 Bridge Layer
|
||||
|
||||
In `crates/chanora_bridge/src/api.rs`, add:
|
||||
|
||||
```rust
|
||||
pub async fn download_avatar(&self, avatar_path: String) -> Result<Vec<u8>, BridgeError> {
|
||||
self.protocol
|
||||
.download_avatar(avatar_path)
|
||||
.await
|
||||
.map(|file| file.data)
|
||||
.map_err(BridgeError::Protocol)
|
||||
}
|
||||
```
|
||||
|
||||
### 5.9 Flutter Integration
|
||||
|
||||
Flutter side:
|
||||
|
||||
1. Call `clientProfile()` to get `ClientProfile` (already exists).
|
||||
2. Check if `avatarPath` is non-empty.
|
||||
3. Call bridge `downloadAvatar(avatarPath)` to get `Uint8List`.
|
||||
4. Display with `Image.memory(bytes)`.
|
||||
|
||||
Caching strategy:
|
||||
|
||||
- In-memory: Use Flutter's standard `ImageCache` or a simple `Map<String, Uint8List>` keyed by avatar path.
|
||||
- Disk: Consider caching to local storage for offline display. This is a follow-up decision, not MVP scope.
|
||||
- The avatar path already encodes the UID, so it can serve as a cache key.
|
||||
|
||||
## 6. Avatar Path Computation
|
||||
|
||||
Chanora already has this implemented in `adapter.rs`:
|
||||
|
||||
```rust
|
||||
fn uid_to_avatar_path(uid_b64: &str) -> String {
|
||||
let decoded = BASE64_STANDARD.decode(uid_b64).unwrap_or_default();
|
||||
let mut rendered = String::with_capacity(decoded.len() * 2);
|
||||
for byte in decoded {
|
||||
rendered.push((b'a' + (byte >> 4)) as char);
|
||||
rendered.push((b'a' + (byte & 0x0f)) as char);
|
||||
}
|
||||
rendered
|
||||
}
|
||||
```
|
||||
|
||||
This maps each nibble to `a` through `p` (0→a, 1→b, ..., 15→p), matching the canonical TeamSpeak implementation.
|
||||
|
||||
The full avatar path is constructed as:
|
||||
|
||||
```rust
|
||||
let avatar_path = if client.avatar_hash.is_empty() || unique_id.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!("/avatar_{}", uid_to_avatar_path(&unique_id))
|
||||
};
|
||||
```
|
||||
|
||||
This is already correct and used in `ClientProfile.avatar_path`. No changes needed.
|
||||
|
||||
## 7. Threading and Concurrency
|
||||
|
||||
| Concern | Design |
|
||||
|---|---|
|
||||
| TCP read blocking the event loop | For avatar/icon sizes (< 100 KB typically), inline async read is acceptable. Spawn a reader task for larger files when channel file browsing is added. |
|
||||
| Multiple concurrent downloads | `pending_downloads` is a HashMap keyed by `FiletransferHandle`. Multiple downloads can be in flight simultaneously. tsclientlib assigns unique handles. |
|
||||
| Download timeout | Add a deadline to pending downloads (e.g., 30 seconds). Sweep expired entries similar to the existing `pending_moves` sweep. |
|
||||
| Cancellation on disconnect | When the connection task exits, all pending oneshot senders are dropped, which resolves the caller's await with a `RecvError`. The caller maps this to `ProtocolError::Lost`. |
|
||||
|
||||
## 8. Diagnostic and Security Considerations
|
||||
|
||||
### 8.1 Diagnostic Redaction
|
||||
|
||||
- File transfer paths may contain user-identifying information (UID-derived avatar names). These should be registered for diagnostic redaction if they appear in log output.
|
||||
- File contents (avatar images) must not appear in log output or diagnostic exports.
|
||||
|
||||
### 8.2 Security
|
||||
|
||||
- The `ftkey` is a one-time token and must not be logged.
|
||||
- TCP file transfer connections are not encrypted. This is a TeamSpeak protocol limitation, not a Chanora design choice. Avatar data is public (visible to anyone on the server), so the risk is acceptable.
|
||||
- File download does not require secrets beyond the existing authenticated connection.
|
||||
|
||||
### 8.3 Privacy
|
||||
|
||||
- Avatar downloads reveal to the server that the user is viewing a specific client's avatar. This is inherent in the protocol.
|
||||
- Chanora should not download avatars proactively for all clients. Download only when the UI needs to display a specific avatar (lazy/on-demand).
|
||||
|
||||
## 9. Out of Scope
|
||||
|
||||
The following are explicitly deferred:
|
||||
|
||||
- File upload (avatar upload, channel file upload).
|
||||
- Channel file browser (listing, creating directories, deleting, renaming).
|
||||
- Resumable downloads (seek position > 0).
|
||||
- File transfer progress reporting.
|
||||
- myTeamSpeak avatar resolution (the `client_myteamspeak_avatar` field).
|
||||
- In-memory hot cache in Rust (Flutter's `ImageCache` handles decoded image caching; add Rust-side layer only if profiling shows need).
|
||||
- Upload, file browser, and channel file management.
|
||||
|
||||
## 10. Cache Architecture
|
||||
|
||||
### 10.1 Layer Ownership
|
||||
|
||||
| Layer | Responsibility | Storage |
|
||||
|---|---|---|
|
||||
| `chanora_protocol` | Download raw bytes from server. No caching logic. | None |
|
||||
| `chanora_cache` | Content-addressed blob store backed by `cacache`: crash-safe writes, SSRI integrity verification, key validation, eviction, clear. Separate crate from `chanora_storage`. | Platform cache directory |
|
||||
| `chanora_core` | Session-aware cache orchestration: check freshness, coalesce requests, rate-limit downloads, persist to disk via `chanora_cache`. | Delegates to `chanora_cache` |
|
||||
| `chanora_bridge` | Expose typed `download_avatar` / `clear_file_cache` / `file_cache_size` to Flutter. | None |
|
||||
| Flutter | Display via `Image.memory`. Standard `ImageCache` for hot memory caching. Evict from `ImageCache` when hash changes. | In-memory only |
|
||||
|
||||
### 10.2 Why Separate `chanora_cache` Crate
|
||||
|
||||
`chanora_cache` is a separate crate from `chanora_storage` for three reasons:
|
||||
|
||||
1. **Different durability semantics.** `chanora_storage` holds identity, bookmarks, and connection profiles — data the user explicitly created. `chanora_cache` holds downloaded blobs that are fully reconstructible from the server. Losing the cache is an inconvenience, not data loss.
|
||||
2. **Different backup semantics.** Cache should be excluded from backups; persistent storage should be included. Platform conventions (iOS `Library/Caches/` vs `Library/Application Support/`) reflect this distinction.
|
||||
3. **Different directory placement.** Cache lives in the platform's cache directory (OS may evict under storage pressure on mobile). Persistent storage lives in the support directory.
|
||||
|
||||
The cache wraps the `cacache` crate for production-tested crash safety and integrity verification. It does not share `chanora_storage`'s crate or directory, and does not reimplement cacache's atomic write or content-addressing logic.
|
||||
|
||||
### 10.3 Why Hybrid (Rust Disk + Flutter Memory)
|
||||
|
||||
- Flutter's built-in `ImageCache` is an LRU in-memory cache (default 1000 images / 100 MiB). It handles hot display caching automatically when you use `MemoryImage`.
|
||||
- Flutter has no built-in disk cache. `cached_network_image` / `flutter_cache_manager` are designed for HTTP URLs, not custom binary protocol data.
|
||||
- Rust already owns the protocol, the connection state, and the anti-flood budget. Putting disk cache here avoids a feedback loop across the bridge.
|
||||
|
||||
### 10.4 Cache Storage
|
||||
|
||||
`chanora_cache` wraps the `cacache` crate for its on-disk storage. The physical layout is managed by `cacache`:
|
||||
|
||||
```
|
||||
<app_cache_dir>/chanora/
|
||||
blobs/ ← cacache content store root
|
||||
content-v2/ ← content-addressed by SHA-512
|
||||
<sha512-hex>/
|
||||
data ← raw blob bytes
|
||||
tmp/ ← temp files (in-flight writes)
|
||||
index-v2/ ← entry index (key → content mapping)
|
||||
```
|
||||
|
||||
Chanora's `BlobCache` maps protocol keys to `cacache` string keys:
|
||||
|
||||
| Protocol key | cacache key | Example |
|
||||
|---|---|---|
|
||||
| Avatar MD5 | `"av_<md5hex>"` | `"av_a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6"` |
|
||||
| Icon CRC32 | `"ic_<crc32u>"` | `"ic_123456789"` |
|
||||
|
||||
**Why `cacache`:**
|
||||
|
||||
1. **Crash safety.** Production-tested atomic writes (temp → rename). Handles partial writes, power loss, crash mid-write. No custom crash safety code to maintain.
|
||||
2. **Integrity verification.** SSRI integrity check on every `read()`. Detects corruption, bit rot, partial writes automatically. Better than custom "delete on read failure".
|
||||
3. **Content dedup.** Same bytes stored once regardless of key. Same avatar on two servers = stored once automatically.
|
||||
4. **Less code to maintain.** ~120 LOC wrapper vs ~200 LOC custom implementation. Crash safety and integrity are the hard parts — `cacache` owns them.
|
||||
|
||||
**Why no `<server_uid>` subdirectory:** The protocol uses content-addressed identifiers. `client_flag_avatar` is the MD5 of the avatar bytes — a given avatar hash always maps to the same bytes regardless of which server the user is on. Same avatar on two servers = same content = stored once by `cacache`. This is a deliberate dedup advantage over per-server namespacing.
|
||||
|
||||
**Why no metadata sidecars:** Content is immutable (a given hash always maps to the same bytes). `cacache` manages its own entry index with timestamps. No custom metadata files needed.
|
||||
|
||||
**Key validation rules:**
|
||||
|
||||
| Prefix | Key format | Validation |
|
||||
|---|---|---|
|
||||
| `av_` | `av_<32 hex chars>` | MD5 is exactly 32 hex characters |
|
||||
| `ic_` | `ic_<1-10 digit number>` | CRC32 unsigned, 0–4294967295 |
|
||||
|
||||
Keys failing validation are rejected at the `BlobCache` API boundary. This prevents path traversal or malformed filenames on disk.
|
||||
|
||||
**Platform paths** (Flutter passes the base directory into Rust at startup, matching the existing `initStorage` pattern):
|
||||
|
||||
| Platform | Cache directory |
|
||||
|---|---|
|
||||
| Android | `context.cacheDir/chanora/` (via `getCacheDir()`) |
|
||||
| iOS | `Library/Caches/chanora/` (via `getApplicationCacheDirectory()`) |
|
||||
| macOS | `~/Library/Caches/chanora/` |
|
||||
| Windows | `%LOCALAPPDATA%/chanora/cache/` |
|
||||
| Linux | `$XDG_CACHE_HOME/chanora/` or `~/.cache/chanora/` |
|
||||
|
||||
Flutter already resolves platform-specific paths. The same `getApplicationCacheDirectory()` call that is available in `path_provider` across all Chanora target platforms should be used. This follows the existing pattern where `app_bootstrap.dart` calls `getApplicationSupportDirectory()` for persistent storage; avatar/icon cache uses the cache-equivalent directory instead.
|
||||
|
||||
The cache init call is separate from storage init:
|
||||
|
||||
```rust
|
||||
// Bridge init (Flutter calls these at startup)
|
||||
pub fn init_storage(support_dir: String) -> Result<(), BridgeError>; // existing
|
||||
pub fn init_cache(cache_dir: String) -> Result<(), BridgeError>; // new
|
||||
```
|
||||
|
||||
### 10.5 Cache Freshness Strategy
|
||||
|
||||
The `client_flag_avatar` field on each client is the authoritative freshness signal:
|
||||
|
||||
```
|
||||
On connect / on client list update:
|
||||
For each visible client with non-empty avatar_hash:
|
||||
cache_key = "av_<avatar_hash>"
|
||||
if cacache entry exists for "av_<avatar_hash>":
|
||||
use cached file (zero downloads)
|
||||
else:
|
||||
enqueue download for avatar_path with expected hash = avatar_hash
|
||||
|
||||
On avatar_hash change for a client:
|
||||
The new hash produces a different cacache key.
|
||||
The old entry remains until eviction or manual clear.
|
||||
The new file is downloaded on demand.
|
||||
```
|
||||
|
||||
This means:
|
||||
|
||||
- **First connect:** No cache hits. Downloads happen lazily as the UI requests avatars.
|
||||
- **Reconnect to same server:** All avatars hit cache instantly (hashes match keys). Zero downloads.
|
||||
- **User changes avatar:** New hash = new cache key. Old entry becomes orphan. New file downloads on next UI request.
|
||||
- **Same user on different server:** Same avatar hash = same cached file. Cross-server dedup for free.
|
||||
|
||||
### 10.6 Anti-Flood and Download Timing
|
||||
|
||||
TeamSpeak servers enforce anti-flood rate limiting. Downloading all avatars eagerly on connect would trigger it on servers with many users.
|
||||
|
||||
**Strategy: lazy + throttled prefetch**
|
||||
|
||||
| Phase | What | Rate |
|
||||
|---|---|---|
|
||||
| Connect settle (first 2-5 s) | Do nothing. Let the initial state snapshot and channel tree arrive. | — |
|
||||
| After settle | UI requests avatars for visible clients in the current channel. These trigger downloads one at a time. | Max 1-2 concurrent downloads per server |
|
||||
| Channel switch | UI requests avatars for newly visible clients. | Same throttle |
|
||||
| Background prefetch (optional, future) | Low-priority downloads for clients in adjacent channels. | 1 request per 500 ms |
|
||||
|
||||
**Anti-flood handling:**
|
||||
|
||||
- If the server responds with an anti-flood error (TS3 error code `0x0701` = `client_could_not_be_banned` / flood-related), back off the download queue.
|
||||
- Implement a simple semaphore in `chanora_core`: max 1-2 concurrent downloads.
|
||||
- If a download gets a flood error, pause the queue for 5 seconds, then resume at reduced rate.
|
||||
|
||||
### 10.7 Retry on Failure
|
||||
|
||||
| Failure type | Strategy |
|
||||
|---|---|
|
||||
| Transient (network timeout, TCP reset) | Retry with exponential backoff: 5 s, 30 s, 2 min, 10 min. Cap at 10 min. |
|
||||
| Server flood limit hit | Pause queue 5 s, then resume at reduced rate. Do not count as a per-file retry. |
|
||||
| Permission denied (no download power) | Do not retry. Record negative cache entry. Only retry if hash changes. |
|
||||
| File not found (avatar removed) | Do not retry. Record negative cache entry. Clear when hash changes or becomes empty. |
|
||||
| Connection lost | All pending downloads fail. On reconnect, cache check runs fresh with current hashes. |
|
||||
|
||||
**Negative cache:** In-memory `HashMap<String, Instant>` with 5-minute TTL. Keys like `"av_<hash>"` or `"ic_<id>"` that received permanent errors are stored with an expiry. On lookup, expired entries are treated as absent. Cleared entirely on reconnect.
|
||||
|
||||
### 10.8 Request Coalescing
|
||||
|
||||
Multiple UI widgets may request the same avatar simultaneously (e.g., channel list + chat view + client info sheet).
|
||||
|
||||
**Pattern:** In `chanora_core`'s `FileTransferService`, maintain an in-flight map:
|
||||
|
||||
```rust
|
||||
HashMap<String, tokio::task::JoinHandle<Result<Vec<u8>, FileTransferError>>>
|
||||
```
|
||||
|
||||
- First request: start download, store handle.
|
||||
- Subsequent requests for same key: await the same handle.
|
||||
- When handle completes: write to cache, wake all waiters, remove from map.
|
||||
|
||||
### 10.9 Cache Eviction and Size Limits
|
||||
|
||||
**MVP approach:**
|
||||
|
||||
- No automatic size-based eviction in MVP. Avatars are small (typically 10-100 KB). Even 1000 avatars = ~50-100 MB.
|
||||
- Rely on platform cache directory semantics (OS may evict under storage pressure on mobile).
|
||||
- Old hash files accumulate but are harmless.
|
||||
|
||||
**Post-MVP:**
|
||||
|
||||
- `BlobCache::evict(max_bytes)` — walk `cacache::ls()` entries, sort by timestamp (oldest first), delete until total size < `max_bytes`. `cacache` manages timestamps internally. No metadata sidecars needed.
|
||||
- Or simpler: `BlobCache::evict_older_than(duration)` — delete entries with timestamp older than N days.
|
||||
- Call on startup and periodically (e.g., every 24 hours or on app resume).
|
||||
|
||||
### 10.10 User-Initiated Cache Clear
|
||||
|
||||
Add a bridge method:
|
||||
|
||||
```rust
|
||||
pub fn clear_file_cache(&self) -> Result<(), BridgeError> {
|
||||
// Delete the entire blobs/ directory contents
|
||||
// Flutter evicts all avatar/icon-related entries from ImageCache
|
||||
}
|
||||
|
||||
pub fn file_cache_size(&self) -> Result<u64, BridgeError> {
|
||||
// Walk blobs/ and sum file sizes
|
||||
}
|
||||
```
|
||||
|
||||
Flutter side:
|
||||
|
||||
```dart
|
||||
// In settings or storage management UI:
|
||||
onPressed: () async {
|
||||
await api.clearFileCache();
|
||||
PaintingBinding.instance.imageCache.clear();
|
||||
}
|
||||
```
|
||||
|
||||
This should be exposed in the app's settings UI under a "Clear cache" or "Storage management" section.
|
||||
|
||||
### 10.11 Storage Clear Across Servers
|
||||
|
||||
Since the cache is flat with content-addressed keys (no server namespacing):
|
||||
|
||||
- Connecting to a different server does not conflict — same avatar hash = same file.
|
||||
- Avatars unique to the old server remain cached. If a user on the new server has the same avatar (same hash), it hits cache instantly (cross-server dedup).
|
||||
- Cache clear removes all cached data regardless of which server it came from.
|
||||
|
||||
### 10.12 Flutter Display Strategy
|
||||
|
||||
**Option A: Bytes across bridge (simpler, recommended for MVP)**
|
||||
|
||||
Rust returns `Vec<u8>` across the bridge. Flutter uses `Image.memory(bytes)`.
|
||||
|
||||
```dart
|
||||
final bytes = await api.downloadAvatar(clientUid: uid);
|
||||
if (bytes != null && bytes.isNotEmpty) {
|
||||
return Image.memory(Uint8List.fromList(bytes));
|
||||
} else {
|
||||
return CircleAvatar(child: Text(initials)); // fallback
|
||||
}
|
||||
```
|
||||
|
||||
Flutter's `ImageCache` caches the decoded image in memory automatically. Same avatar bytes = cache hit in memory.
|
||||
|
||||
**Option B: File path across bridge (better for large images, future)**
|
||||
|
||||
Rust writes to disk and returns the file path. Flutter uses `FileImage`.
|
||||
|
||||
```dart
|
||||
final path = await api.getAvatarPath(avatarHash: hash);
|
||||
if (path != null) {
|
||||
return Image.file(File(path));
|
||||
} else {
|
||||
return CircleAvatar(child: Text(initials));
|
||||
}
|
||||
```
|
||||
|
||||
`FileImage` does not watch for file changes. When the hash changes, the UI must evict the old entry from `ImageCache` using `PaintingBinding.instance.imageCache.evict(key)`.
|
||||
|
||||
**Recommendation:** Start with Option A for MVP. It avoids file-path cross-platform complications and works well for small avatar files. The bridge already returns `Vec<u8>` for the download result.
|
||||
|
||||
## 11. Implementation Sequence
|
||||
|
||||
| Phase | Scope | What |
|
||||
|---|---|---|
|
||||
| Phase 1 | Protocol download | `Request::DownloadFile`, `StreamItem::FileDownload` handling, `ProtocolClient::download_avatar()` / `download_icon()`. No caching. |
|
||||
| Phase 2 | Bridge + Flutter display | Bridge `downloadAvatar()`, Flutter `Image.memory()`, initials fallback. Still no caching — every view re-downloads. |
|
||||
| Phase 3 | Rust disk cache | New `chanora_cache` crate: `cacache`-backed content-addressed blob store (`BlobCache`), key validation, mtime-based eviction, `init_cache` bridge call. |
|
||||
| Phase 4 | Session orchestration | `chanora_core` `FileTransferService`: request coalescing, rate limiter (semaphore), negative cache (5 min TTL), retry backoff. |
|
||||
| Phase 5 | Cache management | Bridge `clearFileCache()` + `fileCacheSize()`, Flutter settings UI, eviction on startup. |
|
||||
|
||||
Phase 1 and 2 deliver visible value (avatars in the UI). Phase 3-5 add robustness.
|
||||
|
||||
## 12. References
|
||||
|
||||
| Reference | Use |
|
||||
|---|---|
|
||||
| `ReSpeak/tsdeclarations` `Messages.toml` lines 828-830 | `ftinitdownload` command declaration |
|
||||
| `ReSpeak/tsdeclarations` `Messages.toml` lines 590 | `FileDownload` response structure |
|
||||
| `ReSpeak/tsdeclarations` `ts3protocol.md` | Low-level TeamSpeak protocol specification |
|
||||
| `ReSpeak/tsclientlib` `src/lib.rs` lines 956-1005 | `download_file` / `upload_file` public API |
|
||||
| `ReSpeak/tsclientlib` `src/lib.rs` lines 1371-1427 | `StreamItem::FileDownload` handling |
|
||||
| `ReSpeak/tsclientlib` `src/lib.rs` lines 1630-1672 | Outgoing init commands |
|
||||
| `Multivit4min/TS3-NodeJS-Library` `src/transport/FileTransfer.ts` | Reference TCP transfer implementation |
|
||||
| `Speckmops/ts3admin.class` `lib/ts3admin.class.php` lines 1352-1370 | Reference avatar download flow |
|
||||
| `docs/architecture/sad.md` SAD-067, SDD-MOD-009 | Protocol adapter boundary rules |
|
||||
| `crates/chanora_protocol/src/adapter.rs` lines 1561-1568 | Existing `uid_to_avatar_path` implementation |
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,770 @@
|
||||
# File Transfer Cache Research
|
||||
|
||||
**Date:** 2026-06-10
|
||||
**Status:** Research complete, design implications noted (updated with TeaSpeak server findings)
|
||||
**Companion to:** `docs/architecture/file-transfer-design.md`
|
||||
**Purpose:** Factual findings from protocol analysis, existing client implementations, and cross-platform research that inform the cache architecture decision.
|
||||
|
||||
---
|
||||
|
||||
## 1. TS3 Protocol Identity Semantics
|
||||
|
||||
### 1.1 Avatar Identity
|
||||
|
||||
| Aspect | Value |
|
||||
|---|---|
|
||||
| Protocol field | `client_flag_avatar` |
|
||||
| Type | `TYPE_STRING` (TeaSpeakLibrary `PropertyDefinition.h:200`) |
|
||||
| Meaning | MD5 hash of the avatar file bytes |
|
||||
| Scope | Per-client per-server — a user can have different avatars on different servers |
|
||||
| Freshness | Automatically up-to-date for any client "in view" (`FLAG_CLIENT_VIEW`) |
|
||||
| Empty value | No avatar set |
|
||||
|
||||
**Key fact:** Identical avatar image bytes produce the **same** `client_flag_avatar` hash on any TS3 server. The hash is a content fingerprint, not a server-assigned identifier.
|
||||
|
||||
**Download path on server:** `/avatar_<base64HashClientUID>` — the filename is derived from the client's unique identifier (UID), not from the content hash. The content hash is communicated separately via `client_flag_avatar`.
|
||||
|
||||
**Sources:**
|
||||
- TeaSpeakLibrary `PropertyDefinition.h:200`: `PropertyDescription{CLIENT_FLAG_AVATAR, "client_flag_avatar", "", TYPE_STRING, FLAG_CLIENT_VIEW | FLAG_SAVE | FLAG_USER_EDITABLE}`
|
||||
- TS3AudioBot avatar upload: computes MD5 of image bytes, then sets `client_flag_avatar` to that hash ([`TS3AudioBot/TSLib/TsBaseFunctions.cs:324-341`](https://github.com/Splamy/TS3AudioBot/blob/a69a38d8cba5a4d671dbe06505506f6b46f1d947/TSLib/TsBaseFunctions.cs#L324-L341))
|
||||
- TS3 NodeJS Library: avatar filename is `avatar_${clientBase64HashClientUID}` ([`TS3-NodeJS-Library/src/node/Client.ts:300-313`](https://github.com/Multivit4min/TS3-NodeJS-Library/blob/0c69b7ee80fa5b74e9175cf4ae3018346f7eb300/src/node/Client.ts#L300-L313))
|
||||
- TS3 PHP Framework: avatar name derivation from UID ([`ts3phpframework/src/Node/Client.php:288-307`](https://github.com/planetteamspeak/ts3phpframework/blob/87046b3d493c4d3d8064c639ea4269571192e476/src/Node/Client.php#L288-L307))
|
||||
|
||||
### 1.2 Icon Identity
|
||||
|
||||
| Aspect | Value |
|
||||
|---|---|
|
||||
| Protocol fields | `channel_icon_id`, `client_icon_id`, `virtualserver_icon_id` |
|
||||
| Type | `TYPE_UNSIGNED_NUMBER` (TeaSpeakLibrary `PropertyDefinition.h:83,146,217`) |
|
||||
| Meaning | CRC32 (unsigned) of the icon file bytes |
|
||||
| Scope | Per-entity per-server — but CRC32 is content-derived |
|
||||
| Download path | `/icon_<unsigned_crc32>` |
|
||||
|
||||
**Key fact:** Identical icon bytes produce the **same** CRC32 on any TS3 server. The icon ID is a content fingerprint. The upload process computes `crc32.unsigned(data)` and stores at `/icon_<id>`.
|
||||
|
||||
**CRC32 collision caveat:** CRC32 is only 32 bits. Different icon content can theoretically produce the same CRC32. Qint's `filecache.rs` explicitly notes this: "there could be collisions because only CRC-32 is used." ForChanora's purposes (small icons, not security-critical), this is acceptable.
|
||||
|
||||
**Sources:**
|
||||
- TeaSpeakLibrary `PropertyDefinition.h:83,146,217`: all icon IDs are `TYPE_UNSIGNED_NUMBER`
|
||||
- TS3 NodeJS Library `uploadIcon()`: computes `crc32.unsigned(data)`, uploads to `/icon_<id>` ([`TS3-NodeJS-Library/src/TeamSpeak.ts:2234-2241`](https://github.com/Multivit4min/TS3-NodeJS-Library/blob/0c69b7ee80fa5b74e9175cf4ae3018346f7eb300/src/TeamSpeak.ts#L2234-L2241))
|
||||
- TS3 PHP Framework: icon path uses `/icon_<unsigned id>` ([`ts3phpframework/src/Node/Node.php:137-146`](https://github.com/planetteamspeak/ts3phpframework/blob/87046b3d493c4d3d8064c639ea4269571192e476/src/Node/Node.php#L137-L146))
|
||||
- TS3 community forum: "The filename itself is the result of the CRC32 checksum" (TeamSpeak staff)
|
||||
|
||||
### 1.3 Implication for Cache Design
|
||||
|
||||
Both avatars and icons are **content-addressed by the protocol itself**:
|
||||
|
||||
| Asset | Content hash source | Same content across servers? |
|
||||
|---|---|---|
|
||||
| Avatar | `client_flag_avatar` = MD5 of bytes | Same bytes → same hash → same ID |
|
||||
| Icon | `icon_id` = CRC32 of bytes | Same bytes → same CRC32 → same ID |
|
||||
|
||||
This means a **flat content-addressed blob store** can achieve zero-duplication without any per-server directories, hardlinks, or ref-counting.
|
||||
|
||||
---
|
||||
|
||||
## 2. Virtual Server Identity
|
||||
|
||||
### 2.1 Server UID
|
||||
|
||||
| Aspect | Value |
|
||||
|---|---|
|
||||
| Protocol field | `virtualserver_unique_identifier` |
|
||||
| Type | `TYPE_STRING` (TeaSpeakLibrary `PropertyDefinition.h:22`) |
|
||||
| Generated by | The server instance, on creation |
|
||||
| Globally unique? | **Not guaranteed** — locally generated, no central registry |
|
||||
| Stable? | Yes — persists across restarts of the same virtual server |
|
||||
|
||||
**Key fact:** `virtualserver_unique_identifier` is generated by each TS3 server. Two physically different servers could theoretically produce the same UID. It is **not safe as a global cache key**.
|
||||
|
||||
### 2.2 What Chanora Currently Tracks
|
||||
|
||||
| Layer | Server identity fields | Source |
|
||||
|---|---|---|
|
||||
| Protocol adapter (`adapter.rs:1752-1755`) | `server_name`, `welcome_message`, `platform`, `version` | `state.server.*` from tsclientlib |
|
||||
| DTO (`ServerSnapshot`) | `server_name`, `welcome_message`, `platform`, `version` | No UID field |
|
||||
| Bridge (`BridgeSnapshot`) | Same as DTO | Same |
|
||||
| Storage (bookmarks) | Keyed by `host` (hostname:port) | SQLite `WHERE host = ?1` |
|
||||
| Core (recent servers) | `cfg.address` as host | Auto-saved on connect |
|
||||
|
||||
**Chanora does not currently plumb `virtualserver_unique_identifier` through the DTO stack.** The field exists in tsclientlib's state but is not extracted.
|
||||
|
||||
### 2.3 Implication for Cache Design
|
||||
|
||||
Using `virtualserver_unique_identifier` as the sole cache key is risky (not globally unique). Using connection address (`host:port`) is safe but duplicates cache entries when the same server is accessed via different addresses.
|
||||
|
||||
**Recommendation:** For a content-addressed blob store, server identity is only needed for per-server metadata (eviction, "clear cache for this server"), not for the blob key itself. The blob key is the content hash.
|
||||
|
||||
---
|
||||
|
||||
## 3. Existing TS3 Client Cache Implementations
|
||||
|
||||
### 3.1 Qint (tsclientlib-based, Tauri + Rust)
|
||||
|
||||
**Architecture:** Per-server directory with SQLite metadata.
|
||||
|
||||
```
|
||||
<cache>/files/<server-uid>/<channel-id>/<base64(path)>
|
||||
```
|
||||
|
||||
**Avatar handling:**
|
||||
- Avatar state stored per `(server, client)` row in SQLite
|
||||
- On avatar hash change, deletes the cached `/avatar_<uid>` file for that server
|
||||
- Avatar download path: `/avatar_<uid_base64>`
|
||||
|
||||
**Icon handling:**
|
||||
- Icons path-cached with CRC32
|
||||
- Code comments note CRC32 collisions and freshness by mtime
|
||||
- Qint explicitly deletes and re-downloads when icon mtime changes
|
||||
|
||||
**Dedup:** None. Same avatar on 5 servers = 5 stored copies.
|
||||
|
||||
**Sources:**
|
||||
- [`Qint/proxy/src/filecache.rs`](https://github.com/ReSpeak/Qint/blob/7efe949adfa1a1ecb9d185e7740e015da18cc41b/proxy/src/filecache.rs#L1-L6): "Stores files transferred via the TS3 file transfer protocol. This includes icons and avatars."
|
||||
- [`Qint/proxy/src/db/mod.rs`](https://github.com/ReSpeak/Qint/blob/7efe949adfa1a1ecb9d185e7740e015da18cc41b/proxy/src/db/mod.rs#L1158-L1176): avatar hash change triggers delete
|
||||
- [`Qint/src-tauri/src/cmd.rs`](https://github.com/ReSpeak/Qint/blob/7efe949adfa1a1ecb9d185e7740e015da18cc41b/src-tauri/src/cmd.rs#L524-L539): file download command
|
||||
|
||||
### 3.2 TS3 Official Client (closed source)
|
||||
|
||||
**Architecture:** Lazy cache with SDK callbacks.
|
||||
|
||||
- `getAvatar()` returns cached path if present; otherwise triggers download
|
||||
- `onAvatarUpdated` callback fires when avatar is downloaded or deleted
|
||||
- Cache paths (from community documentation):
|
||||
- Windows: `%LOCALAPPDATA%\TeamSpeak\Cache\Default`
|
||||
- Linux: `~/.cache/TeamSpeak/Default`
|
||||
- macOS: `~/Library/Caches/TeamSpeak/Default`
|
||||
- SDK also exposes `CLIENT_MYTS_AVATAR` / `client_myteamspeak_avatar` for cross-server myTeamSpeak avatars
|
||||
|
||||
**Sources:**
|
||||
- [`ts3client-pluginsdk/src/plugin.c`](https://github.com/teamspeak/ts3client-pluginsdk/blob/4aa90a53aa150cbf81e13bc97e68c0431b26499f/src/plugin.c#L384-L396): `getAvatar()` and `onAvatarUpdated`
|
||||
- [`ts3client-pluginsdk/public_rare_definitions.h`](https://github.com/teamspeak/ts3client-pluginsdk/blob/4aa90a53aa150cbf81e13bc97e68c0431b26499f/include/teamspeak/public_rare_definitions.h#L284-L313): `CLIENT_FLAG_AVATAR`, `CLIENT_MYTS_AVATAR`
|
||||
- Community: [clear cache](https://community.teamspeak.com/t/clear-cache/41511), [broken icons](https://community.teamspeak.com/t/server-icons-are-displaying-a-broken-image-issues-with-local-cache/58680)
|
||||
|
||||
### 3.3 TeaSpeak Client (TypeScript + C++ native)
|
||||
|
||||
**Architecture:** Browser Cache API for images, per-server own-avatar storage.
|
||||
|
||||
**Key components (from `.d.ts` type declarations):**
|
||||
|
||||
- `AvatarManager` — per-connection (`FileManager`) avatar handler
|
||||
- `cachedAvatars` (private) — in-memory cache of `ClientAvatar` objects
|
||||
- `updateCache(clientAvatarId, clientAvatarHash)` — updates cache when hash changes
|
||||
- `resolveAvatar(clientAvatarId, avatarHash?, cacheOnly?)` — resolves avatar by ID
|
||||
- `flush_cache()` — clears cache
|
||||
- `create_avatar_download(client_avatar_id)` — initiates file transfer
|
||||
|
||||
- `ClientAvatar` — tracks individual avatar state
|
||||
- `clientAvatarId` — derived from client UID via `uniqueId2AvatarId()`
|
||||
- `currentAvatarHash` — the `client_flag_avatar` value
|
||||
- State machine: `unset` → `loading` → `loaded` / `errored`
|
||||
- `loadingTimestamp` — when download started
|
||||
|
||||
- `ImageCache` — generic image cache using browser Cache API
|
||||
- `resolveCached(key, maxAge?)` — check if cached
|
||||
- `putCache(key, value, type?, headers?)` — store
|
||||
- `cleanup(maxAge)` — evict old entries
|
||||
- `reset()` — clear all
|
||||
- `isPersistent()` — whether cache persists to disk
|
||||
|
||||
- `OwnAvatarStorage` — user's own avatar, keyed by `serverUniqueId + mode`
|
||||
- `loadAvatarImage(serverUniqueId, mode)` — load own avatar for a server
|
||||
- `updateAvatar(serverUniqueId, mode, target)` — update own avatar
|
||||
- `avatarUploadSucceeded(serverUniqueId)` — move from "uploading" to "server" state
|
||||
- Stores `LocalAvatarInfo`: fileName, fileSize, **fileHashMD5**, timestamps, contentType
|
||||
|
||||
- `FileManager` — per-connection file transfer manager
|
||||
- `MAX_CONCURRENT_TRANSFERS` — transfer concurrency limit
|
||||
- `avatars: AvatarManager` — avatar subsystem
|
||||
- `initializeFileDownload(options)` — start download (path, name, channel, target)
|
||||
- `deleteIcon(iconId: number)` — delete icon by ID
|
||||
|
||||
- `FileTransfer` — transfer state machine
|
||||
- States: `PENDING → INITIALIZING → CONNECTING → RUNNING → FINISHED / ERRORED / CANCELED`
|
||||
- `InitializedTransferProperties`: serverTransferId, transferKey, **addresses[]**, protocol, seekOffset, fileSize
|
||||
- Multiple addresses returned by server for file transfer (failover)
|
||||
|
||||
- `localIconCache: ImageCache` — global icon cache (singleton)
|
||||
|
||||
**Sources:**
|
||||
- TeaSpeak-Client `imports/shared-app/file/Avatars.d.ts` — ClientAvatar, AbstractAvatarManager
|
||||
- TeaSpeak-Client `imports/shared-app/file/LocalAvatars.d.ts` — AvatarManager
|
||||
- TeaSpeak-Client `imports/shared-app/file/LocalIcons.d.ts` — localIconCache
|
||||
- TeaSpeak-Client `imports/shared-app/file/ImageCache.d.ts` — ImageCache (browser Cache API)
|
||||
- TeaSpeak-Client `imports/shared-app/file/FileManager.d.ts` — FileManager, transfer API
|
||||
- TeaSpeak-Client `imports/shared-app/file/Transfer.d.ts` — FileTransfer, state machine, error types
|
||||
- TeaSpeak-Client `imports/shared-app/file/OwnAvatarStorage.d.ts` — own avatar per-server storage
|
||||
- TeaSpeak-Client `native/serverconnection/test/js/ft.ts` — file transfer test (TCP + ftkey protocol)
|
||||
|
||||
### 3.4 TS3AudioBot (C#)
|
||||
|
||||
**Architecture:** No local avatar cache. Avatar upload is hash-driven.
|
||||
|
||||
- Uploads avatar bytes to `/avatar`, computes MD5, sets `client_flag_avatar` to that hash
|
||||
- Bot avatar selection reads local files from an `avatars/` directory
|
||||
- No caching of other users' avatars
|
||||
|
||||
**Sources:**
|
||||
- [`TS3AudioBot/TSLib/TsBaseFunctions.cs:324-341`](https://github.com/Splamy/TS3AudioBot/blob/a69a38d8cba5a4d671dbe06505506f6b46f1d947/TSLib/TsBaseFunctions.cs#L324-L341)
|
||||
- [`TS3AudioBot/Bot.cs:420-470`](https://github.com/Splamy/TS3AudioBot/blob/a69a38d8cba5a4d671dbe06505506f6b46f1d947/TS3AudioBot/Bot.cs#L420-L470)
|
||||
|
||||
### 3.5 TS3 NodeJS Library
|
||||
|
||||
**Architecture:** No local cache. Downloads on demand.
|
||||
|
||||
- Avatar filename: `avatar_${clientBase64HashClientUID}`
|
||||
- `getAvatar()` downloads directly — no caching layer
|
||||
- Tests assert the exact `/avatar_<base64uid>` path
|
||||
|
||||
**Sources:**
|
||||
- [`TS3-NodeJS-Library/src/node/Client.ts:300-313`](https://github.com/Multivit4min/TS3-NodeJS-Library/blob/0c69b7ee80fa5b74e9175cf4ae3018346f7eb300/src/node/Client.ts#L300-L313)
|
||||
- [`TS3-NodeJS-Library/tests/Client.spec.ts:342-358`](https://github.com/Multivit4min/TS3-NodeJS-Library/blob/0c69b7ee80fa5b74e9175cf4ae3018346f7eb300/tests/Client.spec.ts#L342-L358)
|
||||
|
||||
### 3.6 Summary Table
|
||||
|
||||
| Client | Cache Key Strategy | Dedup Across Servers? | Icon Cache |
|
||||
|---|---|---|---|
|
||||
| **Qint** | `<server-uid>/<channel-id>/<path>` | No | Yes (CRC32, mtime freshness) |
|
||||
| **TS3 Official** | Lazy cache (path-based) | Unknown | Yes |
|
||||
| **TeaSpeak** | UID-derived avatar ID + browser Cache API | Implicit (same hash = same cache) | Yes (global ImageCache) |
|
||||
| **TS3AudioBot** | None | N/A | No |
|
||||
| **TS3 NodeLib** | None | N/A | No |
|
||||
| **Chanora (decided)** | Content hash (MD5/CRC32), flat `blobs/` in `chanora_cache` crate | **Yes** | Yes |
|
||||
|
||||
---
|
||||
|
||||
## 4. TeaSpeak Protocol Definitions (Authoritative)
|
||||
|
||||
From TeaSpeakLibrary `src/PropertyDefinition.h` — the most complete open-source reference for TS3 protocol property types:
|
||||
|
||||
### 4.1 Avatar Properties
|
||||
|
||||
```cpp
|
||||
// Line 200
|
||||
PropertyDescription{CLIENT_FLAG_AVATAR, "client_flag_avatar", "",
|
||||
TYPE_STRING, FLAG_CLIENT_VIEW | FLAG_SAVE | FLAG_USER_EDITABLE}
|
||||
// "automatically up-to-date for any manager 'in view', this manager got an avatar"
|
||||
```
|
||||
|
||||
### 4.2 Icon Properties
|
||||
|
||||
```cpp
|
||||
// Line 83 — server icon
|
||||
PropertyDescription{VIRTUALSERVER_ICON_ID, "virtualserver_icon_id", "0",
|
||||
TYPE_UNSIGNED_NUMBER, FLAG_SERVER_VVSS | FLAG_USER_EDITABLE}
|
||||
|
||||
// Line 146 — channel icon
|
||||
PropertyDescription{CHANNEL_ICON_ID, "channel_icon_id", "0",
|
||||
TYPE_UNSIGNED_NUMBER, FLAG_CHANNEL_VIEW | FLAG_SS | FLAG_USER_EDITABLE}
|
||||
|
||||
// Line 217 — client icon
|
||||
PropertyDescription{CLIENT_ICON_ID, "client_icon_id", "0",
|
||||
TYPE_UNSIGNED_NUMBER, FLAG_CLIENT_VIEW | FLAG_CLIENT_VARIABLE}
|
||||
```
|
||||
|
||||
### 4.3 Server Identity
|
||||
|
||||
```cpp
|
||||
// Line 22
|
||||
PropertyDescription{VIRTUALSERVER_UNIQUE_IDENTIFIER,
|
||||
"virtualserver_unique_identifier", "",
|
||||
TYPE_STRING, FLAG_SERVER_VV | FLAG_SNAPSHOT}
|
||||
```
|
||||
|
||||
### 4.4 File Transfer Permissions
|
||||
|
||||
```cpp
|
||||
// From PermissionManager.cpp
|
||||
PermissionType::i_client_max_avatar_filesize // "Max avatar filesize in bytes"
|
||||
PermissionType::b_client_avatar_delete_other // "Allow deletion of avatars from other clients"
|
||||
PermissionType::b_ft_transfer_list // "Retrieve list of running filetransfers"
|
||||
```
|
||||
|
||||
### 4.5 File Transfer Error Codes
|
||||
|
||||
```cpp
|
||||
// From Error.h
|
||||
channel_no_filetransfer_supported = 0x30C
|
||||
file_transfer_connection_timeout = 0x80E
|
||||
file_transfer_complete = 0x811
|
||||
file_transfer_canceled = 0x812
|
||||
file_transfer_interrupted = 0x813
|
||||
file_transfer_server_quota_exceeded = 0x814
|
||||
file_transfer_client_quota_exceeded = 0x815
|
||||
file_transfer_reset = 0x816
|
||||
file_transfer_limit_reached = 0x817
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Platform Filesystem Research
|
||||
|
||||
### 5.1 Hardlink Support
|
||||
|
||||
| Platform | Filesystem | Hardlinks in App-Private Storage? | Gotcha |
|
||||
|---|---|---|---|
|
||||
| Android (API 28+) | ext4 / f2fs | **Yes** | Rust uses `libc::link`; not FUSE-mounted; same-filesystem only |
|
||||
| iOS | APFS | **Yes** (writable sandbox dirs) | App bundle is read-only; avoid hardlinks to bundle assets |
|
||||
| macOS | APFS | **Yes** | — |
|
||||
| Linux | ext4 / btrfs / xfs | **Yes** | — |
|
||||
| Windows | NTFS | **Yes** | Rust uses `CreateHardLinkW` |
|
||||
|
||||
**`std::fs::hard_link` gotchas (all platforms):**
|
||||
- Same filesystem required
|
||||
- Destination must not exist (returns error)
|
||||
- Symlink behavior is platform-specific
|
||||
- All hardlinks share the same inode — modifying one modifies all
|
||||
- Files must be treated as **immutable** for hardlink safety
|
||||
|
||||
**Sources:**
|
||||
- Rust stdlib: `hard_link` maps to `libc::link` (Unix), `CreateHardLinkW` (Windows) ([Rust source](https://github.com/rust-lang/rust/blob/beae781308e9ddef13074a03faf57ca2fac59a5b/library/std/src/fs.rs#L2898-L2900))
|
||||
- Android: internal storage uses ext4/f2fs, not FUSE ([Android scoped storage docs](https://source.android.com/docs/core/storage/scoped))
|
||||
- iOS: APFS supports hardlinks; writable sandbox directories work ([Apple FileSystem basics](https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html))
|
||||
|
||||
### 5.2 Rust Cache Libraries
|
||||
|
||||
**`cacache`** (MIT licensed, production-ready):
|
||||
- Content-addressed disk cache
|
||||
- Automatic dedup, atomic writes, integrity verification
|
||||
- Exposes `hard_link`, `copy`, and `reflink` paths for retrieval
|
||||
- On-disk layout: `content-v2/sha512/...`
|
||||
- Could replace a custom implementation, but adds a dependency
|
||||
|
||||
**Sources:**
|
||||
- [`cacache-rs` README](https://github.com/zkat/cacache-rs/blob/105692a4daa04ce5f5ef3f8688cd3e1c1fb6a7c0/README.md#L39-L60)
|
||||
- [`cacache-rs` content path](https://github.com/zkat/cacache-rs/blob/105692a4daa04ce5f5ef3f8688cd3e1c1fb6a7c0/src/content/path.rs#L6-L19)
|
||||
- [`cacache-rs` hard_link impl](https://github.com/zkat/cacache-rs/blob/105692a4daa04ce5f5ef3f8688cd3e1c1fb6a7c0/src/content/read.rs#L257-L285)
|
||||
|
||||
### 5.3 Flutter Cache Patterns
|
||||
|
||||
Common Flutter packages use **cache-dir + metadata DB**, not hardlink dedup:
|
||||
- `flutter_cache_manager`: files in cache dir + `sqflite` metadata
|
||||
- `super_cache_disk`: file-per-entry (`.dat` + `.meta`) in app cache dir
|
||||
|
||||
**Sources:**
|
||||
- [flutter_cache_manager on pub.dev](https://pub.dev/packages/flutter_cache_manager)
|
||||
- [super_cache_disk on pub.dev](https://pub.dev/packages/super_cache_disk/versions/1.0.0)
|
||||
|
||||
---
|
||||
|
||||
## 6. Chanora Codebase Context
|
||||
|
||||
### 6.1 Storage Patterns
|
||||
|
||||
| Component | Pattern | Location |
|
||||
|---|---|---|
|
||||
| Identity storage | Atomic write (temp + `sync_all` + `rename`), mode 0600 on Unix | `chanora_storage/src/lib.rs:446-476` |
|
||||
| Metadata | Same atomic write pattern | `chanora_storage/src/lib.rs:480-515` |
|
||||
| Bookmarks | SQLite at `<storage_dir>/chanora.db`, keyed by `host` | `chanora_storage/src/lib.rs:782-928` |
|
||||
| Storage root | `getApplicationSupportDirectory()` from Flutter | `app_bootstrap.dart:19-24` |
|
||||
| Bridge init | `rust.initStorage(dir: dir)` | `app_bootstrap.dart:23-25,108-133` |
|
||||
|
||||
### 6.2 Existing Avatar Handling
|
||||
|
||||
| Component | What | Location |
|
||||
|---|---|---|
|
||||
| UID to avatar path | `uid_to_avatar_path()` — base64 decode UID, encode each byte as 2 chars (a-p) | `adapter.rs:1561-1568` |
|
||||
| Client profile DTO | `avatar_path` field — set when `client.avatar_hash` and `unique_id` non-empty | `dto.rs:135-136`, `adapter.rs:1323-1332` |
|
||||
| No download | Currently no file download implementation exists | — |
|
||||
| No icon handling | No icon field/path in protocol DTO or adapter | — |
|
||||
|
||||
### 6.3 Server Identity in Chanora
|
||||
|
||||
Chanora currently tracks servers by **connection address** (`host:port`), not by server UID:
|
||||
|
||||
- Bookmarks: `WHERE host = ?1`
|
||||
- Recent servers: auto-saved by `cfg.address`
|
||||
- Prefetch cache: keyed by normalized host
|
||||
- ServerSnapshot: has `server_name` but no `server_uid`
|
||||
|
||||
The `virtualserver_unique_identifier` field is available from tsclientlib's state but is **not extracted** by the adapter.
|
||||
|
||||
### 6.4 tsclientlib File Transfer API
|
||||
|
||||
| Aspect | Detail |
|
||||
|---|---|
|
||||
| Download method | `Connection::download_file()` |
|
||||
| Stream items | `StreamItem::FileDownload(FileDownloadResult { size, stream })` |
|
||||
| Failure | `StreamItem::FiletransferFailed(handle, error)` |
|
||||
| TCP handling | tsclientlib handles TCP connection + ftkey writing automatically |
|
||||
| Chanora's job | Read `size` bytes from the returned `TcpStream` |
|
||||
| Async behavior | `StreamItem::FileDownload` fires asynchronously, not inline with the request |
|
||||
|
||||
### 6.5 ts-bookkeeping Generated Fields
|
||||
|
||||
From the generated parser in `target/debug/build/ts-bookkeeping-*/out/`:
|
||||
|
||||
- `virtual_server_id: u64` — numeric, per-virtual-server, may change across restarts
|
||||
- `virtual_server_uid` — string, the `virtualserver_unique_identifier`
|
||||
|
||||
Both are available in the `InInitServer` struct from the init handshake but are not currently plumbed through.
|
||||
|
||||
---
|
||||
|
||||
## 7. TeaSpeak Server Internals (Authoritative)
|
||||
|
||||
Source: TeaSpeak Server at `https://git.did.science/TeaSpeak/Server/Server` (branch `new-groups`, commit `b54c6d4e`).
|
||||
|
||||
### 7.1 Avatar ID Derivation — Server Side
|
||||
|
||||
The server derives the avatar filename from the **client UID**, not from the avatar content:
|
||||
|
||||
```cpp
|
||||
// DataClient.cpp:242-244
|
||||
std::string DataClient::getAvatarId() {
|
||||
return hex::hex(base64::validate(this->getUid()) ? base64::decode(this->getUid()) : this->getUid(), 'a', 'q');
|
||||
}
|
||||
```
|
||||
|
||||
The same transform produces `client_base64HashClientUID` (shown to other clients):
|
||||
|
||||
```cpp
|
||||
// client.cpp:1113-1114
|
||||
bulk.put_unchecked("client_base64HashClientUID",
|
||||
hex::hex(base64::validate(info->client_unique_id) ? base64::decode(info->client_unique_id) : info->client_unique_id, 'a', 'q'));
|
||||
```
|
||||
|
||||
**This matches Chanora's existing `uid_to_avatar_path()` in `adapter.rs:1561-1568`.**
|
||||
|
||||
### 7.2 Avatar Upload Path
|
||||
|
||||
When a client uploads an avatar, the server stores it as `/avatar_<avatarId>`:
|
||||
|
||||
```cpp
|
||||
// file.cpp:696-702
|
||||
} else if (cmd["path"].as<std::string>().empty() && cmd["name"].string() == "/avatar") {
|
||||
...
|
||||
info.file_path = "/avatar_" + this->getAvatarId();
|
||||
transfer_response = file::server()->file_transfer().initialize_avatar_transfer(...);
|
||||
}
|
||||
```
|
||||
|
||||
The avatar file path is identity-based (from UID), not content-based.
|
||||
|
||||
### 7.3 `client_flag_avatar` — Who Computes the Hash?
|
||||
|
||||
**The CLIENT computes the MD5 and sends it to the server during upload.** The server stores it as a string property (`FLAG_USER_EDITABLE`). The server does NOT compute or verify the hash.
|
||||
|
||||
This means `client_flag_avatar` is:
|
||||
- Set by the uploading client
|
||||
- Stored verbatim by the server
|
||||
- Broadcast to other clients as part of the client properties
|
||||
- A reliable content fingerprint: same avatar bytes → same MD5 → same `client_flag_avatar` on any server
|
||||
|
||||
### 7.4 Icon IDs — Server Does NOT Compute CRC32
|
||||
|
||||
Icon IDs are **permission values**, not content hashes computed by the server:
|
||||
|
||||
```cpp
|
||||
// ConnectedClient.cpp:186-210 — client icon ID from permissions
|
||||
auto permission_flags = local_permissions->permission_flags(permission::i_icon_id);
|
||||
new_icon_id = value.value;
|
||||
updated_client_properties.emplace_back(property::CLIENT_ICON_ID);
|
||||
```
|
||||
|
||||
```cpp
|
||||
// channel.cpp:1495-1504 — channel icon ID
|
||||
if(key == property::CHANNEL_ICON_ID) {
|
||||
auto icon_id = converter<uint32_t>::from_string_view(value);
|
||||
channel->permissions()->set_permission(permission::i_icon_id, { ... icon_id ... });
|
||||
}
|
||||
```
|
||||
|
||||
```cpp
|
||||
// server.cpp:76-89 — server icon ID
|
||||
SERVEREDIT_CHK_PROP_CACHED("virtualserver_icon_id", permission::b_virtualserver_modify_icon_id, int64_t)
|
||||
```
|
||||
|
||||
**The CLIENT computes the CRC32 during upload and uses it as the filename `/icon_<crc32>`.** The server stores the file and records the ID as a permission value. No server-side CRC32 or MD5 computation exists.
|
||||
|
||||
### 7.5 Per-Server Storage Layout
|
||||
|
||||
Avatars and icons are stored **per virtual server** on the server's filesystem:
|
||||
|
||||
```cpp
|
||||
// LocalFileSystem.cpp:39-45
|
||||
fs::path LocalFileSystem::server_path(const std::shared_ptr<VirtualFileServer> &server) {
|
||||
return fs::u8path(this->root_path_) / fs::u8path("server_" + std::to_string(server->server_id()));
|
||||
}
|
||||
// target_path = this->server_path(server) / "icons" / path;
|
||||
// target_path = this->server_path(server) / "avatars" / path;
|
||||
```
|
||||
|
||||
```
|
||||
<server_root>/
|
||||
server_<sid>/
|
||||
avatars/
|
||||
/avatar_<avatarId> ← one per client who uploaded
|
||||
icons/
|
||||
/icon_<id> ← one per unique icon
|
||||
```
|
||||
|
||||
### 7.6 File Transfer Protocol (Server Side)
|
||||
|
||||
Upload/delete/query routing:
|
||||
|
||||
```cpp
|
||||
// file.cpp:273-341 — delete routing
|
||||
if (first_entry_name.find("/icon_") == 0 && file_path.empty()) { ... delete_icons(...); }
|
||||
else if (first_entry_name.starts_with("/avatar_") && file_path.empty()) { ... delete_avatars(...); }
|
||||
```
|
||||
|
||||
```cpp
|
||||
// file.cpp:483-523 — query routing
|
||||
if (first_entry_name.find("/icon_") == 0 && file_path.empty()) { ... query_icon_info(...); }
|
||||
else if (first_entry_name.starts_with("/avatar_") && file_path.empty()) { ... query_avatar_info(...); }
|
||||
```
|
||||
|
||||
Transfer initialization returns ftkey and metadata:
|
||||
|
||||
```cpp
|
||||
// file.cpp:759-761
|
||||
result.put_unchecked(0, "ftkey", transfer->transfer_key);
|
||||
result.put_unchecked(0, "seekpos", transfer->file_offset);
|
||||
```
|
||||
|
||||
```cpp
|
||||
// file.cpp:887-899
|
||||
result.put_unchecked(0, "ftkey", transfer->transfer_key);
|
||||
result.put_unchecked(0, "proto", "1");
|
||||
result.put_unchecked(0, "size", transfer->expected_file_size);
|
||||
```
|
||||
|
||||
### 7.7 Key Takeaway for Chanora
|
||||
|
||||
| Who does what | Avatar | Icon |
|
||||
|---|---|---|
|
||||
| **Uploader (client)** computes | MD5 of avatar bytes → sets `client_flag_avatar` | CRC32 of icon bytes → filename `/icon_<crc32>` |
|
||||
| **Server** does | Stores file as `/avatar_<uid>`, saves property | Stores file as `/icon_<id>`, saves permission |
|
||||
| **Other clients** receive | `client_flag_avatar` (MD5) as a property update | `icon_id` (CRC32) as a property update |
|
||||
| **Chanora cache key** | `av_<md5>.dat` — content fingerprint | `ic_<crc32>.dat` — content fingerprint |
|
||||
|
||||
The content hash is computed once (by the uploader) and then broadcast as a property. Chanora never needs to hash anything — it just uses the protocol-provided values as cache keys.
|
||||
|
||||
---
|
||||
|
||||
## 8. Design Implications
|
||||
|
||||
### 8.1 The Core Insight
|
||||
|
||||
The TS3 protocol provides content hashes as part of normal server-to-client updates:
|
||||
|
||||
| Event | Data provided by server | What Chanora gets for free |
|
||||
|---|---|---|
|
||||
| Client enters view | `client_flag_avatar` = MD5 of avatar bytes | Content key for blob store |
|
||||
| Channel update | `channel_icon_id` = CRC32 of icon bytes | Content key for blob store |
|
||||
| Server update | `virtualserver_icon_id` = CRC32 of icon bytes | Content key for blob store |
|
||||
|
||||
No hashing needed on the client side. The protocol is **already content-addressed**.
|
||||
|
||||
### 8.2 Recommended Cache Architecture
|
||||
|
||||
```
|
||||
<app_cache_dir>/chanora/
|
||||
blobs/ ← cacache content store root
|
||||
content-v2/ ← content-addressed by SHA-512
|
||||
<sha512-hex>/data ← raw blob bytes
|
||||
index-v2/ ← key → content mapping
|
||||
```
|
||||
|
||||
Where `<app_cache_dir>` is the platform cache directory (not the support directory used by `chanora_storage`). Chanora's `BlobCache` maps protocol keys (`av_<md5>`, `ic_<crc32>`) to `cacache` string keys. Physical layout is managed by `cacache`.
|
||||
|
||||
**Lookup flow:**
|
||||
1. Server sends `client_flag_avatar = "a1b2c3d4..."` for user X
|
||||
2. Check: does `blobs/av_a1b2c3d4....dat` exist?
|
||||
3. Yes → use it, zero downloads (works for ANY server)
|
||||
4. No → download from `/avatar_<uid_base64>` → save as `blobs/av_a1b2c3d4....dat`
|
||||
|
||||
**Same for icons with `ic_<crc32>.dat`.**
|
||||
|
||||
### 8.3 Why This Beats Alternatives
|
||||
|
||||
| Approach | Dedup | Globally unique key | Needs server UID plumbing | Needs hardlinks | Complexity |
|
||||
|---|---|---|---|---|---|
|
||||
| `<server_uid>/<hash>.dat` | No | No (UID not guaranteed unique) | Yes | Optional | Medium |
|
||||
| `<host>_<port>/<hash>.dat` | No | Yes | No | Optional | Medium |
|
||||
| `<host>_<port>/<hash>.dat` + hardlinks | Yes | Yes | No | Yes | Medium-High |
|
||||
| **`blobs/av_<hash>.dat` (flat)** | **Yes** | **Yes (content hash)** | **No** | **No** | **Low** |
|
||||
|
||||
### 8.4 Trade-offs
|
||||
|
||||
| Pro | Con |
|
||||
|---|---|
|
||||
| Zero duplication across all servers | "Clear cache for server X only" requires metadata layer (Phase 3+) |
|
||||
| No hardlinks needed | Orphan cleanup requires scanning for unreferenced blobs |
|
||||
| No server UID plumbing needed | Cannot distinguish same-hash-different-content for icons (CRC32 collision) |
|
||||
| Simplest possible implementation | — |
|
||||
| Freshness = hash change = different filename (automatic) | — |
|
||||
| Cross-platform (just file I/O) | — |
|
||||
|
||||
### 8.5 Phased Implementation
|
||||
|
||||
| Phase | What | Delivers |
|
||||
|---|---|---|
|
||||
| 1 | Protocol download (raw bytes via adapter, no cache) | Working download pipeline |
|
||||
| 2 | Bridge + Flutter display (`Image.memory()`) | Visible avatars in UI |
|
||||
| 3 | `chanora_cache` crate: cacache-backed blob cache, separate crate, cache dir, mtime eviction | Zero re-downloads, zero duplication |
|
||||
| 4 | Session orchestration (coalescing, rate limiting, negative cache) | Anti-flood, robustness |
|
||||
| 5 | Cache management (clear all, orphan cleanup, optional per-server metadata) | User control |
|
||||
|
||||
---
|
||||
|
||||
## 9. Resolved Questions
|
||||
|
||||
### Q1: Icon CRC32 Collisions — Accept with Size Guard
|
||||
|
||||
**Risk assessment:** CRC32 produces a 32-bit hash. For N unique icons, the Birthday paradox gives collision probability ≈ N² / (2 × 2³²).
|
||||
|
||||
| Icons (N) | Collision probability |
|
||||
|---|---|
|
||||
| 100 | ~0.0001% (negligible) |
|
||||
| 1,000 | ~0.01% (negligible) |
|
||||
| 10,000 | ~1.2% (marginal) |
|
||||
| 65,536 | ~50% (likely) |
|
||||
|
||||
A single user typically encounters fewer than 1,000 unique icons across all servers. The practical collision risk is negligible.
|
||||
|
||||
**What happens on collision:** Wrong icon displayed for a channel/client/server. This is a visual glitch, not a security issue. The icon will appear incorrect until the cache is cleared.
|
||||
|
||||
**Existing practice:** Qint explicitly notes CRC32 collisions (`filecache.rs:4`) but does NOT guard against them — they only refresh icons by mtime. No other TS3 client guards against CRC32 collisions.
|
||||
|
||||
**Recommendation:** Accept CRC32 as the cache key. Add a lightweight **file size guard**: when downloading an icon, if `ic_<crc32>.dat` already exists but has a different size than the `ftinitdownload` response reported, re-download. File size is available from the protocol (`msg.size` in `InFileDownloadPart`). This catches most collisions (different content = different size with high probability) without computing a secondary hash.
|
||||
|
||||
**Decision:** CRC32 + file size guard. No SHA256 overhead needed.
|
||||
|
||||
---
|
||||
|
||||
### Q2: `client_myteamspeak_avatar` — Defer Indefinitely
|
||||
|
||||
**What it is:** A string property (`Option<String>` in ts-bookkeeping) broadcast alongside `client_flag_avatar`. It represents a myTeamSpeak cross-server avatar — a user linked to a myTeamSpeak account can set a global avatar that follows them across all servers.
|
||||
|
||||
**Current state in Chanora's dependency chain:**
|
||||
- ts-bookkeeping exposes it: `InInitServer` has `my_team_speak_avatar: Option<String>`
|
||||
- TeaSpeakLibrary tracks `client_myteamspeak_id` but not the avatar
|
||||
- tsclientlib exposes it as a property on client state
|
||||
|
||||
**Value for Chanora:**
|
||||
- myTeamSpeak is a TeamSpeak-specific cloud service (account sync, cross-server features)
|
||||
- Chanora is an independent client — no myTeamSpeak account integration is planned
|
||||
- The property may contain a URL or identifier that requires myTeamSpeak API access to resolve
|
||||
- Without myTeamSpeak integration, the avatar cannot be fetched
|
||||
|
||||
**Recommendation:** Defer indefinitely. If Chanora ever integrates myTeamSpeak accounts, this can be handled as a separate avatar source (URL-based HTTP download) alongside the existing protocol-based avatar download. The cache architecture supports this — just add a different blob prefix (e.g., `mt_<hash>.dat`).
|
||||
|
||||
**Decision:** Out of scope for MVP and foreseeable roadmap.
|
||||
|
||||
---
|
||||
|
||||
### Q3: Cache Backing Store — `cacache` Wrapper in Separate `chanora_cache` Crate
|
||||
|
||||
**Decision:** Use `cacache` as the backing store inside `chanora_cache`. Not a custom flat-file implementation.
|
||||
|
||||
**Why cacache won over custom:**
|
||||
|
||||
1. **Crash safety is production-tested.** `cacache` handles partial writes, power loss, crash mid-write. A custom implementation would need to get `sync_all` + atomic rename right — one bug = corrupted cache. Even though cache data is disposable (reconstructible from server), `cacache` eliminates this entire class of bugs.
|
||||
|
||||
2. **Less code to maintain.** ~120 LOC wrapper vs ~200 LOC custom implementation. The hard parts (atomic writes, integrity, content dedup) are owned by `cacache`, tested by the npm ecosystem.
|
||||
|
||||
3. **Integrity verification on every read.** SSRI verification detects corruption, bit rot, partial writes automatically. A custom impl would need to add this separately or accept silent corruption.
|
||||
|
||||
4. **Content dedup by SHA-512.** Same avatar on two servers = stored once automatically. The protocol's MD5/CRC32 keys map to `cacache` string keys; content dedup happens at the SHA-512 layer underneath.
|
||||
|
||||
**What about the downsides:**
|
||||
|
||||
| Concern | Assessment |
|
||||
|---|---|
|
||||
| ~6 transitive deps | `sha2` already in tree via `chacha20poly1305`. `serde_json`, `tempfile`, `digest` are lightweight. Acceptable for the safety benefit. |
|
||||
| SHA-512 overhead on every write/read | For <100KB avatars, SHA-512 takes ~0.1ms. Negligible. |
|
||||
| Opaque on-disk format | `cacache` provides `ls()` API for enumeration and inspection. Not as simple as `ls blobs/` but adequate. |
|
||||
| `cacache` has no built-in LRU eviction | We write a custom eviction pass using `cacache::ls()` + timestamp sort. ~20 lines. Same complexity as custom impl's eviction. |
|
||||
|
||||
**Separate crate rationale:**
|
||||
|
||||
- `chanora_cache` is separate from `chanora_storage` because cache data has different durability semantics (disposable vs persistent), different backup semantics (excluded vs included), and different directory placement (cache dir vs support dir).
|
||||
- `chanora_cache` lives in the platform's cache directory (`getApplicationCacheDirectory()`). `chanora_storage` lives in the support directory (`getApplicationSupportDirectory()`).
|
||||
- Bridge init is separate: `init_cache(cache_dir)` vs `init_storage(support_dir)`.
|
||||
|
||||
**API design:**
|
||||
|
||||
```rust
|
||||
pub struct BlobCache { cache_dir: PathBuf, max_bytes: u64 }
|
||||
impl BlobCache {
|
||||
pub fn new(cache_dir: impl AsRef<Path>, max_bytes: u64) -> Result<Self, BlobCacheError>;
|
||||
pub async fn put(&self, prefix: &str, key: &str, data: &[u8]) -> Result<(), BlobCacheError>;
|
||||
pub async fn get(&self, prefix: &str, key: &str) -> Result<Option<Vec<u8>>, BlobCacheError>;
|
||||
pub async fn remove(&self, prefix: &str, key: &str) -> Result<(), BlobCacheError>;
|
||||
pub async fn clear(&self) -> Result<(), BlobCacheError>;
|
||||
pub async fn total_size(&self) -> Result<u64, BlobCacheError>;
|
||||
pub async fn evict(&self) -> Result<(), BlobCacheError>;
|
||||
}
|
||||
```
|
||||
|
||||
All methods are async (cacache is async-native). Key validation at API boundary (`av_` = 32 hex chars, `ic_` = decimal digits).
|
||||
|
||||
---
|
||||
|
||||
### Q4: Per-Blob Metadata — No Metadata Sidecars (Resolved)
|
||||
|
||||
**Original options:**
|
||||
|
||||
| Approach | Pros | Cons |
|
||||
|---|---|---|
|
||||
| SQLite (chanora.db) | ACID, queryable, already in use | Schema migration, couples cache to bookmark DB |
|
||||
| JSON sidecar files | Simple, self-contained, easy to debug | Write amplification (2 files per blob), concurrent write risk |
|
||||
| In-memory only | Simplest | Lost on restart, can't do orphan cleanup offline |
|
||||
| **No metadata (mtime-based)** | **Simplest, zero write amplification, 1 file per blob** | **No per-blob metadata beyond mtime** |
|
||||
|
||||
**Why no metadata is sufficient:**
|
||||
|
||||
1. **Content is immutable.** A given hash (MD5 or CRC32) always maps to the same bytes. There is no "stale content" problem — if the hash changes, it's a new file with a new name. No invalidation needed.
|
||||
|
||||
2. **mtime = insertion time.** Since content is never modified after write, the filesystem mtime equals the time the blob was cached. This is sufficient for "delete oldest files first" eviction.
|
||||
|
||||
3. **Write amplification avoided.** One file per blob (just the data) instead of two (data + JSON sidecar). For a cache that may hold thousands of small files, this matters.
|
||||
|
||||
4. **Eviction is simple.** `walk dir → stat → sort by mtime → delete oldest`. No JSON parsing, no schema, no migration.
|
||||
|
||||
5. **Per-server metadata deferred.** "Clear cache for server X only" and orphan cleanup are post-MVP features. If needed, a refs-layer can be added later without changing the blob layout.
|
||||
|
||||
**Oracle consultation:** Oracle recommended this approach explicitly — no metadata files, mtime-based eviction, separate crate. The immutability guarantee makes metadata redundant.
|
||||
|
||||
**Decision:** No metadata sidecars. One file per blob. Mtime-based eviction. Per-server metadata deferred to post-MVP.
|
||||
|
||||
---
|
||||
|
||||
### Q5: File Transfer Address Failover — Not Needed
|
||||
|
||||
**What the protocol provides:**
|
||||
|
||||
The TeaSpeak client's `InitializedTransferProperties` returns `addresses[]` — an array of `{serverAddress, serverPort}`. The official TS3 client can try multiple addresses for failover.
|
||||
|
||||
**What tsclientlib provides:**
|
||||
|
||||
```rust
|
||||
// tsclientlib/src/lib.rs:1373-1375
|
||||
let ip = msg.ip.unwrap_or_else(|| self.client.address.ip());
|
||||
let addr = SocketAddr::new(ip, msg.port);
|
||||
TcpStream::connect(&addr).await
|
||||
```
|
||||
|
||||
tsclientlib's `InFileDownloadPart` has `ip: Option<IpAddr>` — **single IP only**, not an array. If the server provides an IP, it uses that. Otherwise, it falls back to the connection address. **No multi-address failover.**
|
||||
|
||||
**What ts-bookkeeping parses:**
|
||||
|
||||
```rust
|
||||
pub struct InFileDownloadPart {
|
||||
pub client_filetransfer_id: u16,
|
||||
pub server_filetransfer_id: u16,
|
||||
pub filetransfer_key: String,
|
||||
pub port: u16,
|
||||
pub size: u64,
|
||||
pub protocol: u8,
|
||||
pub ip: Option<IpAddr>, // ← single optional IP
|
||||
}
|
||||
```
|
||||
|
||||
**The server's `notifystartdownload` response** sends `ip` as an optional single value, not an array. The TeaSpeak client's `addresses[]` is a higher-level abstraction (likely the client's own fallback logic), not a protocol feature.
|
||||
|
||||
**Recommendation:** Chanora follows tsclientlib's existing behavior — use `msg.ip` or fallback to connection address. No custom failover logic needed. If the TCP connection fails, the download fails and retries follow the exponential backoff strategy from the design doc.
|
||||
|
||||
**Decision:** Single address (from tsclientlib). No failover needed.
|
||||
Reference in New Issue
Block a user