Files
chanora/docs/architecture/file-transfer-research.md
T
Edison Jwa aa796d7395 feat: file transfer system (avatar/icon download with cacache) (#40)
* docs(architecture): add file transfer design, research, and implementation plan

* feat(cache): add chanora_cache crate with cacache-backed blob cache

- New chanora_cache crate: content-addressed blob store wrapping cacache
- BlobCache API: async put/get/remove/clear/total_size/evict
- Key validation: av_ prefix (32 hex chars), ic_ prefix (decimal digits)
- Cacache provides crash safety, SSRI integrity, content dedup
- Mtime-based eviction via cacache::list_sync + sort by timestamp
- 7 unit tests all passing
- Added to workspace members

* feat(protocol): add file download support for avatars and icons

- Add Request::DownloadFile variant with oneshot reply
- Add ProtocolClient::download_avatar(client_uid) and download_icon(icon_id)
- Track pending file downloads by FiletransferHandle
- Handle StreamItem::FileDownload: read bytes from TCP stream
- Handle StreamItem::FiletransferFailed: map to ProtocolError
- Add ProtocolError::FileTransfer(String) variant
- Add path helper tests for avatar/icon download paths
- No tsclientlib types leak across the adapter boundary

* feat(core): add blob cache wiring and avatar download orchestration

- Add chanora_cache dependency to Cargo.toml
- Add blob_cache field to ChanoraSession (Arc<Mutex<Option<BlobCache>>>)
- Add init_cache() method: creates BlobCache, runs eviction
- Add get_avatar() method: cache-first, download on miss, store in cache
- Add clear_cache() and cache_size() methods for cache management
- Add CoreError::Cache variant for BlobCacheError conversion
- Add avatar_cache integration test

* feat(bridge): add init_cache, download_avatar, and cache management functions

- Add init_cache(dir) bridge function
- Add download_avatar(avatar_hash, client_uid) bridge function
- Add clear_file_cache() and file_cache_size() bridge functions
- Map CoreError::Cache and ProtocolError::FileTransfer in BridgeError

* feat(flutter): add cache initialization wiring and avatar download shims

- Add wireCache() to app_bootstrap using getApplicationCacheDirectory()
- Call wireCache() after wireStorage() in main bootstrap flow
- Add Dart-side initCache and downloadAvatar wrapper shims in api.dart
- Update Cargo.lock for new chanora_cache dependency

* feat(core): FileTransferService with coalescing, throttling, negative cache

- New file_transfer module with FileTransferService struct
- Semaphore(2) throttles concurrent downloads
- In-flight HashMap coalesces duplicate avatar requests
- 5-min negative cache short-circuits ServerRejected misses
- ChanoraSession delegates get_avatar through the service
- connect/disconnect update shared protocol handle
- clear_cache/cache_size delegate to service
- 2 new unit tests (cached hit, negative cache)

* feat(core,bridge): add get_icon with coalescing and negative cache

- FileTransferService::get_icon() mirrors get_avatar pattern
- ChanoraSession::get_icon() delegates through FileTransferService
- Bridge download_icon() exposed for Flutter
- Dart downloadIcon() shim added
- Uses PREFIX_ICON (ic_<crc32u>) cache key format
- 1 new unit test (cached icon hit)

* fix(core,protocol): simplify store_protocol and add download size cap

- store_protocol: always write to shared Arc<Mutex<Option<ProtocolClient>>>;
  the FileTransferService holds the same Arc so it sees updates automatically
- read_download_bytes: reject downloads exceeding 10 MB to prevent
  malicious servers from causing OOM
2026-06-10 11:45:14 +09:00

37 KiB
Raw Blame History

File Transfer Cache Research

Date: 2026-06-10 Status: Research complete, design implications noted (updated with TeaSpeak server findings) Companion to: docs/architecture/file-transfer-design.md Purpose: Factual findings from protocol analysis, existing client implementations, and cross-platform research that inform the cache architecture decision.


1. TS3 Protocol Identity Semantics

1.1 Avatar Identity

Aspect Value
Protocol field client_flag_avatar
Type TYPE_STRING (TeaSpeakLibrary PropertyDefinition.h:200)
Meaning MD5 hash of the avatar file bytes
Scope Per-client per-server — a user can have different avatars on different servers
Freshness Automatically up-to-date for any client "in view" (FLAG_CLIENT_VIEW)
Empty value No avatar set

Key fact: Identical avatar image bytes produce the same client_flag_avatar hash on any TS3 server. The hash is a content fingerprint, not a server-assigned identifier.

Download path on server: /avatar_<base64HashClientUID> — the filename is derived from the client's unique identifier (UID), not from the content hash. The content hash is communicated separately via client_flag_avatar.

Sources:

1.2 Icon Identity

Aspect Value
Protocol fields channel_icon_id, client_icon_id, virtualserver_icon_id
Type TYPE_UNSIGNED_NUMBER (TeaSpeakLibrary PropertyDefinition.h:83,146,217)
Meaning CRC32 (unsigned) of the icon file bytes
Scope Per-entity per-server — but CRC32 is content-derived
Download path /icon_<unsigned_crc32>

Key fact: Identical icon bytes produce the same CRC32 on any TS3 server. The icon ID is a content fingerprint. The upload process computes crc32.unsigned(data) and stores at /icon_<id>.

CRC32 collision caveat: CRC32 is only 32 bits. Different icon content can theoretically produce the same CRC32. Qint's filecache.rs explicitly notes this: "there could be collisions because only CRC-32 is used." ForChanora's purposes (small icons, not security-critical), this is acceptable.

Sources:

1.3 Implication for Cache Design

Both avatars and icons are content-addressed by the protocol itself:

Asset Content hash source Same content across servers?
Avatar client_flag_avatar = MD5 of bytes Same bytes → same hash → same ID
Icon icon_id = CRC32 of bytes Same bytes → same CRC32 → same ID

This means a flat content-addressed blob store can achieve zero-duplication without any per-server directories, hardlinks, or ref-counting.


2. Virtual Server Identity

2.1 Server UID

Aspect Value
Protocol field virtualserver_unique_identifier
Type TYPE_STRING (TeaSpeakLibrary PropertyDefinition.h:22)
Generated by The server instance, on creation
Globally unique? Not guaranteed — locally generated, no central registry
Stable? Yes — persists across restarts of the same virtual server

Key fact: virtualserver_unique_identifier is generated by each TS3 server. Two physically different servers could theoretically produce the same UID. It is not safe as a global cache key.

2.2 What Chanora Currently Tracks

Layer Server identity fields Source
Protocol adapter (adapter.rs:1752-1755) server_name, welcome_message, platform, version state.server.* from tsclientlib
DTO (ServerSnapshot) server_name, welcome_message, platform, version No UID field
Bridge (BridgeSnapshot) Same as DTO Same
Storage (bookmarks) Keyed by host (hostname:port) SQLite WHERE host = ?1
Core (recent servers) cfg.address as host Auto-saved on connect

Chanora does not currently plumb virtualserver_unique_identifier through the DTO stack. The field exists in tsclientlib's state but is not extracted.

2.3 Implication for Cache Design

Using virtualserver_unique_identifier as the sole cache key is risky (not globally unique). Using connection address (host:port) is safe but duplicates cache entries when the same server is accessed via different addresses.

Recommendation: For a content-addressed blob store, server identity is only needed for per-server metadata (eviction, "clear cache for this server"), not for the blob key itself. The blob key is the content hash.


3. Existing TS3 Client Cache Implementations

3.1 Qint (tsclientlib-based, Tauri + Rust)

Architecture: Per-server directory with SQLite metadata.

<cache>/files/<server-uid>/<channel-id>/<base64(path)>

Avatar handling:

  • Avatar state stored per (server, client) row in SQLite
  • On avatar hash change, deletes the cached /avatar_<uid> file for that server
  • Avatar download path: /avatar_<uid_base64>

Icon handling:

  • Icons path-cached with CRC32
  • Code comments note CRC32 collisions and freshness by mtime
  • Qint explicitly deletes and re-downloads when icon mtime changes

Dedup: None. Same avatar on 5 servers = 5 stored copies.

Sources:

3.2 TS3 Official Client (closed source)

Architecture: Lazy cache with SDK callbacks.

  • getAvatar() returns cached path if present; otherwise triggers download
  • onAvatarUpdated callback fires when avatar is downloaded or deleted
  • Cache paths (from community documentation):
    • Windows: %LOCALAPPDATA%\TeamSpeak\Cache\Default
    • Linux: ~/.cache/TeamSpeak/Default
    • macOS: ~/Library/Caches/TeamSpeak/Default
  • SDK also exposes CLIENT_MYTS_AVATAR / client_myteamspeak_avatar for cross-server myTeamSpeak avatars

Sources:

3.3 TeaSpeak Client (TypeScript + C++ native)

Architecture: Browser Cache API for images, per-server own-avatar storage.

Key components (from .d.ts type declarations):

  • AvatarManager — per-connection (FileManager) avatar handler

    • cachedAvatars (private) — in-memory cache of ClientAvatar objects
    • updateCache(clientAvatarId, clientAvatarHash) — updates cache when hash changes
    • resolveAvatar(clientAvatarId, avatarHash?, cacheOnly?) — resolves avatar by ID
    • flush_cache() — clears cache
    • create_avatar_download(client_avatar_id) — initiates file transfer
  • ClientAvatar — tracks individual avatar state

    • clientAvatarId — derived from client UID via uniqueId2AvatarId()
    • currentAvatarHash — the client_flag_avatar value
    • State machine: unsetloadingloaded / errored
    • loadingTimestamp — when download started
  • ImageCache — generic image cache using browser Cache API

    • resolveCached(key, maxAge?) — check if cached
    • putCache(key, value, type?, headers?) — store
    • cleanup(maxAge) — evict old entries
    • reset() — clear all
    • isPersistent() — whether cache persists to disk
  • OwnAvatarStorage — user's own avatar, keyed by serverUniqueId + mode

    • loadAvatarImage(serverUniqueId, mode) — load own avatar for a server
    • updateAvatar(serverUniqueId, mode, target) — update own avatar
    • avatarUploadSucceeded(serverUniqueId) — move from "uploading" to "server" state
    • Stores LocalAvatarInfo: fileName, fileSize, fileHashMD5, timestamps, contentType
  • FileManager — per-connection file transfer manager

    • MAX_CONCURRENT_TRANSFERS — transfer concurrency limit
    • avatars: AvatarManager — avatar subsystem
    • initializeFileDownload(options) — start download (path, name, channel, target)
    • deleteIcon(iconId: number) — delete icon by ID
  • FileTransfer — transfer state machine

    • States: PENDING → INITIALIZING → CONNECTING → RUNNING → FINISHED / ERRORED / CANCELED
    • InitializedTransferProperties: serverTransferId, transferKey, addresses[], protocol, seekOffset, fileSize
    • Multiple addresses returned by server for file transfer (failover)
  • localIconCache: ImageCache — global icon cache (singleton)

Sources:

  • TeaSpeak-Client imports/shared-app/file/Avatars.d.ts — ClientAvatar, AbstractAvatarManager
  • TeaSpeak-Client imports/shared-app/file/LocalAvatars.d.ts — AvatarManager
  • TeaSpeak-Client imports/shared-app/file/LocalIcons.d.ts — localIconCache
  • TeaSpeak-Client imports/shared-app/file/ImageCache.d.ts — ImageCache (browser Cache API)
  • TeaSpeak-Client imports/shared-app/file/FileManager.d.ts — FileManager, transfer API
  • TeaSpeak-Client imports/shared-app/file/Transfer.d.ts — FileTransfer, state machine, error types
  • TeaSpeak-Client imports/shared-app/file/OwnAvatarStorage.d.ts — own avatar per-server storage
  • TeaSpeak-Client native/serverconnection/test/js/ft.ts — file transfer test (TCP + ftkey protocol)

3.4 TS3AudioBot (C#)

Architecture: No local avatar cache. Avatar upload is hash-driven.

  • Uploads avatar bytes to /avatar, computes MD5, sets client_flag_avatar to that hash
  • Bot avatar selection reads local files from an avatars/ directory
  • No caching of other users' avatars

Sources:

3.5 TS3 NodeJS Library

Architecture: No local cache. Downloads on demand.

  • Avatar filename: avatar_${clientBase64HashClientUID}
  • getAvatar() downloads directly — no caching layer
  • Tests assert the exact /avatar_<base64uid> path

Sources:

3.6 Summary Table

Client Cache Key Strategy Dedup Across Servers? Icon Cache
Qint <server-uid>/<channel-id>/<path> No Yes (CRC32, mtime freshness)
TS3 Official Lazy cache (path-based) Unknown Yes
TeaSpeak UID-derived avatar ID + browser Cache API Implicit (same hash = same cache) Yes (global ImageCache)
TS3AudioBot None N/A No
TS3 NodeLib None N/A No
Chanora (decided) Content hash (MD5/CRC32), flat blobs/ in chanora_cache crate Yes Yes

4. TeaSpeak Protocol Definitions (Authoritative)

From TeaSpeakLibrary src/PropertyDefinition.h — the most complete open-source reference for TS3 protocol property types:

4.1 Avatar Properties

// Line 200
PropertyDescription{CLIENT_FLAG_AVATAR, "client_flag_avatar", "",
    TYPE_STRING, FLAG_CLIENT_VIEW | FLAG_SAVE | FLAG_USER_EDITABLE}
// "automatically up-to-date for any manager 'in view', this manager got an avatar"

4.2 Icon Properties

// Line 83 — server icon
PropertyDescription{VIRTUALSERVER_ICON_ID, "virtualserver_icon_id", "0",
    TYPE_UNSIGNED_NUMBER, FLAG_SERVER_VVSS | FLAG_USER_EDITABLE}

// Line 146 — channel icon
PropertyDescription{CHANNEL_ICON_ID, "channel_icon_id", "0",
    TYPE_UNSIGNED_NUMBER, FLAG_CHANNEL_VIEW | FLAG_SS | FLAG_USER_EDITABLE}

// Line 217 — client icon
PropertyDescription{CLIENT_ICON_ID, "client_icon_id", "0",
    TYPE_UNSIGNED_NUMBER, FLAG_CLIENT_VIEW | FLAG_CLIENT_VARIABLE}

4.3 Server Identity

// Line 22
PropertyDescription{VIRTUALSERVER_UNIQUE_IDENTIFIER,
    "virtualserver_unique_identifier", "",
    TYPE_STRING, FLAG_SERVER_VV | FLAG_SNAPSHOT}

4.4 File Transfer Permissions

// From PermissionManager.cpp
PermissionType::i_client_max_avatar_filesize    // "Max avatar filesize in bytes"
PermissionType::b_client_avatar_delete_other    // "Allow deletion of avatars from other clients"
PermissionType::b_ft_transfer_list              // "Retrieve list of running filetransfers"

4.5 File Transfer Error Codes

// From Error.h
channel_no_filetransfer_supported     = 0x30C
file_transfer_connection_timeout      = 0x80E
file_transfer_complete                = 0x811
file_transfer_canceled                = 0x812
file_transfer_interrupted             = 0x813
file_transfer_server_quota_exceeded   = 0x814
file_transfer_client_quota_exceeded   = 0x815
file_transfer_reset                   = 0x816
file_transfer_limit_reached           = 0x817

5. Cross-Platform Filesystem Research

Platform Filesystem Hardlinks in App-Private Storage? Gotcha
Android (API 28+) ext4 / f2fs Yes Rust uses libc::link; not FUSE-mounted; same-filesystem only
iOS APFS Yes (writable sandbox dirs) App bundle is read-only; avoid hardlinks to bundle assets
macOS APFS Yes
Linux ext4 / btrfs / xfs Yes
Windows NTFS Yes Rust uses CreateHardLinkW

std::fs::hard_link gotchas (all platforms):

  • Same filesystem required
  • Destination must not exist (returns error)
  • Symlink behavior is platform-specific
  • All hardlinks share the same inode — modifying one modifies all
  • Files must be treated as immutable for hardlink safety

Sources:

5.2 Rust Cache Libraries

cacache (MIT licensed, production-ready):

  • Content-addressed disk cache
  • Automatic dedup, atomic writes, integrity verification
  • Exposes hard_link, copy, and reflink paths for retrieval
  • On-disk layout: content-v2/sha512/...
  • Could replace a custom implementation, but adds a dependency

Sources:

5.3 Flutter Cache Patterns

Common Flutter packages use cache-dir + metadata DB, not hardlink dedup:

  • flutter_cache_manager: files in cache dir + sqflite metadata
  • super_cache_disk: file-per-entry (.dat + .meta) in app cache dir

Sources:


6. Chanora Codebase Context

6.1 Storage Patterns

Component Pattern Location
Identity storage Atomic write (temp + sync_all + rename), mode 0600 on Unix chanora_storage/src/lib.rs:446-476
Metadata Same atomic write pattern chanora_storage/src/lib.rs:480-515
Bookmarks SQLite at <storage_dir>/chanora.db, keyed by host chanora_storage/src/lib.rs:782-928
Storage root getApplicationSupportDirectory() from Flutter app_bootstrap.dart:19-24
Bridge init rust.initStorage(dir: dir) app_bootstrap.dart:23-25,108-133

6.2 Existing Avatar Handling

Component What Location
UID to avatar path uid_to_avatar_path() — base64 decode UID, encode each byte as 2 chars (a-p) adapter.rs:1561-1568
Client profile DTO avatar_path field — set when client.avatar_hash and unique_id non-empty dto.rs:135-136, adapter.rs:1323-1332
No download Currently no file download implementation exists
No icon handling No icon field/path in protocol DTO or adapter

6.3 Server Identity in Chanora

Chanora currently tracks servers by connection address (host:port), not by server UID:

  • Bookmarks: WHERE host = ?1
  • Recent servers: auto-saved by cfg.address
  • Prefetch cache: keyed by normalized host
  • ServerSnapshot: has server_name but no server_uid

The virtualserver_unique_identifier field is available from tsclientlib's state but is not extracted by the adapter.

6.4 tsclientlib File Transfer API

Aspect Detail
Download method Connection::download_file()
Stream items StreamItem::FileDownload(FileDownloadResult { size, stream })
Failure StreamItem::FiletransferFailed(handle, error)
TCP handling tsclientlib handles TCP connection + ftkey writing automatically
Chanora's job Read size bytes from the returned TcpStream
Async behavior StreamItem::FileDownload fires asynchronously, not inline with the request

6.5 ts-bookkeeping Generated Fields

From the generated parser in target/debug/build/ts-bookkeeping-*/out/:

  • virtual_server_id: u64 — numeric, per-virtual-server, may change across restarts
  • virtual_server_uid — string, the virtualserver_unique_identifier

Both are available in the InInitServer struct from the init handshake but are not currently plumbed through.


7. TeaSpeak Server Internals (Authoritative)

Source: TeaSpeak Server at https://git.did.science/TeaSpeak/Server/Server (branch new-groups, commit b54c6d4e).

7.1 Avatar ID Derivation — Server Side

The server derives the avatar filename from the client UID, not from the avatar content:

// DataClient.cpp:242-244
std::string DataClient::getAvatarId() {
    return hex::hex(base64::validate(this->getUid()) ? base64::decode(this->getUid()) : this->getUid(), 'a', 'q');
}

The same transform produces client_base64HashClientUID (shown to other clients):

// client.cpp:1113-1114
bulk.put_unchecked("client_base64HashClientUID",
    hex::hex(base64::validate(info->client_unique_id) ? base64::decode(info->client_unique_id) : info->client_unique_id, 'a', 'q'));

This matches Chanora's existing uid_to_avatar_path() in adapter.rs:1561-1568.

7.2 Avatar Upload Path

When a client uploads an avatar, the server stores it as /avatar_<avatarId>:

// file.cpp:696-702
} else if (cmd["path"].as<std::string>().empty() && cmd["name"].string() == "/avatar") {
    ...
    info.file_path = "/avatar_" + this->getAvatarId();
    transfer_response = file::server()->file_transfer().initialize_avatar_transfer(...);
}

The avatar file path is identity-based (from UID), not content-based.

7.3 client_flag_avatar — Who Computes the Hash?

The CLIENT computes the MD5 and sends it to the server during upload. The server stores it as a string property (FLAG_USER_EDITABLE). The server does NOT compute or verify the hash.

This means client_flag_avatar is:

  • Set by the uploading client
  • Stored verbatim by the server
  • Broadcast to other clients as part of the client properties
  • A reliable content fingerprint: same avatar bytes → same MD5 → same client_flag_avatar on any server

7.4 Icon IDs — Server Does NOT Compute CRC32

Icon IDs are permission values, not content hashes computed by the server:

// ConnectedClient.cpp:186-210 — client icon ID from permissions
auto permission_flags = local_permissions->permission_flags(permission::i_icon_id);
new_icon_id = value.value;
updated_client_properties.emplace_back(property::CLIENT_ICON_ID);
// channel.cpp:1495-1504 — channel icon ID
if(key == property::CHANNEL_ICON_ID) {
    auto icon_id = converter<uint32_t>::from_string_view(value);
    channel->permissions()->set_permission(permission::i_icon_id, { ... icon_id ... });
}
// server.cpp:76-89 — server icon ID
SERVEREDIT_CHK_PROP_CACHED("virtualserver_icon_id", permission::b_virtualserver_modify_icon_id, int64_t)

The CLIENT computes the CRC32 during upload and uses it as the filename /icon_<crc32>. The server stores the file and records the ID as a permission value. No server-side CRC32 or MD5 computation exists.

7.5 Per-Server Storage Layout

Avatars and icons are stored per virtual server on the server's filesystem:

// LocalFileSystem.cpp:39-45
fs::path LocalFileSystem::server_path(const std::shared_ptr<VirtualFileServer> &server) {
    return fs::u8path(this->root_path_) / fs::u8path("server_" + std::to_string(server->server_id()));
}
// target_path = this->server_path(server) / "icons" / path;
// target_path = this->server_path(server) / "avatars" / path;
<server_root>/
  server_<sid>/
    avatars/
      /avatar_<avatarId>     ← one per client who uploaded
    icons/
      /icon_<id>             ← one per unique icon

7.6 File Transfer Protocol (Server Side)

Upload/delete/query routing:

// file.cpp:273-341 — delete routing
if (first_entry_name.find("/icon_") == 0 && file_path.empty()) { ... delete_icons(...); }
else if (first_entry_name.starts_with("/avatar_") && file_path.empty()) { ... delete_avatars(...); }
// file.cpp:483-523 — query routing
if (first_entry_name.find("/icon_") == 0 && file_path.empty()) { ... query_icon_info(...); }
else if (first_entry_name.starts_with("/avatar_") && file_path.empty()) { ... query_avatar_info(...); }

Transfer initialization returns ftkey and metadata:

// file.cpp:759-761
result.put_unchecked(0, "ftkey", transfer->transfer_key);
result.put_unchecked(0, "seekpos", transfer->file_offset);
// file.cpp:887-899
result.put_unchecked(0, "ftkey", transfer->transfer_key);
result.put_unchecked(0, "proto", "1");
result.put_unchecked(0, "size", transfer->expected_file_size);

7.7 Key Takeaway for Chanora

Who does what Avatar Icon
Uploader (client) computes MD5 of avatar bytes → sets client_flag_avatar CRC32 of icon bytes → filename /icon_<crc32>
Server does Stores file as /avatar_<uid>, saves property Stores file as /icon_<id>, saves permission
Other clients receive client_flag_avatar (MD5) as a property update icon_id (CRC32) as a property update
Chanora cache key av_<md5>.dat — content fingerprint ic_<crc32>.dat — content fingerprint

The content hash is computed once (by the uploader) and then broadcast as a property. Chanora never needs to hash anything — it just uses the protocol-provided values as cache keys.


8. Design Implications

8.1 The Core Insight

The TS3 protocol provides content hashes as part of normal server-to-client updates:

Event Data provided by server What Chanora gets for free
Client enters view client_flag_avatar = MD5 of avatar bytes Content key for blob store
Channel update channel_icon_id = CRC32 of icon bytes Content key for blob store
Server update virtualserver_icon_id = CRC32 of icon bytes Content key for blob store

No hashing needed on the client side. The protocol is already content-addressed.

<app_cache_dir>/chanora/
  blobs/                         ← cacache content store root
    content-v2/                  ← content-addressed by SHA-512
      <sha512-hex>/data          ← raw blob bytes
    index-v2/                    ← key → content mapping

Where <app_cache_dir> is the platform cache directory (not the support directory used by chanora_storage). Chanora's BlobCache maps protocol keys (av_<md5>, ic_<crc32>) to cacache string keys. Physical layout is managed by cacache.

Lookup flow:

  1. Server sends client_flag_avatar = "a1b2c3d4..." for user X
  2. Check: does blobs/av_a1b2c3d4....dat exist?
  3. Yes → use it, zero downloads (works for ANY server)
  4. No → download from /avatar_<uid_base64> → save as blobs/av_a1b2c3d4....dat

Same for icons with ic_<crc32>.dat.

8.3 Why This Beats Alternatives

Approach Dedup Globally unique key Needs server UID plumbing Needs hardlinks Complexity
<server_uid>/<hash>.dat No No (UID not guaranteed unique) Yes Optional Medium
<host>_<port>/<hash>.dat No Yes No Optional Medium
<host>_<port>/<hash>.dat + hardlinks Yes Yes No Yes Medium-High
blobs/av_<hash>.dat (flat) Yes Yes (content hash) No No Low

8.4 Trade-offs

Pro Con
Zero duplication across all servers "Clear cache for server X only" requires metadata layer (Phase 3+)
No hardlinks needed Orphan cleanup requires scanning for unreferenced blobs
No server UID plumbing needed Cannot distinguish same-hash-different-content for icons (CRC32 collision)
Simplest possible implementation
Freshness = hash change = different filename (automatic)
Cross-platform (just file I/O)

8.5 Phased Implementation

Phase What Delivers
1 Protocol download (raw bytes via adapter, no cache) Working download pipeline
2 Bridge + Flutter display (Image.memory()) Visible avatars in UI
3 chanora_cache crate: cacache-backed blob cache, separate crate, cache dir, mtime eviction Zero re-downloads, zero duplication
4 Session orchestration (coalescing, rate limiting, negative cache) Anti-flood, robustness
5 Cache management (clear all, orphan cleanup, optional per-server metadata) User control

9. Resolved Questions

Q1: Icon CRC32 Collisions — Accept with Size Guard

Risk assessment: CRC32 produces a 32-bit hash. For N unique icons, the Birthday paradox gives collision probability ≈ N² / (2 × 2³²).

Icons (N) Collision probability
100 ~0.0001% (negligible)
1,000 ~0.01% (negligible)
10,000 ~1.2% (marginal)
65,536 ~50% (likely)

A single user typically encounters fewer than 1,000 unique icons across all servers. The practical collision risk is negligible.

What happens on collision: Wrong icon displayed for a channel/client/server. This is a visual glitch, not a security issue. The icon will appear incorrect until the cache is cleared.

Existing practice: Qint explicitly notes CRC32 collisions (filecache.rs:4) but does NOT guard against them — they only refresh icons by mtime. No other TS3 client guards against CRC32 collisions.

Recommendation: Accept CRC32 as the cache key. Add a lightweight file size guard: when downloading an icon, if ic_<crc32>.dat already exists but has a different size than the ftinitdownload response reported, re-download. File size is available from the protocol (msg.size in InFileDownloadPart). This catches most collisions (different content = different size with high probability) without computing a secondary hash.

Decision: CRC32 + file size guard. No SHA256 overhead needed.


Q2: client_myteamspeak_avatar — Defer Indefinitely

What it is: A string property (Option<String> in ts-bookkeeping) broadcast alongside client_flag_avatar. It represents a myTeamSpeak cross-server avatar — a user linked to a myTeamSpeak account can set a global avatar that follows them across all servers.

Current state in Chanora's dependency chain:

  • ts-bookkeeping exposes it: InInitServer has my_team_speak_avatar: Option<String>
  • TeaSpeakLibrary tracks client_myteamspeak_id but not the avatar
  • tsclientlib exposes it as a property on client state

Value for Chanora:

  • myTeamSpeak is a TeamSpeak-specific cloud service (account sync, cross-server features)
  • Chanora is an independent client — no myTeamSpeak account integration is planned
  • The property may contain a URL or identifier that requires myTeamSpeak API access to resolve
  • Without myTeamSpeak integration, the avatar cannot be fetched

Recommendation: Defer indefinitely. If Chanora ever integrates myTeamSpeak accounts, this can be handled as a separate avatar source (URL-based HTTP download) alongside the existing protocol-based avatar download. The cache architecture supports this — just add a different blob prefix (e.g., mt_<hash>.dat).

Decision: Out of scope for MVP and foreseeable roadmap.


Q3: Cache Backing Store — cacache Wrapper in Separate chanora_cache Crate

Decision: Use cacache as the backing store inside chanora_cache. Not a custom flat-file implementation.

Why cacache won over custom:

  1. Crash safety is production-tested. cacache handles partial writes, power loss, crash mid-write. A custom implementation would need to get sync_all + atomic rename right — one bug = corrupted cache. Even though cache data is disposable (reconstructible from server), cacache eliminates this entire class of bugs.

  2. Less code to maintain. ~120 LOC wrapper vs ~200 LOC custom implementation. The hard parts (atomic writes, integrity, content dedup) are owned by cacache, tested by the npm ecosystem.

  3. Integrity verification on every read. SSRI verification detects corruption, bit rot, partial writes automatically. A custom impl would need to add this separately or accept silent corruption.

  4. Content dedup by SHA-512. Same avatar on two servers = stored once automatically. The protocol's MD5/CRC32 keys map to cacache string keys; content dedup happens at the SHA-512 layer underneath.

What about the downsides:

Concern Assessment
~6 transitive deps sha2 already in tree via chacha20poly1305. serde_json, tempfile, digest are lightweight. Acceptable for the safety benefit.
SHA-512 overhead on every write/read For <100KB avatars, SHA-512 takes ~0.1ms. Negligible.
Opaque on-disk format cacache provides ls() API for enumeration and inspection. Not as simple as ls blobs/ but adequate.
cacache has no built-in LRU eviction We write a custom eviction pass using cacache::ls() + timestamp sort. ~20 lines. Same complexity as custom impl's eviction.

Separate crate rationale:

  • chanora_cache is separate from chanora_storage because cache data has different durability semantics (disposable vs persistent), different backup semantics (excluded vs included), and different directory placement (cache dir vs support dir).
  • chanora_cache lives in the platform's cache directory (getApplicationCacheDirectory()). chanora_storage lives in the support directory (getApplicationSupportDirectory()).
  • Bridge init is separate: init_cache(cache_dir) vs init_storage(support_dir).

API design:

pub struct BlobCache { cache_dir: PathBuf, max_bytes: u64 }
impl BlobCache {
    pub fn new(cache_dir: impl AsRef<Path>, max_bytes: u64) -> Result<Self, BlobCacheError>;
    pub async fn put(&self, prefix: &str, key: &str, data: &[u8]) -> Result<(), BlobCacheError>;
    pub async fn get(&self, prefix: &str, key: &str) -> Result<Option<Vec<u8>>, BlobCacheError>;
    pub async fn remove(&self, prefix: &str, key: &str) -> Result<(), BlobCacheError>;
    pub async fn clear(&self) -> Result<(), BlobCacheError>;
    pub async fn total_size(&self) -> Result<u64, BlobCacheError>;
    pub async fn evict(&self) -> Result<(), BlobCacheError>;
}

All methods are async (cacache is async-native). Key validation at API boundary (av_ = 32 hex chars, ic_ = decimal digits).


Q4: Per-Blob Metadata — No Metadata Sidecars (Resolved)

Original options:

Approach Pros Cons
SQLite (chanora.db) ACID, queryable, already in use Schema migration, couples cache to bookmark DB
JSON sidecar files Simple, self-contained, easy to debug Write amplification (2 files per blob), concurrent write risk
In-memory only Simplest Lost on restart, can't do orphan cleanup offline
No metadata (mtime-based) Simplest, zero write amplification, 1 file per blob No per-blob metadata beyond mtime

Why no metadata is sufficient:

  1. Content is immutable. A given hash (MD5 or CRC32) always maps to the same bytes. There is no "stale content" problem — if the hash changes, it's a new file with a new name. No invalidation needed.

  2. mtime = insertion time. Since content is never modified after write, the filesystem mtime equals the time the blob was cached. This is sufficient for "delete oldest files first" eviction.

  3. Write amplification avoided. One file per blob (just the data) instead of two (data + JSON sidecar). For a cache that may hold thousands of small files, this matters.

  4. Eviction is simple. walk dir → stat → sort by mtime → delete oldest. No JSON parsing, no schema, no migration.

  5. Per-server metadata deferred. "Clear cache for server X only" and orphan cleanup are post-MVP features. If needed, a refs-layer can be added later without changing the blob layout.

Oracle consultation: Oracle recommended this approach explicitly — no metadata files, mtime-based eviction, separate crate. The immutability guarantee makes metadata redundant.

Decision: No metadata sidecars. One file per blob. Mtime-based eviction. Per-server metadata deferred to post-MVP.


Q5: File Transfer Address Failover — Not Needed

What the protocol provides:

The TeaSpeak client's InitializedTransferProperties returns addresses[] — an array of {serverAddress, serverPort}. The official TS3 client can try multiple addresses for failover.

What tsclientlib provides:

// tsclientlib/src/lib.rs:1373-1375
let ip = msg.ip.unwrap_or_else(|| self.client.address.ip());
let addr = SocketAddr::new(ip, msg.port);
TcpStream::connect(&addr).await

tsclientlib's InFileDownloadPart has ip: Option<IpAddr>single IP only, not an array. If the server provides an IP, it uses that. Otherwise, it falls back to the connection address. No multi-address failover.

What ts-bookkeeping parses:

pub struct InFileDownloadPart {
    pub client_filetransfer_id: u16,
    pub server_filetransfer_id: u16,
    pub filetransfer_key: String,
    pub port: u16,
    pub size: u64,
    pub protocol: u8,
    pub ip: Option<IpAddr>,    // ← single optional IP
}

The server's notifystartdownload response sends ip as an optional single value, not an array. The TeaSpeak client's addresses[] is a higher-level abstraction (likely the client's own fallback logic), not a protocol feature.

Recommendation: Chanora follows tsclientlib's existing behavior — use msg.ip or fallback to connection address. No custom failover logic needed. If the TCP connection fails, the download fails and retries follow the exponential backoff strategy from the design doc.

Decision: Single address (from tsclientlib). No failover needed.