Edison Jwa
7968f90f7d
tsdeclarations eval: direct codegen redundant (already via tsproto), but permission ID generation and error code enum have high value. Permission mapping: 253 TS3 permissions identified, 7 P1 for chat/moderation. Encoding verification: tsclientlib uses strict UTF-8, UCS-2 is edge case.
103 lines
4.4 KiB
Markdown
103 lines
4.4 KiB
Markdown
# tsclientlib Encoding Handling Verification
|
|
|
|
**Date:** 2026-06-11
|
|
**Scope:** Verification of encoding handling in Chanora's tsclientlib fork
|
|
**Checkout analyzed:** `~/.cargo/git/checkouts/tsclientlib-b9d0d0212095a0c5/3fbfa26/`
|
|
|
|
---
|
|
|
|
## 1. How tsclientlib Handles Encoding
|
|
|
|
### Command Layer (`tsproto-packets/src/commands.rs`)
|
|
|
|
The `CommandParser` operates on raw `&[u8]` bytes. String conversion happens at the value level:
|
|
|
|
- **`CommandArgumentValue::get_str()`** (line 157): Returns `Result<Cow<'a, str>>` using `str::from_utf8()` on raw bytes or `String::from_utf8()` on unescaped bytes.
|
|
- **`CommandArgumentValue::get_parse()`** (line 166): Calls `get_str()` then `.parse()` for typed values.
|
|
- Both methods return `Utf8Error` on invalid UTF-8 — **strict validation, not lossy**.
|
|
|
|
### Packet Layer (`tsproto/src/`)
|
|
|
|
| File | Line | Usage | Behavior |
|
|
|------|------|-------|----------|
|
|
| `license.rs` | 482, 550 | `str::from_utf8(&data[...])` | Strict — returns `Error::DeserializeString` on failure |
|
|
| `log.rs` | 84, 101 | `if let Ok(s) = str::from_utf8(packet.content())` | Silent skip — invalid UTF-8 packets are not logged |
|
|
| `client.rs` | 363 | `str::from_utf8(name)` | Used in error message formatting only |
|
|
| `algorithms.rs` | 450 | `String::from_utf8_lossy(&dec)` | Lossy — only in debug `println!` |
|
|
|
|
### Bookkeeping Layer (`utils/ts-bookkeeping/`)
|
|
|
|
- All string fields in `Connection`, `Server`, `Client`, `Channel` structs are `String` (Rust's UTF-8 type).
|
|
- Generated message parsers (via `MessageDeclarations.tt` line 47) use `str::from_utf8(s)?.to_string()` — strict UTF-8 validation.
|
|
- `ParseError::StringParse` wraps `std::str::Utf8Error` (messages.rs:26).
|
|
|
|
---
|
|
|
|
## 2. UCS-2 and CESU-8 Handling
|
|
|
|
**No UCS-2 or CESU-8 code exists anywhere in the fork.**
|
|
|
|
Search for `ucs`, `UCS`, `cesu`, `CESU`, `utf16`, `UTF16`, `utf-16`, `UTF-16` across all `.rs` files returned zero results.
|
|
|
|
The fork treats all protocol data as UTF-8 bytes. There is no:
|
|
- UCS-2 to UTF-8 conversion
|
|
- CESU-8 decoding
|
|
- UTF-16 handling
|
|
- Encoding detection or negotiation
|
|
|
|
---
|
|
|
|
## 3. What Happens with Non-UTF-8 Data
|
|
|
|
If the server sends non-UTF-8 bytes (e.g., UCS-2 encoded emoji from an older client):
|
|
|
|
| Layer | Behavior |
|
|
|-------|----------|
|
|
| `CommandParser::get_str()` | Returns `Err(Utf8Error)` — call site decides |
|
|
| License parsing | Returns `Err(Error::DeserializeString)` — connection fails |
|
|
| Log parsing | `if let Ok(s)` — silently skipped, packet not logged |
|
|
| Message parsing | Returns `Err(ParseError::StringParse)` — command rejected |
|
|
|
|
**Critical:** There is no graceful fallback. Non-UTF-8 data causes errors at the protocol layer.
|
|
|
|
---
|
|
|
|
## 4. What the TS3 Protocol Actually Sends
|
|
|
|
Per ReSpeak protocol reference §1.8.3: "Command string encoded in UTF-8"
|
|
|
|
The TS3 protocol itself uses UTF-8 for command strings. The UCS-2 concern applies to:
|
|
- **Older mobile clients** (pre-3.2.0) that may send CESU-8 encoded supplementary characters
|
|
- **Server-side storage** where some servers may store UCS-2 for certain fields
|
|
- **Emoji support** added in Server 3.2.0+ (UTF-8 with full Unicode)
|
|
|
|
In practice, the protocol wire format is UTF-8. The UCS-2 issue is a server/client implementation detail, not a protocol encoding issue.
|
|
|
|
---
|
|
|
|
## 5. Risks for Chanora
|
|
|
|
### Low Risk
|
|
- **Normal operation:** Modern TS3 servers and clients send valid UTF-8. No issues expected.
|
|
- **Chanora's own strings:** Always valid UTF-8 (Rust `String` type guarantees this).
|
|
|
|
### Medium Risk
|
|
- **Legacy clients:** Older mobile clients sending CESU-8 will cause `Utf8Error` in command parsing. Commands will be rejected, not silently corrupted.
|
|
- **Server names/channel names with emoji:** If a server stores names in UCS-2 format (some older databases), Chanora will fail to parse them.
|
|
|
|
### Mitigation
|
|
- The fork uses **strict** UTF-8 validation (not lossy). This is correct behavior — rejecting malformed data is safer than silently corrupting it.
|
|
- If UCS-2 compatibility is needed, the fix belongs in the `EdisonJwa/tsclientlib` fork, not in Chanora's protocol layer.
|
|
|
|
---
|
|
|
|
## 6. Conclusion
|
|
|
|
tsclientlib handles encoding correctly for the TS3 protocol:
|
|
- All string parsing uses `str::from_utf8()` — strict UTF-8 validation
|
|
- No UCS-2/CESU-8 code exists — the fork assumes UTF-8 wire format
|
|
- Invalid UTF-8 causes errors, not silent corruption
|
|
- The protocol itself is UTF-8; UCS-2 is an edge case from legacy implementations
|
|
|
|
**Status:** VERIFIED — encoding handling is correct as-is.
|