chanora/docs/references/encoding-verification.md

tsclientlib Encoding Handling Verification

Date: 2026-06-11 Scope: Verification of encoding handling in Chanora's tsclientlib fork Checkout analyzed: ~/.cargo/git/checkouts/tsclientlib-b9d0d0212095a0c5/3fbfa26/

---

1. How tsclientlib Handles Encoding

Command Layer (tsproto-packets/src/commands.rs)

The CommandParser operates on raw &[u8] bytes. String conversion happens at the value level:

• CommandArgumentValue::get_str() (line 157): Returns Result<Cow<'a, str>> using str::from_utf8() on raw bytes or String::from_utf8() on unescaped bytes.
• CommandArgumentValue::get_parse() (line 166): Calls get_str() then .parse() for typed values.
• Both methods return Utf8Error on invalid UTF-8 — strict validation, not lossy.

Packet Layer (tsproto/src/)

File	Line	Usage	Behavior
license.rs	482, 550	str::from_utf8(&data[...])	Strict — returns Error::DeserializeString on failure
log.rs	84, 101	if let Ok(s) = str::from_utf8(packet.content())	Silent skip — invalid UTF-8 packets are not logged
client.rs	363	str::from_utf8(name)	Used in error message formatting only
algorithms.rs	450	String::from_utf8_lossy(&dec)	Lossy — only in debug println!

Bookkeeping Layer (utils/ts-bookkeeping/)

• All string fields in Connection, Server, Client, Channel structs are String (Rust's UTF-8 type).
• Generated message parsers (via MessageDeclarations.tt line 47) use str::from_utf8(s)?.to_string() — strict UTF-8 validation.
• ParseError::StringParse wraps std::str::Utf8Error (messages.rs:26).

---

2. UCS-2 and CESU-8 Handling

No UCS-2 or CESU-8 code exists anywhere in the fork.

Search for ucs, UCS, cesu, CESU, utf16, UTF16, utf-16, UTF-16 across all .rs files returned zero results.

The fork treats all protocol data as UTF-8 bytes. There is no:

• UCS-2 to UTF-8 conversion
• CESU-8 decoding
• UTF-16 handling
• Encoding detection or negotiation

---

3. What Happens with Non-UTF-8 Data

If the server sends non-UTF-8 bytes (e.g., UCS-2 encoded emoji from an older client):

Layer	Behavior
CommandParser::get_str()	Returns Err(Utf8Error) — call site decides
License parsing	Returns Err(Error::DeserializeString) — connection fails
Log parsing	if let Ok(s) — silently skipped, packet not logged
Message parsing	Returns Err(ParseError::StringParse) — command rejected

Critical: There is no graceful fallback. Non-UTF-8 data causes errors at the protocol layer.

---

4. What the TS3 Protocol Actually Sends

Per ReSpeak protocol reference §1.8.3: "Command string encoded in UTF-8"

The TS3 protocol itself uses UTF-8 for command strings. The UCS-2 concern applies to:

• Older mobile clients (pre-3.2.0) that may send CESU-8 encoded supplementary characters
• Server-side storage where some servers may store UCS-2 for certain fields
• Emoji support added in Server 3.2.0+ (UTF-8 with full Unicode)

In practice, the protocol wire format is UTF-8. The UCS-2 issue is a server/client implementation detail, not a protocol encoding issue.

---

5. Risks for Chanora

Low Risk

• Normal operation: Modern TS3 servers and clients send valid UTF-8. No issues expected.
• Chanora's own strings: Always valid UTF-8 (Rust String type guarantees this).

Medium Risk

• Legacy clients: Older mobile clients sending CESU-8 will cause Utf8Error in command parsing. Commands will be rejected, not silently corrupted.
• Server names/channel names with emoji: If a server stores names in UCS-2 format (some older databases), Chanora will fail to parse them.

Mitigation

• The fork uses strict UTF-8 validation (not lossy). This is correct behavior — rejecting malformed data is safer than silently corrupting it.
• If UCS-2 compatibility is needed, the fix belongs in the EdisonJwa/tsclientlib fork, not in Chanora's protocol layer.

---

6. Conclusion

tsclientlib handles encoding correctly for the TS3 protocol:

• All string parsing uses str::from_utf8() — strict UTF-8 validation
• No UCS-2/CESU-8 code exists — the fork assumes UTF-8 wire format
• Invalid UTF-8 causes errors, not silent corruption
• The protocol itself is UTF-8; UCS-2 is an edge case from legacy implementations

Status: VERIFIED — encoding handling is correct as-is.