Trained Dictionaries

Trained dictionaries compress small, highly correlated message families by referencing a shared string table (dict_id) instead of sending literals on every message. They are an optional stateful feature — enabled by default in session encoders but invisible when payloads do not match the candidate profile.

When dictionaries help

Signal	Threshold (spec Rule ST4)
Payload family	Stable schema/shape across messages
Encoded size	128 B – 8 KB per message
String repetition	Same vocabulary across the family

Typical workloads:

• Paginated API responses with repeated enum/status strings
• Log events sharing level, service, region values
• Telemetry ticks with stable field names and low-cardinality dimensions
• Small control messages in a long-lived session

Dictionaries do not help one-shot large batches (use Batch profile) or unrelated cache keys (use stateless Dynamic).

Wire format

A trained dictionary reference encodes as:

[dict_id][compressed block]

The compressed block maps column string values to dictionary indices. Dictionary metadata (id, version, hash, invalidation rule) travels in the control stream so a fresh decoder can load the profile before decoding references.

See Spec §13.5.7 and §15.4 trained dictionary transport.

Session configuration

Dictionaries are controlled by enableTrainedDictionary on the session encoder:

import { createSessionEncoder } from "@twilic/core";

const enc = createSessionEncoder({
  enableTrainedDictionary: true, // default
});

use twilic::{create_session_encoder, SessionOptions};

let enc = create_session_encoder(SessionOptions {
    enable_trained_dictionary: true,
    ..Default::default()
});

import twilic

enc = twilic.create_session_encoder(
    twilic.SessionOptions(enable_trained_dictionary=True)
)

Disable when debugging wire bytes or when message families are too heterogeneous:

createSessionEncoder({ enableTrainedDictionary: false });

How dictionaries are learned

Within a session, the encoder:

1. Observes string columns in batch or repeated single messages
2. Builds a dictionary when unique ratio is low (spec simplified heuristic: unique_ratio <= 0.25)
3. Assigns a dict_id and transports the dictionary profile to the decoder
4. Subsequent messages reference dict_id + compressed block

Fresh decoders receive the dictionary profile through the control stream before any dict_id reference is decoded. Invalid hash or version mismatch is rejected — see conformance tests in twilic-rust.

Dictionary scope

Separate dictionaries per data family (spec §15.3):

user-records-dict     → paginated user API
log-events-dict       → structured logging
telemetry-tick-dict   → live metrics stream

Do not share one dictionary across unrelated message types. Encoder auto-selection assigns dictionary IDs per family within a session.

Invalidation and recovery

Dictionary state is session-local. After disconnect or RESET_STATE:

• All dict_id references become invalid
• Sender must re-transport dictionary profiles or fall back to stateless encoding
• Receiver discards cached dictionaries

enc.reset(); // clears dictionary state along with bases and templates

If a decoder receives an unknown dict_id:

Policy	Behavior
failFast	Decode error
statelessRetry	Signal full-frame retry

See Stateful Decoding and Session Encoder.

vs column DICTIONARY codec

Twilic has two dictionary mechanisms:

Mechanism	Scope	When
Column DICTIONARY codec	Single batch message	Low-cardinality string column in col_batch
Trained dictionary	Session-wide	Repeated small messages in same family

Column dictionary is stateless within the batch. Trained dictionary persists across the session for incremental streams.

Measuring impact

1. Run Playground schema-first view on your record shape
2. Compare stateless encode() vs session encodeMicroBatch() on log fixtures
3. Use pnpm example:logs in examples for batch vs micro-batch size tables

For patch + dictionary combined savings, run pnpm example:websocket:simulate.

Related

• Encoder Selection — Rule ST4 thresholds
• Stateful Streams
• Batch & Columnar
• Session Encoder reference