FAQ

General

Is Twilic stable?

The v2 wire format is a stable, versioned specification. Implementations are required to be deterministic and interoperable within the v2 profile. The spec does not change wire behavior between minor releases.

v2 is a clean break from v1. A v2 decoder is not required to decode v1 payloads.

Is Twilic production-ready?

The specification is finalized. The eighteen official SDKs (Rust, Go, Python, JavaScript/TypeScript, Java, Scala, Ruby, R, Zig, PHP, Kotlin, Dart, Elixir, Lua, C, C++, C#, and Swift) are conformance-tested against shared binary fixtures generated by the Rust reference implementation.

Does Twilic support streaming?

Twilic operates on messages, not streams. However:

• Any reliable, ordered channel (WebSocket, gRPC stream, TCP) can carry a sequence of Twilic messages.
• The Stateful Profile provides session-level compression across messages on such channels.
• Twilic does not define a framing protocol — use length-prefixed framing or a transport that handles message boundaries.

Can I use Twilic over HTTP?

Yes. Use the Dynamic or Batch Profile in stateless mode. Set Content-Type: application/octet-stream (or a custom MIME type like application/x-twilic) for Twilic payloads.

Stateful Profile is not appropriate for HTTP request/response because there is no shared persistent session state.

Is there a text/debug format?

Twilic is a binary format. There is no normative text representation. For debugging, decode to the value tree and serialize to JSON for inspection. The SDKs provide decode → standard value tree → JSON serialization as a standard pattern.

---

Encoding

How is null encoded?

null is the single byte 0xC0. This is the same as MessagePack's nil.

How are integers encoded?

Integers use the smallest valid width:

• -32..127: fixint (1 byte)
• -128..-33: i8 (2 bytes)
• 128..255: u8 (2 bytes)
• And so on, up to i64 / u64 (9 bytes)

This means small common values like 0, 1, true, false are very compact.

How are 64-bit integers handled in JavaScript?

JavaScript's Number type cannot represent all u64 values safely (values above 2^53 - 1 lose precision). The JS SDK decodes u64 and i64 as bigint by default. Encode with BigInt literals:

const value = { counter: 9007199254740993n }; // safe as bigint

What is varuint?

Twilic uses a variable-length unsigned integer encoding (Twilic-PV) for metadata fields: lengths, IDs, and counts. It works like LEB128:

• If the high bit of a byte is 0, that byte completes the value.
• If the high bit is 1, more bytes follow.

This is only used for metadata (lengths, key_id, shape_id, etc.), not for integer values in the payload. Payload integers always use fixed-width tags.

Can I encode arbitrary binary data?

Yes. Use the bin8, bin16, or bin32 types. There is no base64 step. Binary data is sent as raw bytes with a length prefix.

let value = Value::Binary(vec![0xDE, 0xAD, 0xBE, 0xEF]);

Does the encoder choose codecs automatically?

Yes. In Dynamic Profile, the encoder detects homogeneous arrays and selects typed_vec automatically. In col_batch, each column's codec is chosen based on the column's value statistics. The selection is deterministic — given the same input, the encoder always picks the same codec.

---

Interning

When do intern tables reset?

Intern tables (key_id, str_id, shape_id) reset at each top-level message boundary. They are never shared across messages. This is a hard requirement for stateless decoding.

How many keys can be interned per message?

There is no hard limit in the spec. IDs are varuint-encoded, so any number of keys can be interned. In practice, most messages have fewer than 64 distinct keys.

Does string interning affect keys or values?

Both. key_ref interns map key strings. str_ref interns string values. They use separate ID spaces.

What if a key_ref or str_ref references an unknown ID?

This is a hard decode error. The decoder must fail immediately. Unknown references are never silently accepted.

---

Stateful Mode

What happens if I lose sync with the receiver?

Send RESET_STATE immediately. Both sender and receiver invalidate all session state. The sender then emits a stateless full frame (or fresh base/template registration) before resuming stateful references.

Can I use stateful mode over a message queue?

Generally no. Stateful mode requires ordered, reliable delivery with no silent drops of reset or registration frames. Most message queues can reorder or drop messages. Use stateless mode for queues.

Is stateful mode backward-compatible?

Receivers that do not implement stateful mode will fail to decode state_patch and template_batch frames. Stateful mode must be negotiated out-of-band (e.g., via connection handshake or application configuration). Implementations that don't support stateful mode MUST reject stateful frames with a clear error, not silently ignore them.

---

Interoperability

How do I verify cross-SDK compatibility?

The benchmark and twilic repositories include a set of binary fixtures generated by the Rust reference implementation. Each SDK runs conformance tests against these fixtures. If your implementation decodes all fixtures correctly and your encoder output is decoded correctly by the Rust implementation, you are v2-interoperable.

Can I use Twilic between services written in different languages?

Yes. This is a primary design goal. All eighteen official SDKs target the v2 wire format and are tested for interoperability. A payload encoded by the Rust SDK is correctly decoded by the other official SDKs.

Does field order matter?

In Dynamic Profile, map keys are sent in the order the encoder emits them. There is no canonical ordering requirement for maps. Decoders must accept any key order.

In Bound Profile, field order is fixed by schema. A decoder relying on field position for a schema-aware message must use the schema-defined order.

Can a v2 decoder decode v1 payloads?

No. v2 is a clean break. The tag-table wire model of v2 is incompatible with v1's message-kind envelope. If you need to support both, detect the version from a framing header or connection handshake and dispatch accordingly.

---

Performance

Is Twilic faster than JSON?

For encoding and decoding, binary formats are generally faster than text formats because they avoid UTF-8 parsing, number-to-string conversion, and escape handling. The JS SDK's N-API backend (native Rust) is typically 3–5× faster than JSON.stringify/JSON.parse for equivalent payloads.

However, for a single small object, the difference may be negligible compared to network latency. Twilic's performance advantage is most pronounced on batch encoding/decoding where per-record overhead dominates.

Should I use row batch or column batch?

Use row batch when:

• Latency matters and batches are small (< 64 records)
• Columns are mixed types or have low regularity
• You want simpler encoder logic

Use column batch when:

• Throughput matters more than latency
• Batches are large (> 64 records)
• Columns are numeric and have high regularity (time series, metrics)
• You need maximum compression ratio

How much memory does the encoder use?

The Dynamic encoder maintains message-local intern tables (key table, string table, shape table). Memory is proportional to the number of distinct keys, strings, and shapes in a single message — typically a few KB for realistic messages.

The SessionEncoder additionally maintains session state (base snapshots, templates). Memory depends on the size and number of registered state objects.