Why Twilic?

The Problem with Existing Formats

JSON

JSON is human-readable, universally supported, and easy to debug. But it has well-known costs:

• Field names are repeated verbatim in every object, even when the schema is fixed.
• No native binary type — binary data requires base64 encoding.
• No compact integer encoding — 1000000 costs 7 bytes as text.
• No mechanism for structural reuse across a stream of similar records.

MessagePack

MessagePack addresses the encoding inefficiency of JSON without changing the data model: binary types, compact integer widths, and smaller overall payload size. It is a good default binary format.

But MessagePack still sends field names in every map, has no native shape-interning, and provides no batch or columnar modes. On workloads where the same object shape appears hundreds or thousands of times — telemetry, logs, API responses, database rows — the key overhead dominates.

Protocol Buffers / FlatBuffers / Cap'n Proto

Schema-first formats achieve the smallest possible payloads by agreeing on field positions and types ahead of time. But they require a shared schema and a code generation step. They cannot encode ad-hoc or schema-less data without workarounds, and they cannot gracefully degrade to a self-describing form when no schema is available.

What Twilic Does Differently

Twilic occupies the middle ground: schema-less usability with progressive schema-aware compression.

Scenario	JSON	MessagePack	Twilic
One-shot ad-hoc value	✓ verbose	✓ compact	✓ compact
Repeated object shape	✗ key overhead	✗ key overhead	✓ shape interning
Repeated field names	✗ verbose	✗ verbose	✓ key interning
Repeated string values	✗ verbose	✗ verbose	✓ string interning
Homogeneous int arrays	✗ text ints	○ fixed-width	✓ bitpack/delta/RLE
Tabular batch	✗	✗	✓ columnar batch
Stateful stream	✗	✗	✓ state patch
No schema needed	✓	✓	✓
Schema speeds it up	✗	✗	✓

When to Use Twilic

Twilic has a strong advantage when:

• Objects with the same shape appear repeatedly — API response lists, telemetry events, log records, database rows.
• Field names are long or numerous — REST APIs with verbose naming conventions.
• String values repeat — status fields, enum-like categories, user IDs in a session.
• Numeric arrays are dense — time series, sensor data, coordinates.
• Batching is available — sending multiple records together instead of one at a time.
• A long-lived stream exists — WebSocket connections, gRPC streams, message queue consumers.

Twilic's advantage over MessagePack shrinks on one-shot, non-repeating payloads. For those use cases, MessagePack or even JSON may be adequate.

Comparison with Benchmarks

Pinned sizes from the benchmark harness (@twilic/core 3.1.0, Node.js 24, N-API). Byte counts are deterministic for these fixtures; try them live in the Playground.

Dynamic vs MessagePack

Homogeneous 256-record batch (batch-homogeneous-256):

Format	Bytes	vs MessagePack
Twilic Dynamic (encodeBatch)	5,316	72.7% smaller
MessagePack	19,505	—
JSON (UTF-8)	28,202	—

Single-record single-small is a tie: Twilic and MessagePack both encode to 140 bytes (JSON 180). Twilic’s advantage appears when the same shape repeats.

Bound / Batch vs Protobuf / Avro

UserRecordV1 ×256, schema shared out of band (v3 contract):

Format	Bytes	vs Protobuf	vs Avro
Twilic BOUND_STREAM	2,395	30.7% smaller	16.0% smaller
Twilic SCHEMA_BATCH	798	76.9% smaller	72.0% smaller
Protobuf stream	3,458	—	—
Avro raw stream	2,852	—	—

• Single-record encoding: Twilic Dynamic ≈ MessagePack.
• Homogeneous batches: Twilic Dynamic is substantially smaller than MessagePack via shape and key interning.
• Schema-fixed streams: Twilic BOUND_STREAM beats schema-shared Protobuf/Avro raw streams on this fixture.
• Tabular packs: Twilic SCHEMA_BATCH is the strongest size mode when a shared schema is available.
• Stateful patch on hot object streams: Twilic can send only changed fields after the first full frame.

Full tables, assumptions, and regenerate commands: Benchmark.