System Design

Day 4 of 10: Encoding, Schema Evolution & Data Flow (DDIA)

by Kaushal Bhatt
5 min read

🚀 Day 4 of 10: Designing Data-Intensive Applications

Software is constantly changing. New features ship. Business rules evolve. Schemas must change.

But in distributed systems, code cannot update everywhere at the same time.

During rolling upgrades, old and new versions of your application run side by side. If your data encoding is not designed for this, you get downtime or data loss.

Here’s what Chapter 4 of *Designing Data-Intensive Applications* teaches:

### 1. Two Representations of Data - In-memory: Objects, structs, pointers — optimized for the CPU - On the wire / on disk: Byte sequences — self-contained and portable

Moving between them is called encoding (serialization) and decoding (deserialization).

2. Textual vs Binary Encodings

Textual formats (JSON, XML, CSV) - Human-readable and widely supported - But: ambiguous numbers, no native binary, and very verbose

Binary schema-driven formats (Protocol Buffers, Thrift, Avro) - Much smaller and faster - Use field tags or schema resolution instead of repeating field names

Key differences: - Protobuf & Thrift → use numeric field tags. You can rename fields freely, but never change a tag number. - Avro → no field tags at all. Decoding works by comparing the writer’s schema with the reader’s schema. Perfect for dynamically generated schemas.

3. The Two Compatibility Rules You Must Follow

  • Backward Compatibility: Newer code can read data written by older code
  • Forward Compatibility: Older code can read data written by newer code (by ignoring unknown fields)

Best practices: 1. Never change existing field tags 2. Only add optional fields (or fields with defaults) 3. Always preserve unknown fields when reading → modifying → writing data

4. How Data Actually Flows

There are three main patterns:

  • Through Databases → Data outlives code. You will have records written by many different schema versions.
  • Through Services (REST / RPC) → REST + JSON for public APIs. Binary RPC (gRPC, Thrift) for internal high-speed communication.
  • Through Message Brokers → Asynchronous (Kafka, RabbitMQ). Producers and consumers are decoupled in time.

Key Takeaway

Evolvability is not optional in distributed systems. Choose schema-driven binary encodings, enforce strict backward + forward compatibility, and design your data flow so that old and new code can coexist peacefully.

Tomorrow (Day 5) we dive into Replication — how to keep data copies consistent across machines.

Drop a 🔥 if this helped. Comment “schema” if you’ve ever broken production with a schema change.

Who’s currently dealing with schema evolution pain? Let’s discuss.

#SystemDesign #SchemaEvolution #ProtocolBuffers #Avro #DistributedSystems #DesigningDataIntensiveApplications