Definition:
Apache Avro is a data serialization framework that provides compact, fast, and language-neutral data exchange. It uses JSON-based schemas to define data structures and serializes data into a compact binary format. Unlike other serialization formats, Avro includes the schema with the data or provides schema evolution capabilities, making it ideal for systems where data structures change over time while maintaining backward/forward compatibility.
Key Properties:
- Schema-based: Data structure defined in JSON schemas
- Compact binary format: Efficient storage and transmission
- Schema evolution: Add, remove, or modify fields while maintaining compatibility
- Language-neutral: Works across different programming languages
- Self-describing: Schema can be embedded with data or referenced separately
- Dynamic typing: Can handle data without compile-time schema knowledge
Core Concepts:
- Schema: JSON definition of data structure and types
- Schema registry: Central repository for schema management
- Schema evolution: Rules for compatible schema changes
- Serialization: Convert objects to binary format using schema
- Deserialization: Convert binary data back to objects
- Schema fingerprinting: Unique identification of schema versions
Example:
Apache Kafka + Avro:
Event Streaming Pipeline:
User Action → Kafka (Avro) → Stream Processing → Analytics DB
Benefits:
• Schema registry ensures data quality
• Backward/forward compatibility
• Compact message size
• Strong typing validation