Why Kafka for Data Pipelines
According to Confluent, Kafka is used by over 80% of Fortune 100 companies. LinkedIn processes over 7 trillion messages per day. Kafka dominates because of durability, ordering, and throughput — a single broker handles millions of messages per second.
Kafka Deployment Options
- Amazon MSK — Managed Apache Kafka. MSK Serverless for auto-scaling. Available in EU regions.
- Confluent Cloud — Kafka-as-a-service with Schema Registry, ksqlDB, managed connectors.
- Self-managed on Kubernetes — Using Strimzi or Confluent Operator. Most control, most burden.
Topic Design
Use the naming pattern: <domain>.<entity>.<event-type> (e.g., orders.order.created). Choose partition keys that evenly distribute load while maintaining ordering. Start with 6-12 partitions per topic.
Schema Management
Use Avro or Protobuf with Schema Registry. Set BACKWARD compatibility. Add new fields with defaults, never remove fields.
Data Ingestion with Kafka Connect
Change Data Capture with Debezium
Debezium captures row-level changes from databases and streams them into Kafka. The gold standard for database-to-Kafka integration.
Application Events
Produce directly using KafkaJS (TypeScript), with idempotent producers and LZ4 compression for optimal performance.
Stream Processing Options
- Kafka Streams — Java/Kotlin library, no separate cluster needed.
- Apache Flink — Distributed framework for complex event processing. Available as Amazon Managed Flink.
- Lambda consumers — AWS Lambda can consume from MSK. Best for simple transformations.
Sink Destinations
- S3 Sink (Parquet/Avro for data lake)
- Elasticsearch/OpenSearch (real-time search)
- JDBC Sink (relational databases)
- Snowflake/BigQuery (data warehouse)
Production Monitoring
Monitor consumer lag (most critical), under-replicated partitions (should always be zero), request latency p99, and disk utilisation. Use replication factor 3 with min.insync.replicas=2. For cross-region DR, use MirrorMaker 2.
Our DevOps consulting team has built streaming architectures for European enterprises. Contact us to discuss your data pipeline requirements.