Case Study: Designing a Real-Time Data Processing System

Table of Contents

1. Background: The Problem Statement
2. Key Requirements and Constraints
3. System Design: Architecture Overview
4. Component Deep-Dive
   • Data Ingestion Layer
   • Stream Processing Layer
   • Storage Layer
   • Consumption & Visualization Layer
5. Technical Challenges and Solutions
6. Implementation Phases
7. Performance Metrics and Results
8. Lessons Learned and Future Enhancements
9. References

Background: The Problem Statement

MealMover’s legacy infrastructure relied on batch processing: Data from user apps, driver GPS trackers, and restaurant POS systems was collected hourly, processed overnight, and used to update ETAs, inventory levels, and driver assignments the next day. This led to three critical pain points:

• Stale ETAs: Customers received order updates every 5–10 minutes, leading to frustration when drivers were delayed.
• Inefficient Driver Routing: Driver locations were outdated, causing suboptimal assignment of orders (e.g., a driver 2 miles away might be assigned instead of one 0.5 miles away).
• Inventory Stockouts: Restaurants often ran out of ingredients because inventory alerts were delayed by hours, leading to canceled orders.

The goal was to replace batch processing with a real-time system that could process data in milliseconds to seconds, enabling dynamic ETAs, efficient routing, and proactive inventory management.

Key Requirements and Constraints

To guide the design, the team defined clear requirements and constraints:

Functional Requirements

1. Real-Time Order Tracking: Update order status (e.g., “preparing,” “on the way”) and driver location to customers every 10–30 seconds.
2. Dynamic ETA Calculation: Compute ETAs based on live traffic, driver speed, and restaurant preparation time (latency < 1 second).
3. Inventory Alerts: Notify restaurants when ingredient stock drops below a threshold (e.g., < 5 servings of a dish).
4. Driver Assignment Optimization: Match incoming orders to the nearest available driver in real time.

Non-Functional Requirements

• Latency: End-to-end processing latency < 1 second (99th percentile).
• Throughput: Handle 10,000+ events/second (peak load during lunch/dinner rushes).
• Reliability: 99.9% uptime (max 8.76 hours of downtime/year).
• Scalability: Scale horizontally to support 2x growth in orders over 6 months.

Constraints

• Hybrid Infrastructure: MealMover uses a mix of on-premises servers (for sensitive customer data) and cloud (AWS) for scalable processing.
• Budget: $500k annual cost cap for new infrastructure (excluding labor).
• Compliance: GDPR/CCPA compliance for user location and order data (data retention < 30 days).

System Design: Architecture Overview

The team designed a streaming-first architecture with four core layers:

Figure 1: High-level architecture of MealMover’s real-time data processing system.

1. Data Ingestion Layer: Collects raw data from sources (apps, GPS, POS) and streams it to processing.
2. Stream Processing Layer: Processes data in real time (filtering, transforming, aggregating) and enriches it with context (e.g., traffic data).
3. Storage Layer: Stores processed data for real-time access (hot data) and long-term analysis (cold data).
4. Consumption Layer: Delivers insights to end-users (customers, drivers, restaurants) via APIs and dashboards.

Component Deep-Dive

1. Data Ingestion Layer

Goal: Reliably collect and buffer high-velocity data from diverse sources.

Sources

• User Mobile Apps: Order placements, status updates (e.g., “order received”).
• Driver GPS Trackers: Location (latitude/longitude), speed, battery level (sent every 5 seconds).
• Restaurant POS Systems: Ingredient usage, order preparation status (e.g., “cooking started”).
• Third-Party APIs: Traffic data (Google Maps), weather (OpenWeatherMap).

Tools & Architecture

The team chose Apache Kafka as the central message broker for its high throughput, durability, and scalability. Key design choices:

• Topics: Organized data into Kafka topics (e.g., driver-locations, order-events, inventory-updates).
• Partitioning: Topics were partitioned by key (e.g., driver-id for driver-locations) to ensure ordered processing of a driver’s location history.
• Producers: Custom Kafka producers in Python (for POS systems) and Java (for mobile apps) with retries and idempotent delivery to avoid duplicates.
• Connectors: Used Kafka Connect with JDBC connectors to ingest data from on-prem POS databases (MySQL) and S3 connectors for cloud-based logs.

Result: Ingestion latency < 100ms, with support for 15,000 events/second (exceeding the 10k requirement).

2. Stream Processing Layer

Goal: Process streaming data in real time to generate actionable insights.

Tools & Workflows

The team evaluated Apache Spark Streaming and Apache Flink. Flink was chosen for its stateful processing capabilities (critical for tracking order/driver state) and low latency.

Key workflows included:

• Order-Driver Matching:

  • Input: order-events (new orders) and driver-locations (live GPS).
  • Logic: For each new order, compute the distance from nearby drivers (using Haversine formula) and assign the closest available driver.
  • Output: assigned-orders topic.

• Dynamic ETA Calculation:

  • Input: driver-locations, order-events, and traffic-data.
  • Logic: Enrich driver location with real-time traffic (e.g., “5 mph in downtown”) to compute ETA using historical speed patterns (e.g., “driver A averages 15 mph in this neighborhood”).
  • Output: eta-updates topic (pushed to customer apps via WebSockets).

• Inventory Alerting:

  • Input: inventory-updates (ingredient usage from POS).
  • Logic: Track ingredient stock levels in a Flink state store; trigger alerts when stock < threshold (e.g., “5 servings left of ‘Margherita Pizza’”).
  • Output: inventory-alerts topic (sent to restaurant dashboards).

State Management

To track state (e.g., current driver availability, ingredient stock), Flink used a RocksDB state backend with incremental checkpoints. This reduced memory usage by storing state on disk and only keeping hot data in memory.

Result: Processing latency < 500ms (99th percentile), with support for 10k events/second.

3. Storage Layer

Goal: Store data for real-time access (hot) and long-term analytics (cold).

Hot Storage (Real-Time Access)

• Redis: Caches frequently accessed data (e.g., current ETA for an order, active driver locations) with a TTL of 5 minutes.
• InfluxDB: Time-series database for metrics (e.g., driver speed over time, order latency) with high write throughput.

Cold Storage (Long-Term Analysis)

• Apache Cassandra: Distributed NoSQL database for order history and driver logs (scales horizontally; supports 100M+ rows).
• Amazon S3: Archives raw data (JSON/Parquet) for batch analytics (e.g., weekly sales reports) with lifecycle policies to move old data to S3 Glacier.

Data Flow

Processed data from Flink is written to both hot (Redis/InfluxDB) and cold (Cassandra/S3) storage via Kafka Connect sinks.

Result: Hot data access latency < 10ms; cold storage scales to 10TB/year.

4. Consumption Layer

Goal: Deliver real-time insights to end-users.

APIs & Dashboards

• REST APIs: Built with Node.js/Express to serve ETA updates and order status to mobile apps (e.g., GET /orders/{id}/eta).
• WebSockets: Used Socket.io to push real-time ETA updates to customer browsers (e.g., “Your order is 2 minutes away!”).
• Grafana Dashboards: For restaurant managers (inventory levels, alert status) and operations teams (system health: latency, throughput, error rates).

Example Workflow: Customer ETA Update

1. Driver’s GPS sends location to Kafka (driver-locations topic).
2. Flink processes the location, computes new ETA, and writes to eta-updates topic.
3. Kafka Connect streams eta-updates to Redis.
4. Node.js API reads from Redis and pushes via WebSocket to the customer’s app.

Result: End-to-end latency (GPS → customer app) < 1 second.

Technical Challenges and Solutions

Challenge 1: Backpressure During Traffic Spikes

Issue: Lunch/dinner rushes caused 2x spikes in events (20k events/sec), overwhelming Flink consumers.
Solution: Kafka topic partitioning + consumer group scaling. The team increased Kafka partitions for high-traffic topics (e.g., driver-locations from 10 to 20 partitions) and configured Flink to auto-scale consumer parallelism via Kubernetes HPA.

Challenge 2: Duplicate Order Events

Issue: Mobile apps occasionally resend order events (due to network flakiness), leading to duplicate orders.
Solution: Idempotent processing. Flink used order IDs as unique keys and stored processed IDs in a Redis set to filter duplicates. Kafka producers were configured with enable.idempotence=true to avoid duplicate writes to topics.

Challenge 3: State Size Explosion

Issue: Flink’s state store (tracking 100k active orders + 50k drivers) grew to 50GB, slowing checkpoints.
Solution: State TTL and compaction. The team set a TTL of 24 hours for order state (orders are delivered within 2 hours) and used RocksDB’s compaction to prune old data.

Challenge 4: Data Consistency Across Storage

Issue: Inconsistent ETA values between Redis (hot) and Cassandra (cold) due to write failures.
Solution: Kafka exactly-once semantics (EOS). Flink’s EOS integration with Kafka ensured that writes to storage were atomic—either fully committed or rolled back.

Implementation Phases

Phase 1: Prototyping (Weeks 1–4)

• Built a minimal viable pipeline: Kafka → Flink (simple ETA calculation) → Redis → WebSocket API.
• Tested with simulated data (1k events/sec) to validate latency (< 1s) and throughput.

Phase 2: Testing (Weeks 5–8)

• Load Testing: Used JMeter to simulate 20k events/sec; validated system stability.
• Fault Injection: Killed Kafka brokers/Flink tasks to test recovery (mean time to recover < 2 minutes).
• Compliance Audit: Verified GDPR compliance (data encryption in transit/rest, 30-day retention).

Phase 3: Deployment (Weeks 9–12)

• Phased Rollout: Migrated 10% of drivers/restaurants to the new system, then scaled to 100% over 4 weeks.
• Orchestration: Deployed components on Kubernetes (EKS for cloud, on-prem K8s for data ingestion) for auto-scaling.

Phase 4: Monitoring (Ongoing)

• Metrics: Prometheus collected latency (Kafka, Flink), throughput, and error rates.
• Logs: ELK Stack (Elasticsearch, Logstash, Kibana) centralized logs for debugging.
• Alerting: PagerDuty alerts for critical issues (e.g., latency > 1s for 5 minutes).

Performance Metrics and Results

Lessons Learned and Future Enhancements

Key Lessons

• State Management is Critical: Underestimating state size (e.g., driver history) led to initial Flink performance issues. Invest in state backends (RocksDB) and TTL early.
• Test for Spikes: Simulating traffic spikes (e.g., 2x normal load) during testing prevented production outages.
• Hybrid Storage Works: Combining Redis (hot) and Cassandra (cold) balanced speed and cost.

Future Enhancements

• Edge Processing: Deploy Flink on edge devices (drivers’ phones) to process GPS data locally, reducing cloud latency.
• ML-Driven ETAs: Train a model on historical ETA accuracy to predict delays (e.g., “driver A is 10% slower in rain”).
• Cost Optimization: Migrate cold storage to S3 Glacier Deep Archive for long-term data (> 90 days).

References

1. Apache Kafka Documentation. kafka.apache.org/documentation/
2. Apache Flink Stateful Stream Processing. flink.apache.org/features/stateful-stream-processing.html
3. InfluxDB Time-Series Data Guide. docs.influxdata.com/influxdb/v2.0/write-data/
4. Confluent. “Building a Real-Time ETA System with Kafka and Flink.” confluent.io/blog/real-time-eta-system-kafka-flink/
5. AWS. “Hybrid Cloud Architecture Patterns.” aws.amazon.com/architecture/hybrid/