Menu
GitHub LinkedIn
Back to projects

Real-Time Market Analytics Pipeline

A path for every event.

A distributed AWS pipeline for live market data: Kafka ingestion, Spark Structured Streaming features, an S3 data lake, 43 daily ML features, and a FastAPI dashboard, all provisioned with Terraform.

At a glance

  • 43Daily ML features per symbol
  • 4Windows: 1m, 5m, 15m, 60m
  • 30 sMicro-batches
  • GoalLearn to split ingestion, streaming, storage and serving across machinesBuiltKafka, Spark, S3 and an API on four EC2 instances
  • 2 minWatermark for late candles
  • Daily batch: EventBridge → Lambda → SSM.
    1. Kafka
    2. Spark
    3. S3 + PostgreSQL
    4. FastAPI
    • Kafka
    • Spark
    • PostgreSQL
    • FastAPI
    • Terraform
Real-Time Market Analytics Pipeline · Streaming and daily batch on AWS
  1. Producer — EC2 · yfinance, every 60 s
  2. EventBridge — daily, 11 p.m. UTC
  3. Kafka — 3 partitions, by symbol
  4. Lambda → SSM — starts the job
  5. Spark Streaming — 1m / 5m / 15m / 60m windows
  6. Spark batch — 43 daily features
  7. S3 data lake — Parquet · symbol / date
  8. PostgreSQL — serving store
  9. FastAPI — API + dashboard

The story

The goal

The goal was to learn how a real distributed pipeline splits ingestion, streaming, storage and serving across separate machines, instead of running everything on one.

Ingestion

A producer on EC2 polls 1-minute candles every 60 seconds and publishes them to Kafka, keyed by symbol across three partitions so each stock's candles stay in order.

Streaming

Spark Structured Streaming runs micro-batches every 30 seconds over 1-minute, 5-minute, 15-minute and 60-minute windows, with a 2-minute watermark for late candles. It writes Parquet to an S3 data lake, partitioned by symbol and date, and writes each window to PostgreSQL: 1-minute windows are appended, longer ones upserted.

Daily batch

Every evening, EventBridge triggers a Lambda that uses SSM to start a Spark batch job. It computes 43 lagged features and forward-return labels for machine learning.

Serving

PostgreSQL keeps a short hot window for serving, and a FastAPI service with a Jinja2 dashboard shows live prices, streaming features and daily features.

Infrastructure

Terraform provisions the VPC and four EC2 instances: Kafka, the producer, Spark, and the API with its database. The course deployment uses a single broker and single-node Spark; the layout is designed so it could move to MSK and EMR without changing the pipeline code.

The code is private. I'm happy to walk through it.