Home

Storm Data ETL

A Go service that consumes raw weather storm reports from Kafka, enriches and normalizes the data, and produces transformed events to a downstream Kafka topic. Part of the storm data pipeline.

Pipeline Position

Collector --> Kafka --> [ETL] --> Kafka --> API --> PostgreSQL + GraphQL

Upstream: The collector service publishes raw CSV events to the raw-weather-reports topic.

Downstream: The API service consumes enriched events from the transformed-weather-data topic.

Uses the storm-data-shared library for logging, health endpoints, config parsing, and retry logic. For the full pipeline architecture, see the system wiki.

Pages

• Architecture -- Hexagonal design, package layout, configuration, design decisions, and capacity
• Development -- Build, test, lint, CI, and project conventions
• Enrichment Rules -- Transformation pipeline, normalization, and severity classification