Billion-Scale Data Analytics Engine
Distributed analytics engine processing 1B+ telecom & telemetry records, cutting query latency from minutes to milliseconds.
Overview
A distributed analytics engine designed to ingest, transform, and query datasets exceeding one billion telecom and telemetry records. The system replaced legacy batch-processing pipelines that took hours to produce reports, delivering the same insights with sub-second query latency.
At its core, the platform uses ClickHouse as a columnar data store optimized for analytical workloads. Data ingestion is parallelized across Dask clusters that handle partitioning, deduplication, and schema normalization before records land in ClickHouse. Prefect orchestrates the end-to-end pipeline, providing retry logic, dependency graphs, and observability for every workflow run.
A graph layer powered by Neo4j maps relationships between telecom entities — subscribers, cell towers, and network events — enabling multi-hop traversals that traditional SQL cannot express efficiently. DuckDB serves as an embedded analytical engine for ad-hoc exploration during development and QA cycles.
Key Achievements
- 1B+ telecom & telemetry records processed with sub-second query latency
- Query latency reduced from minutes to milliseconds (reports that once took hours now run interactively)
- Parallelized ingestion with partitioning, deduplication, and schema normalization at billion-record volume
- Zero-downtime deployments in air-gapped environments
Tech Stack
- ClickHouse — columnar storage with vectorized execution for analytical queries at telecom scale
- Dask — distributed parallel computation across worker clusters for ETL at billion-record volume
- Prefect — workflow orchestration with built-in retry, caching, and observability
- FastAPI — async Python API layer for low-latency service endpoints
- Neo4j — graph database for modeling telecom entity relationships and multi-hop traversals
- DuckDB — embedded analytics engine for fast ad-hoc exploration during development