CDAT-AI
Distributed telecom analytics engine processing 100Cr+ records with sub-second query latency.
Overview
CDAT-AI is a distributed telecom analytics engine designed to ingest, transform, and query datasets exceeding one billion records. The system was built from the ground up to replace legacy batch-processing pipelines that took hours to produce reports, delivering the same insights in sub-second query times.
At its core, the platform uses ClickHouse as a columnar data store optimized for analytical workloads. Data ingestion is parallelized across Dask clusters that handle partitioning, deduplication, and schema normalization before records land in ClickHouse. Prefect orchestrates the end-to-end pipeline, providing retry logic, dependency graphs, and observability for every workflow run.
A graph layer powered by Neo4j maps relationships between telecom entities -- subscribers, cell towers, and network events -- enabling multi-hop traversals that traditional SQL cannot express efficiently. DuckDB serves as an embedded analytical engine for ad-hoc exploration during development and QA cycles.
Key Achievements
- Processing 100Cr+ (1 billion+) telecom records with sub-second query latency
- Reduced report generation time from hours to milliseconds
- Built 200+ MCP tool integrations for AI agent interaction
- Zero-downtime deployments in air-gapped environments
Tech Stack
- ClickHouse -- columnar storage with vectorized execution for analytical queries at telecom scale
- Dask -- distributed parallel computation across worker clusters for ETL at billion-record volume
- Prefect -- workflow orchestration with built-in retry, caching, and observability
- FastAPI -- async Python API layer for low-latency service endpoints
- Neo4j -- graph database for modeling telecom entity relationships and multi-hop traversals
- DuckDB -- embedded analytics engine for fast ad-hoc exploration during development