How Enterprise Teams Scale to 20M+ Statements

The Challenge

A global enterprise learning program spanning dozens of business units generates tens of millions of xAPI statements a year, from compliance training to onboarding to simulation-based assessments. At that volume, the bottleneck shifts from "can we store this" to "can we query it fast enough to matter."

Architecture Decisions That Made the Difference

• ClickHouse as the analytical query layer, purpose-built for column-oriented aggregation at scale
• AWS S3 or Azure Storage for durable, cost-efficient object storage of raw statements and attachments
• Statement validation and normalization at ingestion, not query time
• Registration-based grouping to avoid expensive joins across related statements

Query Latency Under 250ms

By pre-aggregating common report dimensions and keeping the hot query path entirely inside ClickHouse, even dashboards spanning hundreds of millions of rows stay responsive — most interactive queries return in well under 250ms. The platform operations view is where that latency is actually watched.

Results

The program now processes statement volume at enterprise scale without a dedicated data engineering team managing the pipeline, freeing L&D leaders to focus on what the data shows rather than how to query it. Moving existing history in at this volume is its own exercise: bulk import covers the mechanics.