Pinot

Apache Pinot is an open-source real-time distributed OLAP datastore, originally developed at LinkedIn in 2013 to power user-facing analytics at scale and open-sourced in 2015. Pinot was donated to the Apache Software Foundation in 2018. Pinot's design philosophy is user-facing real-time analytics — powering LinkedIn's 'Who Viewed Your Profile,' 'Company Analytics,' and 'Job Analytics' features that serve hundreds of millions of members. Pinot ingests data in real-time from Kafka and Kinesis, stores it in columnar segments on deep storage (S3, HDFS), and serves multi-dimensional queries with consistent sub-second latency regardless of query complexity. Pinot's segment format uses star-tree indexing — a specialized multi-dimensional pre-aggregation structure that dramatically speeds up high-cardinality GROUP BY queries. Unlike Druid (which favors approximate analytics), Pinot provides exact real-time aggregations, and its upsert support (real-time record deduplication and updates) is stronger than Druid's. StarTree, founded by ex-LinkedIn engineers, offers Pinot as a managed cloud service (StarTree Cloud). Pinot integrates with Superset, Grafana, and Tableau for visualization. Major users include LinkedIn, Walmart, Etsy, Stripe, Uber, and Slack. Pinot's multi-stage query engine (MSQE) enables complex SQL joins across segments, addressing a historical limitation. Apache Pinot 1.0 shipped in 2023, followed by continued improvements in index types, real-time upserts, and cloud-native deployments.