Hadoop

About Hadoop

Apache Hadoop is an open-source framework for distributed storage and processing of large datasets, developed by Doug Cutting and Mike Cafarella and first released in 2006, inspired by Google's MapReduce and GFS papers. Hadoop's two core components are HDFS (Hadoop Distributed File System) for storing data across commodity hardware clusters, and YARN (Yet Another Resource Negotiator) for cluster resource management. MapReduce (the original processing model) writes intermediate results to disk between steps, which proved slower than in-memory alternatives. The Hadoop ecosystem grew vast: Hive (SQL on HDFS), HBase (NoSQL on HDFS), Pig (data flow scripting), Sqoop (RDBMS import/export), Oozie (workflow scheduling), and Zookeeper (distributed coordination). Apache Spark replaced MapReduce as the primary processing engine due to in-memory computation — 100x faster for iterative algorithms. The term 'Hadoop' now often refers to the broader ecosystem rather than MapReduce specifically. On-premises Hadoop clusters (Cloudera, Hortonworks — now merged into CDP) were the dominant enterprise data platform from 2010–2018. Cloud data lakes (S3 + Athena, GCS + BigQuery, Azure Data Lake + Synapse) and managed Spark (Databricks, EMR) have largely replaced on-premises Hadoop. HDFS remains relevant as the storage layer for many Spark workloads, and the Hadoop ecosystem concepts (YARN, Hive metastore) underpin modern cloud data platforms.