Skip to main content

Table 2 Description of the Hadoop related projects/ecosystems

From: Applications of the MapReduce programming framework to clinical big data analysis: current landscape and future trends

Hadoop related project and technology Description Download URL
Avro • Avro is a framework for performing remote procedure calls and data serialization. http://avro.apache.org
Flume • Flume is a tool for harvesting, aggregating and moving large amounts of log data in and out of Hadoop. http://flume.apache.org
HBase • Based on Google’s Bigtable, HBase is an open-source, distributed, versioned, column-oriented store that sits on top of HDFS. HBase is column-based rather than row-based, which enables high-speed execution of operations performed over similar values across massive datasets. http://hbase.apache.org
HCatalog • An incubator-level project at Apache, HCatalog is a metadata and table storage management service for HDFS. http://Incubator.apache.org/hcatalog/
Hive • Hive provides a warehouse structure and SQL-like access for data in HDFS and other Hadoop input sources http://hive.apache.org
Mahout • Mahout is a scalable machine-learning and data mining library. http://mahout.apache.org
Oozie • Oozie is a job coordinator and workflow manager for jobs executed in Hadoop, which can include non-MapReduce jobs. http://oozie.apache.org
Pig • Pig is a framework consisting of a high-level scripting language (Pig Latin) and a run-time environment that allows users to execute MapReduce on a Hadoop cluster. http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html
Sqoop • Sqoop (SQL-to-Hadoop) is a tool which transfers data in both directions between relational systems and HDFS or other Hadoop data stores, e.g. Hive or HBase. http://sqoop.apache.org
ZooKeeper • ZooKeeper is a service for maintaining configuration information, naming, providing distributed synchronization and providing group services. http://zookeeper.apache.org
YARN • YARN is a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users’ applications. http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/YARN.html
Cascading • Cascading is an alternative API to Hadoop MapReduce. Cascading now has support for reading and writing data to and from a HBase cluster. http://wiki.apache.org/hadoop/Hbase/Cascading
Twitter Storm • Twitter Storm is a free and open source distributed real time computation system. http://storm.incubator.apache.org/
High performance computing cluster (HPCC) • HPCC is an open source, data-intensive computing system platform developed by LexisNexis Risk Solutions http://hpccsystems.com/
Dremel • Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data http://research.google.com/pubs/pub36632.html