by mike o’brien, mongodb kernel tools lead and maintainer of mongo-hadoop, the hadoop adapter for mongodb hadoop is a powerful, jvm-based platform for running map/reduce jobs on clusters of many machines, and it excels at doing analytics
by mike o’brien, mongodb kernel tools lead and maintainer of mongo-hadoop, the hadoop adapter for mongodb
hadoop is a powerful, jvm-based platform for running map/reduce jobs on clusters of many machines, and it excels at doing analytics and processing tasks on very large data sets.
since mongodb excels at storing large operational data sets for applications, it makes sense to explore using these together - mongodb for storage and querying, and hadoop for batch processing.
the mongodb connector for hadoopwe recently released the 1.1 release of the mongodb connector for hadoop. the mongodb connector for hadoop makes it easy to use mongo databases, or mongodb backup files in .bson format, as the input source or output destination for hadoop map/reduce jobs. by inspecting the data and computing input splits, hadoop can process the data in parallel so that very large datasets can be processed quickly.
the mongodb connector for hadoop also includes support for pig and hive, which allow very sophisticated mapreduce workflows to be executed just by writing very simple scripts.
pig is a high-level scripting language for data analysis and building map/reduce workflowshive is a sql-like language for ad-hoc queries and analysis of data sets on hadoop-compatible file systems.hadoop streaming is also supported, so map/reduce functions can be written in any language besides java. right now the mongodb connector for hadoop supports streaming in ruby, node.js and python.
how it workshow the hadoop connector works
the adapter examines the mongodb collection and calculates a set of splits from the dataeach of the splits gets assigned to a node in hadoop clusterin parallel, hadoop nodes pull data for their splits from mongodb (or bson) and process them locallyhadoop merges results and streams output back to mongodb or bsoni’ll be giving an hour-long webinar on what’s new with the mongo-hadoop integration. the webinar will cover
using java mapreduce with the mongodb connector for hadoopusing hadoop streaming for other non-jvm languageswriting pig scripts with the mongodb connector for hadoopmongodb and hadoop usage with elastic mapreduce to easily kick off your hadoop jobs
overview of mongoupdatewriteable: using the result output from hadoop to modify an existing output collection
the webinar will be offered twice on august 8:
8 am pdt / 11 am edt / 3pm utc11am pdt / 2pm edt / 6pm utcregister for the webinar on august 8
update: watch the webinar recording
原文地址：mongodb connector for hadoop, 感谢原作者分享。