资源下载 1.下载hadoop-2.5.2.tar.gzhttp://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz 2.下载hadoop2.5.2-eclipse-pluginhttp://download.csdn.net/detail/lzm1340458776/8519515 3.下载hadoop-common-2.2.0-binhttps://git
资源下载
1.下载hadoop-2.5.2.tar.gz http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
2.下载hadoop2.5.2-eclipse-plugin http://download.csdn.net/detail/lzm1340458776/8519515
3.下载hadoop-common-2.2.0-bin https://github.com/srccodes/hadoop-common-2.2.0-bin
开始安装
1.解压hadoop-2.5.2.tar.gz到win7下某个路径。
2.配置本地hadoop的环境
3.window-->preferences--->hadoop map/reduce 关联步骤1中解压的hadoop路径
4.window--->show view选择map/reduce locations
5.配置连接参数
注:host可以填写ip地址,也可以填写对应的主机名(要在c盘hosts文件配置对应的主机名),mapreduce的port可以默认,hdfs的port为core-site.xml文件中配置的端口。
6.查看文件系统
相关问题:
7.编写wordcount程序,运行可能会出现如下错误:
java.lang.nullpointerexception at java.lang.processbuilder.start(unknown source) at org.apache.hadoop.util.shell.runcommand(shell.java:482) at org.apache.hadoop.util.shell.run(shell.java:455) at org.apache.hadoop.util.shell$shellcommandexecutor.execute(shell.java:702) at org.apache.hadoop.util.shell.execcommand(shell.java:791) at org.apache.hadoop.util.shell.execcommand(shell.java:774) at org.apache.hadoop.fs.rawlocalfilesystem.setpermission(rawlocalfilesystem.java:646) at org.apache.hadoop.fs.rawlocalfilesystem.mkdirs(rawlocalfilesystem.java:434) at org.apache.hadoop.fs.filterfilesystem.mkdirs(filterfilesystem.java:281) at org.apache.hadoop.mapreduce.jobsubmissionfiles.getstagingdir(jobsubmissionfiles.java:125) at org.apache.hadoop.mapreduce.jobsubmitter.submitjobinternal(jobsubmitter.java:348) at org.apache.hadoop.mapreduce.job$10.run(job.java:1285) at org.apache.hadoop.mapreduce.job$10.run(job.java:1282) at java.security.accesscontroller.doprivileged(native method) at javax.security.auth.subject.doas(unknown source) at org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1614) at org.apache.hadoop.mapreduce.job.submit(job.java:1282) at org.apache.hadoop.mapreduce.job.waitforcompletion(job.java:1303) at com.lixue.mapreduce.chain.wordcountmapreduce.main(wordcountmapreduce.java:70)
出现这个问题的原因是window下缺少运行mr程序的工具。解决方案:
解压hadoop-common-2.2.0-bin,将解压后的bin目录覆盖hadoop解压后目录中的bin文件夹。
此时运行mapreduce程序,又会发现控制台只输出了如下信息:
log4j:warn no appenders could be found for logger (com.hp.hpl.jena.util.filemanager).log4j:warn please initialize the log4j system properly.log4j:warn see http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
解决方案:把hadoop-2.5.2/etc/hadoop/log4j.properties文件拷贝到项目的src目录下。
再次运行mapreduce程序,又出现了一个问题:
java.lang.runtimeexception: java.lang.instantiationexception at org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:131) at org.apache.hadoop.mapreduce.jobsubmitter.writenewsplits(jobsubmitter.java:490) at org.apache.hadoop.mapreduce.jobsubmitter.writesplits(jobsubmitter.java:510) at org.apache.hadoop.mapreduce.jobsubmitter.submitjobinternal(jobsubmitter.java:394) at org.apache.hadoop.mapreduce.job$10.run(job.java:1285) at org.apache.hadoop.mapreduce.job$10.run(job.java:1282) at java.security.accesscontroller.doprivileged(native method) at javax.security.auth.subject.doas(unknown source) at org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1614) at org.apache.hadoop.mapreduce.job.submit(job.java:1282) at org.apache.hadoop.mapreduce.job.waitforcompletion(job.java:1303) at com.lixue.mapreduce.chain.wordcountmapreduce.main(wordcountmapreduce.java:72)caused by: java.lang.instantiationexception at sun.reflect.instantiationexceptionconstructoraccessorimpl.newinstance(unknown source) at java.lang.reflect.constructor.newinstance(unknown source) at org.apache.hadoop.util.reflectionutils.newinstance(reflectionutils.java:129) ... 11 more
解决方案:把驱动类中的:
job.setinputformatclass(fileinputformat.class);
改成:job.setinputformatclass(textinputformat.class);