执行了一个简单的官方提供的作业:
iie4bu@swarm-worker1:~/app/hadoop-2.6.0-cdh5.15.1/share/hadoop/mapreduce$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar pi 2 3
报错信息如下:

Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:392)
	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:307)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:366)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)

解决办法:
yarn.nodemanager.local-dirs对应的路径必须在hdfs-site.xml中的hadoop.tmp.dir路径下,否则yarn会报错找不到缓存文件

    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/home/iie4bu/app/tmp/hadoop-tmp/nm-local-dir</value>
    </property>
Logo

一站式 AI 云服务平台

更多推荐