spark作业提交失败分析

版权声明：本文为博主原创文章，未经博主允许不得转载。https://www.jianshu.com/p/a4bf0f7173f5

提交一个spark作业，报错：

error cluster.yarnclientschedulerbackend：the yarn application has already ended！It might have been killed or the Application Master may have failed to start.

error spark.sparkcontext：error inilializing sparkcontext.

再提交一个yarn作业，

hadoop jar /opt/cloudera/parcels/CDH-6.1-/jars/hadoop-examples.jar pi 10 1

报错：

Application application_1560473958049_0005 failed 2 times due to AM Container

for appattempt_1560473958049_0005_000002 exited with exitCode: -1000

Diagnostics: Not able to initialize app directories in any of the configured local directories for app application_1560473958049_0005.

可以看出，是在yarn初始化作业的时候就报错了，其实是一个权限问题。未启用kerberos前目录权限为yarn:yarn，启用后变成wangkuan:yarn，导致权限不兼容。查看yarn.nodemanager.local-dirs，rm -rf 该目录/usercache/*，之后再重启yarn即可。