[Spark]本地调试遇到的坑
2019-09-18 本文已影响0人
帅可儿妞
- scc.checkpoint 设置成本地路径, 报错:
- Exception in thread "main" java.io.FileNotFoundException: File file:/sparklink/statis/checkpoint/XXXX/does not exist
- scc.checkpoint在配置文件中设置成自己创建的目录, 但是报错:
Exception in thread "main" java.io.FileNotFoundException: File file:/sparklink/statis/checkpoint/test/dacad09a-6a0e-4ee8-9820-cb832a9cd515 does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:537) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:750) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:527) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409) at org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2118) at org.apache.spark.SparkContext$$anonfun$setCheckpointDir$2.apply(SparkContext.scala:2114) at scala.Option.map(Option.scala:146) at org.apache.spark.SparkContext.setCheckpointDir(SparkContext.scala:2114) at org.apache.spark.streaming.StreamingContext.checkpoint(StreamingContext.scala:241) at com.xylink.ai.consumer.Kafka2HiveXxx$.main(Kafka2HiveXxx.scala:27) at com.xylink.ai.consumer.Kafka2HiveXxx.main(Kafka2HiveXxx.scala) - 解决: 创建的目录的权限问题: chown -R shrekernil:admin /sparklink
