spark window下 源码编译 hdp-2.6.5.0-2

1. 下载git，运行git bash，打包编译全部在git bash里进行
2. 配置 window环境
   JAVA_HOEM= C:\Program Files\Java\jdk1.8.0_51
MAVEN_HOME=C:\Program Files\Java\apache-maven-3.3.9
CLASSPAH = .;%JAVA_HOME%\lib\tools.jar;%JAVA_HOME%\lib\dt.jar
PATH =%JAVA_HOEM%\bin;%MAVEN_HOME%\bin;
   配置apache-maven-3.3.9\conf\settings.xml，添加阿里云maven镜像

<mirror>
  <id>aliyun</id>
  <mirrorOf>central</mirrorOf>
  <name>aliyun</name>
  <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

在spark源码根目录下修改pom.xml，添加hdp仓库

<repository>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <id>hortonworks.extrepo</id>
      <name>Hortonworks HDP</name>
      <url>http://repo.hortonworks.com/content/repositories/releases</url>
    </repository>

    <repository>
      <releases>
        <enabled>true</enabled>
      </releases>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <id>hortonworks.other</id>
      <name>Hortonworks Other Dependencies</name>
      <url>http://repo.hortonworks.com/content/groups/public</url>
    </repository>

3. 在git bash 环境下，进入到spark源码根目录
   执行export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
4. 在git bash 环境下 执行编译命令
   ./build/mvn -Pyarn -Phive -Phive-thriftserver -Phadoop-2.7 -Dhadoop.version=2.7.3.2.6.5.0-292 -DskipTests clean package

[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.871 s]
[INFO] Spark Project Tags ................................. SUCCESS [  4.643 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  5.465 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  3.399 s]
[INFO] Spark Project Networking ........................... SUCCESS [  6.305 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  3.803 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  6.325 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  8.329 s]
[INFO] Spark Project Core ................................. SUCCESS [03:34 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [  8.510 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 17.058 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 48.022 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:39 min]
[INFO] Spark Project SQL .................................. SUCCESS [04:57 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:34 min]
[INFO] Spark Project Tools ................................ SUCCESS [  1.197 s]
[INFO] Spark Project Hive ................................. SUCCESS [02:02 min]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 24.269 s]
[INFO] Spark Project REPL ................................. SUCCESS [  6.952 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [  6.469 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 23.803 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 11.613 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 11.489 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 17.401 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 26.982 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  3.199 s]
[INFO] Spark Avro ......................................... SUCCESS [ 10.273 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20:06 min
[INFO] Finished at: 2020-07-02T14:15:49+08:00
[INFO] Final Memory: 90M/1014M
[INFO] ------------------------------------------------------------------------

官网
https://spark.apache.org/docs/2.4.0/building-spark.html#apache-maven