Hadoop

win10编译hadoop3.0.0-beta1源码，获取win

2017-10-24  本文已影响0人  夜雨辉晨

最近在阿里云主机上安装了hadoop3.0.0伪分布式，但是在windows使用eclipse连接时，没找到对应版本的hadoop.dll和winutils.exe，便萌生了自己编译的想法。在此记录一下主要步骤和踩过的坑。

安装需要的编译工具

首先肯定要按照源码根目录中附带的BUILDING文档安装编译所需的各种工具。以下为BUILDING文件中对windows编译的部分要求。建议通读该文档后，再着手hadoop源码的编译。

Building on Windows

Requirements:

  • Windows System
  • JDK 1.8+
  • Maven 3.0 or later
  • ProtocolBuffer 2.5.0
  • CMake 3.1 or newer
  • Windows SDK 7.1 or Visual Studio 2010 Professional
  • Windows SDK 8.1 (if building CPU rate control for the container executor)
  • zlib headers (if building native code bindings for zlib)
  • Internet connection for first build (to fetch all Maven and Hadoop dependencies)
  • Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
    tools must be present on your PATH.
  • Python ( for generation of docs using 'mvn site')

在安装相关软件的过程中，可以参考Windows 10编译Hadoop 2.6.0源码和在Windows 10 64-bit上安装Windows SDK 7.1和.NET4。这两篇博文已经十分详细，我就不再赘述，只在这里写一下我碰到的问题。

1.关于zlib的编译安装。建议在安装make工具之后，下载到zlib1.2.11源代码，将zlib1.2.11/win32中的Makefile.gcc复制到zlib1.2.11目录下，然后在命令行中使用
make -f makefile.gcc
进行编译，得到zlib1.dll，然后在新建系统变量ZLIB_HOME="你的zlib目录（例如C:\program1\zlib-1.2.11）"。
2.关于Maven仓库的位置。BUILDING文档建议hadoop源码的目录路径尽量短，事实上，最好Maven仓库的位置也设置到根目录。最开始时，由于Maven仓库路径过长，导致我在使用windowsSDK7编译hadoop源码的过程中，在某一处产生的命令行长度超过限制而始终失败。
在此处产生的报错信息为：

Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]

详细报错信息为：

[INFO] --- native-maven-plugin:1.0-alpha-8:javah (default) @ hadoop-common ---
[INFO] cmd.exe /X /C "C:\Java\jdk1.8.0_151\bin\javah -d C:\hadoop\hadoop-common-project\hadoop-common\target\native\javah -classpath C:\hadoop\hadoop-common-project\hadoop-common\target\classes;C:\hadoop\hadoop-common-project\hadoop-annotations\target\hadoop-annotations-3.0.0-beta1.jar;C:\Java\jdk1.8.0_151\jre\..\lib\tools.jar;C:\Users\fyyc\.m2\repository\com\google\guava\guava\11.0.2\guava-11.0.2.jar;C:\Users\fyyc\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;C:\Users\fyyc\.m2\repository\org\apache\httpcomponents\httpclient\4.5.2\httpclient-4.5.2.jar;C:\Users\fyyc\.m2\repository\org\apache\httpcomponents\httpcore\4.4.4\httpcore-4.4.4.jar;C:\Users\fyyc\.m2\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;C:\Users\fyyc\.m2\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;C:\Users\fyyc\.m2\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;C:\Users\fyyc\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\fyyc\.m2\repository\javax\servlet\javax.servlet-api\3.1.0\javax.servlet-api-3.1.0.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-server\9.3.19.v20170502\jetty-server-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-http\9.3.19.v20170502\jetty-http-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-io\9.3.19.v20170502\jetty-io-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-util\9.3.19.v20170502\jetty-util-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-servlet\9.3.19.v20170502\jetty-servlet-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-security\9.3.19.v20170502\jetty-security-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-webapp\9.3.19.v20170502\jetty-webapp-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\org\eclipse\jetty\jetty-xml\9.3.19.v20170502\jetty-xml-9.3.19.v20170502.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-core\1.19\jersey-core-1.19.jar;C:\Users\fyyc\.m2\repository\javax\ws\rs\jsr311-api\1.1.1\jsr311-api-1.1.1.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-servlet\1.19\jersey-servlet-1.19.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-json\1.19\jersey-json-1.19.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;C:\Users\fyyc\.m2\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;C:\Users\fyyc\.m2\repository\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-jaxrs\1.9.13\jackson-jaxrs-1.9.13.jar;C:\Users\fyyc\.m2\repository\org\codehaus\jackson\jackson-xc\1.9.13\jackson-xc-1.9.13.jar;C:\Users\fyyc\.m2\repository\com\sun\jersey\jersey-server\1.19\jersey-server-1.19.jar;C:\Users\fyyc\.m2\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;C:\Users\fyyc\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\fyyc\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;C:\Users\fyyc\.m2\repository\commons-beanutils\commons-beanutils\1.9.3\commons-beanutils-1.9.3.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-configuration2\2.1.1\commons-configuration2-2.1.1.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar;C:\Users\fyyc\.m2\repository\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar;C:\Users\fyyc\.m2\repository\org\slf4j\slf4j-log4j12\1.7.25\slf4j-log4j12-1.7.25.jar;C:\Users\fyyc\.m2\repository\org\apache\avro\avro\1.7.4\avro-1.7.4.jar;C:\Users\fyyc\.m2\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\fyyc\.m2\repository\org\xerial\snappy\snappy-java\1.0.4.1\snappy-java-1.0.4.1.jar;C:\Users\fyyc\.m2\repository\com\google\re2j\re2j\1.1\re2j-1.1.jar;C:\Users\fyyc\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\fyyc\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\hadoop\hadoop-common-project\hadoop-auth\target\hadoop-auth-3.0.0-beta1.jar;C:\Users\fyyc\.m2\repository\com\nimbusds\nimbus-jose-jwt\4.41.1\nimbus-jose-jwt-4.41.1.jar;C:\Users\fyyc\.m2\repository\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;C:\Users\fyyc\.m2\repository\net\minidev\json-smart\2.3\json-smart-2.3.jar;C:\Users\fyyc\.m2\repository\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;C:\Users\fyyc\.m2\repository\org\ow2\asm\asm\5.0.4\asm-5.0.4.jar;C:\Users\fyyc\.m2\repository\org\apache\curator\curator-framework\2.12.0\curator-framework-2.12.0.jar;C:\Users\fyyc\.m2\repository\com\jcraft\jsch\0.1.54\jsch-0.1.54.jar;C:\Users\fyyc\.m2\repository\org\apache\curator\curator-client\2.12.0\curator-client-2.12.0.jar;C:\Users\fyyc\.m2\repository\org\apache\curator\curator-recipes\2.12.0\curator-recipes-2.12.0.jar;C:\Users\fyyc\.m2\repository\com\google\code\findbugs\jsr305\3.0.0\jsr305-3.0.0.jar;C:\Users\fyyc\.m2\repository\org\apache\htrace\htrace-core4\4.1.0-incubating\htrace-core4-4.1.0-incubating.jar;C:\Users\fyyc\.m2\repository\org\apache\zookeeper\zookeeper\3.4.9\zookeeper-3.4.9.jar;C:\Users\fyyc\.m2\repository\io\netty\netty\3.10.5.Final\netty-3.10.5.Final.jar;C:\Users\fyyc\.m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;C:\Users\fyyc\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-simplekdc\1.0.1\kerb-simplekdc-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-client\1.0.1\kerb-client-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-config\1.0.1\kerby-config-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-core\1.0.1\kerb-core-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-pkix\1.0.1\kerby-pkix-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-asn1\1.0.1\kerby-asn1-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-util\1.0.1\kerby-util-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-common\1.0.1\kerb-common-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-crypto\1.0.1\kerb-crypto-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-util\1.0.1\kerb-util-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\token-provider\1.0.1\token-provider-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-admin\1.0.1\kerb-admin-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-server\1.0.1\kerb-server-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerb-identity\1.0.1\kerb-identity-1.0.1.jar;C:\Users\fyyc\.m2\repository\org\apache\kerby\kerby-xdr\1.0.1\kerby-xdr-1.0.1.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.7.8\jackson-databind-2.7.8.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.7.8\jackson-annotations-2.7.8.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.7.8\jackson-core-2.7.8.jar;C:\Users\fyyc\.m2\repository\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;C:\Users\fyyc\.m2\repository\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar org.apache.hadoop.io.compress.zlib.ZlibCompressor org.apache.hadoop.io.compress.zlib.ZlibDecompressor org.apache.hadoop.security.JniBasedUnixGroupsMapping org.apache.hadoop.io.nativeio.NativeIO org.apache.hadoop.security.JniBasedUnixGroupsNetgroupMapping org.apache.hadoop.io.compress.snappy.SnappyCompressor org.apache.hadoop.io.compress.snappy.SnappyDecompressor org.apache.hadoop.io.compress.zstd.ZStandardCompressor org.apache.hadoop.io.compress.zstd.ZStandardDecompressor org.apache.hadoop.io.compress.lz4.Lz4Compressor org.apache.hadoop.io.compress.lz4.Lz4Decompressor org.apache.hadoop.io.erasurecode.ErasureCodeNative org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawDecoder org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder org.apache.hadoop.crypto.OpensslCipher org.apache.hadoop.crypto.random.OpensslSecureRandom org.apache.hadoop.util.NativeCrc32"
命令行太长。
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 27.710 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 15.316 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  2.078 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.250 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.392 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.499 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  6.374 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.876 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  9.691 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.084 s]
[INFO] Apache Hadoop Common ............................... FAILURE [ 14.188 s]
[INFO] Apache Hadoop NFS .................................. SKIPPED
[INFO] Apache Hadoop KMS .................................. SKIPPED
[INFO] Apache Hadoop Common Project ....................... SKIPPED
[INFO] Apache Hadoop HDFS Client .......................... SKIPPED
[INFO] Apache Hadoop HDFS ................................. SKIPPED
[INFO] Apache Hadoop HDFS Native Client ................... SKIPPED
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:32 min
[INFO] Finished at: 2017-10-24T11:15:25+08:00
[INFO] Final Memory: 96M/848M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

可以看到，在此处由于命令行太长而导致编译失败。因此需要将maven仓库的默认路径设置为短路径，并且被编译的hadoop源码的路径也尽量短。

3.关于windowsSDK7的安装。可能需要修改注册表，在修改注册表时，可能需要修改该注册表数值的所属用户。具体可以参考在Windows 10 64-bit上安装Windows SDK 7.1和.NET4。

4.在编译时，最好以管理员模式打开编译工具。

进行编译

我使用的是WindowsSDK7,以管理员模式打开，执行如下命令编译。
mvn package -Pdist,native-win -DskipTests -Dtar

获取hadoop.dll和winutils.exe

编译成功后，在"hadoop-common-project\hadoop-common\target\bin"中，可以找到编译好的hadoop.dll和winutils.exe

本人还是学生，所知甚少，文中若有错误，请不吝指出。

上一篇下一篇

猜你喜欢

热点阅读