hadoop和hbase接入snappy编译
2022-06-20 本文已影响0人
至垚
- 下载snappy源码包
http://pkgs.fedoraproject.org/repo/pkgs/snappy/ - 编译
对jdk1.8.0_191/jre/lib/amd64/server/libjvm.so二进制包软链到/usr/local/lib/libjvm.so
进入对应snappy包内
./autogen.sh
./configure
make -j4 && make install - 编译hadoop对应的snappy包
下载
https://code.google.com/archive/p/hadoop-snappy/source/default/source
编译
cd trunk
mvn clean package -DskipTests -Dsnappy.prefix=/usr/local/ - 复制hadoop的snappy二进制包到hadoop的lib目录
hadoop-snappy-0.0.1-SNAPSHOT.tar.gz解压
cp -r hadoop-snappy-0.0.1-SNAPSHOT/lib/* <HADOOP_HOME>/lib - 修改环境变量
hadoop-env.sh
export LD_LIBRARY_PATH=LD_LIBRARY_PATH
export JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/Linux-amd64-64/
测试
/data/hadoop/bin/hadoop checknative -a
- 对hbase新增snappy压缩
将hadoop-snappy-0.0.1-SNAPSHOT.jar复制到{HADOOP_HOME}/lib/native
LD_LIBRARY_PATH:
HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/
测试
bin/hbase --config /data/hbase/conf org.apache.hadoop.util.NativeLibraryChecker
