Hive源码解读（2）shell脚本解读

2019-04-09 本文已影响0人井地儿

在上文中我们得知，Hive的入口脚本都在bin/etc目录下，启动hive客户端的过程就是通过这些脚本启动的。细心一点的话，我们在hive脚本中看到了服务列表（SERVICE_LIST），这些脚本正是Hive Server清单。

Hive 服务清单（Hive Server List）

选项	名称	描述
cli	命令行界面	hive 的命令行交互界面
beeline	beeline连接	beeline连接服务
hiveserve	Hive Serve	监听来自其他进程的thrift连接的守护进程
hiveserve2	Hive Serve 2	另一个版本的Hive Server
hwi	Hive Web Interface	Hive的Web界面服务
jar		hadoop jar命令的包装，提供Hive环境下的hadoop jar支持
lineage	血缘关系	提供hive血缘关系分析
metastore	元数据存储	元数据存储服务
metatool	元数据工具	提供元数据操作的一些服务
orcfiledump	-	-
rcfilecat	-	-
schemaTool	-	-
version	-	-

beeline.sh

我们可以看出，脚本里主要定义了beeline函数。

beeline主类。org.apache.hive.beeline.Beeline
beeline启动jar。hive-beeline-*.jar

THISSERVICE=beeline
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

beeline () {
  CLASS=org.apache.hive.beeline.BeeLine;

  # include only the beeline client jar and its dependencies
  beelineJarPath=`ls ${HIVE_LIB}/hive-beeline-*.jar`
  superCsvJarPath=`ls ${HIVE_LIB}/super-csv-*.jar`
  jlineJarPath=`ls ${HIVE_LIB}/jline-*.jar`
  jdbcStandaloneJarPath=`ls ${HIVE_LIB}/../jdbc/hive-jdbc-*-standalone.jar`
  hadoopClasspath=""
  if [[ -n "${HADOOP_CLASSPATH}" ]]
  then
    hadoopClasspath="${HADOOP_CLASSPATH}:"
  fi
  export HADOOP_CLASSPATH="${hadoopClasspath}${HIVE_CONF_DIR}:${beelineJarPath}:${superCsvJarPath}:${jlineJarPath}:${jdbcStandaloneJarPath}"
  export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dlog4j.configuration=beeline-log4j.properties "

  exec $HADOOP jar ${beelineJarPath} $CLASS $HIVE_OPTS "$@"
}

beeline_help () {
  beeline "--help"
}

cli.sh

Command line interface。命令行界面。cli.sh脚本主要定义了cli函数。

cli主类。org.apache.hadoop.hive.cli.CliDriver

THISSERVICE=cli
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

cli () {
  CLASS=org.apache.hadoop.hive.cli.CliDriver
  execHiveCmd $CLASS "$@"
}

cli_help () {
  CLASS=org.apache.hadoop.hive.cli.CliDriver
  execHiveCmd $CLASS "--help"
}

hiveburninclient.sh

hiveburninclient.sh定义了hiveburninclient函数。

hiveburninclient主类。org.apache.hive.testutils.jdbc.HiveBurnInClient

THISSERVICE=hiveburninclient
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

hiveburninclient() {
  echo "Starting hiveburninclient"
  CLASS=org.apache.hive.testutils.jdbc.HiveBurnInClient
  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi
  JAR=${HIVE_LIB}/hive-service-*.jar
  exec $HADOOP jar $JAR $CLASS $HIVE_OPTS "$@"
}

hiveburninclient_help() {
  hiveburninclient -H
}

hiveserver.sh

Hive Server。hiveserver.sh定义了hiveserver函数。

hiveserver主类。org.apache.hadoop.hive.service.HiveServer

THISSERVICE=hiveserver
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

hiveserver() {
  echo "Starting Hive Thrift Server"
  CLASS=org.apache.hadoop.hive.service.HiveServer
  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi
  JAR=${HIVE_LIB}/hive-service-*.jar

  # hadoop 20 or newer - skip the aux_jars option and hiveconf

  exec $HADOOP jar $JAR $CLASS $HIVE_OPTS "$@"
}

hiveserver_help() {
  hiveserver -h
}

hiveserver2.sh

Hive Server2。hiveserver2.sh中定义了hiveserver函数。

hiveserver2主类。org.apache.hive.service.server.HiveServer2

THISSERVICE=hiveserver2
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

hiveserver2() {
  CLASS=org.apache.hive.service.server.HiveServer2
  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi
  JAR=${HIVE_LIB}/hive-service-*.jar

  exec $HADOOP jar $JAR $CLASS $HIVE_OPTS "$@"
}

hiveserver2_help() {
  hiveserver2 -H
}

hwi.sh

Hive Web界面。hwi.sh定义了hwi函数。

hwi主类。org.apache.hadoop.hive.hwi.HWIServer

THISSERVICE=hwi
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

hwi() {

  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi

  CLASS=org.apache.hadoop.hive.hwi.HWIServer
  # The ls hack forces the * to be expanded which is required because 
  # System.getenv doesn't do globbing
  export HWI_JAR_FILE=$(ls ${HIVE_LIB}/hive-hwi-*.jar)
  export HWI_WAR_FILE=$(ls ${HIVE_LIB}/hive-hwi-*.war)

  #hwi requires ant jars
  if [ "$ANT_LIB" = "" ] ; then
    ANT_LIB=/opt/ant/lib
  fi
  for f in ${ANT_LIB}/*.jar; do
    if [[ ! -f $f ]]; then
      continue;
    fi
    HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f
  done

  export HADOOP_CLASSPATH
  
  # hadoop 20 or newer - skip the aux_jars option and hiveconf
  exec $HADOOP jar ${HWI_JAR_FILE} $CLASS $HIVE_OPTS "$@"
}

hwi_help(){
  echo "Usage ANT_LIB=XXXX hive --service hwi"  
}

jar.sh

这是一个hadoop jar命令的包装，用于支持Hive环境的hadoop jar命令。

THISSERVICE=jar
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

jar () {
  RUNJAR=$1
  shift

  RUNCLASS=$1
  shift

  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi

  if [ -z "$RUNJAR" ] ; then
    echo "RUNJAR not specified"
    exit 3
  fi

  if [ -z "$RUNCLASS" ] ; then
    echo "RUNCLASS not specified"
    exit 3
  fi

  # hadoop 20 or newer - skip the aux_jars option and hiveconf
  exec $HADOOP jar $RUNJAR $RUNCLASS $HIVE_OPTS "$@"
}

jar_help () {
  echo "Used for applications that require Hadoop and Hive classpath and environment."
  echo "./hive --service jar <yourjar> <yourclass> HIVE_OPTS <your_args>"
}

lineage.sh

lineage.sh脚本是用来做hive表的血缘分析。在脚本里定义了lineage函数。

lineage主类。org.apache.hadoop.hive.ql.tools.LineageInfo

THISSERVICE=lineage
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

lineage () {
  CLASS=org.apache.hadoop.hive.ql.tools.LineageInfo

  # cli specific code
  if [ ! -f ${HIVE_LIB}/hive-exec-*.jar ]; then
    echo "Missing Hive exec Jar"
    exit 3;
  fi

  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi

  exec $HADOOP jar ${HIVE_LIB}/hive-exec-*.jar $CLASS  "$@"
}

lineage_help () {
  echo "usage ./hive 'hql' "
}

metastore.sh

元数据存储服务。metastore.sh中定义了metastore函数。

metastore主类。org.apache.hadoop.hive.metastore.HiveMetaStore

THISSERVICE=metastore
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

metastore() {
  echo "Starting Hive Metastore Server"
  CLASS=org.apache.hadoop.hive.metastore.HiveMetaStore
  if $cygwin; then
    HIVE_LIB=`cygpath -w "$HIVE_LIB"`
  fi
  JAR=${HIVE_LIB}/hive-service-*.jar

  # hadoop 20 or newer - skip the aux_jars option and hiveconf

  export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS $HADOOP_OPTS"
  exec $HADOOP jar $JAR $CLASS "$@"
}

metastore_help() {
  metastore -h
}

metatool.sh

元数据工具。
主类：org.apache.hadoop.hive.metastore.tools.HiveMetaTool

THISSERVICE=metatool
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

metatool () {
  HIVE_OPTS=''
  CLASS=org.apache.hadoop.hive.metastore.tools.HiveMetaTool
  execHiveCmd $CLASS "$@"
}

metatool_help () {
  HIVE_OPTS=''
  CLASS=org.apache.hadoop.hive.metastore.tools.HiveMetaTool
  execHiveCmd $CLASS "--help"
}

orcfiledump.sh

后续再补充。

rcfilecat.sh

后续再补充。

schemaTool.sh

后续再补充。

version.sh

版本信息。
主类：org.apache.hadoop.hive.common.util.HiveVersionInfo

THISSERVICE=version
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

version () {
  JAR=$1
  if [ -z "$JAR" ] ; then
    JAR=${HIVE_LIB}/hive-exec-*.jar
  else
    JAR=${HIVE_LIB}/$1
  fi

  # hadoop 20 or newer - skip the aux_jars option and hiveconf
  CLASS=org.apache.hive.common.util.HiveVersionInfo
  exec $HADOOP jar $JAR $CLASS
}

version_help () {
  echo "Show Version information of hive jars"
  echo "./hive --version [hiveJar]"
}

至此，我们大概对hive的服务有了一个初步的概念。

Hive源码解读（2）shell脚本解读

Hive 服务清单（Hive Server List）

beeline.sh

cli.sh

hiveburninclient.sh

hiveserver.sh

hiveserver2.sh

hwi.sh

jar.sh

lineage.sh

metastore.sh

metatool.sh

orcfiledump.sh

rcfilecat.sh

schemaTool.sh

version.sh

猜你喜欢

热点阅读