Hadoop Single Node Cluster

安装JDK（JRE）

• 查看java版本

  java -version
  

• 查询jdk安装路径

  update-alternatives --display java
  

• 安装jdk

  sudo apt install openjdk-8-jdk-headless
  

设置SSH免密码登录

• 安装openssh-server

  sudo apt-get install openssh-server
  

• 配置认证

  ssh localhost #选择yes即可连接成功
  exit  #退出ssh连接
  cd ~/.ssh/   
  ssh-keygen -t rsa              # 会有提示，都按回车就可以
  cat ./id_rsa.pub >> ./authorized_keys  # 加入授权
  
  

  ​

下载安装Hadoop

• 进入Hadoop官网

  http://hadoop.apache.org/

• 点击Releases

• 找到想要的版本

  如hadoop-2.9.0.tar.gz的下载链接：

  http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz

• 下载Hadoop

  wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz
  

• 解压，移动到安装目录

  tar -zxvf hadoop-2.9.0.tar.gz #解压
  sudo mv ./hadoop-2.9.0 /usr/local/hadoop/ #移动到安装目录
  

• 设置Hadoop环境变量

  • 编辑~/.bashrc文件

    #设置JDK安装路径
    export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
    #设置HADOOP_HOME为Hadoop的安装路径
    export HADOOP_HOME=/usr/local/hadoop
    #设置PATH
    export PATH=$PATH:$HADOOP_HOME/bin
    export PATH=$PATH:$HADOOP_HOME/sbin
    #设置Hadoop其他环境变量
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export YARN_HOME=$HADOOP_HOME
    #链接库相关设置
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
    export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
    

• 使环境变量生效

  source ~/.bashrc
  

  ​

配置Hadoop

• hadoop-env.sh

  export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  

• core-site.xml

  <configuration>
  <property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:9000</value>
  </property>
  </configuration>
  

• yarn-site.xml

  <configuration>
  
  <!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  </configuration>
  

• mapred-site.xml (从mapred-site.xml.template里面复制)

  <configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  </configuration>
  

• hdfs-site.xml

  <configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/local/hadoop/hadoop_data/hdfs/datanode</value>
  </property>
  </configuration>
  

  ​

创建并格式化HDFS目录

#创建namenode目录
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode 
#创建datanode目录
sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode 
#将Hadoop目录的所有者改为hduser
sudo chown hduser:hduser -R /usr/local/hadoop 
#将HDFS格式化
hadoop namenode -format

启动Hadoop

#启动HDFS
start-dfs.sh
#启动Yarn
start-yarn.sh
#同时启动HDFS和YARN
start-all.sh
#查看Hadoop是否启动
jps

Hadoop Web 界面

• Resource-Manager 界面

  http://localhost:8088/

• Namenode HDFS界面

  http://localhost:50070/