搭建Hadoop-HA + ZooKeeper + Yarn环境

2019-01-16  本文已影响0人  上杉丶零

前提：搭建Hadoop-HA + ZooKeeper环境

node01 node02 node03 node04
NameNode01 NameNode02 NameNode03
DataNode01 DataNode02 DataNode03
JournalNode01 JournalNode02 JournalNode03
ZooKeeper01 ZooKeeper02 ZooKeeper03
ZooKeeperFailoverController01 ZooKeeperFailoverController02 ZooKeeperFailoverController03
ResourceManager01 ResourceManager02
NodeManager01 NodeManager02 NodeManager03
  1. 配置node01、node02、node03、node04上的Yarn

在node01上修改/opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml：
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml
添加：

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> 
  </property>
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>yarnResourceManagerHACluster</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>RM01,RM02</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.RM01</name>
    <value>node03</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.RM02</name>
    <value>node04</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.RM01</name>
    <value>node03:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.RM02</name>
    <value>node04:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>node02:2181,node03:2181,node04:2181</value>
  </property>
</configuration>

在node01上修改/opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml：
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml
添加：

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>

将node01上的/opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml、/opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml拷贝到node02、node03、node04：
scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node02:/opt/hadoop/hadoop-3.1.1/etc/hadoop/ && scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node03:/opt/hadoop/hadoop-3.1.1/etc/hadoop/ && scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node04:/opt/hadoop/hadoop-3.1.1/etc/hadoop/

  1. 配置node01、node02、node03、node04上的环境变量

在node01上修改/etc/profile：
vim /etc/profile
添加：

export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node02上修改/etc/profile
vim /etc/profile
添加：

export YARN_NODEMANAGER_USER=root
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node03上修改/etc/profile
vim /etc/profile
添加：

export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node04上修改/etc/profile
vim /etc/profile
添加：

export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node01、node02、node03、node04上运行：
. /etc/profile

  1. 启动ZooKeeper

在node02、node03、node04上运行：
zkServer.sh start

  1. 启动ResourceManager和NodeManager

在node3/node4上运行：
start-yarn.sh

  1. 启动Hadoop

在node01/node02/node03/node04上运行：
start-dfs.sh

  1. 查看进程

在node01、node02、node03、node04上运行：
jps

  1. 访问网页

NameNode01：http://192.168.163.191:9870
NameNode02：http://192.168.163.192:9870
NameNode03：http://192.168.163.193:9870
DataNode01：http://192.168.163.192:9864
DataNode02：http://192.168.163.193:9864
DataNode03：http://192.168.163.194:9864
ResourceManager01：http://192.168.163.193:8088
ResourceManager02：http://192.168.163.194:8088

上一篇下一篇

猜你喜欢

热点阅读