CDH5.6的RPM安装

2020-04-13 本文已影响0人叫小七的剑客

1.将系统源换为本地源

2.启动httpd服务

3.配置jdk、mysql、ssh

4.将cdh5.6的rpm包上传到虚拟机

5.关闭防火墙和SELINUX

service iptables stop
vim /etc/sysconfig/selinux
SELINUX=disabled

6.将CDH文件复制到/var/www/html/下面

cp -r CDH /var/www/html/

CDH下面的文件有：

1、5.6.0 
2、cloudera-cdh5.repo  
3、RPM-GPG-KEY-cloudera

7.修改cloudera-cdh5.repo

[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on SLES 11 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=http://192.168.102.128/CDH/5.6.0
gpgkey=http://192.168.102.128/CDH/RPM-GPG-KEY-cloudera
gpgcheck = 1

RPM-GPG-KEY-cloudera中的内容

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.5 (GNU/Linux)

mQGiBEpBgEURBAC+CL1a6BfVEoKAX1KcOHqq9Z10WdPGOgTM+AtnOVPJdJvIZcDk
YGUmycpaGxY3+xX1x8ZvxNb7WXiei8FMPm4sR/xQC/CF2iS5399tjLJqcDEjdqTV
/whQ4Rrg1JLGaHUjR0YmrOteT71xikEwlCalToxQuhBz7Nz4aBeDDPf9lwCgvG+x
CaOxict+He03g4HNSTZ0T0UEAIxKITpCA6ZvUPoEGhpn+Gt+wJK/ScB0FKCfW8Au
QQZP6tgxDEg0baasT8MxuXXE2+opaaWPTVa64ws7OvbyH5z1xhBOx4qRVBx8bZsF
YQUk/1PBvg6yA4Rmaqi7nTToHatP69/JMLfTyH8sXETMQ8z5T0LAD6a5ELAYBqql
bJWRA/4lkbaGIwkyLcOAop/g0SCERHt66ML1pwdjxvzE2rRKFUbjUbRZsHTqVq5E
BgpcTIeTuRy02yQ+Bh+JaBtYhn0AY5+t7jcCdJeTahS/7RKJPYPiSfbgI6zwpHM9
kX4FT+0yDgnVF1H/h9p19Uv/3ahIgt7op/M1eAdH0/eP6Dv04rQnWXVtIE1haW50
YWluZXIgPHdlYm1hc3RlckBjbG91ZGVyYS5jb20+iGAEExECACAFAkpBgEUCGwMG
CwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRD5DA2P6PhqzRo1AKCIHNWJSd7OipbZ
qp58f/BWaIBlDACggNRH4Hvg92t3xtwYFdohRWF2Xbi5Ag0ESkGARxAIAMaPPGfQ
vsLkyLyM3ePtkkHi0bew0XGW1CYxWOZLMu8wnJgMHpfPD2dLgp6PEh+zpi2SM1ie
QGAW6K040TSuC9P+LcZB7SxanIE7lONHjz7spGQift30WFZcaIgF+MuyZIihNh7v
tZ9ip8JZYPA88XRNU1CKuXx4r8iCDJ4ICksFKeOwQUuzf/IRJapzEZ0ixfVTwx91
yG10TvHK63BRLXYHBML4Og9FaPZgFq2N9Yz4Wpu/Pn6tjZAMeSJXm2qNO2PSoTC/
kapubpMwSmOBlZqrHi9lcIWricXE9dcyaGVRAf3CJRlX4ZNuwcQjyks5BFibU3/z
qlzP6KgwTgDmaaMAAwUH/04KRM3k6Ow2KkDt2BKWveOI24mkIQahUJ7/iZlKsL27
3VcGQZ7jU28GT0FH9iYeAgbpLrrEuDAFZpGm9RoOVJGnxWX3DVL1+qkiS56pXfU+
8atZlkCGx09IilJgf0ATlmYxbTtYliTRPK4lQYOfNB1v23bdlBwISjcDRkWu22ao
atSBzr/FARL6fdZZqp2qfWOmcteiLagioo6s0ogxKNQH5PldUQy9n2W/oOXss5sC
lnUNvzKlzzx/pFkT8ZUAvuLY0v8gykk586vbjiuPkg8uAOBhtnsSWwJ6nEPaRCnu
iwlqGxgXmnJ7UMzOimkuf0XvqavhkMEEAqRJkNLyWVuISQQYEQIACQUCSkGARwIb
DAAKCRD5DA2P6PhqzUV2AJ0eV3C407Y3Xi4d27clLsz/wW0HMgCghcxCmiOT2kWH
6Ya7d9nkKz2UM+Y=
=+VR8
-----END PGP PUBLIC KEY BLOCK-----

8.在/var/www/html下下载cloudera-cdh5.repo

wget http://192.168.102.128/CDH/cloudera-cdh5.repo

9.将cloudera-cdh5.repo复制到/etc/yum.repos.d/下

cp cloudera-cdh5.repo /etc/yum.repos.d/

安装hdfs

安装NameNode、JobTracker等

yum --disablerepo=* --enablerepo=cloudera-cdh5 install hadoop-hdfs-namenode hadoop-yarn-resourcemanager hadoop-mapreduce-historyserver hadoop-yarn-proxyserver

安装datanode、journalnode、secondarynamenode、nodemanager

cd /var/www/html/CDH/5.6.0/RPMS/x86_64/
rpm -ivh hadoop-hdfs-datanode-2.6.0+cdh5.6.0+1023-1.cdh5.6.0.p0.120.el6.x86_64.rpm
rpm -ivh hadoop-hdfs-secondarynamenode-2.6.0+cdh5.6.0+1023-1.cdh5.6.0.p0.120.el6.x86_64.rpm
rpm -ivh hadoop-hdfs-journalnode-2.6.0+cdh5.6.0+1023-1.cdh5.6.0.p0.120.el6.x86_64.rpm
rpm -ivh hadoop-yarn-nodemanager-2.6.0+cdh5.6.0+1023-1.cdh5.6.0.p0.120.el6.x86_64.rpm

3.修改四个配置文件

3.1. core-site.xml

/etc/hadoop/conf

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://CDH1:8020/</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/var/hadoop/log</value>
        </property>

</configuration>

3.2 hdfs-site.xml

<configuration>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/mnt/drbd/hdfs/dfs/name</value>
        </property>
        <property>
                <name>dfs.permissions.superusergroup</name>
                <value>hadoop</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/mnt/diskb/dfs</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
</configuration>

3.3 mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>CDH1:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>CDH1:19888</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/user/history/done_intermediate</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/user/history/done</value>
    </property>

</configuration>

3.4yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>

    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>

    <property>
        <description>List of directories to store localized files in.</description>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
    </property>

    <property>
        <description>Where to store container logs.</description>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/var/log/hadoop-yarn/containers</value>
    </property>

    <property>
        <description>Where to aggregate logs to.</description>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>hdfs://CDH1/var/log/hadoop-yarn/apps</value>
    </property>
    <property>
        <description>Classpath for typical applications.</description>
        <name>yarn.application.classpath</name>
        <value>
            $HADOOP_CONF_DIR,
            $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
            $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
            $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
            $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
        </value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>CDH1:8050</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>CDH1:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>CDH1:8141</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>CDH1:8088</value>
    </property>
    <property>
        <name>yarn.web-proxy.address</name>
        <value>CDH1:19999</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.staging-dir</name>
        <value>/user</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>CDH1</value>
    </property>



</configuration>

4.创建namenode目录，赋予权限

mkdir -p /mnt/drbd/hdfs/dfs/name
chown -R hdfs:hdfs /mnt/drbd/hdfs/dfs/name
chmod 700 /mnt/drbd/hdfs/dfs/name
mkdir -p /var/hadoop/log/dfs/namesecondary
chown -R hdfs:hdfs /var/hadoop/log/dfs/namesecondary
mkdir -p /mnt/diskb/dfs
 chown -R hdfs:hdfs /mnt/diskb/dfs

5.格式化namenode

su - hdfs 
hdfs namenode –format

6.启动namenode（root用户）

cdh5.7需要在/etc/default/hadoop里面配置java环境变量

service hadoop-hdfs-namenode start

8.创建hdfs临时目录

su - hdfs 
hadoop fs -mkdir -p /tmp
hadoop fs -chmod -R 1777 /tmp
hadoop fs -mkdir -p /var/log/hadoop-yarn
hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
hadoop fs -mkdir -p /user/history
hadoop fs -chmod -R 1777 /user/history
hadoop fs -mkdir -p /user/history/done_intermediate
hadoop fs -chown mapred:hadoop /user/history/done_intermediate
hadoop fs -chmod -R 1777 /user/history/done_intermediate
hadoop fs -mkdir -p /user/history/done
hadoop fs -chown mapred:hadoop /user/history/done
hadoop fs -chmod -R 750 /user/history/done
hadoop fs -mkdir -p /user/mapreduce
hadoop fs -chown mapreduce /user/mapreduce

9.启动其他服务

service hadoop-yarn-resourcemanager start
service hadoop-mapreduce-historyserver start
service hadoop-yarn-proxyserver start
service hadoop-hdfs-journalnode start
service hadoop-hdfs-secondarynamenode start
service hadoop-hdfs-datanode start

7.检查namenode是否启动成功

service --status-all | grep Hadoop

安装hive

1.安装hive

yum --disablerepo=* --enablerepo=cloudera-cdh5 install hive

2.配置hive

将mysql的驱动jar包放入/usr/lib/hive/lib中。

3.在mysql数据库中创建metastore数据库

mysql -uroot -proot
CREATE DATABASE metastore;
USE metastore;
SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.14.0.mysql.sql;

4.修改hive的配置文件

<configuration>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://CDH1/metastore</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>root</value>
    </property>
    <property>
        <name>datanucleus.autoCreateSchema</name>
        <value>true</value>
    </property>
    <property>
        <name>datanucleus.fixedDatastore</name>
        <value>true</value>
    </property>
</configuration>

5.在hdfs上创建hive的用户目录

sudo - hdfs 
hdfs dfs -mkdir /user/hive
hdfs dfs -chmod a+w  /user/hive

6.安装hive-metadata

yum --disablerepo=* --enablerepo=cloudera-cdh5 install hive-metastore

7.安装hive-server2

yum --disablerepo=* --enablerepo=cloudera-cdh5 install hive-server2

安装zookeeper

1.安装zookeeper

yum --disablerepo=* --enablerepo=cloudera-cdh5 install zookeeper-server

2.配置zookeeper

修改zoo.cfg文件

#在最后一行加入
server.1=CDH1:2888:3888

3.初始化zookeeper

/etc/init.d/zookeeper-server init --myid=1

4.在/usr/lib/zookeeper/bin/zkServer.sh中添加java环境变量

4.启动zookeeper

service zookeeper-server start

5.检查zookeeper是否成功

service --status-all | grep zookeeper

安装HBase

1.安装HBase-Master

yum --disablerepo=* --enablerepo=cloudera-cdh5 install hbase-master
yum --disablerepo=* --enablerepo=cloudera-cdh5 install hbase-regionserver

2.修改hbase-site.xml

<configuration>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://CDH1:8020/hbase</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>CDH1</value>
    </property>
    <property>
        <name>hbase.hregion.majorcompaction.cron</name>
        <value>0 0 1 * * ?</value>
    </property>

</configuration>

3.在hdfs上创建hbase目录

sudo - hdfs 
hdfs dfs -mkdir /hbase
hdfs dfs -chown hbase /hbase

4.启动服务：

service hbase-master start
service hbase-regionserver start

5.检查服务是否启动

service --status-all | grep HBase

CDH5.6的RPM安装

安装hdfs

安装hive

安装zookeeper

安装HBase

猜你喜欢

热点阅读