（大数据平台搭建）--CDH6.0.0配置安装文档

2019-02-07 本文已影响0人 lumanjiao

CDH6.0.0配置安装文档（根据官网）

I文档概况

①安装准备工作

②CDH安装具体步骤

③可能遇到的问题及解决方案

II安装准备工作

（master和slave都需要做的工作）

一、系统准备

操作系统：CentOS7.5（64位）

Cloudera Manager：6.0.0

CDH: 6.0.0

二、配置网络环境

注意：CDH需要ipv4，不支持ipv6

(1)CentOS 修改IP地址

# vi /etc/sysconfig/network-scripts/ifcfg-ens160

TYPE=Ethernet

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=static

BROADCAST=192.168.0.255

IPADDR=192.168.1.156

NETMASK=255.255.255.0

NETWORK=192.168.1.0

DNS1=8.8.8.8

DNS2=8.8.4.4

DEFROUTE=yes

IPV6INIT=no

IPV6_AUTOCONF=no

NAME=ens160

UUID=87fdb2a5-9eb0-4299-ac25-cb039053c0cd

DEVICE=ens160

ONBOOT=yes

(2)CentOS 修改网关

# vi /etc/sysconfig/network

NETWORKING=yes(表示系统是否使用网络，一般设置为yes。如果设为no，则不能使用网络，而且很多系统服务程序将无法启动)

HOSTNAME=fwq157(设置本机的主机名，这里设置的主机名要和/etc/hosts中设置的主机名对应)

GATEWAY=192.168.1.1(设置本机连接的网关的IP地址。例如，网关为10.0.0.2)

(3)CentOS 修改DNS

# vi /etc/resolv.conf

nameserver 8.8.8.8 #google域名服务器

nameserver 8.8.4.4 #google域名服务器

(4)将主机名设置为唯一名称

sudo hostnamectl set-hostname fwq157

(5)CentOS修改主机名

# vi /etc/hosts（master节点需要添加master和slave所有节点的ip和主机名，slave节点只需要填写本机的和master的ip和主机名）

192.168.1.157 fwq157

(6)重新启动网络配置# service network restart

#ping www.baidu.com

（7）重新启动

#reboot

三、安装依赖软件包

# yum install -y vim wget lrzsz

四、禁用防火墙

关闭防火墙

#systemctl stop firewalld.service

开启防火墙

#systemctl start firewalld.service

关闭开机启动

#systemctl disable firewalld.service

开启开机启动

#systemctl enable firewalld.service

五、设置SELinux模式

关闭selinux

#setenforce 0 // 将/etc/selinux/config设置成SELINUX=permissive

六、开启NTP服务

yum install ntp

systemctl enable ntpd

systemctl start ntpd

hwclock --systohc

七、设置ssh无密码登陆

（1）在主节点上执行

cd /root

mkdir .ssh

cd .ssh

ssh-keygen -t rsa

cat id_rsa.pub >> authorized_keys

ssh localhost //测试是否生效

修改文件夹/root/.ssh 的权限：

（本地机和远程机都要修改）

chmod 700 /root/.ssh

修改文件的属性：

chmod 600 /root/.ssh/authorized_keys

（2）生成的密钥copy到所有slave节点：

scp authorized_keys远程用户名@远程服务器IP:/root/.ssh/

测试ssh本机ip和ssh slave的ip是否成功。

八、安装JDK

新建tool目录

#cd /home

#mkdir tool

将安装包中jdk-8u172-linux-x64.rpm到tool目录

rpm -ivh jdk-8u181-linux-x64.rpm

配置环境变量，在/root/.bashrc中添加：

export JAVA_HOME=/usr/java/jdk1.8.0_172-amd64/

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

配置完需要#source /root/.bashrc

III安装具体步骤

一、Configure a Repository for Cloudera Manager

1、把cloudera-manager.repo文件复制到/etc/yum.repos.d/目录下

或者执行

wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

2、 Import the repository signing GPG key:

RHEL 7 compatible:

#sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

二、Install Cloudera Manager Packages

#sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

三、Install and Configure MySQL for Cloudera Software

1、在tool目录下执行：

#wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

#sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm

#sudo yum update

#sudo yum install mysql-server

#sudo systemctl start mysqld

2、Configuring and Starting the MySQL Server

Note: If you are making changes to an existing database, make sure to stop any services that use the database before continuing.

#sudo systemctl stop mysqld

[if !supportLists](1)[endif]Move old InnoDB log files /var/lib/mysql/ib_logfile0 and /var/lib/mysql/ib_logfile1 out of /var/lib/mysql/ to a backup location.

[if !supportLists](2)[endif]把/etc/my.cnf备份

[if !supportLists](3)[endif]Vim my.cnf,修改如下：

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

transaction-isolation = READ-COMMITTED

# Disabling symbolic-links is recommended to prevent assorted security risks;

# to do so, uncomment this line:

symbolic-links = 0

key_buffer_size = 32M

max_allowed_packet = 32M

thread_stack = 256K

thread_cache_size = 64

query_cache_limit = 8M

query_cache_size = 64M

query_cache_type = 1

max_connections = 550

#expire_logs_days = 10

#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.

#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your

#system and chown the specified folder to the mysql user.

log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set

#a server_id, MySQL will not start. The server_id must be unique within

#the replicating group.

server_id=1

binlog_format = mixed

read_buffer_size = 2M

read_rnd_buffer_size = 16M

sort_buffer_size = 8M

join_buffer_size = 8M

# InnoDB settings

innodb_file_per_table = 1

innodb_flush_log_at_trx_commit = 2

innodb_log_buffer_size = 64M

innodb_buffer_pool_size = 4G

innodb_thread_concurrency = 8

innodb_flush_method = O_DIRECT

innodb_log_file_size = 512M

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

3、Ensure the MySQL server starts at boot:

#sudo systemctl enable mysqld

#sudo systemctl start mysqld

4、初始化：

#sudo /usr/bin/mysql_secure_installation

5、Installing the MySQL JDBC Driver

#wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

#tar zxvf mysql-connector-java-5.1.46.tar.gz

#sudo mkdir -p /usr/share/java/

#cd mysql-connector-java-5.1.46

#sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

6、Creating Databases for Cloudera Software

#mysql -u root -p

Enter password:

mysql>CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

mysql>GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';

mysql>flush privileges;

mysql>SHOW DATABASES;

mysql>SHOW GRANTS FOR '<user>'@'%';

mysql>exit;

需要建的库有

scm、amon、rman、hue、metastore、sentry、nav、navms、oozie

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm@123';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY ' amon@123';

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman@123';

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue@123';

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore@123';

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry@123';

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav@123';

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms@123';

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie@123';

四、Set up the Cloudera Manager Database

1、Run the scm_prepare_database.sh script on the Cloudera Manager Server host, using the database name, username, and password you created in Step 10.5: Install and Configure Databases:

#sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

When prompted, enter the password.

出现如下字样则SCM数据库配置成功。

五、Install CDH and Other Software

若要离线安装：则

1、从https://archive.cloudera.com/cdh6/6.0.0/parcels/下载三个cdh安装文件

2、把这三个文件拷贝到/opt/cloudera/parcel-repo/路径下

3、#mv CDH-6.0.0-1.cdh6.0.0.p0.537114-sles12.parcel.sha 256 CDH-6.0.0-1.cdh6.0.0.p0.537114-sles12.parcel.sha

若在线安装则直接则行：

1、Start Cloudera Manager Server:

#sudo systemctl start cloudera-scm-server

2、Wait several minutes for the Cloudera Manager Server to start. To observe the startup process, run the following on the Cloudera Manager Server host:

#sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

When you see this log entry, the Cloudera Manager Admin Console is ready:

3、In a web browser, go to http://<server_host>:7180, where <server_host> is the FQDN or IP address of the host where the Cloudera Manager Server is running.

4、Log into Cloudera Manager Admin Console. The default credentials are:

Username: admin

Password: admin

5、进入页面

Specify Hosts

如果已经安装过jdk，则不需要勾选

提供SSH 登录凭据

Install Agents

正在安装选定Parcel

检查主机正确性

看是否都为绿色对勾，把不正确的改正。

可能遇到的问题和改正方式：

1、大内存页设置

1、echo never > /sys/kernel/mm/transparent_hugepage/defrag

2、echo never > /sys/kernel/mm/transparent_hugepage/enabled

3、然后将同一命令添加到/etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。

2、升级软件依赖版本

解决方法：可以忽略。

yum -y install epel-release

yum install python-pip

pip install --upgrade psycopg2

3、虚拟内存设置

临时修改：sysctl vm.swappiness=10

永久修改：

vim /etc/sysctl.conf

添加vm.swappiness=10

六、Set Up a Cluster

1、Select Services

2、自定义角色分配

按默认的来。

3、数据库设置

需要进入数据库执行：

CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'hive'@'%' IDENTIFIED BY 'hive@123';

CREATE DATABASE report DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

4、审核更改

注意：关于/solr的目录不要修改，不然后面会报错。

5、首次运行命令

6、Summary

七、完成效果。

IV可能遇到的问题及解决方案

1、有时会遇到CDH集群的7180页面无法访问

#rm -rf /var/run/cloudera-scm-server