【CDH安装】环境准备
一、基础环境设置
注意：基础环境需在每台集群服务器设置。
1.1、软件环境
| 编号 | 软件名称 | 软件版本 |
|---|---|---|
| 1 | 操作系统 | CentOS Linux release 7.5.1804 (Core) |
| 2 | JDK | jdk-8u144-linux-x64 |
| 3 | Scala | scala-2.11.8 |
1.2、服务器规划
| 编号 | 服务器名称 | 内网IP | 内存 | 用途 |
|---|---|---|---|---|
| 1 | cdhserver1 | 192.168.26.21 | 20G | CM-server、CM-agent、MySQL、HTTP-server |
| 2 | cdhserver2 | 192.168.26.22 | 9G | CM-agent |
| 3 | cdhserver3 | 192.168.26.23 | 9G | CM-agent |
1.3、配置主机名IP映射
注意：CDH中谨慎使用FQDN,与主机名保持一致
vi /etc/hosts
#添加如下地址映射
192.168.26.21 cdhserver1
192.168.26.22 cdhserver2
192.168.26.23 cdhserver3
1.4、SSH免密登录配置
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver1
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver2
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver3
1.5、防火墙配置
为了安装时简单，可以执行以下命令暂时关闭所有防火墙。
systemctl stop firewalld.service systemctl disable firewalld.service firewall-cmd --state
安装完成开启防火墙，设置防火墙策略。
#开启防火墙
systemctl start firewalld.service
#设置可以访问的端口
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.21' port protocol='tcp' port='0-65535' accept"
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.22' port protocol='tcp' port='0-65535' accept"
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.23' port protocol='tcp' port='0-65535' accept"
firewall-cmd –permanent –add-port=0-65535/tcp firewall-cmd –reload
1.6、关闭SELinux
vi /etc/sysconfig/selinux
#修改SELINUX为disabled
SELINUX=disabled
1.7、开启NTP服务实现时间同步
yum install -y ntp systemctl enable ntpd systemctl start ntpd
1.8、安装JDK环境
话说JDK8最新更新的本版不开源了，下载历史本版：jdk-8u181-linux-x64.rpm
https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html
若系统自带openJDK，首先卸载
java -version rpm -qa | grep jdk
yum -y remove java-xxx-openjdk-xxx.el7_0.x86_64
yum -y remove java-xxx-openjdk-headless-xxx.el7_0.x86_64
安装JDK
rpm -ivh jdk-8u181-linux-x64.rpm
配置JDK环境变量
vi /etc/profile source /etc/bashrc
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin
1.9 安装SCALA环境
下载scala-2.11.8 :
https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.rpm 或
https://www.scala-lang.org/files/archive/scala-2.11.8.rpm
rpm -ivh scala-2.11.8.rpm
1.10 修改Linux内核参数
Cloudera 建议将 swappiness 设置为 10（控制换出运行时内存权重）
#查看swappiness
cat /proc/sys/vm/swappiness
#永久性修改，执行下面两条命令
sysctl -w vm.swappiness=10
echo vm.swappiness = 10 >> /etc/sysctl.conf
修改文件句柄数
vi /etc/security/limits.conf
在limits.conf文件末尾添加如下配置：
# End of file
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
vi /etc/security/limits.d/20-nproc.conf
修改 ‘* soft nproc 4096’ 成更大的值或注释掉，因为该文件的配置会覆盖limits.conf的配置。
* soft nproc 131072
透明大页面压缩
临时关闭：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久关闭：
vi /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local
在文件后添加下面内容: ，每次添加完记得执行chmod +x /etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
查看状态
cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/enabled
1.11 重启集群服务器
init 6
二、MYSQL安装
2.1 卸载centos7自带的mariadb
rpm -qa | grep mariadb rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
2.2 安装依赖
yum install -y net-tools perl wget
2.3 下载MySQL安装包并解压
wget https://cdn.mysql.com//archives/mysql-5.7/mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar
tar -xvf mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar
mysql-community-server-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-client-5.7.18-1.el7.x86_64.rpm
mysql-community-common-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-5.7.18-1.el7.x86_64.rpm
mysql-community-server-minimal-5.7.18-1.el7.x86_64.rpm
mysql-community-test-5.7.18-1.el7.x86_64.rpm
mysql-community-minimal-debuginfo-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm
2.4 依次安装mysql rpm包
按common–>libs–>client–>server->devel->libs-compat的顺序安装
rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-devel-5.7.18-1.el7.x86_64.rpm
#安装下面包，否则Hue安装报连接错误
rpm -ivh mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm
2.5 修改MySQL端口和数据存放路径
mkdir -p /data/mysql/data
vi /etc/my.cnf
image.png
2.6 启动mysql服务及查看状态
systemctl start mysqld.service systemctl status mysqld.service
2.7 初始化数据库密码
#查看初始密码
grep "password" /var/log/mysqld.log
mysql -uroot -p
#默认密码检查策略要求密码必须包含：大小写字母、数字和特殊符号，并且长度不能少于8位 ALTER USER 'root'@'localhost' IDENTIFIED BY 'Root@2019';
#远程连接数据库的时候需要输入用户名和密码
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Root@2019' WITH GRANT OPTION;
FLUSH PRIVILEGES;
#查看端口
show global variables like 'port';
方式二：
修改配置文件/etc/my.cnf令MySQL跳过登录时的权限检验，在[mysqld]下加入一行：
skip-grant-tables
使用 mysql登陆
mysql> use mysql;
mysql> UPDATE user SET authentication_string = password('root') WHERE host = 'localhost' AND user = 'root';
mysql> select host,user, authentication_string, password_expired from user;
mysql> update user set password_expired='N' where password_expired='Y' //密码不过期
mysql> update user set host='%' where user='root' and host='localhost'; //远程可访问
mysql> flush privileges; //刷新
mysql> exit;//退出
删除skip-grant-tables
2.8 设置开机启动
systemctl enable mysqld systemctl daemon-reload
