docker下搭建cdh
2023-05-05 本文已影响0人
小KKKKKKKK
此次过程为：通过centos镜像，创建三个容器，在三个容器中搭建一个小型的cdh集群。
centos已经具备，接下来就是创建三个容器
docker网络
由于cdh集群需要三个容器之间网络互通，所以先创建docker网络
Docker 提供三种网络驱动：bridge、overlay、macvlan
overlay 和 macvlan 用于创建跨主机的网络
这里三个容器是在同一个主机上，所以直接使用默认的网络驱动即可，后续跨主机再继续深究其他两种网络驱动。
查看docker网络
docker network ls
返回
NETWORK ID NAME DRIVER SCOPE
3be276e8c0b7 bridge bridge local
6bdb390e9d18 host host local
d7df99850908 none null local
创建docker网络
docker network create --driver bridge --subnet=172.21.0.0/24 --gateway=172.21.0.1 cdh_net
返回
569e6db77a6b3e1a063949fee0d3be9565f8cd621256993b51311ad8a497eb67
参数说明：
--driver 网络驱动（可不写，默认为bridge）
--subnet 子网ip
--gateway 网关
cdh_net 网络名称
再次查看网络
docker network ls
返回
NETWORK ID NAME DRIVER SCOPE
3be276e8c0b7 bridge bridge local
569e6db77a6b cdh_net bridge local
6bdb390e9d18 host host local
d7df99850908 none null local
说明需要的网络已经创建好了
创建容器
网络创建好，接下来需要基于创建好的网络创建容器，镜像为centos的镜像
查看容器
docker container list
返回
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
docker中没有容器
查看镜像
docker images
返回
REPOSITORY TAG IMAGE ID CREATED SIZE
centos latest 5d0da3dc9764 19 months ago 231MB
选择的镜像为centos:latest
创建容器
docker run -it -d --name cdh01 --network=cdh_net --ip=172.21.0.101 -p2227:22 -p8080:8080 -p8000:8000/udp -p7183:7183 -p7180:7180 -p9870:9870 -p16010:16010 -p18088:18088 --restart always --privileged=true centos:latest /usr/sbin/init
返回
e399964a01bdd5855dffb9aba355ee9bdcc3edc89b0159cba027e067d6bcfdca
docker run -it -d --name cdh02 --network=cdh_net --ip=172.21.0.102 -p2228:22 --restart always --privileged=true centos:latest /usr/sbin/init
返回
666d932c563f327e032416f7fb02a8c62befd0f0b9896da8da0486bf0765f028
docker run -it -d --name cdh03 --network=cdh_net --ip=172.21.0.103 -p2229:22 --restart always --privileged=true centos:latest /usr/sbin/init
返回
f02d6a292ddd12aba34c1090569bbb2e32c18e39dbc919d50969648193750ef8
查看容器，看是否创建成功
docker container list
返回
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f02d6a292ddd centos:latest "/usr/sbin/init" About a minute ago Up About a minute 0.0.0.0:2229->22/tcp cdh03
666d932c563f centos:latest "/usr/sbin/init" 2 minutes ago Up 2 minutes 0.0.0.0:2228->22/tcp
cdh02
e399964a01bd centos:latest "/usr/sbin/init" 3 minutes ago Up 3 minutes 0.0.0.0:7180->7180/tcp, 0.0.0.0:7183->7183/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:9870->9870/tcp, 0.0.0.0:16010->16010/tcp, 0.0.0.0:18088->18088/tcp, 0.0.0.0:8000->8000/udp, 0.0.0.0:2227->22/tcp cdh01
三个容器创建完成
问题
问题1-docker的dnf makecache启动报错
但是在docker desktop中点进容器，在日志中发现有报错，报错内容为：
image-20230426153952824.png
根据提示，进入命令行中执行“systemctl status dnf-makecache.service”命令看问题
点进Terminal
执行
systemctl status dnf-makecache.service
61e7deca063417e478405b967d34a12.png
根据告警告诉无法下载metadata，没有准备好跟mirrorlist连接
应该是跟yum有关系
执行
yum list
返回
CentOS Linux 8 - AppStream 85 B/s | 38 B 00:00
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: No URLs in mirrorlist
故进入“/etc/yum.repo.d”中进行修改，我这里把路径下其他文件都修改了文件名，只留下CentOS-Linux-BaseOS.repo，把文件中的镜像路径改为阿里的镜像，“mirrors.aliyun.com”阿里镜像地址。
修改完成后
yum clean all
返回
0 files removed
yum list
哗啦啦刷一大堆
说明镜像通了，接下来重启容器，查看日志
image-20230426155512259.png
可发现之前的报错，现在ok了，这个问题解决了
此时此刻，理论上应该可以远程容器了
问题2-未安装openssh服务
但是，我通过xshell，ssh或者telnet连接容器，都是在报错
image-20230426155629156.png
image-20230426155656649.png
经过查询发现，原来是容器没有安装openssh服务，openssh服务是负责ssh登录的，一坑接一坑
现在镜像有了，天下我有，安装openssh即可
查询需要看装的服务
yum search openssh
返回
Last metadata expiration check: 0:06:51 ago on Wed Apr 26 07:53:29 2023.
====================================================================================================== Name Exactly Matched: openssh =======================================================================================================
openssh.x86_64 : An open source implementation of SSH protocol version 2
===================================================================================================== Name & Summary Matched: openssh ======================================================================================================
openssh-keycat.x86_64 : A mls keycat backend for openssh
========================================================================================================== Name Matched: openssh ===========================================================================================================
openssh-cavs.x86_64 : CAVS tests for FIPS validation
openssh-clients.x86_64 : An open source SSH client applications
openssh-ldap.x86_64 : A LDAP support for open source SSH server daemon
openssh-server.x86_64 : An open source SSH server daemon
经查询只需要安装openssh-clients跟openssh-server就行
安装
yum install openssh-clients -y
yum install openssh-server -y
安装完毕后，还需要启动服务
systemctl restart sshd
查看是否启动成功
systemctl status sshd
返回
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-04-26 08:03:29 UTC; 2s ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 192 (sshd)
Tasks: 1 (limit: 50160)
Memory: 1.1M
CGroup: /docker/f02d6a292ddd12aba34c1090569bbb2e32c18e39dbc919d50969648193750ef8/system.slice/sshd.service
└─192 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-12>
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Changed dead -> start
Apr 26 08:03:29 f02d6a292ddd systemd[1]: Starting OpenSSH server daemon...
Apr 26 08:03:29 f02d6a292ddd systemd[192]: sshd.service: Executing: /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-25>
Apr 26 08:03:29 f02d6a292ddd sshd[192]: Server listening on 0.0.0.0 port 22.
Apr 26 08:03:29 f02d6a292ddd sshd[192]: Server listening on :: port 22.
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Got notification message from PID 192 (READY=1)
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Changed start -> running
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Job sshd.service/start finished, result=done
Apr 26 08:03:29 f02d6a292ddd systemd[1]: Started OpenSSH server daemon.
Apr 26 08:03:29 f02d6a292ddd systemd[1]: sshd.service: Failed to send unit change signal for sshd.service: Connection reset by peer
telnet检测一下
telnet 10.10.20.198 2229
返回
Connecting to 10.10.20.198:2229...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.
SSH-2.0-OpenSSH_8.0
说明端口可以连接
ssh远程登陆
但是让我输入密码，但是我创建容器之后没有地方创建密码的地方
问题3-远程未创建密码
在docker desktop点进对应的容器中创建密码
passwd
返回
sh: passwd: command not found
没有对应服务，继续安装对应服务
yum install passwd -y
安装好后
passwd
返回
Changing password for user root.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
连续输入两次需要更改的密码即可
再通过xshell远程登陆，这次没有问题，顺利登陆容器
image-20230426161131382.png
至此，三个容器已经顺利登录了，接下来就是按照服务器搭建cdh的方式开始搭建
搭建cdh
前面什么检查防火墙、selinux那些都不需要了，openssh都没有的容器，怎么可能有防火墙。
修改hosts文件
修改
vi /etc/hosts
发送其他容器
scp /etc/hosts 172.21.0.102:/etc
scp /etc/hosts 172.21.0.103:/etc
需要输入密码
配置免密码登录
进入root目录
cd /root
生成公钥与私钥
ssh-keygen -t rsa
连续点三次回车
在/root/.ssh目录下生成两个文件id_rsa、id_rsa.pub（公钥、私钥）
每个节点都要生成！！！
将每台的公钥发送至第一台服务器，在其他服务器上执行
ssh-copy-id e399964a01bd
需要输密码
复制第一台服务器的认证到其他服务器
scp /root/.ssh/authorized_keys 666d932c563f:/root/.ssh
scp /root/.ssh/authorized_keys f02d6a292ddd:/root/.ssh
需要输最后一次密码
验证
ssh 666d932c563f
ssh f02d6a292ddd
已经不需要输入密码了，完成
设置时钟同步
无需查看是否安装过ntpdate，容器肯定没安装过，直接安装
先通过yum源查看一下有没有
yum search ntpdate
返回
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 1:18:04 ago on Wed Apr 26 07:35:55 2023.
No matches found.
居然没有
但是查看centos7的iso文件里面是由对应的包，此时需要使用我内部使用的centos7的iso的源。
在/etc/yum.repo.d中在创建一个我内部使用的源
再次搜索
yum search ntpdate
返回
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:08 ago on Wed Apr 26 08:58:29 2023.
========================================================================================= Name Exactly Matched: ntpdate ==========================================================================================
ntpdate.x86_64 : Utility to set the date and time via NTP
[root@e399964a01bd yum.repos.d]# yum search ntpdate
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:43 ago on Wed Apr 26 08:58:29 2023.
========================================================================================= Name Exactly Matched: ntpdate ==========================================================================================
ntpdate.x86_64 : Utility to set the date and time via NT
有了，安装
yum install ntpdate -y
返回
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:03:34 ago on Wed Apr 26 08:58:29 2023.
Error:
Problem: problem with installed package
-7.61.1-18.el8.x86_64
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libcrypto.so.1.1()(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libcrypto.so.1.1(OPENSSL_1_1_0)(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libssl.so.1.1()(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libssl.so.1.1(OPENSSL_1_1_0)(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires libssl.so.1.1(OPENSSL_1_1_1)(64bit), but none of the providers can be installed
- package libcurl-minimal-7.61.1-18.el8.x86_64 requires openssl-libs(x86-64) >= 1:1.1.1g, but none of the providers can be installed
- package libcurl-minimal-7.61.1-22.el8.x86_64 requires openssl-libs(x86-64) >= 1:1.1.1k, but none of the providers can be installed
- cannot install both openssl-libs-1:1.0.2k-19.el7.x86_64 and openssl-libs-1:1.1.1g-15.el8_3.x86_64
- cannot install both openssl-libs-1:1.0.2k-19.el7.x86_64 and openssl-libs-1:1.1.1k-4.el8.x86_64
- cannot install both openssl-libs-1:1.0.2k-19.el7.x86_64 and openssl-libs-1:1.1.1k-5.el8_5.x86_64
- package ntpdate-4.2.6p5-29.el7.centos.2.x86_64 requires libcrypto.so.10()(64bit), but none of the providers can be installed
- package ntpdate-4.2.6p5-29.el7.centos.2.x86_64 requires libcrypto.so.10(libcrypto.so.10)(64bit), but none of the providers can be installed
- package ntpdate-4.2.6p5-29.el7.centos.2.x86_64 requires libcrypto.so.10(OPENSSL_1.0.2)(64bit), but none of the providers can be installed
- conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
报错，嘤嘤嘤
问题看起来应该是安装libcurl-minimal-7.61.1-18.el8.x86_64的时候，缺少了很多这个库的依赖，libcrypto.so.1.1、libssl.so.1.1、openssl-libs
查看了yum源的问题，我从docker hub拉取的是centos8的镜像，但是阿里的源提供的是centos7的源，导致无法使用，所以需要从头再来一次，在docker hub找到一个centos7的镜像，再来过。
在docker hub找镜像
image-20230427144921136.png
这次找一个centos7，应该没问题了
拉取镜像
docker pull centos:centos7@sha256:dead07b4d8ed7e29e98de0f4504d87e8880d4347859d839686a31da35a3b532f
查看镜像
docker images
网络创建过了就不需要创建网络了，直接创建容器
重复前面的步骤
在yum中查找ntp
yum search ntp
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.tuna.tsinghua.edu.cn
* updates: ftp.riken.jp
================================================================================================ N/S matched: ntp ================================================================================================
ntp.x86_64 : The NTP daemon and utilities
ntp-doc.noarch : NTP documentation
ntp-perl.noarch : NTP utilities written in Perl
ntpdate.x86_64 : Utility to set the date and time via NTP
python-ntplib.noarch : Python module that offers a simple interface to query NTP servers
chrony.x86_64 : An NTP client/server
cyrus-imapd.x86_64 : A high-performance mail server with IMAP, POP3, NNTP and SIEVE support
fontpackages-devel.noarch : Templates and macros used to create font packages
fontpackages-filesystem.noarch : Directories used by font packages
fontpackages-tools.noarch : Tools used to check fonts and font packages
sntp.x86_64 : Standard Simple Network Time Protocol program
Name and summary matches only, use "search all" for everything.
这次直接有了
通过yum安装
yum install ntp -y
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.cqu.edu.cn
* extras: mirrors.tuna.tsinghua.edu.cn
* updates: ftp.riken.jp
Resolving Dependencies
--> Running transaction check
---> Package ntp.x86_64 0:4.2.6p5-29.el7.centos.2 will be installed
--> Processing Dependency: libopts.so.25()(64bit) for package: ntp-4.2.6p5-29.el7.centos.2.x86_64
--> Running transaction check
---> Package autogen-libopts.x86_64 0:5.18-5.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================================================================================================
Package Arch Version Repository Size
==================================================================================================================================================================================================================
Installing:
ntp x86_64 4.2.6p5-29.el7.centos.2 base 549 k
Installing for dependencies:
autogen-libopts x86_64 5.18-5.el7 base 66 k
Transaction Summary
==================================================================================================================================================================================================================
Install 1 Package (+1 Dependent package)
Total download size: 615 k
Installed size: 1.5 M
Downloading packages:
(1/2): autogen-libopts-5.18-5.el7.x86_64.rpm | 66 kB 00:00:00
(2/2): ntp-4.2.6p5-29.el7.centos.2.x86_64.rpm | 549 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.2 MB/s | 615 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : autogen-libopts-5.18-5.el7.x86_64 1/2
Installing : ntp-4.2.6p5-29.el7.centos.2.x86_64 2/2
Verifying : autogen-libopts-5.18-5.el7.x86_64 1/2
Verifying : ntp-4.2.6p5-29.el7.centos.2.x86_64 2/2
Installed:
ntp.x86_64 0:4.2.6p5-29.el7.centos.2
Dependency Installed:
autogen-libopts.x86_64 0:5.18-5.el7
Complete!
编辑ntp服务节点配置文件
vi /etc/ntp.conf
修改后文件内容
1683255544796.png
重启ntp服务
systemctl start ntpd
查看状态
systemctl status ntpd
确认没有问题
设置开机启动
systemctl enable ntpd
编辑ntp客户端节点配置文件
vi /etc/ntp.conf
修改后文件内容
1683256090185.png
重启ntp服务
systemctl start ntpd
查看状态
systemctl status ntpd
查看配置是否生效
ntpq -p
返回
remote refid st t when poll reach delay offset jitter
==============================================================================
*a6d039150d75 LOCAL(0) 6 u 20 64 17 0.216 0.093 0.073
确认没有问题
设置开机启动
systemctl enable ntpd
主节点部署MySQL
查看是否安装过MySQL
rpm -qa | grep -i mysql
没有返回，说明没有安装过
安装MySQL
下载rpm包
wget https://repo.mysql.com/mysql57-community-release-el7-10.noarch.rpm
安装mysql源
yum localinstall -y mysql57-community-release-el7-10.noarch.rpm
检查mysql源
yum repolist enabled | grep "mysql.*-community.*"
返回
mysql-connectors-community/x86_64 MySQL Connectors Community 203
mysql-tools-community/x86_64 MySQL Tools Community 129
mysql57-community/x86_64 MySQL 5.7 Community Server 504
安装
yum -y install mysql-community-server
查看状态
systemctl status mysqld.service
返回
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
启动
systemctl restart mysqld.service
查看状态
systemctl status mysqld.service
说明安装正常
修改MySQL登录密码
查看默认密码
grep 'temporary password' /var/log/mysqld.log
返回
2023-05-05T08:19:46.844203Z 1 [Note] A temporary password is generated for root@localhost: tBGvK=)I8oy?
密码为：tBGvK=)I8oy?
修改登录密码
mysql -u root -p
输入一次密码
进入MySQL命令行
ALTER USER 'root'@'localhost' IDENTIFIED BY '（要改的密码）';
小插曲，这里修改的是‘123456’的密码，返回告诉我密码不够安全
返回
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
这里需要修改MySQL的密码强度要求
set global validate_password_policy=LOW;
再去修改，告诉我还是不够安全，原因是密码长度要求为8位，再把长度修改为6位
set global validate_password_length=6;
再再次修改，成功
ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
返回
Query OK, 0 rows affected (0.01 sec)
退出，再次登录验证是否修改成功
quit
返回
Bye
登陆验证
mysql -u root -p
输入修改的新密码，登陆成功
MySQL密码修改完毕
修改MySQL配置文件
vi /etc/my.cnf
在文件中添加内容
validate_password=off
character_set_server=utf8
init_connect='SET NAMES utf8'
重启mysql，将配置进行生效
systemctl restart mysqld
此配置内容不修改，创建CDH相关库时会出现问题
允许root用户远程登录
进入MySQL命令行
mysql -u root -p
输入密码
执行
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '（密码）' WITH GRANT OPTION;
FLUSH PRIVILEGES;
创建CDH相关mysql库
CREATE DATABASE scm DEFAULT CHARACTER SET utf8;
CREATE USER 'scm'@'%'IDENTIFIED BY 'scm';
GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL PRIVILEGES ON *.* TO 'scm'@'%' IDENTIFIED BY 'scm' WITH GRANT OPTION;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8;
CREATE USER 'amon'@'%'IDENTIFIED BY 'amon';
GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL PRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY 'amon' WITH GRANT OPTION;
CREATE DATABASE hive DEFAULT CHARACTER SET utf8;
CREATE USER 'hive'@'%'IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8;
CREATE USER 'hue'@'%'IDENTIFIED BY 'hue';
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL PRIVILEGES ON *.* TO 'hue'@'%' IDENTIFIED BY 'hue' WITH GRANT OPTION;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8;
CREATE USER 'oozie'@'%'IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY 'oozie' WITH GRANT OPTION;
FLUSH PRIVILEGES;
验证数据库是否创建
show databases;
返回
+--------------------+
| Database |
+--------------------+
| information_schema |
| amon |
| hive |
| hue |
| mysql |
| oozie |
| performance_schema |
| scm |
| sys |
+--------------------+
9 rows in set (0.00 sec)
创建成功
配置cdh相关的yum源
我这里有之前配置过的yum源，直接拿配置用，如果没有配置过请参考https://www.jianshu.com/p/6f120a99cbae第8节
所有服务器安装JDK
在所有节点执行
yum install -y oracle-j2sdk1.8-1.8.0+update181-1.x86_64
配置jdk环境
vi /etc/profile
添加内容
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=:$JAVA_HOME/bin:$PATH
保存退出
生效
source /etc/profile
安装cloudera-agent
对所有节点安装daemons、agent
yum install -y python27 python2
yum install -y cloudera-manager-daemons
yum install -y cloudera-manager-agent.x86_64
修改所有节点agent配置
vi /etc/cloudera-scm-agent/config.ini
修改server_host值
server_host=a6d039150d75
在主节点安装cloudera-manager-server及scm数据库初始化
安装server服务
yum install -y cloudera-manager-server
初始化scm数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm
小插曲，这里有个报错
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/
../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.[ main] DbCommandExecutor INFO Unable to find JDBC driver for database type: MySQL
[ main] DbCommandExecutor ERROR JDBC Driver com.mysql.jdbc.Driver not found.
[ main] DbCommandExecutor ERROR Exiting with exit code 3
--> Error 3, giving up (use --force if you wish to ignore the error)
连接MySQL缺少一个jar包，jar包的路径在“/usr/share/java”下，顺着目录查找就根本没有这个目录，秉着没有啥就给啥的路子
就创建/usr/share/java目录，把mysql-connector-java-5.1.46.jar，上传，改成mysql-connector-java.jar名字
再执行，通过
返回
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/
../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Sat May 06 05:42:40 UTC 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be estab
lished by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
启动cloudera-manager-server及所有cloudera-manager-agent
在主节点启动
systemctl start cloudera-scm-server
查看状态
systemctl status cloudera-scm-server
返回
● cloudera-scm-server.service - Cloudera CM Server Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
Active: inactive (dead)
[root@a6d039150d75 java]# systemctl restart cloudera-scm-server
[root@a6d039150d75 java]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2023-05-06 05:48:50 UTC; 1s ago
Process: 6155 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
Main PID: 6157 (java)
CGroup: /docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/system.slice/cloudera-scm-server.service
└─6157 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server...
‣ 6157 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server...
May 06 05:48:50 a6d039150d75 systemd[1]: cloudera-scm-server.service: control process exited, code=exited status=0
May 06 05:48:50 a6d039150d75 systemd[1]: cloudera-scm-server.service got final SIGCHLD for state start-pre
May 06 05:48:50 a6d039150d75 systemd[1]: About to execute: /opt/cloudera/cm/bin/cm-server
May 06 05:48:50 a6d039150d75 systemd[1]: Forked /opt/cloudera/cm/bin/cm-server as 6157
May 06 05:48:50 a6d039150d75 systemd[1]: Cleaning 6157 from watches.
May 06 05:48:50 a6d039150d75 systemd[1]: Watching 6157 through watch_pids1.
May 06 05:48:50 a6d039150d75 systemd[1]: cloudera-scm-server.service changed start-pre -> running
May 06 05:48:50 a6d039150d75 systemd[1]: Job cloudera-scm-server.service/start finished, result=done
May 06 05:48:50 a6d039150d75 systemd[1]: Started Cloudera CM Server Service.
May 06 05:48:50 a6d039150d75 systemd[1]: Watching 6157 through watch_pids1.
所有节点启动
systemctl start cloudera-scm-agent
查看状态
systemctl status cloudera-scm-agent
返回
● cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2023-05-06 05:49:56 UTC; 21s ago
Main PID: 6356 (cmagent)
CGroup: /docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/docker/a6d039150d7557748adae3b2eb48f51a476b75252b843ef6dbbc46b758641872/system.slice/cloudera-scm-agent.service
└─6356 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
‣ 6356 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread tmpfs INFO Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
May 06 05:49:57 a6d039150d75 cm[6356]: [06/May/2023 05:49:57 +0000] 6356 MainThread logging INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
May 06 05:50:02 a6d039150d75 cm[6356]: status_server: added process group
May 06 05:50:02 a6d039150d75 cm[6356]: flood: added process group
May 06 05:50:02 a6d039150d75 cm[6356]: /opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_pslinux.py:477: RuntimeWarning: dirty, writeback, mapped, commit_limit memory stats couldn't b...d were set to 0
May 06 05:50:02 a6d039150d75 cm[6356]: warnings.warn(msg, RuntimeWarning)
Hint: Some lines were ellipsized, use -l to show in full.
集群搭建
进入本机ip+7180端口
1683352347845.png
1683352384040.png
1683352406081.png
1683352437631.png
1683352481227.png
1683352526645.png
1683352553412.png
给集群起个名字
1683352553412.png
通过ip搜索可以，因为提前给所有主机安装了agent，直接可以查找到
1683352598065.png
1683352716262.png
1683352728669.png
1683352746988.png
1683353530953.png
主要是修改swap分区的问题
临时修改
sysctl vm.swappiness=10
永久修改
vi /etc/sysctl.conf
添加内容
vm. swappiness=10
重新检查
通过
基本已经搭建完成，还有许多后续问题，继续记录
