Ambari 2.6.2安装Hadoop hbase集群

2018-06-08 本文已影响907人大道至简非简

一、环境准备

1、集群规划

	hadoop01/192.168.1.151	hadoop02/192.168.1.152	hadoop03/192.168.1.153	hadoop04/192.168.1.154	hadoop05/192.168.1.155
ambari server	是	否	否	否	否
ambari agent	是	是	是	是	是
namenode	是	是	否	否	否
datanode	否	否	是	是	是
resourcemanager	是	是	否	否	否
journalnode	是	是	是	是	是
zookeeper	是	是	是	否	否

01和02主要做管理机，03-05做数据数据节点，01-03做zookeeper集群。

2、安装一台lnmp备用

选择hadoop01

https://lnmp.org/install.html

wget -c http://soft.vpser.net/lnmp/lnmp1.5.tar.gz && tar zxf lnmp1.5.tar.gz && cd lnmp1.5 && ./install.sh lnmp

选择安装mysql5.7版本用于支持ambari

3、下载离线版所需文件

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.0/bk_ambari-installation/content/ch_obtaining-public-repos.html

下载备用。

二、安装配置ambari

1、配置离线仓库地址

mkdir /home/wwwroot/default/ambari
mkdir /home/wwwroot/default/ambari/HDP-UTILS
cd /home/wwwroot/default/ambari

复制下载文件到相关目录并解压

tar -zxvf ambari-2.6.2.0-ubuntu16.tar.gz
tar -zxvf HDP-GPL-2.6.5.0-ubuntu16-gpl.tar.gz
tar -zxvf HDP-2.6.5.0-ubuntu16-deb.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-ubuntu16.tar.gz -C HDP-UTILS

开启nginx对ambari目录的浏览功能。

2、制作list文件并复制到各台agent主机

cd /etc/apt/sources.list.d
ambari.list

deb http://192.168.1.151/ambari/ambari/ubuntu16/2.6.2.0-155/ Ambari main

ambari-hdp.list

deb http://192.168.1.151/ambari/HDP/ubuntu16/2.6.5.0-292/ HDP main
deb http://192.168.1.151/ambari/HDP-GPL/ubuntu16/2.6.5.0-292/ HDP-GPL main
deb http://192.168.1.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu16/1.1.0.22/ HDP-UTILS main

更新list

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
sudo apt-get update
apt-cache showpkg ambari-server
apt-cache showpkg ambari-agent
apt-cache showpkg ambari-metrics-assembly

3、安装ambari

sudo apt-get install ambari-server


root@hadoop01:/home# sudo apt-get install ambari-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libsensors4 postgresql postgresql-9.5 postgresql-client-9.5 postgresql-client-common postgresql-common postgresql-contrib-9.5 ssl-cert sysstat
Suggested packages:
  lm-sensors postgresql-doc locales-all postgresql-doc-9.5 libdbd-pg-perl openssl-blacklist isag
The following NEW packages will be installed:
  ambari-server libsensors4 postgresql postgresql-9.5 postgresql-client-9.5 postgresql-client-common postgresql-common postgresql-contrib-9.5 ssl-cert sysstat
0 upgraded, 10 newly installed, 0 to remove and 48 not upgraded.
Need to get 757 MB of archives.
After this operation, 831 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://192.168.1.151/ambari/ambari/ubuntu16/2.6.2.0-155 Ambari/main amd64 ambari-server amd64 2.6.2.0-155 [752 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 libsensors4 amd64 1:3.4.0-2 [28.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-common all 173ubuntu0.1 [28.4 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-9.5 amd64 9.5.13-0ubuntu0.16.04 [871 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 ssl-cert all 1.0.37 [16.9 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-common all 173ubuntu0.1 [154 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-9.5 amd64 9.5.13-0ubuntu0.16.04 [3,001 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql all 9.5+173ubuntu0.1 [5,450 B]                                                                                                                                                                
Get:9 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-contrib-9.5 amd64 9.5.13-0ubuntu0.16.04 [448 kB]                                                                                                                                              
Get:10 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 sysstat amd64 11.2.0-1ubuntu0.2 [243 kB]                                                                                                                                                                
Fetched 757 MB in 12s (59.8 MB/s)                                                                                                                                                                                                                                            
Preconfiguring packages ...
Selecting previously unselected package libsensors4:amd64.
(Reading database ... 70302 files and directories currently installed.)
Preparing to unpack .../libsensors4_1%3a3.4.0-2_amd64.deb ...
Unpacking libsensors4:amd64 (1:3.4.0-2) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_173ubuntu0.1_all.deb ...
Unpacking postgresql-client-common (173ubuntu0.1) ...
Selecting previously unselected package postgresql-client-9.5.
Preparing to unpack .../postgresql-client-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package ssl-cert.
Preparing to unpack .../ssl-cert_1.0.37_all.deb ...
Unpacking ssl-cert (1.0.37) ...
Selecting previously unselected package postgresql-common.
Preparing to unpack .../postgresql-common_173ubuntu0.1_all.deb ...
Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'
Unpacking postgresql-common (173ubuntu0.1) ...
Selecting previously unselected package postgresql-9.5.
Preparing to unpack .../postgresql-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql.
Preparing to unpack .../postgresql_9.5+173ubuntu0.1_all.deb ...
Unpacking postgresql (9.5+173ubuntu0.1) ...
Selecting previously unselected package postgresql-contrib-9.5.
Preparing to unpack .../postgresql-contrib-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-contrib-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package sysstat.
Preparing to unpack .../sysstat_11.2.0-1ubuntu0.2_amd64.deb ...
Unpacking sysstat (11.2.0-1ubuntu0.2) ...
Selecting previously unselected package ambari-server.
Preparing to unpack .../ambari-server_2.6.2.0-155_amd64.deb ...
Unpacking ambari-server (2.6.2.0-155) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libsensors4:amd64 (1:3.4.0-2) ...
Setting up postgresql-client-common (173ubuntu0.1) ...
Setting up postgresql-client-9.5 (9.5.13-0ubuntu0.16.04) ...
update-alternatives: using /usr/share/postgresql/9.5/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up ssl-cert (1.0.37) ...
Setting up postgresql-common (173ubuntu0.1) ...
Adding user postgres to group ssl-cert

Creating config file /etc/postgresql-common/createcluster.conf with new version

Creating config file /etc/logrotate.d/postgresql-common with new version
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up postgresql-9.5 (9.5.13-0ubuntu0.16.04) ...
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?<!\\)\${ <-- HERE ([^}]+)}/ at /usr/sbin/pam_getenv line 78.
Creating new cluster 9.5/main ...
  config /etc/postgresql/9.5/main
  data   /var/lib/postgresql/9.5/main
  locale en_US.UTF-8
  socket /var/run/postgresql
  port   5432
update-alternatives: using /usr/share/postgresql/9.5/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
Setting up postgresql (9.5+173ubuntu0.1) ...
Setting up postgresql-contrib-9.5 (9.5.13-0ubuntu0.16.04) ...
Setting up sysstat (11.2.0-1ubuntu0.2) ...

Creating config file /etc/default/sysstat with new version
update-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode
Setting up ambari-server (2.6.2.0-155) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...

5、初始配置

sudo ambari-server setup

使用自定义jdk
采用mysql数据库（生产环境采用内置postgresql不合适）

需要相关jdbc。
https://dev.mysql.com/downloads/connector/j/

在主节点把mysql数据库连接包拷贝在/var/lib/ambary-server/resources目录下并改名为mysql-jdbc-driver.jar

vim /etc/ambari-server/conf/ambari.properties

Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
......ambari-admin-2.6.2.0.155.jar
.....
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

三、配置ambari

1、启动ambari

root@hadoop01:~# sudo ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........Unable to determine server PID. Retrying...
......Unable to determine server PID. Retrying...
......Unable to determine server PID. Retrying...
ERROR: Exiting with exit code -1. 
REASON: Ambari Server java process died with exitcode 1. Check /var/log/ambari-server/ambari-server.out for more information.

出现错误。
导入mysql脚本

mysql -uroot -p

mysql> use ambari;
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

再次启动

root@hadoop01:~# sudo ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

采用admin/admin登录

四、配置hadoop

选择本地repository，填入本地url；

本地hdp-utils缺失文件错误（2.6文件几十k，2.5版本文件32mb，估计官网问题）。更换成公网地址。
或者用2.5的文件替换2.6的文件dist目录。

cat ~/.ssh/id_rsa
获取ssh密钥。

解决问题
etc/ambari-agent/conf/ambari-agent.ini
[security]
force_https_protocol=PROTOCOL_TLSv1_2

关闭thp
vi /etc/init.d/disable-transparent-hugepages

#!/bin/sh

### BEGIN INIT INFO

# Provides:  disable-transparent-hugepages

# Required-Start:$local_fs

# Required-Stop:

# X-Start-Before:mongod mongodb-mms-automation-agent

# Default-Start: 2 3 4 5

# Default-Stop:  0 1 6

# Short-Description: Disable Linux transparent huge pages

# Description:   Disable Linux transparent huge pages, to improve

#database performance.

### END INIT INFO

case $1 in

　　start)

　　　　if [ -d /sys/kernel/mm/transparent_hugepage ]; then

　　　　　　thp_path=/sys/kernel/mm/transparent_hugepage

　　　　elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then

　　　　　　thp_path=/sys/kernel/mm/redhat_transparent_hugepage

　　　　else

　　　　　　return 0

　　　　fi

　　　　echo 'never' > ${thp_path}/enabled

　　　　echo 'never' > ${thp_path}/defrag

　　　　unset thp_path

　　;;

esac

sudo chmod 755 /etc/init.d/disable-transparent-hugepages

sudo update-rc.d disable-transparent-hugepages defaults

选择需要的组件
hbase不是最新2.1.0版本。

经过各种坑，顺利完成。

配置高可用

高可用配置完成。