mysql主主复制+nginx负载均衡+zookeeper+ka
2020-08-11 本文已影响0人
天才一般的幼稚
一、服务器配置
| 192.168.2.247 | 192.168.2.248 | 192.168.2.249 |
|---|---|---|
| jdk | jdk | jdk |
| tomcat | tomcat | |
| nginx | ||
| mysql | mysql | |
| kafka | ||
| elasticsearch+head | ||
| activemq | activemq | |
| redis | redis |
二、安装jdk
1、解压 jdk压缩包
[root@247 local]# tar -zxf jdk-8u11-linux-x64.tar.gz
2、添加环境变量
[root@247 local]# vim /etc/profile
3、在最后添加
JAVA_HOME=/usr/local/jdk1.8.0_11/
export JRE_HOME=/usr/local/jdk1.8.0_11/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
4、编译、查看
[root@247 local]# source /etc/profile
[root@247 local]# java -version
java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)
5、其余同理，不再赘述
二、安装activemq
1、解压activemq
[root@247 local]# tar -zxf apache-activemq-5.15.9-bin.tar.gz
2、启动activemq
[root@247 apache-activemq-5.15.9]# cd bin
[root@247 bin]# ./activemq start
INFO: Loading '/usr/local/apache-activemq-5.15.9//bin/env'
INFO: Using java '/usr/local/jdk1.8.0_11/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/usr/local/apache-activemq-5.15.9//data/activemq.pid' (pid '1495')
三、安装redis
1、解压
[root@247 ~]# tar -zxf redis-5.0.5.tar.gz
2、编译、安装
2.1、编译
[root@247 redis-5.0.5]# make
发现报错：报错，缺少依赖，需要安装gcc
make[3]: Entering directory `/usr/local/redis-5.0.5/deps/hiredis'
gcc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb net.c
make[3]: gcc: Command not found
make[3]: *** [net.o] Error 127
make[3]: Leaving directory `/usr/local/redis-5.0.5/deps/hiredis'
make[2]: *** [hiredis] Error 2
make[2]: Leaving directory `/usr/local/redis-5.0.5/deps'
make[1]: [persist-settings] Error 2 (ignored)
CC adlist.o
/bin/sh: cc: command not found
make[1]: *** [adlist.o] Error 127
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'
make: *** [all] Error 2
安装gcc依赖
[root@247 redis-5.0.5]# yum install gcc
重新编译，成功
CC listpack.o
CC localtime.o
CC lolwut.o
CC lolwut5.o
LINK redis-server
INSTALL redis-sentinel
CC redis-cli.o
LINK redis-cli
CC redis-benchmark.o
LINK redis-benchmark
INSTALL redis-check-rdb
INSTALL redis-check-aof
Hint: It's a good idea to run 'make test' ;)
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'
2.2、安装
[root@247 redis-5.0.5]# make install PREFIX=/usr/local/redis
cd src && make install
make[1]: Entering directory `/usr/local/redis-5.0.5/src'
CC Makefile.dep
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'
make[1]: Entering directory `/usr/local/redis-5.0.5/src'
Hint: It's a good idea to run 'make test' ;)
INSTALL install
INSTALL install
INSTALL install
INSTALL install
INSTALL install
make[1]: Leaving directory `/usr/local/redis-5.0.5/src'
3、移动配置文件到安装目录下
[root@247 redis-5.0.5]# mkdir /usr/local/redis/etc
[root@247 redis-5.0.5]# mv redis.conf /usr/local/redis/etc
4、修改redis.conf配置文件
# 507行 密码
# requirepass foobared 取消注释，输入密码
requirepass 123 指定密码123
5、配置redis为后台启动
vim /usr/local/redis/etc/redis.conf
# 将daemonize no 改成daemonize yes
6、将redis加入到开机启动
[root@247 redis-5.0.5]# vim /etc/rc.local
#在最后一行添加
/usr/local/redis/bin/redis-server
7、添加环境变量
[root@247 redis-5.0.5]# vim /etc/profile
JAVA_HOME=/usr/local/jdk1.8.0_11/
REDIS_HOME=/usr/local/redis/ # 添加REDIS_HOME
export JRE_HOME=/usr/local/jdk1.8.0_11/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$REDIS_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
[root@247 redis-5.0.5]# source /etc/profile
8、开机启动
[root@247 redis-5.0.5]# redis-server /usr/local/redis/etc/redis.conf
16161:C 06 Aug 2020 03:45:13.727 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16161:C 06 Aug 2020 03:45:13.727 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=16161, just started
16161:C 06 Aug 2020 03:45:13.727 # Configuration loaded
四、安装tomcat
1、解压
[root@247 local]# tar -zxf apache-tomcat-8.5.45.tar.gz
五、安装mysql
1、安装mysql
1.1、安装mysql源镜像
[root@247 ~]# yum localinstall mysql57-community-release-el7-8.noarch.rpm
1.2、检查mysql源是否安装成功
[root@247 ~]# yum repolist enabled | grep "mysql.*-community.*"
mysql-connectors-community/x86_64 MySQL Connectors Community 165
mysql-tools-community/x86_64 MySQL Tools Community 115
mysql57-community/x86_64 MySQL 5.7 Community Server 444
1.3、修改vim /etc/yum.repos.d/mysql-community.repo源
此处默认使用mysql57，所以这里不做改动
1.4、安装Mysql
[root@247 ~]# yum install mysql-community-server
1.5、启动MySQL服务
[root@247 ~]# systemctl start mysqld
[root@247 ~]# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2020-08-06 04:20:55 EDT; 8s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 16414 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
Process: 16364 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 16416 (mysqld)
CGroup: /system.slice/mysqld.service
└─16416 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Aug 06 04:20:51 247 systemd[1]: Starting MySQL Server...
Aug 06 04:20:55 247 systemd[1]: Started MySQL Server.
1.6、设置开机启动
[root@247 ~]# systemctl enable mysqld
[root@247 ~]# systemctl daemon-reload
1.7、修改密码策略
此处关闭密码的校验，否则需要输入复杂的密码
[root@247 ~]# vim /etc/my.cnf
# 底部添加配置
validate_password = off
1.8、重新启动mysql服务
[root@247 ~]# systemctl restart mysqld
1.9、查看默认密码
[root@247 ~]# grep 'temporary password' /var/log/mysqld.log
2020-08-06T08:20:53.105154Z 1 [Note] A temporary password is generated for root@localhost: klh(HlZ7PHh_
1.10、使用默认密码登录
[root@247 ~]# mysql -uroot -p
Enter password:
1.11、修改密码
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)
1.12、允许外部用户访问
mysql> grant all privileges on *.* to root@'%' identified by 'root';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
1.13、执行sql文件报错
ERROR 1418 (HY000): This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)
此处报错导致存储函数没有执行，set global log_bin_trust_function_creators=TRUE;
1.14、另一台同理配置
2、配置mysql主主复制
2.1、修改配置文件
[root@247 ~]# vim /etc/my.cnf
# 在底部添加一下参数
server-id=1
innodb_flush_log_at_trx_commit=2
sync_binlog=1
log-bin=mysql-bin-1
配置说明：
设置主服务 的ID (id可以自己随便设置但是要保证和slave的id不一样)
server-id=1
设为1当然是最安全的，但性能也是最差的（相对其他两个参数而言，但不是不能接受）。如果对数据一致性和完整性要求不高，完全可以设为2，如果只最求性能，例如高并发写的日志服务器，设为0来获得更高性能
innodb_flush_log_at_trx_commit=2
开启binlog 志同步功能
sync_binlog=1
binlog 日志文件名
log-bin=mysql-bin-200
这个表示只同步某个库 (如果没有此项，表示同步所有的库)
binlog-do-db=xxxx
2.2、重启mysql
[root@247 ~]# systemctl restart mysqld
2.3、进入mysq输入授权命令
GRANT REPLICATION SLAVE ON *.* to 'root'@'%' identified by '123456';
2.4、查看master节点的状态
mysql> show master status;
+--------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+--------------------+----------+--------------+------------------+-------------------+
| mysql-bin-1.000001 | 437 | | | |
+--------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
2.5、修改从节点配置
[root@248 ~]# vim /etc/my.cnf
server-id=201 #和master不一样
innodb_flush_log_at_trx_commit=2
sync_binlog=1
log-bin=mysql-bin-201
[root@248 ~]# systemctl restart mysqld
2.6、指定master节点
mysql> change master to master_host='192.168.2.247',master_user='root' ,master_password='123456', master_log_file='mysql-bin-1.000001' ,master_log_pos=437;
Query OK, 0 rows affected, 2 warnings (0.02 sec)
命令说明：
master_host: 主机的ip
master_user : 主机授权的用户.
master_password : 主机授权时候填写的密码
master_log_file : 主机show master status;中的File
master_log_pos: 主机show master status;中的Position.
2.7、启动slave
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.2.247
Master_User: root
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin-1.000001
Read_Master_Log_Pos: 437
Relay_Log_File: 248-relay-bin.000002
Relay_Log_Pos: 322
Relay_Master_Log_File: mysql-bin-1.000001
Slave_IO_Running: Yes #IO线程启动
Slave_SQL_Running: Yes #SQL线程启动
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 437
Relay_Log_Space: 527
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_UUID: bdbe37d5-d7bd-11ea-b10d-000c29b348db
Master_Info_File: /var/lib/mysql/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.01 sec)
ERROR:
No query specified
至此，已完成了247和248的主从复制，重复操作，完成248和247的主从复制即可实现主主复制了。
六、配置Nginx负载均衡
1、安装nginx
1.1、解压nginx
[root@247 local]# tar -zxf nginx-1.18.0.tar.gz
1.2、安装依赖包
[root@247 nginx-1.18.0]# yum -y install gcc gcc-c++ make libtool zlib zlib-devel openssl openssl-devel pcre pcre-devel
1.3、配置nginx
[root@247 nginx-1.18.0]# ./configure --prefix=/usr/local/nginx
1.4、编译、安装
make && make install
1.5、启动
进入nginx/sbin下，执行nginx
[root@247 nginx]# ./nginx
2、配置负载均衡
2.1、配置nginx.conf
nginx_conf.png
2.2、启动tomcat
247-tomcat.jpg
248-tomcat.jpg
2.3、查看负载均衡是否成功
247-nginx.jpg
七、配置Kafka、Elasticsearch、Connect连接器
具体参考这位大神的：
使用Kafka的Connect监听Mysql数据并同步到ElasticSearch-刘宇
1、安装zookeeper
1.1、解压zookeeper
[root@249 ~]# tar -xzvf zookeeper-3.4.14.tar.gz -C /usr/local
1.2、创建快照和日志存储路径
[root@249 zookeeper-3.4.14]# mkdir dataDir
[root@249 zookeeper-3.4.14]# mkdir dataDirLog
1.3、修改配置文件
[root@249 zookeeper-3.4.14]# cd conf/
[root@249 conf]# cp zoo_sample.cfg zoo.cfg
[root@249 conf]# vim zoo.cfg
zoo.cfg.png
1.4、添加zookeeper唯一标识
[root@249 conf]# cd ../dataDir
[root@249 dataDir]# echo "1" > myid
1.5、启动zookeeper
[root@249 dataDir]# cd /usr/local/zookeeper-3.4.14/bin
[root@249 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
2、安装kafka
2.1、解压
[root@249 local]# tar -zxf /root/kafka_2.11-2.2.1.tgz -C ./
[root@249 local]# mv kafka_2.11-2.2.1/ kafka
2.2、修改配置文件
[root@249 local]# cd kafka/
[root@249 kafka]# cd config/
[root@249 config]# vim server.properties
# The id of the broker. This must be set to a unique integer for each broker.
# 此处唯一
broker.id=0
# The address the socket server listens on. It will get the value returned from
listeners=PLAINTEXT://192.168.2.249:9092
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.2.249:2181
2.3、后台启动kafka
[root@249 kafka]# nohup bin/kafka-server-start.sh config/server.properties &
[root@249 kafka]# jps
14386 Kafka
14722 Jps
14359 QuorumPeerMain
3、安装Elasticsearch
3.1、解压
[root@249 ~]# tar -zxf elasticsearch-5.6.8.tar.gz -C /usr/local
3.2、修改配置文件
[root@249 ~]# vim /usr/local/elasticsearch-5.6.8/config/elasticsearch.yml
elasticsearch.yml.png
3.3、新建路径
[root@249 elasticsearch-5.6.8]# mkdir data
[root@249 elasticsearch-5.6.8]# mkdir logs
3.4、创建新用户
[root@249 elasticsearch-5.6.8]# useradd es
3.5、设置目录归属
[root@249 elasticsearch-5.6.8]# chown -R es:es /usr/local/elasticsearch-5.6.8
[root@249 elasticsearch-5.6.8]# ll
total 224
drwxr-xr-x 2 es es 4096 Aug 11 02:27 bin
drwxr-xr-x 2 es es 75 Aug 11 02:43 config
drwxr-xr-x 2 es es 6 Aug 11 02:32 data
drwxr-xr-x 2 es es 4096 Feb 16 2018 lib
-rw-r--r-- 1 es es 11358 Feb 16 2018 LICENSE.txt
drwxr-xr-x 2 es es 6 Aug 11 02:33 logs
drwxr-xr-x 13 es es 236 Feb 16 2018 modules
-rw-r--r-- 1 es es 194187 Feb 16 2018 NOTICE.txt
drwxr-xr-x 2 es es 6 Feb 16 2018 plugins
-rw-r--r-- 1 es es 9490 Feb 16 2018 README.textile
3.6、启动elasticsearch
[root@249 elasticsearch-5.6.8]# su es
[es@249 elasticsearch-5.6.8]$ cd bin
[es@249 bin]$ nohup ./elasticsearch &
[1] 14921
[es@249 bin]$ nohup: ignoring input and appending output to ‘nohup.out’
[es@249 bin]$ jps
14960 Jps
14921 Elasticsearch
3.7、报错
1、
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
root用户编辑/etc/security/limits.conf，尾部添加这两行，并重启服务器
image.png
2、
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
root用户编辑/etc/sysctl.conf
添加vm.max_map_count=262144
[root@249 bin]# vim /etc/sysctl.conf
[root@249 bin]# sysctl -p
vm.max_map_count = 262144
3.8、测试
[es@249 bin]$ curl http://192.168.2.249:9200
{
"name" : "node-1",
"cluster_name" : "my-application",
"cluster_uuid" : "dN7-77B7Qn2yrxXgfKk_cw",
"version" : {
"number" : "5.6.8",
"build_hash" : "688ecce",
"build_date" : "2018-02-16T16:46:30.010Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
八、配置kafka中的Connect，实现将MySQL数据同步到Elasticsearch中
8.1、在kafka的config文件中配置mysql到kafka的连接器
mysql-kafka-connect.png
8.2、在kafka的config文件中配置kafka到elasticsearch的连接器
image.png
8.3、运行connect
[root@249 bin]# nohup ./connect-standalone.sh ../config/connect-standalone.properties ../config/es-sensorData.properties ../config/sensorData.properties &
