Heartbeat+DRBD+MySQL高可用读写分离架构方案
2017-05-03 本文已影响155人
素白流殇
声明：本案例仅为评估测试版本
注意：所有服务器之间必须做好时间同步
架构拓扑
架构图
IP信息：
IP地址
Heartbeat安装部署
1.安装heartbeat（主备节点同时安装）
[root@master1 ~]# yum install heartbeat* -y
[root@master2 ~]# yum install heartbeat* -y
heartbeat的yum安装完的默认路径为 /etc/ha.d/
[root@master1 ~]# cd /etc/ha.d/
[root@master1 ha.d]# ls -al
total 28
drwxr-xr-x 4 root root 4096 May 3 19:08 .
drwxr-xr-x. 67 root root 4096 May 3 19:08 ..
-rwxr-xr-x 1 root root 745 Dec 3 2013 harc
drwxr-xr-x 2 root root 4096 May 3 19:08 rc.d
-rw-r--r-- 1 root root 692 Dec 3 2013 README.config
drwxr-xr-x 2 root root 4096 May 3 19:08 resource.d
-rw-r--r-- 1 root root 2082 Mar 24 21:01 shellfuncs
一般安装任何软件注意阅读README文件
从readme文件我们可以获取到三个重要文件：
- ha.cf：主配置文件
- haresources：资源配置文件
- authkeys：认证信息文件
此三类文件在主备服务器上保持完全一致
[root@master1 ha.d]# cat README.config
You need three configuration files to make heartbeat happy,
and they all go in this directory.
They are:
ha.cf Main configuration file
haresources Resource configuration file
authkeys Authentication information
These first two may be readable by everyone, but the authkeys file
must not be.
The good news is that sample versions of these files may be found in
the documentation directory (providing you installed the documentation).
If you installed heartbeat using rpm packages then
this command will show you where they are on your system:
rpm -q heartbeat -d
If you installed heartbeat using Debian packages then
the documentation should be located in /usr/share/doc/heartbeat
需要将此文件从 /usr/share/doc/heartbeat 下copy到 /etc/ha.d/ 目录下
[root@master1 ~]# cd /usr/share/doc/heartbeat-3.0.4/
[root@master1 heartbeat-3.0.4]# cp authkeys ha.cf haresources /etc/ha.d/
2.配置ha.cf
ha.cf文件详解请参考：http://blog.csdn.net/lgstudyvc/article/details/48341185
[root@master1 ha.d]# egrep -v "^$|#" ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
mcast eth0:2 225.0.0.1 694 1 0
auto_failback on
node master1
node master2
3.配置authkeys
[root@master1 ha.d]# cat authkeys
#
# Authentication file. Must be mode 600
#
#
# Must have exactly one auth directive at the front.
# auth send authentication using this method-id
#
# Then, list the method and key that go with that method-id
#
# Available methods: crc sha1, md5. Crc doesn't need/want a key.
#
# You normally only have one authentication method-id listed in this file
#
# Put more than one to make a smooth transition when changing auth
# methods and/or keys.
#
#
# sha1 is believed to be the "best", md5 next best.
#
# crc adds no security, except from packet corruption.
# Use only on physically secure networks.
#
auth 2
#1 crc
2 sha1 HI!
#3 md5 Hello!
[root@master1 ha.d]# chmod 600 authkeys #必须修改文件权限为600
4.配置haresources
master1 IPaddr::10.0.0.130/24/eth0:3 drbddisk::data Filesystem::/dev/drbd1::/data::ext3 mysqld
#添加主节点为master设置vip为10.0.0.130/24,网卡为eth0:3
#drbddisk挂在data目录
#fliesystem将drbd1目录挂在到/data目录下，并且启动mysql服务
DRBD安装部署
1.两台主库数据分区信息
数据分区信息
2.对/dev/sdb进行分区
[root@master1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x0e4d1c28.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130): +500M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (66-130, default 66):
Using default value 66
Last cylinder, +cylinders or +size{K,M,G} (66-130, default 130):
Using default value 130
Command (m for help): p
Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0e4d1c28
Device Boot Start End Blocks Id System
/dev/sdb1 1 65 522081 83 Linux
/dev/sdb2 66 130 522112+ 83 Linux
3.为新分区创建文件系统
注意：sdb2作为drbd的状态信息分区不需要格式化创建文件系统，所以只对sdb1进行格式化创建文件系统即可
[root@master1 ~]# mkfs.ext3 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
130560 inodes, 522080 blocks
26104 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
64 block groups
8192 blocks per group, 8192 fragments per group
2040 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@master1 ~]# tune2fs -c -1 /dev/sdb1 #只允许一个设备进行挂载
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
4.挂在/dev/sdb1分区到/data目录下，给后续mysql服务器存放data数据
[root@master1 ~]# mount /dev/sdb
sdb sdb1 sdb2
[root@master1 ~]# mount /dev/sdb1 /data/
[root@master1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_master1-lv_root
6.5G 2.2G 4.1G 35% /
tmpfs 491M 0 491M 0% /dev/shm
/dev/sda1 477M 28M 425M 7% /boot
/dev/sdb1 494M 11M 458M 3% /data
5.安装drbd内核模块
[root@master1 ~]# rpm -ivh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
[root@master1 ~]# yum install drbd84* kmod-drbd84
6.确认drbd模块已经加载到内核
[root@master1 ~]# modprobe drbd
[root@master1 ~]# lsmod | grep drbd
drbd 374888 0
libcrc32c 1246 1 drbd
7.查看drbd的配置文件
查看配置文件可知，需要单独创建响应的.res资源文件，且有相关的实例文件
[root@master1 etc]# cat drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
[root@master1 drbd.d]# cat /etc/drbd.d/mysql.res
global {
# minor-count 64;
# dialog-refresh 5; # 5 seconds
# disable-ip-verification;
usage-count no;
}
common {
protocol C;
disk {
on-io-error detach;
#size 454G;
no-disk-flushes;
no-md-flushes;
}
net {
sndbuf-size 512k;
# timeout 60; # 6 seconds (unit = 0.1 seconds)
# connect-int 10; # 10 seconds (unit = 1 second)
# ping-int 10; # 10 seconds (unit = 1 second)
# ping-timeout 5; # 500 ms (unit = 0.1 seconds)
max-buffers 8000;
unplug-watermark 1024;
max-epoch-size 8000;
# ko-count 4;
# allow-two-primaries;
cram-hmac-alg "sha1";
shared-secret "hdhwXes23sYEhart8t";
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
# data-integrity-alg "md5";
# no-tcp-cork;
}
syncer {
rate 120M;
al-extents 517;
}
}
resource data {
on master1 {
device /dev/drbd1;
disk /dev/sdb1;
address 10.0.2.110:7788;
meta-disk /dev/sdb2 [0];
}
on master2 {
device /dev/drbd1;
disk /dev/sdb1;
address 10.0.0.2.120:7788;
meta-disk /dev/sdb2 [0];
}
}
8.初始化meta分区
[root@master1 ~]# drbdadm create-md data
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
9.初始化设备同步(覆盖备节点，保持数据一致)
[root@master1 ~]# drbdadm -- --overwrite-data-of-peer primary data
10.启动drbd
[root@master1 ~]# drbdadm up all
[root@master1 ~]# chkconfig drbd off
11.挂在drbd分区到data目录下
[root@master1 ~]# drbdadm primary all
[root@master1 ~]# mount /dev/drbd1 /data
说明：/data目录为数据库的数据目录
安装MySQL数据库（两台主服务器上同时安装）
1.将mysql数据的/data文件夹建立在/dev/sdb1分区的data目录下
#到MySQL官网下载相应的mysql安装包，可以选择rpm，yum方式，此处选择二进制方式，方便定制安装路径
[root@master1 ~]# tar zxvf mysql-5.6.32-linux-glibc2.5-x86_64.tar.gz
#直接将解压后的mysql文件copy到安装目录即可
[root@master1 ~]# mv mysql-5.6.32-linux-glibc2.5-x86_64 /usr/local/mysql-5.6.32
#创建软连接否则后续利用二进制脚本启动过程中会保存，因为mysql.server脚本中引用的路径为/usr/local/mysql
[root@master1 ~]# ln -s /usr/local/mysql-5.6.32/ /usr/local/mysql
#使用mysql自带的mysql_install_db脚本进行初始化安装
[root@master1 scripts]# useradd -s /sbin/nologin -M mysql
[root@master1 scripts]# ./mysql_install_db -user=mysql --basedir=/usr/local/mysql-5.6.32/ --datadir=/data/
2.修改my.cnf配置文件
[root@master1 mysql-5.6.32]# vim my.cnf
[root@master1 mysql-5.6.32]# cat my.cnf
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html
[mysqld]
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
# These are commonly set, remove the # and set as required.
basedir = /usr/local/mysql-5.6.32/
datadir = /data
port = 3306
server_id = 110
socket = /tmp/mysql.sock
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
3.复制二进制mysql启动文件到/etc/init.d/下，方便heartbeat调用
[root@master1 support-files]# cp mysql.server /etc/init.d/mysqld
4.启动msyqld时出现报错请删除系统自带的/etc/my.cnf配置文件
[root@master1 ~]# /etc/init.d/mysqld start
Starting MySQL... ERROR! The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
[root@master1 ~]# rm -rf /etc/my.cnf
[root@master1 ~]# /etc/init.d/mysqld start
Starting MySQL. SUCCESS!
5.设置mysql数据库密码
#复制mysql安装文件下的bin文件到系统环境路径下
[root@master1 bin]# mysqladmin -uroot password
New password:
Confirm new password:
[root@master1 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.32 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+---------------------+
| Database |
+---------------------+
| information_schema |
| #mysql50#lost+found |
| mysql |
| performance_schema |
| test |
+---------------------+
5 rows in set (0.01 sec)
Heartbeat+Mysql+DRBD故障测试：
1.主备节点IP信息：
主节点IP：
[root@master1 /]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:8A:B3:73
inet addr:10.0.0.110 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe8a:b373/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4055 errors:0 dropped:0 overruns:0 frame:0
TX packets:3888 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:494029 (482.4 KiB) TX bytes:1890864 (1.8 MiB)
eth0:1 Link encap:Ethernet HWaddr 00:0C:29:8A:B3:73
inet addr:10.0.1.110 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:2 Link encap:Ethernet HWaddr 00:0C:29:8A:B3:73
inet addr:10.0.2.110 Bcast:10.0.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:3 Link encap:Ethernet HWaddr 00:0C:29:8A:B3:73
inet addr:10.0.0.130 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
备节点IP：
[root@master2 /]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:85:D5:3D
inet addr:10.0.0.120 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe85:d53d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:41412 errors:0 dropped:0 overruns:0 frame:0
TX packets:16511 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:47131293 (44.9 MiB) TX bytes:3064356 (2.9 MiB)
eth0:1 Link encap:Ethernet HWaddr 00:0C:29:85:D5:3D
inet addr:10.0.1.120 Bcast:10.0.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth0:2 Link encap:Ethernet HWaddr 00:0C:29:85:D5:3D
inet addr:10.0.2.120 Bcast:10.0.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
2.主备节点drbd信息
主节点：
[root@master1 /]# cat /proc/drbd
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by mockbuild@Build64R6, 2016-12-13 18:38:15
1: cs:Connected ro:Primary/Secondary ds:UpToDate/Diskless C r-----
ns:1326 nr:26 dw:594 dr:28007 al:16 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:522084
备节点：
[root@master2 /]# cat /proc/drbd
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by mockbuild@Build64R6, 2016-12-13 18:38:15
1: cs:Connected ro:Secondary/Primary ds:Diskless/UpToDate C r-----
ns:26 nr:1326 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
3.主备节点data目录信息
主节点：
[root@master2 data]# ll
total 110616
-rw-rw---- 1 mysql mysql 56 May 3 12:59 auto.cnf
-rw-rw---- 1 mysql mysql 12582912 May 3 14:08 ibdata1
-rw-rw---- 1 mysql mysql 50331648 May 3 14:08 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 May 3 12:58 ib_logfile1
-rw-r----- 1 mysql root 5849 May 3 14:08 master2.err
drwx------ 2 mysql mysql 4096 May 3 12:58 mysql
drwx------ 2 mysql mysql 4096 May 3 12:58 performance_schema
drwx------ 2 mysql mysql 4096 May 3 12:58 test
备节点：
[root@master1 data]# ll
total 111081
-rw-rw---- 1 mysql mysql 56 May 3 12:47 auto.cnf
-rw-rw---- 1 mysql mysql 12582912 May 3 14:18 ibdata1
-rw-rw---- 1 mysql mysql 50331648 May 3 14:18 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 May 3 2017 ib_logfile1
drwx------ 2 root root 12288 May 3 2017 lost+found
-rw-r----- 1 mysql root 24692 May 3 14:18 master1.err
-rw-rw---- 1 mysql mysql 5 May 3 14:18 master1.pid
-rw-r----- 1 500 root 4907 May 3 14:08 master2.err
drwx------ 2 mysql mysql 4096 May 3 2017 mysql
drwx------ 2 mysql mysql 4096 May 3 12:41 performance_schema
drwx------ 2 mysql mysql 1024 May 3 2017 test
