基于k8s+kubedb，搭建高可用mysql-replicat
2020-10-21 本文已影响0人
喝不醉的Solace
kubedb 是一个快速搭建高可用存储组件的框架，其内部定义了许多高级Custom Resource Definitions（CRD），它以Kubernetes本机方式为MySQL提供了声明式配置。您只需要在MySQL对象中描述所需的数据库配置，KubeDB Operator将为你创建处于所需状态的Kubernetes对象。
mysql-group-replication
1.环境及版本
1.kubernetes: 1.19.1 集群
2.helm：3.3.1
3.kubedb: 0.12.0
2.安装kubedb
参考：https://kubedb.com/docs/0.12.0/setup/install/
3 开始部署
3.1创建storageclass
- 创建storageclass方式很多，可以根据场景具体选择，这里我们选用基于nfs的storageclass
nfs搭建方法参考百度，这里不再说明 - nfs服务器搭建好后，使用helm安装动态Pv provisioner（根据pvc template 动态创建pv）
- 提前创建你的namespace,这里我提前创建为midware
$ helm repo add stable https://charts.helm.sh/stable
$ helm install --set nfs.server=10.20.128.217 --set nfs.path=/usr/local/nfs/storage -n midware nfs-common stable/nfs-client-provisioner
[root@k8s-master storage]# helm ls -n midware
nfs-common midware 1 2020-10-20 15:44:47.129746856 +0800 CST deployed nfs-client-provisioner-1.2.9 3.1.0
[root@k8s-master mysql]# kubectl get sc -n midware
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage fuseim.pri/ifs Delete Immediate false 7d23h
nfs-client cluster.local/nfs-common-nfs-client-provisioner Delete Immediate true
3.2创建config-map
如果你不需要定制mysql.cnf中的某些属性，这一步可忽略
apiVersion: v1
kind: ConfigMap
metadata:
name: my-custom-config
namespace: midware
data:
my-config.cnf: |
[mysqld]
max_connections = 2048
read_buffer_size = 4194304
skip-name-resolve
innodb_lru_scan_depth = 256
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
3.3 创建存储mysql账号密码的secret
注意，这里需要提前进行base64编码
apiVersion: v1
data:
password: MXFhekNERSM1dGdi
user: cm9vdA==
kind: Secret
metadata:
name: m1-auth
namespace: midware
type: Opaque
3.4 创建mysql版本声明的CRD
cat mysql-v.yaml
apiVersion: catalog.kubedb.com/v1alpha1
kind: MySQLVersion
metadata:
name: "5.7.25"
namespace: midware
labels:
app: kubedb
spec:
version: "5.7.25"
db:
image: "kubedb/mysql:5.7.25"
exporter:
image: "kubedb/mysqld-exporter:v0.11.0"
tools:
image: "kubedb/mysql-tools:5.7.25"
podSecurityPolicies:
databasePolicyName: "mysql-db"
snapshotterPolicyName: "mysql-snapshot"
initContainer:
image: "kubedb/mysql-tools:5.7.25"
kubectl create -f mysql-v.yaml
3.5 创建mysql集群
apiVersion: kubedb.com/v1alpha1
kind: MySQL
metadata:
name: mysql
namespace: midware
spec:
version: "5.7.25"
replicas: 3
#databaseSecret:
# secretName: m1-auth
topology:
mode: GroupReplication
group:
name: "dc002fc3-c412-4d18-b1d4-66c1fbfbbc9b"
baseServerID: 100
storageType: Durable
podTemplate:
spec:
livenessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 15
exec:
command:
- "bash"
- "-c"
- 'mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h localhost -nsLNE -e "select member_state from performance_schema.replication_group_members where member_id=@@server_uuid;" 2>/dev/null | grep -v "*" | egrep -v "ERROR|OFFLINE"'
readinessProbe:
initialDelaySeconds: 120
timeoutSeconds: 5
periodSeconds: 15
exec:
command:
- "bash"
- "-c"
- 'mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h localhost -nsLNE -e "select member_state from performance_schema.replication_group_members where member_id=@@server_uuid;" 2>/dev/null | grep -v "*" | egrep -v "ERROR|OFFLINE"'
configSource:
configMap:
name: my-custom-config
storage:
storageClassName: "nfs-client"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
terminationPolicy: WipeOut
updateStrategy:
type: RollingUpdate
测试
创建时间可能有多久，目前kubedb集群只支持到了mysql:5.7.25这个版本
创建完之后
[root@k8s-master storage]# kubectl get statefulset -n midware
NAME READY AGE
mysql 3/3 56m
[root@k8s-master storage]# kubectl get pvc -n midware
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-mysql-0 Bound pvc-29383e83-a72f-4f60-a1a1-e023da5af19c 10Gi RWO nfs-client 57m
data-mysql-1 Bound pvc-e44bc02f-53aa-4273-9720-e2b796b6b591 10Gi RWO nfs-client 54m
data-mysql-2 Bound pvc-54b98831-da8f-43bc-8551-883133144da2 10Gi RWO nfs-client 51m
可以看到，对应的pvc也被相应的创建好，因为我们上面注释掉了databaseSecret,所以这里使用的是默认的 mysql-auth /(mysql object name)-auth
#获取密码
kubectl get secrets -n midware mysql-auth -o jsonpath='{.data.\password}' | base64 -d
#查看集群主节点ID
kubectl exec -it -n midware mysql-0 -- mysql -u root --password=K-JjDOGgBuiwLI3C --host=mysql-0.mysql-gvr.midware -e "show status like '%primary%'"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------------------------+--------------------------------------+
| Variable_name | Value |
+----------------------------------+--------------------------------------+
| group_replication_primary_member | e6f40b38-137f-11eb-ba96-fa46838d1905 |
+----------------------------------+--------------------------------------+
#查看集群节点
[root@k8s-master storage]# kubectl exec -it -n midware mysql-0 -- mysql -u root --password=K-JjDOGgBuiwLI3C --host=mysql-0.mysql-gvr.midware -e "use performance_schema;select * from replication_group_members"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------------------------+--------------------------------------+---------------------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+---------------------------+-------------+--------------+
| group_replication_applier | 551265bc-1380-11eb-90e0-fe12a187e6b2 | mysql-1.mysql-gvr.midware | 3306 | ONLINE |
| group_replication_applier | a5dd7243-1380-11eb-8f54-d61d5a9c417d | mysql-2.mysql-gvr.midware | 3306 | ONLINE |
| group_replication_applier | e6f40b38-137f-11eb-ba96-fa46838d1905 | mysql-0.mysql-gvr.midware | 3306 | ONLINE |
+---------------------------+--------------------------------------+---------------------------+-------------+--------------+
到这里，高可用集群就创建完成了.
常见问题
- 相对官网来说，上述yaml解决了官网例子，集群还没启动完检查就失败了，导致k8s杀掉mysql无限重启
- kubedb目前开源版本个人感觉不够成熟，PS（0.12.0就出商业版了，github一堆issue，回复较少）
- 其他问题留言吧
