kubernetes高可用用kubeadm方式部署实战

2019-11-22 本文已影响0人每天进步一典

简介

使用kubeadm方式部署kubernetes HA

架构信息

系统版本：CentOS 7
内核：3.10.0-957.el7.x86_64
Kubernetes: v1.16.0
Docker-ce: 18.06
推荐硬件配置：2核4G

Keepalived保证apiserever服务器的IP高可用
Haproxy实现apiserver的负载均衡
为了减少服务器数量，haproxy、keepalived配置在master-01和master-02。

节点名称角色 IP 安装软件
负载VIP VIP 10.1.1.16

节点	角色	IP地址	kubeadm、kubelet、kubectl、docker、haproxy、keepalived
master-01	master	10.1.1.10	kubeadm、kubelet、kubectl、docker、haproxy、keepalived
master-02	master	10.1.1.3	kubeadm、kubelet、kubectl、docker、haproxy、keepalived
master-03	master	10.1.1.4	kubeadm、kubelet、kubectl、docker、haproxy、keepalived
node-01	node	10.1.1.5	kubeadm、kubelet、kubectl、docker
Pod网段		10.224.0.0/16
service网段		10.96.0/16

部署前准备工作

关闭selinux和防火墙

sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0
systemctl disable firewalld
systemctl stop firewalld

关闭swap

swapoff –a
/etc/fstab挂载的也要注释。

为每台服务器添加host解析记录

cat >> /etc/hosts <<EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.1.10 master-01
10.1.1.3 master-02
10.1.1.4 node-01
10.1.1.5 master-03
EOF

创建并分发密钥
在master-01创建ssh密钥。

root@master-01 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:+QAoEdGjIo9NWonLEzM9LMEQjposaeuB9Oc+ShJUXNk root@master-01
The key's randomart image is:
+---[RSA 2048]----+
|=.==..o          |
|oo.ooo E         |
|.+=o...          |
|BB==   . .       |
|B#= .   S        |
|Oo*      o       |
|.+.o .    .      |
|. + o.           |
| . .oo.          |
+----[SHA256]-----+
[root@master-01 ~]# ls .ssh/
id_rsa  id_rsa.pub  known_hosts
分发到其他服务器

[root@master-01 ~]# for n in `seq -w 01 02 03`;do ssh-copy-id -p43999 master-$n;done

[root@master-01 ~]# for n in `seq -w 01`;do ssh-copy-id node-$n;done

部署keepalived和haproxy

在master-01和master-02安装keepalived和haproxy

yum install -y keepalived haproxy

修改配置
keepalived配置
master-01的priority为100，master-02的priority为90，其他配置一样。


! Configuration File for keepalived

global_defs {
   notification_email {
        mrli@163.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_1
}

vrrp_instance VI_1 {
    state MASTER          
    interface ens33
    lvs_sync_daemon_inteface ens33     #按实际网口修改
    virtual_router_id 88
    advert_int 1
    priority 100         
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
      10.1.1.16/24
    }
}

[root@master-01 k8s]# cat  <<EOF  >  /etc/keepalived/keepalived.conf 

! Configuration File for keepalived

global_defs {
   notification_email {
        mrli@163.com
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_1
}

vrrp_instance VI_1 {
    state MASTER          
    interface ens33
    lvs_sync_daemon_inteface ens33
    virtual_router_id 88
    advert_int 1
    priority 100         
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
      10.1.1.16/24
    }
}
EOF

haproxy配置
master-01和master-02的haproxy配置是一样的。因为haproxy是和k8s apiserver是部署在同一台服务器上，都用6443会冲突，此处我们监听的是10.1.1.16的8443端口。

[root@master-01 k8s]# cat  <<EOF > /etc/haproxy/haproxy.cfg

global
        chroot  /var/lib/haproxy
        daemon
        group haproxy
        user haproxy
        log 127.0.0.1:514 local0 warning
        pidfile /var/lib/haproxy.pid
        maxconn 20000
        spread-checks 3
        nbproc 8

defaults
        log     global
        mode    tcp
        retries 3
        option redispatch

listen https-apiserver
        bind 10.1.1.16:8443
        mode tcp
        balance roundrobin
        timeout server 900s
        timeout connect 15s
      server apiserver01 10.1.1.10:6443 check port 6443 inter 5000 fall 5
        server apiserver02 10.1.1.3:6443 check port 6443 inter 5000 fall 5
EOF

在master-01和master-02上启动服务

systemctl enable keepalived && systemctl start keepalived 
systemctl enable haproxy && systemctl start haproxy

部署kubernetes

配置yum安装源

cat <<EOF  > /etc/yum.repos.d/kubernetes.repo 
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

1、安装相关软件

yum install -y kubelet-1.16.0 kubeadm-1.16.0 kubectl-1.16.0 ipvsadm ipset docker-ce-18.06.1.ce

2、启动相关服务

systemctl enable kubelet

3、创建初始化配置文件


[root@master-01 k8s]# cat  <<EOF > kubeadm-config.yaml 
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
controlPlaneEndpoint: "10.1.1.16:6443"
imageRepository: "registry.aliyuncs.com/google_containers"
networking:
  podSubnet: "10.244.0.0/16"
apiServer:
  certSANs:
  - "k8s.mytest.com"
EOF

注意：podSubet应该和后面的fannel网络保持一致，否则创建fannel会失败

4、预下载镜像

[root@master-01 k8s]# kubeadm config images pull --config kubeadm-init.yaml

5、初始化master-01

[root@master-01 k8s]# kubeadm init --config kubeadm-init.yaml

6、根据提示准备kubeconfig

mkdir -p \$HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf \$HOME/.kube/config
sudo chown \$(id -u):\$(id -g) \$HOME/.kube/config

7、其他master加入到集群方法（根据上一步提示获得）
复制master-01的证书到其他master节点

[root@master-01 k8s]# cat base/cp-cert.sh 
#!/bin/bash
##拷贝证书到其他master节点
USER=root
CONTROL_PLANE_IPS="master01 master02 master03"
for host in $CONTROL_PLANE_IPS; do
    ssh -p43999 "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
    scp -P43999 /etc/kubernetes/pki/ca.* "${USER}"@$host:/etc/kubernetes/pki/
    scp -P43999 /etc/kubernetes/pki/sa.* "${USER}"@$host:/etc/kubernetes/pki/
    scp -P43999 /etc/kubernetes/pki/front-proxy-ca.* "${USER}"@$host:/etc/kubernetes/pki/
    scp -P43999 /etc/kubernetes/pki/etcd/ca.* "${USER}"@$host:/etc/kubernetes/pki/etcd/

复制完成在其他master上加入集群的方法

kubeadm join 10.1.1.16:6443 --token sigq9b.m6lwwk40n1piqfbh \
    --discovery-token-ca-cert-hash sha256:9c3602a14cf4717202acf6ae004629188551a35603007c8bd1fc3dfc7f19061b \
--control-plane --certificate-key b0be6c57cb55983dabf8df1a94d4d32eeb2c7efd1966ac5b37ae515f8a0a99c9

8、其他node加入集群方法（根据上一步提示获得）

kubeadm join 10.1.1.16:6443 --token sigq9b.m6lwwk40n1piqfbh \
--discovery-token-ca-cert-hash sha256:9c3602a14cf4717202acf6ae004629188551a35603007c8bd1fc3dfc7f19061b

9、创建fannel网络
脚本yaml如下(可在官网下载)

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

[root@master-01 k8s]# kubectl apply -f kube-flannel.yml

10、检查组件状态

[root@master-01 k8s]# kubectl get pod -n kube-system
NAME                                READY   STATUS    RESTARTS   AGE
coredns-58cc8c89f4-24s7h            1/1     Running   0          6h28m
coredns-58cc8c89f4-d47h4            1/1     Running   0          6h28m
etcd-master-01                      1/1     Running   0          6h27m
etcd-master-02                      1/1     Running   2          6h19m
etcd-master-03                       1/1     Running   0          6h18m
kube-apiserver-master-01            1/1     Running   1          6h27m
kube-apiserver-master-02            1/1     Running   4          6h19m
kube-apiserver-master-03              1/1     Running   1          6h18m
kube-controller-manager-master-01   1/1     Running   3          6h27m
kube-controller-manager-master-02   1/1     Running   2          6h19m
kube-controller-manager-master-03    1/1     Running   3          6h17m
kube-flannel-ds-amd64-6pddd         1/1     Running   0          6h22m
kube-flannel-ds-amd64-72dp8         1/1     Running   0          6h18m
kube-flannel-ds-amd64-bn5th         1/1     Running   0          6h19m
kube-flannel-ds-amd64-lt7jr         1/1     Running   0          5h58m
kube-proxy-75vdx                    1/1     Running   0          6h19m
kube-proxy-8lptw                    1/1     Running   0          6h18m
kube-proxy-bz9lb                    1/1     Running   0          5h58m
kube-proxy-ffgnk                    1/1     Running   0          6h28m
kube-scheduler-master-01            1/1     Running   1          6h27m
kube-scheduler-master-02            1/1     Running   3          6h19m
kube-scheduler-master-03              1/1     Running   3          6h17m

都为runnning表示正常
11、检查集群状态

[root@master-01 k8s]# kubectl get  node
NAME        STATUS   ROLES    AGE     VERSION
master-01   Ready    master   6h30m   v1.16.0
master-02   Ready    master   6h20m   v1.16.0
node-01    Ready    node   6h19m   v1.16.2
master-03       Ready    master   6h19m   v1.16.2

12、验证高可用

在master-01上创建一个简单的pod

[root@master-01 k8s]# cat test.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80

查看状态

[root@master-01 k8s]# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
myapp-deploy-798dc9b584-fdmlp   1/1     Running   0          102m
myapp-deploy-798dc9b584-fq546   1/1     Running   0          102m

当前的vip在master-01上