k8sv1.18.2多master节点高可用集群搭建
虚拟机： VMware Workstation Pro 15
Linux 版本：CentOS Linux release 7.8.2003 (Core)
docker版本：19.03.11
kubectl版本: v1.18.2
操作系统：centos7.6以及更高版本都可以配置：4核cpu，4G内存，1块50G硬盘
==================================所有主机执行如下基础配置：============================================
systemctl stop firewalld && systemctl disable firewalld
yum -y install wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate
yum install iptables-services -y
service iptables stop && systemctl disable iptables
#时间同步
ntpdate cn.pool.ntp.org
1）crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
2）重启crond服务进程：
service crond restart
关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0 #关闭selinux
swapoff -a #关闭swap
关闭交换分区
swapoff -a# 永久禁用，打开/etc/fstab注释掉swap那一行。sed -i 's/.*swap.*/#&/' /etc/fstab
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
hostnamectl set-hostname master1
hostnamectl set-hostname master2
hostnamectl set-hostname master3
hostnamectl set-hostname node1
cat >> /etc/hosts << EOF
192.168.180.135 master1
192.168.180.136 master2
192.168.180.137 master3
192.168.180.138 node1
EOF
reboot -f
# 配置master1到node无密码登陆，配置master1到master2、master3无密码登陆
在master1上操作
ssh-keygen -t rsa
#一直回车就可以
ssh-copy-id -i .ssh/id_rsa.pub root@master2
#上面需要输入yes之后，输入密码，输入master2物理机密码即可
ssh-copy-id -i .ssh/id_rsa.pubroot@master3
#上面需要输入yes之后，输入密码，输入master3物理机密码即可
ssh-copy-id -i .ssh/id_rsa.pubroot@node1
#上面需要输入yes之后，输入密码，输入node1物理机密码即可
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 >/proc/sys/net/bridge/bridge-nf-call-ip6tables
echo """
vm.swappiness = 0
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
""" > /etc/sysctl.conf
sysctl -p
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in \${ipvs_modules}; do
/sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1 if [ $? -eq 0 ];
then /sbin/modprobe \${kernel_module} fidone
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
1.所有机器安装docker环境
===========================================安装docker===============================================
sudo yum update
curl -sSL https://get.docker.com/ | sh
sudo service docker start
docker run hello-world
docker version
yum -y install epel-release
yum -y install python-pip
sudo pip install --upgrade pip
pip -V
sudo pip install docker-compose
安装过程中如果出现如下错误：
Cannot uninstall ‘requests’. It is adistutils installed project and thus we cannot accurately determine which filesbelong to it which would lead to only a partial uninstall.
解决办法：pip install docker-compose--ignore-installed requests
sudo docker-compose --version
######## subprocess32报错解决方案
yum -y install gcc gcc-c++
yum install python-devel
sudo pip install subprocess32 -U
sudo pip install docker-compose
2.所有主机安装k8s环境(kubelet kubeadm kubectl)
===========================================安装k8s===============================================
2.1 按照顺序执行如下
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
指定版本方式(本次安装使用这种方式)
yum install -y kubelet-1.18.2 kubectl-1.18.2 kubeadm-1.18.2
或者可以选择安装最新版本方式
yum makecache fast
yum install -y kubelet kubeadm kubectl
# 跳过密钥安装如下
yum install kubectl-1.18.2-0.x86_64 --nogpgcheck
yum install kubelet-1.18.2-0.x86_64 --nogpgcheck
yum install kubeadm-1.18.2-0.x86_64 --nogpgcheck
systemctl enable kubelet.service
2.2 创建一个目录mkdir kubernetes上传all包文件 (官方下载的镜像)
执行如下命令安装：
docker load -i 1-18-kube-apiserver.tar.gz
docker load -i 1-18-kube-scheduler.tar.gz
docker load -i 1-18-kube-controller-manager.tar.gz
docker load -i 1-18-pause.tar.gz
docker load -i 1-18-cordns.tar.gz
docker load -i 1-18-etcd.tar.gz
docker load -i 1-18-kube-proxy.tar.gz
2.3 部署keepalive+lvs实现master节点高可用-对apiserver做高可用
2.3.1 部署keepalived+lvs，在各master节点操作
yum install -y socat keepalived ipvsadm conntrack
2.3.2 修改配置
<master 1 >节点修改之后的keepalived.conf如下所示:
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface ens33
virtual_router_id 80
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass just0kk
}
virtual_ipaddress {
192.168.180.200
}
}
virtual_server 192.168.180.200 6443 {
delay_loop 6
lb_algo loadbalance
lb_kind DR
net_mask 255.255.255.0
persistence_timeout 0
protocol TCP
real_server 192.168.180.135 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.180.136 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.180.137 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
<master 2 >节点修改之后的keepalived.conf如下所示:
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface ens33
virtual_router_id 80
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass just0kk
}
virtual_ipaddress {
192.168.180.200
}
}
virtual_server 192.168.180.200 6443 {
delay_loop 6
lb_algo loadbalance
lb_kind DR net_mask 255.255.255.0
persistence_timeout 0
protocol TCP
real_server 192.168.180.135 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.180.136 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.180.137 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
<master 3 >节点修改之后的keepalived.conf如下所示:
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
nopreempt
interface ens33
virtual_router_id 80
priority 30
advert_int 1
authentication {
auth_type PASS
auth_pass just0kk
}
virtual_ipaddress {
192.168.180.200
}
}
virtual_server 192.168.180.200 6443 {
delay_loop 6
lb_algo loadbalance
lb_kind DR
net_mask 255.255.255.0
persistence_timeout 0
protocol TCP
real_server 192.168.180.135 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.180.136 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.180.137 6443 {
weight 1
SSL_GET {
url {
path /healthz
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
2.3.3 提示：重要知识点，必看，否则生产会遇到巨大的坑
keepalive需要配置BACKUP，而且是非抢占模式nopreempt，假设master1宕机，启动之后vip不会自动漂移到master1，这样可以保证k8s集群始终处于正常状态，因为假设master1启动，apiserver等组件不会立刻运行，如果vip漂移到master1，那么整个集群就会挂掉，这就是为什么我们需要配置成非抢占模式了
2.3.4 启动顺序master1->master2->master3，在master1、master2、master3依次执行如下命令
systemctl enable keepalived && systemctl start keepalived && systemctl status keepalived
2.3.5 ip addr 查看ens33网卡下是否有192.168.180.200 如果只是master1有VIP 则正常 如果出现master节点都有的情况 大概率是vrrp组播问题
则需要执行如下处理：
yum install tcpdump
sudo tcpdump -i ens33 vrrp -n
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens33 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
#刷新防火墙
firewall-cmd --reload;
2.4 创建kubeadm-config.yaml 初始化文件 在master1上操作如下
cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.2
controlPlaneEndpoint: 192.168.180.200:6443
apiServer:
certSANs:
- 192.168.180.135
- 192.168.180.136
- 192.168.180.137
- 192.168.180.138
- 192.168.180.200
networking:
podSubnet: 10.244.0.0/16
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
初始化命令：
kubeadm init --config kubeadm-config.yaml
初始化命令执行成功之后显示如下内容，说明初始化成功了
在master1节点执行如下，这样才能有权限操作k8s资源
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes
master1节点是NotReady
kubectl get pods -n kube-system
cordns也是处于pending状态
安装网络插件解决该问题
docker load -i cni.tar.gz
docker load -i calico-node.tar.gz
可以访问下面的github地址，把下面的目录clone和下载下来，解压之后，在把文件传到master1节点即可
https://github.com/luckylucky421/kubernetes1.17.3/tree/master
GIT链接：https://github.com/luckylucky421/kubernetes1.17.3.git
kubectl apply -f calico.yaml
把master1节点的证书拷贝到master2和master3上
（1）在master2和master3上创建证书存放目录
cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/
（2）在master1节点把证书拷贝到master2和master3上，在master1上操作如下：
scp /etc/kubernetes/pki/ca.crt root@master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key root@master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key root@master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub root@master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt root@master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key root@master2:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt root@master2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key root@master2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/ca.crt master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key master3:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt master3:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key master3:/etc/kubernetes/pki/etcd/
在master2和master3上执行：
kubeadm join 192.168.180.200:6443 --token lazpeh.w7ofsulkv2ccsz52 \
--discovery-token-ca-cert-hash sha256:a5ac0c822dcac9f74339768a2173f0aac460dffd04364d4eaad70f77573d5d18 \
--control-plane
备注：--control-plane：这个参数表示加入到k8s集群的是master节点
在master2和master3上操作：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g)$HOME/.kube/config
在node1上执行：（不需要拷贝证书）
kubeadm join 192.168.180.200:6443 --token lazpeh.w7ofsulkv2ccsz52 \
--discovery-token-ca-cert-hash sha256:a5ac0c822dcac9f74339768a2173f0aac460dffd04364d4eaad70f77573d5d18
安装traefik
每个节点下执行
docker load -i traefik_1_7_9.tar.gz
1）生成traefik证书，在master1上操作
mkdir ~/ikube/tls/ -p
echo """
[req]
distinguished_name = req_distinguished_name
prompt = yes
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_value = CN
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_value = Beijing
localityName = Locality Name (eg, city)
localityName_value = Haidian
organizationName = Organization Name (eg, company)
organizationName_value = Channelsoft
organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_value = R & D Department
commonName = Common Name (eg, your name or your server\'s hostname)
commonName_value = *.multi.io
emailAddress = Email Address
emailAddress_value = lentil1016@gmail.com
""" > ~/ikube/tls/openssl.cnf
openssl req -newkey rsa:4096 -nodes -config ~/ikube/tls/openssl.cnf -days 3650 -x509 -out ~/ikube/tls/tls.crt -keyout ~/ikube/tls/tls.key
kubectl create -n kube-system secret tls ssl --cert ~/ikube/tls/tls.crt --key ~/ikube/tls/tls.key
2）执行yaml文件 github仓库下载
kubectl apply -f traefik.yaml
3）查看traefik是否部署成功：
kubectl get pods -n kube-system
安装kubernetes-dashboard 2.0版本（kubernetes的web ui界面）
每个节点下执行
docker load -i dashboard_2_0_0.tar.gz
docker load -i metrics-scrapter-1-0-1.tar.gz
只在master1节点操作
kubectl apply -f kubernetes-dashboard.yaml
查看dashboard是否安装成功：
kubectl get pods -n kubernetes-dashboard
查看dashboard前端的service
kubectl get svc -n kubernetes-dashboard
修改service type类型变成NodePort：
kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
把 type: ClusterIP变成 type: NodePort，保存退出即可
kubectl get svc -n kube-system
https://192.168.180.200:31102/
通过yaml文件里指定的默认的token登陆dashboard
1）查看kubernetes-dashboard名称空间下的secret
kubectl get secret -n kubernetes-dashboard
kubectl describe secret kubernetes-dashboard-token-lrpsj -n kubernetes-dashboard
创建管理员token，可查看任何空间权限
kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:kubernetes-dashboard
安装metrics监控相关的插件
docker load -i metrics-server-amd64_0_3_1.tar.gz
docker load -i addon.tar.gz
只需要在master1上执行
kubectl apply -f metrics.yaml
