一、网络规划

• 所有ECS节点网段：172.16.0.0/16
• Service网段：10.96.0.0/16
• Pod网段：196.16.0.0/16

二、升级Linux内核

三、环境准备

1、 查看集群版本

cat /etc/redhat-release

2、修改域名

hostnamectl set-hostname k8s-xxx

3、集群规划

• k8s-master-01
• k8s-master-02
• k8s-master-03
• k8s-master-lb
• k8s-node-01
• k8s-node-02
  ...
• k8s-nodeN

4、每个机器准备域名

vim /etc/hosts

172.16.2.11     k8s-master-01
172.16.1.10     k8s-master-02
172.16.2.10     k8s-master-03
172.16.1.20     k8s-node-01
172.16.1.21     k8s-node-02
172.16.2.20     k8s-node-03
172.16.1.250    k8s-master-lb # 非高可用，可以不用这个。这个使用keepalive配置

5、关闭 selinux

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

6、关闭swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri 's/.*swap.*/#&/' /etc/fstab

7、修改limit

ulimit -SHn 65535
vi /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

8、为了方便以后操作配置ssh免密连接，master1运行

# 生成ssh密钥
ssh-keygen -t rsa
# 循环拷贝ssh密钥
for i in k8s-master-01 k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 k8s-node-03;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

image.png

9、安装yum

yum install wget git jq psmisc net-tools yum-utils device-mapper-persistent-data lvm2  -y

10、安装ipvs工具,方便以后操作ipvs，ipset，conntrack等

yum install ipvsadm ipset sysstat conntrack libseccomp -y

11、所有节点配置ipvs模块,执行以下命令，在内核4.19+版本改为nf_conntrack， 4.18下改为nf_conntrack_ipv4

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

12、修改ipvs配置，加入以下内容

vim /etc/modules-load.d/ipvs.conf

ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

13、执行命令

systemctl enable --now systemd-modules-load.service  #--now = enable+start

14、检测是否加载

lsmod | grep -e ip_vs -e nf_conntrack

15、设置ipv4

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
net.ipv4.conf.all.route_localnet = 1

vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16768
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16768
EOF
sysctl --system

16、所有节点配置完内核后，重启服务器，保证重启后内核依旧加载

reboot
lsmod | grep -e ip_vs -e nf_conntrack

四、安装Docker

1、修改docker配置,新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload && systemctl enable --now docker

2、离线安装

http://mirrors.aliyun.com/docker-ce/linux/centos/7.9/x86_64/stable/Packages/
yum localinstall xxxx