二进制安装-k8s高可用集群05-部署flannel网络
2021-08-26 本文已影响0人
Chris0Yang
flannel 第一次启动时，从 etcd 获取 Pod 网段信息，为本节点分配一个未使用的 /24 段地址，然后创建 flannel.1（也可能是其它名称，如 flannel1 等） 接口。
flannel 将分配的 Pod 网段信息写入 /run/flannel/docker 文件，docker 后续使用这个文件中的环境变量设置 docker0 网桥
1）下载和分发 flanneld 二进制文件
到 https://github.com/coreos/flannel/releases 页面下载最新版本的发布包
mkdir /data/flannel && cd /data/flannel
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C /data/flannel
分发 flanneld 二进制文件到集群所有节点：
cat > magic19_distribute_flanneld.sh << "EOF"
#!/bin/bash
# 分发 flanneld 二进制文件到集群所有节点
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp /data/flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/
ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF
2）创建 flannel 证书和私钥
cat > /data/cert/flanneld-csr.json <<EOF
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "4Paradigm"
}
]
}
EOF
- 该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；
生成证书和私钥：
[root@master cert]#
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
-ca-key=/etc/kubernetes/cert/ca-key.pem \
-config=/etc/kubernetes/cert/ca-config.json \
-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
[root@master cert]# ls flanneld*pem
将生成的证书和私钥分发到所有节点（master 和 worker）：
cat > magic20_distribute_cert_allnode.sh << "EOF"
#!/bin/bash
# 将生成的证书和私钥分发到所有节点（master 和 worker）
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld"
scp /data/cert/flanneld*.pem root@${node_ip}:/etc/flanneld/cert
done
EOF
3）向 etcd 写入集群 Pod 网段信息
注意：本步骤只需执行一次
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
输出结果
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}
- flanneld 当前版本 (v0.10.0) 不支持 etcd v3，故使用 etcd v2 API 写入配置 key 和网段数据；
- 写入的 Pod 网段 ${CLUSTER_CIDR} 必须是 /16 段地址，必须与 kube-controller-manager 的 --cluster-cidr 参数值一致；
4）创建 flanneld 的 systemd unit 文件
source /opt/k8s/bin/environment.sh
cat > /data/template/flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
-etcd-cafile=/etc/kubernetes/cert/ca.pem \\
-etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
-etcd-endpoints=${ETCD_ENDPOINTS} \\
-etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
-iface=${VIP_IF}
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
- mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件，后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥；
- flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口，如上面的 eth0 接口;
- flanneld 运行时需要 root 权限；
5）分发 flanneld systemd unit 文件到所有节点
cat > magic21_distribute_flanneld_systemd_unit.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp /data/template/flanneld.service root@${node_ip}:/etc/systemd/system/
done
EOF
6）启动 flanneld 服务
cat > magic22_start_flanneld_service.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl start flanneld"
done
EOF
7）检查启动结果
cat > magic23_verification_flanneld.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl status flanneld|grep Active"
done
EOF
看到如下输出：
bash magic23_verification_flanneld.sh
>>> 172.68.96.101
Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
>>> 172.68.96.102
Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
>>> 172.68.96.103
Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
则说明正常，如果失败，则用如下命令查看日志：
journalctl -ux flanneld
8）检查分配给各 flanneld 的 Pod 网段信息
查看集群 Pod 网段 (/16)：
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/config
输出结果
{"Network":"172.30.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}
查看已分配的 Pod 子网段列表 (/24):
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
ls ${FLANNEL_ETCD_PREFIX}/subnets
输出结果
/kubernetes/network/subnets/172.30.70.0-24
/kubernetes/network/subnets/172.30.35.0-24
/kubernetes/network/subnets/172.30.58.0-24
查看某一个 Pod 网段对应的节点 IP 和 flannel 接口地址
注意：其中的IP段换成自己的
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.35.0-24
{"PublicIP":"172.68.96.102","BackendType":"vxlan","BackendData":{"VtepMAC":"8e:9b:8c:9e:c4:e5"}}
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.70.0-24
{"PublicIP":"172.68.96.103","BackendType":"vxlan","BackendData":{"VtepMAC":"12:82:83:59:cf:b8"}}
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.58.0-24
{"PublicIP":"172.68.96.104","BackendType":"vxlan","BackendData":{"VtepMAC":"e6:b2:fd:f6:66:96"}}
9）验证各节点能通过 Pod 网段互通
在各节点上部署 flannel 后，检查是否创建了 flannel 接口 (名称可能为 flannel0、flannel.0、flannel.1 等)：
cat > magic24_verification_interflow_pod.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
done
EOF
输出结果：
bash magic24_verification_interflow_pod.sh
>>> 172.68.96.101
inet 172.30.70.0/32 scope global flannel.1
>>> 172.68.96.102
inet 172.30.35.0/32 scope global flannel.1
>>> 172.68.96.103
inet 172.30.58.0/32 scope global flannel.1
在各节点上 ping 所有 flannel 接口 IP，确保能通：
注意：其中的IP段换成自己的
cat > magic25_check_flannel_IP.sh << "EOF"
#!/bin/bash
# 检查各节点上是否能ping通，所有flannel接口IP
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "ping -c 1 172.30.35.0"
ssh ${node_ip} "ping -c 1 172.30.58.0"
ssh ${node_ip} "ping -c 1 172.30.70.0"
done
EOF
输出：
bash magic25_check_flannel_IP.sh
>>> 172.68.96.101
PING 172.30.35.0 (172.30.35.0) 56(84) bytes of data.
64 bytes from 172.30.35.0: icmp_seq=1 ttl=64 time=0.370 ms
--- 172.30.35.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.370/0.370/0.370/0.000 ms
PING 172.30.58.0 (172.30.58.0) 56(84) bytes of data.
64 bytes from 172.30.58.0: icmp_seq=1 ttl=64 time=0.318 ms
--- 172.30.58.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.318/0.318/0.318/0.000 ms
PING 172.30.70.0 (172.30.70.0) 56(84) bytes of data.
64 bytes from 172.30.70.0: icmp_seq=1 ttl=64 time=0.027 ms
--- 172.30.70.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.027/0.027/0.027/0.000 ms
..........
