使用二进制方式搭建Kubernetes集群
2021-02-13 本文已影响0人
BitInterfc
相比于Kubeadmin，使用二进制的方式会费劲很多，但是，我们需要依次搭建etcd，API Server， Kubelet， Kube-Proxy，这对于我们理解K8s的架构，大有裨益
一、准备工作
和使用Kubeadmin 一样，每台环境都需要进行如下操作
1、准备三台虚拟机,需要提前配置好hostname
[root@vitellin1 ~]# cat /etc/hosts
#127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.11.96.168 vitellin1
10.11.96.199 vitellin2
10.11.97.23 vitellin3
2 禁用防火墙
systemctl stop firewalld
systemctl disable firewalld
3 禁用SELinux
修改/etc/selinux/config, 设置SELINUX=disabled. 重启机器.
[root@vitellin1 ~]# sestatus
SELinux status: disabled
4 禁用交换分区
编辑/etc/fstab, 将swap注释掉(最后一行). 重启机器.
[root@vitellin1 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Apr 6 15:18:09 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
## Please take a snapshot of your VM before modifying this file. Modify this file incorrectly most likely will corrupt your system or stop your system from booting up
/dev/mapper/rhel-root / xfs defaults 0 0
UUID=4f3976c1-1696-4787-8618-f52bb1c0c86a /boot xfs defaults 0 0
#/dev/mapper/rhel-swap swap swap defaults 0 0
5 修改网络配置
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
二、部署etcd集群
1 拷贝并解压 TLS.tar.gz
scp TLS.tar.gz root@vitellin1.fyre.ibm.com:/root/.
[root@vitellin1 TLS]# ls
cfssl cfssl-certinfo cfssl.sh cfssljson etcd k8s
2 运行 cfssl.sh
[root@vitellin1 TLS]# cat cfssl.sh
#curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
#curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
#curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
cp -rf cfssl cfssl-certinfo cfssljson /usr/local/bin
chmod +x /usr/local/bin/cfssl*
./cfssl.sh
3 进入TLS/etcd文件夹，依次执行 generate_etcd_cert.sh 里边的每个命令
[root@vitellin1 etcd]# ls
ca-config.json ca-key.pem ca.pem server-csr.json server.csr
ca-csr.json ca.csr generate_etcd_cert.sh server-key.pem server.pem
[root@vitellin1 etcd]# cat generate_etcd_cert.sh
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
3.1 生成ca.pem, ca-config.json 和 server-csr.json
[root@vitellin1 etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/02/12 20:09:06 [INFO] generating a new CA key and certificate from CSR
2021/02/12 20:09:06 [INFO] generate received request
2021/02/12 20:09:06 [INFO] received CSR
2021/02/12 20:09:06 [INFO] generating key: rsa-2048
2021/02/12 20:09:07 [INFO] encoded CSR
2021/02/12 20:09:07 [INFO] signed certificate with serial number 24238789529817309110953484382237664561540666164
3.2 修改 server-csr.json
[root@vitellin1 etcd]# cat server-csr.json
{
"CN": "etcd",
"hosts": [
"10.11.96.168", //master node
"10.11.96.199", //worker node 1
"10.11.97.23" //worker node 2
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
3.3 生成各种key.pem
[root@vitellin1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/02/12 20:11:04 [INFO] generate received request
2021/02/12 20:11:04 [INFO] received CSR
2021/02/12 20:11:04 [INFO] generating key: rsa-2048
2021/02/12 20:11:04 [INFO] encoded CSR
2021/02/12 20:11:04 [INFO] signed certificate with serial number 343880566118958193345285469553913608046569564825
2021/02/12 20:11:04 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@vitellin1 etcd]# ls *.pem
ca-key.pem ca.pem server-key.pem server.pem
4 拷贝并解压 etcd.tar.gz
scp etcd.tar.gz root@vitellin1.fyre.ibm.com:/root/.
[root@vitellin1 ~]# ls | grep etcd
etcd
etcd.service
[root@vitellin1 ~]# cat etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
5 进入etcd 文件夹
5.1 更新 cfg/etcd.conf
[root@vitellin1 cfg]# cat etcd.conf
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.11.96.168:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.11.96.168:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.11.96.168:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.11.96.168:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.11.96.168:2380,etcd-2=https://10.11.96.199:2380,etcd-3=https://10.11.97.23:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
5.2 更新ssl文件夹里边的证书
cd ssl
rm -rf *
cp ~/TLS/etcd/{ca,server,server-key}.pem .
- 拷贝文件到相应的文件夹
cp -r etcd /opt/.
cp etcd.service /usr/lib/systemd/system/.
scp -r etcd/ root@10.11.96.199:/opt/.
scp etcd.service root@10.11.96.199:/usr/lib/systemd/system/.
scp -r etcd/ root@10.11.97.23:/opt/.
scp etcd.service root@10.11.97.23:/usr/lib/systemd/system/.
注意：对于etcd.conf, 每个node节点需要修改相应的ETCD_NAME 和内部address
7 对于每个节点，启动etcd service
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
systemctl status etcd
[root@vitellin3 cfg]# systemctl status etcd
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2021-02-12 06:24:31 PST; 50s ago
Main PID: 29937 (etcd)
CGroup: /system.slice/etcd.service
└─29937 /opt/etcd/bin/etcd --name=etcd-3 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.11.97.23:23...
三、为API Server添加自签证书
以下操作更换了一套VM设备
1、修改server-csr.json
[root@antonymy1 k8s]# cat server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.11.66.181",
"10.11.66.192",
"10.11.67.77"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
2 执行 generate_k8s_cert.sh
[root@antonymy1 k8s]# ./generate_k8s_cert.sh
2021/02/12 20:31:13 [INFO] generating a new CA key and certificate from CSR
2021/02/12 20:31:13 [INFO] generate received request
2021/02/12 20:31:13 [INFO] received CSR
2021/02/12 20:31:13 [INFO] generating key: rsa-2048
2021/02/12 20:31:14 [INFO] encoded CSR
2021/02/12 20:31:14 [INFO] signed certificate with serial number 491229188461810525319895221992191303771907510087
2021/02/12 20:31:14 [INFO] generate received request
2021/02/12 20:31:14 [INFO] received CSR
2021/02/12 20:31:14 [INFO] generating key: rsa-2048
2021/02/12 20:31:15 [INFO] encoded CSR
2021/02/12 20:31:15 [INFO] signed certificate with serial number 73476638067131475182674490991671886698367945401
2021/02/12 20:31:15 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2021/02/12 20:31:15 [INFO] generate received request
2021/02/12 20:31:15 [INFO] received CSR
2021/02/12 20:31:15 [INFO] generating key: rsa-2048
2021/02/12 20:31:15 [INFO] encoded CSR
2021/02/12 20:31:15 [INFO] signed certificate with serial number 166731788942921905349358753121498446060396384435
2021/02/12 20:31:15 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
结果，产生各种pem文件：
[root@antonymy1 k8s]# ls *.pem
ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem server-key.pem server.pem
四、部署Master组件
1 部署 ApiServer
2 部署Controller-manager
3 Scheduler
五、部署Node组件
1 Docker
2 Kubelet
3 KubeProxy
4 批注Kubelet证书申请加入集群
kubectl get csr
六、部署CNI网络
