kubernetes安装
K8s安装
主机部署组件说明（根据文档架构）：
master-1主机对应的是ETCE_NAME="etcd01" IP为192.168.174.30 安装组件包括 kube-apiserver, kube-controller-manager, kube-scheduler, etcd
master-2主机对应的是ETCE_NAME="etcd02" IP为192.168.174.31 安装组件包括 kube-apiserver, kube-controller-manager, kube-scheduler, etcd
node-1主机对应的是ETCE_NAME="etcd03" IP为192.168.174.40 安装组件包括 kubelet, kube-proxy, docker, flannel, etcd
node-1主机对应的IP为192.168.174.41 安装组件包括 kubelet, kube-proxy, docker, flannel
master组件作用
kube-apiserver：
kube-apiserver对外暴露了Kubernetes API。它是的 Kubernetes 前端控制层。它被设计为水平扩展，即通过部署更多实例来缩放
kube-cotroller-manager：
kube-controller-manager运行控制器，它们是处理集群中常规任务的后台线程。逻辑上，每个控制器是一个单独的进程，但为了降低复杂性，它们都被编译成独立的可执行文件，并在单个进程中运行。
etcd：
etc用于kubernetes的后端存储。所有集群数据都存储在此处，始终为kubernetes集群的etcd数据提供备份计划。
kube-scheduler：
kube-scheduler监视没有分配节点的新创建的pod，选择一个节点供他们运行。
node组件作用：
kubelet：
kubelet是主要的节点代理，它监测已分配给其节点的Pod（通过apiserver或通过本地配置文件），提供如下功能：
* 挂载Pod所需要的数据卷（volume）
* 下载Pod的secrets
* 通过Docker运行（或通过rkt）运行Pod的容器
* 周期性的对容器生命周期进行探测
* 如果需要，通过创建镜像Pod（Mirror Pod）将Pod的状态报告回系统的其余部分
* 将节点的状态报告会系统的其余部分
kube-proxy：
kube-proxy通过维护主机上的网络规则并执行连接转发，实现了kubernetes服务抽象
flannel：
flannel网络插件，目前支持UDP、VxLAN、AWS VPC和GCE路由等数据转发方式
工作原理是，数据源容器中发出后，经由所在主机的docker0虚拟网卡转发到flannel0虚拟网卡，node间的flannel0虚拟网卡互为网关，所以当node1上的pod访问node2上的pod时就可以通信了。如果两台node其中一台无法访问公网一台可访问公网，那么无法访问公网的会通过可以访问公网的node而访问公网。
laster负载均衡 10.206.176.19对应192.168.176.19 组件 LVS
1、使用cfssl生成自签证书所需文件已下载到目录中，如需重复下载可用浏览器打开下面链接，或直接wget
https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
#chmod +x cfssl*
#mv cfssl_linux-amd64 /usr/local/bin/cfssl
#mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
#mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
2、创建CA证书配置，生成CA证书和密钥
etcd和kubernetes都需要生成证书
etcd证书存放目录：mkdir -p /etc/etcd/ssl
kubernetes证书存放目录：mkdir -p /etc/kubernetes/ssl
创建证书时临时存放：mkdir /root/ssl
文件有格式要求，首先我们先生成默认文件，然后根据config.json文件的格式创建ca-config.json文件，过期时间设置为87600h。
创建CA配置文件
#cd /root/ssl
#cfssl print-defaults config > config.json
#vim config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
#mv config.json ca-config.json
ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示client可以用该 CA 对server提供的证书进行验证；
client auth：表示server可以用该CA对client提供的证书进行验证；
创建CA证书签名请求
#cfssl print-defaults csr > csr.json
#vim csr.json
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
#mv csr.json ca-csr.json
“CN”：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；
“O”：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；
生成 CA 证书和私钥
#cfssl gencert -initca ca-csr.json | cfssljson -bare ca
创建kubernetes证书
创建kubernetes证书签名请求
#vim server-csr.json
{
"CN": "etcd",
"hosts": [
"192.168.174.30",
"192.168.174.31",
"192.168.174.40"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
生成 kubernetes 证书和私钥
#cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2019/03/01 10:55:26 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
上面的警告忽略
一、安装Etcd
以下配置需要在三台服务器上配置，参考excel文档中的部署架构
这里对应的主机分别是：
master-1主机对应的是ETCE_NAME="etcd01" IP为192.168.174.30
master-2主机对应的是ETCE_NAME="etcd02" IP为192.168.174.31
node-1主机对应的是ETCE_NAME="etcd03" IP为192.168.174.40
除了配置文件需要三台不同，其余操作一致。
1、二进制包下载位置
https://github.com/etcd-io/etcd/releases/tag/v3.2.12
下载etcd-v3.2.12-linux-amd64.tar.gz
#mkdir /opt/etcd/{bin,cfg,ssl} -p
#tar zxvf etcd-v3.2.12-linux-amd64.tar.gz
#mv etcd-v3.2.12-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
2、创建etcd配置文件
#cd /opt/etcd/cfg/
#vim etcd
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.174.31:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.174.31:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.174.31:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.174.31:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.174.31:2380,etcd02=https://192.168.174.30:2380,etcd03=https://192.168.174.40:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
配置文件注解
ETCD_NAME="etcd01" #节点名称
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #数据目录
ETCD_ISTEN_PEER_URLS="https://192.168.174.31:2380" #集群通信监听地址 ，IP为自己服务器IP
ETCD_ISTEN_CLIENT_URLS="https://192.168.174.31:2379" #客户端访问监听地址，IP为自己服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.174.31:2380" #集群通告地址，IP为自己服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.174.31:2379" #客户端通告地址，IP为自己服务器IP
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.174.31:2380,etcd02=https://192.168.174.30:2380,etcd03=https://192.168.174.40:2380" #集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #集群Token
ETCD_INITIAL_CLUSTER_STATE="new" #加入集群的当前状态，new是新集群，existing表示加入已有集群
systemctl管理etcd
#vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=ontify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
把证书拷贝到配置文件中的位置
#cp ca*pem server*pem /opt/etcd/ssl/
关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service #禁止firewall开机启动
setenforce 0
启动etcd
systemctl start etcd
systemctl enable etcd
检查etcd集群状态
#/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.174.31:2379,https://192.168.174.30:2379,https://192.168.174.40:2379" cluster-health
输出以下内容即为成功：
luster-health
member 21cdde3b45cde04f is healthy: got healthy result from https://192.168.174.40:2379
member ca2bc200e194cb1c is healthy: got healthy result from https://192.168.174.30:2379
member dd8d169df81310cc is healthy: got healthy result from https://192.168.174.31:2379
日志查看
/var/log/message
或
journalctl -u etcd
二、在Node节点安装Docker
#yum -y install yum-utils device-mapper-persistent-data lvm2
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#yum install docker-ce -y
#curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://bc437cce.m.daocloud.io
#systemctl start docker
#systemctl enable docker
三、部署Flannel网络
Falnnel要用etcd存储自身一个子网信息，所以要保证能成功连接Etcd，写入预定义字网段
在master-1上执行：
#/opt/etcd/bin/etcdctl \
--ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem \
--endpoints="https://192.168.174.30:2379,https://192.168.174.31:2379,htts://192.168.174.40:2379" \
set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
已下步骤在每个node节点上都操作
1、下载二进制包
#wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
#tar -zxvf flannel-v0.10.0-linux-amd64.tar.gz
#mkdir -p /opt/kubernetes/bin
#mv flanneld mk-docker-opts.sh /opt/kubernetes/bin/
把证书拷贝到配置文件中的位置
#cp ca*pem server*pem /opt/etcd/ssl/ （所说的证书都是master-01上生成的证书）
2、配置Flannel
#mkdir /opt/kubernetes/cfg
#vim /opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.174.30:2379,https://192.168.174.31:2379,https://192.168.174.40:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"
3、systemd管理Flannel
#vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
4、配置docker启动指定子网段
#cp /usr/lib/systemd/system/docker.service /usr/lib/systemd/system/docker.service.bak
#vim /usr/lib/systemd/system/docker.service
#cat /usr/lib/systemd/system/docker.service |egrep -v '^#|^$'
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutStartSec=0
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
5、重启Flannel和docker
#systemctl daemon-reload
#systemctl start flanneld
#systemctl enable flanneld
#systemctl restart docker
6、检查是否生效
#ps -ef |grep docker
root 3761 1 1 15:27 ? 00:00:00 /usr/bin/dockerd --bip=172.17.44.1/24 --ip-masq=false --mtu=1450
#ip addr
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:3a:13:4d:7f brd ff:ff:ff:ff:ff:ff
inet 172.17.44.1/24 brd 172.17.44.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether 02:0e:32:58:7f:75 brd ff:ff:ff:ff:ff:ff
inet 172.17.44.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::e:32ff:fe58:7f75/64 scope link
valid_lft forever preferred_lft forever
docker0和flannel网卡在同一个网段，并且两个node节点直接应该能通
如果ping不通，并且/var/log/message下有类似iptables的告警，需要再关闭防火墙systemctl stop firewalld.service ，之后重启docker
==========================================================================================================================================
四、在Master节点部署组件
1、生成证书
创建CA证书
mkdir /root/kubernetes-ssl
cd /root/kubernetes-ssl
# cat ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
# cat ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# cfssl gencert -initca ca-csr.json |cfssljson -bare ca
生成apiserver证书
# cat server-csr.json
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.176.19",
"192.168.174.30",
"192.168.174.31",
"192.168.174.40",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
生成kube-proxy证书
# cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
最终生成证书文件包括：
# ls *pem
ca-key.pem ca.pem kube-proxy-key.pem kube-proxy.pem server-key.pem server.pem
2、部署apiserver组件
下载二进制包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md （只下载kubernetes-server-linux-amd64.tar.gz）
# mkdir /opt/kubernetes/{bin,cfg,ssl} -p
# tar zxvf kubernetes-server-linux-amd64.tar.gz
# cd kubernetes/server/bin/
# cp kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin/
创建token文件
# cat /opt/kubernetes/cfg/token.csv
674c457d4dcf2eefe4920d7dbb6b0ddc,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
第一列：随机字符串，自己可生成
第二列：用户名
第三列：UID
第四列：用户组
创建apiserver配置文件
vim /opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.174.30:2379,https://192.168.174.31:2379,https://192.168.174.40:2379 \
--bind-address=192.168.174.30 \
--secure-port=6443 \
--advertise-address=192.168.174.30 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
注解：
cat /opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \ #启用日志
--v=4 \ #日志等级
--etcd-servers=https://192.168.174.30:2379,https://192.168.174.31:2379,https://192.168.174.40:2379 \ #集群地址
--bind-address=192.168.174.30 \ #监听地址
--secure-port=6443 \ #安全端口
--advertise-address=192.168.174.30 \ #集群通告地址
--allow-privileged=true \ #启用授权
--service-cluster-ip-range=10.0.0.0/24 \ #Service虚拟IP地址段
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ #准入控制模块
--authorization-mode=RBAC,Node \ #认证授权，启用RBAC授权和节点自管理
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \ #启用TLS bootstrap功能
--service-node-port-range=30000-50000 \ #Service Node类型默认分配端口范围
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
system管理apiserver
# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
#cd /root/kubernetes-ssl/
#cp *.pem /opt/kubernetes/ssl/
启动：
#systemctl daemon-reload
#systemctl enable kube-apiserver
#systemctl restart kube-apiserver
部署schduler组件
创建schduler配置文件
# vim /opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"
注解：
--master 连接本地apiserver
--leader-elect 当该组件启动多个时，自动选举（HA）
systemd管理schduler组件
# vim /opt/kubernetes/cfg/kube-scheduler
KUBE_SCHEDULER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"
# vim /usr/lib/systemd/system/kube-scheduler.service
[Unti]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
启动：
#systemctl daemon-reload
#systemctl enable kube-scheduler
#systemctl restart kube-scheduler
部署controller-manager组件
创建controller-manager配置文件：
# vim /opt/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem"
system管理controller-manager组件
# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
启动：
#systemctl daemon-reload
#systemctl enable kube-controller-manager
#systemctl restart kube-controller-manager
查看集群组件状态：
# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
输出如上内容表示成功
两个节点都部署
在NODE节点部署组件
master apiserver启用TLS认证后，Node节点kubelet组件想要加入集群，必须使用CA签发的有效证书才能与apiserver通信，当Node节点很多时，签署证书是一件很繁琐的事情，因此有了TLS Bootstrapping机制，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。
在Master节点操作：
1。将kubelet-bootstrap用户绑定到系统集群角色
/opt/kubernetes/bin/kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
2.创建kubeconfig文件
进入到生成kubernetes证书的目录
#cd /root/kubernets-ssl
定义环境变量
#KUBE_APISERVER="https://192.168.176.19:6443" (如果配置负载均衡则使用负载均衡地址，如果没有配置则指定master节点地址)
#BOOTSTRAP_TOKEN=674c457d4dcf2eefe4920d7dbb6b0ddc
创建bootstrapping.kubeconfig文件
设置集群参数（直接在终端执行）
#/opt/kubernetes/bin/kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig
设置客户端认证参数（直接在终端执行）
#/opt/kubernetes/bin/kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig
设置上下文参数（直接在终端执行）
#/opt/kubernetes/bin/kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
设置默认上下文（直接在终端执行）
#/opt/kubernetes/bin/kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
创建kube-proxy.kubeconfig文件（直接在终端执行）
#/opt/kubernetes/bin/kubectl config set-cluster kubernetes \
--certificate-authority=./ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig
#/opt/kubernetes/bin/kubectl config set-credentials kube-proxy \
--client-certificate=./kube-proxy.pem \
--client-key=./kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
#/opt/kubernetes/bin/kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
#/opt/kubernetes/bin/kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
#ls
bootstrapping.kubeconfig kube-proxy.kubeconfig
将这两个文件拷贝到Node节点/opt/kubernetes/cfg目录下
3.部署kubelet组件
将之前下载的二进制包中kubele和kube-proxy拷贝到Node节点的/opt/kubernetes/bin目录下，之前下载的二进制包解压后目录/root/kubernetes/server/bin，找不到用find搜
在node节点下创建kubelet配置文件(其中的IP写本机IP)
#cat /opt/kubernetes/cfg/kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.174.41 \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
参数说明
--hostname-override=192.168.174.41 \ #在集群中显示的主机名
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \ #文件位置，会自动生成
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \ #指定刚生成的bootstrap.kubeconfig文件
--cert-dir=/opt/kubernetes/ssl \ #颁发证书存放位置
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0" #管理Pod网络的镜像
编写kubelet.config配置文件
# cat /opt/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.174.41
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
webhook:
enabled: false
systemd管理kubele组件
# cat /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
启动
#systemctl daemon-reload
#systemctl enable kubelet
#systemctl restart kubelet
在Mater审批Node加入集群
需要手动允许节点加入
#/opt/kubernetes/bin/kubectl get csr
#/opt/kubernetes/bin/kubectl certificate approve XXXXXID
#/opt/kubernetes/bin/kubectl get node
4.部署kube-proxy组件
创建kube-proxy配置文件：
# cat /opt/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.174.40 \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
systemd管理kube-proxy组件
cat /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
启动：
#systemctl daemon-reload
#systemctl enable kube-proxy
#systemctl restart kube-proxy
如果启动后报错：
kube-proxy: W0305 10:54:29.666610 31207 server.go:605] Failed to retrieve node info: nodes "192.168.174.40" not found
需要检查：
/opt/kubernetes/cfg/kubelet.config 和 /opt/kubernetes/cfg/kube-proxy 中的IP地址是否都是本机地址。
查看集群状态
# /opt/kubernetes/bin/kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.174.40 Ready <none> 6h v1.11.8
192.168.174.41 Ready <none> 1d v1.11.8
# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
5.运行一个测试示例
创建一个nginx web，判断集群是否正常工作
# /opt/kubernetes/bin/kubectl run nginx --image=nginx --replicas=3
# /opt/kubernetes/bin/kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort
service/nginx exposed
查看Pod，Service：
# /opt/kubernetes/bin/kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-64f497f8fd-9ccgb 1/1 Running 0 6h
nginx-64f497f8fd-bbw97 1/1 Running 0 6h
nginx-64f497f8fd-pxkh8 1/1 Running 0 6h
# /opt/kubernetes/bin/kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 4d
nginx NodePort 10.0.0.16 <none> 88:39427/TCP 57s
记录下最后一条的端口号39427
访问：http://192.168.174.40:39427
6.部署Dashboard（Web UI）
dashboard-deployment.yaml 部署Pod,提供web服务
dashboard-rbac.yaml 授权访问apiserver获取信息
dashboard-service.yaml 发布服务，提供对外访问
# cat dashboard-deployment.yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: kubernetes-dashboard
containers:
- name: kubernetes-dashboard
image: registry.cn-hangzhou.aliyuncs.com/kube_containers/kubernetes-dashboard-amd64:v1.8.1
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 9090
protocol: TCP
livenessProbe:
httpGet:
path: /
port: 9090
initialDelaySeconds: 30
timeoutSeconds: 30
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
# cat dashboard-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
addonmanager.kubernetes.io/mode: Reconcile
name: kubernetes-dashboard
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
# cat dashboard-service.yaml
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
type: NodePort
selector:
k8s-app: kubernetes-dashboard
ports:
- port: 80
targetPort: 9090
创建：
/opt/kubernetes/bin/kubectl create -f dashboard-rbac.yaml
/opt/kubernetes/bin/kubectl create -f dashboard-deployment.yaml
/opt/kubernetes/bin/kubectl create -f dashboard-service.yaml
等待一会，查看资源状态
# /opt/kubernetes/bin/kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/kubernetes-dashboard-d9545b947-jrmmc 1/1 Running 0 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes-dashboard NodePort 10.0.0.6 <none> 80:41545/TCP 16m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/kubernetes-dashboard 1 1 1 1 16m
NAME DESIRED CURRENT READY AGE
replicaset.apps/kubernetes-dashboard-d9545b947 1 1 1 16m
# /opt/kubernetes/bin/kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.6 <none> 80:41545/TCP 16m
访问：http://192.168.174.40:41545
