安装k8s

2018-07-08 本文已影响0人 Nick_4438

前言

通过本文的学习，读者能够安装一个k8s集群；使用软件版本如下：

CentOS Linux release 7.4.1708 (Core)
docker17.03.2-ce
socat-1.7.3.2-2.el7.x86_64
kubelet-1.10.0-0.x86_64
kubernetes-cni-0.6.0-0.x86_64
kubectl-1.10.0-0.x86_64
kubeadm-1.10.0-0.x86_64

网络配置如下：

节点名称	ip	备注
node01	192.168.3.68	master and etcd
node02	192.168.3.69	master and etcd
node03	192.168.3.70	master and etcd
node04	192.168.3.71	node
vip	192.168.3.72

vip节点不需要物理机

安装

设置网络

分别在对应主机上设置主机名

hostnamectl set-hostname node1
hostnamectl set-hostname node2
hostnamectl set-hostname node3
hostnamectl set-hostname node4
hostnamectl set-hostname vip

所有主机上执行，映射域名

cat <<EOF > /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.3.68 node1
192.168.3.69 node2
192.168.3.70 node3
192.168.3.71 node4
192.168.3.73 vip
EOF

node1上执行ssh免密码登陆配置

ssh-keygen  #一路回车即可
ssh-copy-id  node2
ssh-copy-id  node3
ssh-copy-id  node4
ssh-copy-id  vip

四台主机配置、停防火墙、关闭Swap、关闭Selinux、设置内核、K8S的yum源、安装依赖包、配置ntp（配置完后建议重启一次）

systemctl stop firewalld
systemctl disable firewalld

swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce  0 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  

modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl 

systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
 
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

安装、配置keepalived(主节点，node1、node2、node3)

yum install -y keepalived
systemctl enable keepalived

node1的keepalived.conf

cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://192.168.3.72:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface enp0s3
    virtual_router_id 61
    priority 100
    advert_int 1
    mcast_src_ip 192.168.3.68
    nopreempt
    authentication {
        auth_type PASS
        auth_pass sqP05dQgMSlzrxHj
    }
    unicast_peer {
        192.168.150.69
        192.168.150.70
    }
    virtual_ipaddress {
        192.168.150.72/24
    }
    track_script {
        CheckK8sMaster
    }

}
EOF

node2的keepalived.conf

cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_k8s
}

global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://192.168.3.72:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 61
    priority 90
    advert_int 1
    mcast_src_ip 192.168.3.69
    nopreempt
    authentication {
        auth_type PASS
        auth_pass sqP05dQgMSlzrxHj
    }
    unicast_peer {
        192.168.3.68
        192.168.3.70
    }
    virtual_ipaddress {
        192.168.3.72/24
    }
    track_script {
        CheckK8sMaster
    }

}
EOF

node3的keepalived.conf

cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_k8s
}

global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://192.168.3.72:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 61
    priority 90
    advert_int 1
    mcast_src_ip 192.168.3.70
    nopreempt
    authentication {
        auth_type PASS
        auth_pass sqP05dQgMSlzrxHj
    }
    unicast_peer {
        192.168.3.68
        192.168.3.69
    }
    virtual_ipaddress {
        192.168.3.72/24
    }
    track_script {
        CheckK8sMaster
    }

}
EOF

interface 需要根据电脑自己的网卡名字修改；ip地址需要根据实际情况修改；

依次启动keepalived(node1, node2,node3)

systemctl restart keepalived

如果没有错误，则在node1上已经能够看得到，绑定的虚拟ip，如果有异常请停下排查；node1上运行ip a查看
image.png

创建etcd证书(单独在node1上执行)

设置cfssl环境

cd ~
mkdir soft
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH

创建 CA 配置文件（下面配置的IP为etc节点的IP）

mkdir /root/ssl
cd /root/ssl
cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "8760h"
},
"profiles": {
  "kubernetes-Soulmate": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
    "expiry": "8760h"
  }
}
}
}
EOF

cat >  ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
  "C": "CN",
  "ST": "shanghai",
  "L": "shanghai",
  "O": "k8s",
  "OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.3.68",
    "192.168.3.69",
    "192.168.3.70"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

node1分发etcd证书到node2、node3上面

mkdir -p /etc/etcd/ssl
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
ssh -n node2 "mkdir -p /etc/etcd/ssl && exit"
ssh -n node3 "mkdir -p /etc/etcd/ssl && exit"
scp -r /etc/etcd/ssl/*.pem node2:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem node3:/etc/etcd/ssl/

安装配置etcd (三主节点node1,node2,node3）

三个节点分别执行

yum install etcd -y
mkdir -p /var/lib/etcd

node1的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
  --name node1 \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.3.68:2380 \
  --listen-peer-urls https://192.168.3.68:2380 \
  --listen-client-urls https://192.168.3.68:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.3.68:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster node1=https://192.168.3.68:2380,node2=https://192.168.3.69:2380,node3=https://192.168.3.70:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node2的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
  --name node2 \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.3.69:2380 \
  --listen-peer-urls https://192.168.3.69:2380 \
  --listen-client-urls https://192.168.3.69:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.3.69:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster node1=https://192.168.3.68:2380,node2=https://192.168.3.69:2380,node3=https://192.168.3.70:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

node3的etcd.service

cat <<EOF >/etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd \
  --name node3 \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.3.70:2380 \
  --listen-peer-urls https://192.168.3.70:2380 \
  --listen-client-urls https://192.168.3.70:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.3.70:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster node1=https://192.168.3.68:2380,node2=https://192.168.3.69:2380,node3=https://192.168.3.70:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

添加自启动（node1,node2,node3, etc集群最少2个节点才能启动，启动报错看mesages日志）

# mv etcd.service /usr/lib/systemd/system/  ??
cp /etc/systemd/system/etcd.service /usr/lib/systemd/system/
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

第一个节点在执行到systemctl start etcd时会卡住，这时候继续在第二个节点上执行对应命令，第二个节点执行完毕之后，第一个节点会执行成功；

在三个etcd节点执行一下命令检查(node1,node2,node3)

etcdctl --endpoints=https://192.168.3.68:2379,https://192.168.3.69:2379,https://192.168.3.70:2379 \
  --ca-file=/etc/etcd/ssl/ca.pem \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health

image.png

所有节点安装配置docker

安装docker（kubeadm目前支持docker最高版本是17.03.x）

yum install https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm  -y
yum install https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/Packages/docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm  -y

修改配置文件 vim /usr/lib/systemd/system/docker.service

ExecStart=/usr/bin/dockerd   -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock  --registry-mirror=https://ms3cfraz.mirror.aliyuncs.com

https://ms3cfraz.mirror.aliyuncs.com地址需要配置成为自己的地址

启动docker

systemctl daemon-reload
systemctl restart docker
systemctl enable docker
systemctl status docker       # 检查docker安装状态

安装、配置kubeadm

所有节点安装kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet

所有节点修改kubelet配置文件vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

#修改这一行,没看到这行？
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
#添加这一行
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.0"

所有节点修改完配置文件一定要重新加载配置·

systemctl daemon-reload
systemctl enable kubelet

命令补全

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

初始化集群

node1、node2、node3添加集群初始配置文件（集群配置文件一样）

cat <<EOF > config.yaml 
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://192.168.3.68:2379
  - https://192.168.3.69:2379
  - https://192.168.3.70:2379
  caFile: /etc/etcd/ssl/ca.pem
  certFile: /etc/etcd/ssl/etcd.pem
  keyFile: /etc/etcd/ssl/etcd-key.pem
  dataDir: /var/lib/etcd
networking:
  podSubnet: 10.244.0.0/16
kubernetesVersion: 1.11.0
api:
  advertiseAddress: "192.168.3.72"
token: "b99a00.a144ef80536d4344"
tokenTTL: "0s"
apiServerCertSANs:
- node1
- node2
- node3
- 192.168.3.68
- 192.168.3.69
- 192.168.3.70
- 192.168.3.71
- 192.168.3.72
featureGates:
  CoreDNS: true
imageRepository: "registry.cn-hangzhou.aliyuncs.com/google_containers"
EOF

安装k8s

前言

安装

设置网络

安装、配置keepalived(主节点，node1、node2、node3)

创建etcd证书(单独在node1上执行)

安装配置etcd (三主节点node1,node2,node3）

所有节点安装配置docker

安装、配置kubeadm

初始化集群

猜你喜欢

热点阅读