kubeadm集群更改证书

Note: 本文基于使用kubeadm搭建的kubernetes集群进行讲解

---

第一步：获取证书签发信息

• 方式一：通过原有证书进行获取相关信息

  • 获取apiserver签发信息

    $ openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt
    ......
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Extended Key Usage:
            TLS Web Server Authentication
        X509v3 Subject Alternative Name:
            DNS:localhost, DNS:node1, DNS:node2, DNS:node3, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc,   DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:10.96.0.1, IP Address:192.168.12.10, IP   Address:192.168.12.11, IP Address:192.168.12.12, IP Address:192.168.12.13
    ......
    

  • 从上面输出信息可知签发的DNS和IP详情

    DNS.1 = localhost
    # 各 master 节点 nodeName
    DNS.2 = node1
    DNS.3 = node2
    DNS.4 = node3
    # kubenertes svc 地址
    DNS.5 = kubernetes
    DNS.6 = kubernetes.default
    DNS.7 = kubernetes.default.svc
    # 带 dns domain 的 kubenertes svc 地址
    DNS.8 = kubernetes.default.svc.cluster.local
    IP.6 = 127.0.0.1
    # kubenertes svc 的 clusterip
    IP.5 = 10.96.0.1
    # api-server VIP 地址
    IP.4 = 192.168.12.10
    # 各 master 节点 IP
    IP.1 = 192.168.12.11
    IP.2 = 192.168.12.12
    IP.3 = 192.168.12.13
    

• 方式二：自行统计各项信息

  • 在创建证书之前需要获取到以下信息，在签发证书的时候会用到它：

    • 各master节点IP
    • 各master节点nodeName
    • 如果有设置apiserver负载均衡则需要VIP，否则请忽略
    • kubernetes集群dns domain
    • kubenertes.default.svc的clusterip

  • 本文以下面集群信息为例：

    • 节点信息：

      nodeName	ip	relo
      slb	192.168.12.10	slb
      node1	192.168.12.11	master
      node2	192.168.12.12	master
      node3	192.168.12.13	master
      node4	192.168.12.14	worker
      node5	192.168.12.15	worker

    • 获取kubernetes dns domain：

      # CoreDNS 查看方法
      $ kubectl get cm -n kube-system coredns -o yaml | grep kubernetes
      kubernetes cluster.local in-addr.arpa ip6.arpa {
      # 由以上输出可知dns domain为 cluster.local
      
      # kube-dns 查看方法
      $ kubectl get deployment -n kube-system kube-dns -o yaml | grep domain
      - --domain=cluster.local.
      # 由以上输出可知dns domain为 cluster.local
      

    • kubernetes apiserver clusterip

      $ kubectl get svc kubernetes -n default
      NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
      kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   280d
      # 由结果可知apiserver clusterip
      

同理获取etcd集群证书签发的DNS和IP详情，默认etcd证书路径为/etc/kubernetes/pki/etcd

第二步：创建证书

Note: 我们只需要在一个节点上进行证书生成，生成的证书分发到其他节点即可

---

• 创建CA服务端证书签名请求配置文件openssl.conf，内容如下，注意替换alt_names_cluster、alt_names_etcd域中的值

  [ req ]
  default_bits = 2048
  default_md = sha256
  distinguished_name = req_distinguished_name
  
  [req_distinguished_name]
  
  [ v3_ca ]
  basicConstraints = critical, CA:TRUE
  keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign
  
  [ v3_req_server ]
  basicConstraints = CA:FALSE
  keyUsage = critical, digitalSignature, keyEncipherment
  extendedKeyUsage = serverAuth
  
  [ v3_req_client ]
  basicConstraints = CA:FALSE
  keyUsage = critical, digitalSignature, keyEncipherment
  extendedKeyUsage = clientAuth
  
  [ v3_req_apiserver ]
  basicConstraints = CA:FALSE
  keyUsage = critical, digitalSignature, keyEncipherment
  extendedKeyUsage = serverAuth
  subjectAltName = @alt_names_cluster
  
  [ v3_req_etcd ]
  basicConstraints = CA:FALSE
  keyUsage = critical, digitalSignature, keyEncipherment
  extendedKeyUsage = serverAuth, clientAuth
  subjectAltName = @alt_names_etcd
  
  [ alt_names_cluster ]
  DNS.1 = localhost
  DNS.2 = node1
  DNS.3 = node2
  DNS.4 = node3
  DNS.5 = kubernetes
  DNS.6 = kubernetes.default
  DNS.7 = kubernetes.default.svc
  DNS.8 = kubernetes.default.svc.cluster.local
  IP.1 = 127.0.0.1
  IP.2 = 10.96.0.1
  IP.3 = 192.168.12.10
  IP.4 = 192.168.12.11
  IP.5 = 192.168.12.12
  IP.6 = 192.168.12.13
  
  [ alt_names_etcd ]
  DNS.1 = localhost
  DNS.2 = node1
  DNS.3 = node2
  DNS.4 = node3
  IP.1 = 127.0.0.1
  IP.2 = 0:0:0:0:0:0:0:1
  IP.3 = 192.168.12.11
  IP.4 = 192.168.12.12
  IP.5 = 192.168.12.13
  

• 创建集群 key 与 CA

  • 将要创建的 CA

    路径	Common Name	描述
    ca.crt,key	kubernetes	Kubernetes general CA
    etcd/ca.crt,key	kubernetes	For all etcd-related functions
    front-proxy-ca.crt,key	kubernetes	For the front-end proxy

  • 要注意 CA 中 CN(Common Name) 与 O(Organization) 等内容是会影响Kubernetes组件认证的

    • CA (Certificate Authority) 是自签名的根证书，用来签名后续创建的其它证书
    • CN (Common Name), apiserver 会从证书中提取该字段作为请求的用户名 (User Name)
    • O (Organization), apiserver 会从证书中提取该字段作为请求用户所属的组 (Group)

  一般CA根证书有效期为10年，若举例过期时间还长，可跳过本节操作，命令中的3650为3650天，即证书有效期。

  • kubernetes-ca

    openssl genrsa -out ca.key 2048
    openssl req -x509 -new -nodes -key ca.key \
      -subj "/CN=kubernetes" -config openssl.conf \
      -extensions v3_ca -out ca.crt -days 3560
    

  • etcd-ca

    mkdir -p etcd
    openssl genrsa -out etcd/ca.key 2048
    openssl req -x509 -new -nodes -key etcd/ca.key \
      -subj "/CN=kubernetes" -config openssl.conf \
      -extensions v3_ca -out etcd/ca.crt -days 3560
    

  • front-proxy-ca

    openssl genrsa -out front-proxy-ca.key 2048
    openssl req -x509 -new -nodes -key front-proxy-ca.key \
      -subj "/CN=kubernetes" -config openssl.conf \
      -extensions v3_ca -out front-proxy-ca.crt -days 3560
    

• 创建 Certificates

  • 将要创建的 Certificates

    Name	Key	Certificates	Common Name	Organization
    etcd/server	etcd/server.key	etcd/server.crt	master
    etcd/peer	etcd/peer.key	etcd/peer.crt	master
    etcd/healthcheck-client	etcd/healthcheck-client.key	etcd/healthcheck-client.crt	kube-etcd-healthcheck-client	system:masters
    apiserver-etcd-client	apiserver-etcd-client.key	apiserver-etcd-client.crt	kube-apiserver-etcd-client	system:masters
    apiserver	apiserver.key	apiserver.crt	kube-apiserver
    apiserver-kubelet-client	apiserver-kubelet-client.key	apiserver-kubelet-client.crt	kube-apiserver-kubelet-client	system:masters
    front-proxy-client	front-proxy-client.key	front-proxy-client.crt	front-proxy-client
    kube-scheduler	kube-scheduler.key	kube-scheduler.crt	system:kube-scheduler
    sa(kube-controller-manager)	sa.key(sa.pub)	kube-controller-manager.crt	system:kube-controller-manager
    admin(kubectl)	admin.key	admin.crt	kubernetes-admin	system:masters
    kubelet	kubelet.key	kubelet.crt	system:node:master	system:nodes

mkdir etcd
cp /etc/kubernetes/pki/etcd/*.key etcd/
cp /etc/kubernetes/pki/etcd/ca.crt etcd/

• etcd/server

  ~~openssl genrsa -out etcd/server.key 2048~~
  openssl req -new -key etcd/server.key \
    -subj "/CN=master" -out etcd/server.csr
  openssl x509 -in etcd/server.csr -req -CA etcd/ca.crt \
    -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
    -extfile openssl.conf -out etcd/server.crt -days 3560
  

• etcd/peer

  ~~openssl genrsa -out etcd/peer.key 2048~~
  openssl req -new -key etcd/peer.key \
    -subj "/CN=master" -out etcd/peer.csr
  openssl x509 -in etcd/peer.csr -req -CA etcd/ca.crt \
    -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
    -extfile openssl.conf -out etcd/peer.crt -days 3560
  

• etcd/healthcheck-client

  ~~openssl genrsa -out etcd/healthcheck-client.key 2048~~
  openssl req -new -key etcd/healthcheck-client.key \
    -subj "/CN=kube-etcd-healthcheck-client/O=system:masters" \
    -out etcd/healthcheck-client.csr
  openssl x509 -in etcd/healthcheck-client.csr -req -CA etcd/ca.crt \
    -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
    -extfile openssl.conf -out etcd/healthcheck-client.crt -days 3560
  

cp /etc/kubernetes/pki/*.key .
cp /etc/kubernetes/pki/ca.crt .
cp /etc/kubernetes/pki/front-proxy-ca.crt .

• apiserver-etcd-client

  ~~openssl genrsa -out apiserver-etcd-client.key 2048~~
  openssl req -new -key apiserver-etcd-client.key \
    -subj "/CN=kube-apiserver-etcd-client/O=system:masters" \
    -out apiserver-etcd-client.csr
  openssl x509 -in apiserver-etcd-client.csr -req -CA etcd/ca.crt \
    -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
    -extfile openssl.conf -out apiserver-etcd-client.crt -days 3560
  

• apiserver

  ~~openssl genrsa -out apiserver.key 2048~~
  openssl req -new -key apiserver.key \
    -subj "/CN=kube-apiserver" -config openssl.conf \
    -out apiserver.csr
  openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -extensions v3_req_apiserver \
    -extfile openssl.conf -out apiserver.crt -days 3560
  

• apiserver-kubelet-client

  ~~openssl genrsa -out apiserver-kubelet-client.key 2048~~
  openssl req -new -key apiserver-kubelet-client.key \
    -subj "/CN=kube-apiserver-kubelet-client/O=system:masters" \
    -out apiserver-kubelet-client.csr
  openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -extensions v3_req_client \
    -extfile openssl.conf -out apiserver-kubelet-client.crt -days 3560
  

• front-proxy-client

  ~~openssl genrsa -out front-proxy-client.key 2048~~
  openssl req -new -key front-proxy-client.key \
    -subj "/CN=front-proxy-client" \
    -out front-proxy-client.csr
  openssl x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key \
    -CAcreateserial -extensions v3_req_client \
    -extfile openssl.conf -out front-proxy-client.crt -days 3560
  

• kube-scheduler

  openssl genrsa -out kube-scheduler.key 2048
  openssl req -new -key kube-scheduler.key \
    -subj "/CN=system:kube-scheduler" \
    -out kube-scheduler.csr
  openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -extensions v3_req_client \
    -extfile openssl.conf -out kube-scheduler.crt -days 3560
  

cp /etc/kubernetes/pki/sa.pub .

• sa(kube-controller-manager)

  ~~openssl genrsa -out sa.key 2048~~
  ~~openssl rsa -in sa.key -pubout -out sa.pub~~
  openssl req -new -key sa.key \
    -subj "/CN=system:kube-controller-manager" \
    -out kube-controller-manager.csr
  openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -extensions v3_req_client \
    -extfile openssl.conf -out kube-controller-manager.crt -days 3560
  

• admin(kubectl)

  openssl genrsa -out admin.key 2048
  openssl req -new -key admin.key \
    -subj "/CN=kubernetes-admin/O=system:masters" \
    -out admin.csr
  openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key \
    -CAcreateserial -extensions v3_req_client \
    -extfile openssl.conf -out admin.crt -days 3560 
  

• kubelet

  openssl genrsa -out kubelet.key 2048
  # 此处为 master 节点 nodeName，每个 master 生成对应的证书
  openssl req -new -key kubelet.key \
    -subj "/CN=system:node:node1/O=system:nodes" \
    -out kubelet.csr
  openssl x509 -req -CA ca.crt -CAkey ca.key \
    -CAcreateserial -extensions v3_req_client \
    -extfile openssl.conf -days 3560 -in kubelet.csr -out kubelet.crt
  

第三步：生成kubernetes各组件配置文件并应用

• 所要生成的配置文件列表

  配置文件名称	组件证书文件名称	组件秘钥文件名称	根证书文件名称
  admin.conf(kubectl)	admin.crt	admin.key	ca.crt
  kubelet.conf	kubelet.crt	kubelet.key	ca.crt
  scheduler.conf	kube-scheduler.crt	kube-scheduler.key	ca.crt
  controller-manager.conf	kube-controller-manager.crt	sa.key	ca.crt

  • 操作前请先备份原有配置文件
  • 除了kubelet.conf文件需注意配置为对应节点的nodeName，其余配置文件可通用
  • 以下操作请先在一台 master 节点上操作确认没有问题后再进行配置其他节点
  • –certificate-authority：指定根证书
  • –client-certificate、–client-key：指定组件证书及秘钥
  • –embed-certs=true：将组件证书内容嵌入到生成的配置文件中(不加时，写入的是证书文件路径)

• admin.conf(kubectl)

  KUBE_APISERVER="https://192.168.12.10:6443"
  CLUSTER_NAME="kubernetes"
  KUBE_USER="kubernetes-admin"
  KUBE_CERT="admin"
  KUBE_CONFIG="admin.conf"
  
  # 设置集群参数
  kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置客户端认证参数
  kubectl config set-credentials ${KUBE_USER} \
    --client-certificate=${KUBE_CERT}.crt \
    --client-key=${KUBE_CERT}.key \
    --embed-certs=true \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置上下文参数
  kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${KUBE_USER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置当前使用的上下文
  kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
  
  # 查看生成的配置文件
  kubectl config view --kubeconfig=${KUBE_CONFIG}
  

cp /etc/kubernetes/kubelet.conf .

• kubelet.conf（注意配置对应的nodeName）

  KUBE_APISERVER="https://192.168.12.10:6443"
  CLUSTER_NAME="kubernetes"
  # 此处为 master 节点 nodeName，每个 master 生成对应的 kubelet.conf
  KUBE_USER="system:node1:master"
  KUBE_CERT="kubelet"
  KUBE_CONFIG="kubelet.conf"
  
  # 设置集群参数
  kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置客户端认证参数
  kubectl config set-credentials ${KUBE_USER} \
    --client-certificate=${KUBE_CERT}.crt \
    --client-key=kubelet.key \
    --embed-certs=true \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置上下文参数
  kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${KUBE_USER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置当前使用的上下文
  kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
  
  # 查看生成的配置文件
  kubectl config view --kubeconfig=${KUBE_CONFIG}
  

• scheduler.conf

  KUBE_APISERVER="https://192.168.12.10:6443"
  CLUSTER_NAME="kubernetes"
  KUBE_USER="system:kube-scheduler"
  KUBE_CERT="kube-scheduler"
  KUBE_CONFIG="scheduler.conf"
  
  # 设置集群参数
  kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置客户端认证参数
  kubectl config set-credentials ${KUBE_USER} \
    --client-certificate=${KUBE_CERT}.crt \
    --client-key=${KUBE_CERT}.key \
    --embed-certs=true \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置上下文参数
  kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${KUBE_USER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置当前使用的上下文
  kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
  
  # 查看生成的配置文件
  kubectl config view --kubeconfig=${KUBE_CONFIG}
  

• controller-manager.conf

  KUBE_APISERVER="https://192.168.12.10:6443"
  CLUSTER_NAME="kubernetes"
  KUBE_USER="system:kube-controller-manager"
  KUBE_CERT="kube-controller-manager"
  KUBE_CONFIG="controller-manager.conf"
  
  # 设置集群参数
  kubectl config set-cluster ${CLUSTER_NAME} \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置客户端认证参数
  kubectl config set-credentials ${KUBE_USER} \
    --client-certificate=${KUBE_CERT}.crt \
    --client-key=sa.key \
    --embed-certs=true \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置上下文参数
  kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
    --cluster=${CLUSTER_NAME} \
    --user=${KUBE_USER} \
    --kubeconfig=${KUBE_CONFIG}
  
  # 设置当前使用的上下文
  kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG}
  
  # 查看生成的配置文件
  kubectl config view --kubeconfig=${KUBE_CONFIG}
  

  rm -rf *.csr
  cp -r /etc/kubernetes/ /etc/kubernetes.bak
  cd /etc/kubernetes/pki
 \cp -rf /root/key/* .
  cd ..
  \cp pki/*.conf .
   rm -rf openssl.conf

• 应用配置

  • 重启 Docker 和 Kubelet
  • 查看三个kubernetes组件（kubelet，controller-manager，scheduler）的日志，确认是否还有证书过期报错信息。

• Worker节点证书更新操作

  • 停止docker和kubelet

    systemctl stop docker && systemctl stop kubelet
    

  • 删除kubelet.conf文件，文件一般在/etc/kubernetes目录下。

  • 编辑bootstrap-kubelet.conf文件（文件一般在/etc/kubernetes目录下），修改certificate-authority-data内容，与master节点中的admin.conf文件的该区域内容相同。

  • 备份后删除该目录下的文件rm -rf /var/lib/kubelet/pki

  • 重启所有节点docker和kubelet

    systemctl restart docker && systemctl restart kubelet
    

重启服务

• 将全部kube-proxy重启
• 将全部网络插件重启（比如：flannel）

更新权限更新token

• cp /etc/kubernetes/admin.conf ~/.kube/config
• kubeadm token list
• kubeadm token create --ttl=0
• kubeadm token list
  把获取到的token更新到node节点/etc/kubernetes/bootstrap-kubelet.conf
  重启node节点kubelet