搭建高可用Etcd集群 (TLS)

2018-07-05 本文已影响0人 Xiak

etcd集群采用raft算法选举Leader，最小raft集群需要3个参与者，所以一个etcd集群最少需要3台虚拟机.

1. 准备

1.1 准备环境

准备三台CentOS7.3主机，主机名分别为etcd1, etcd2, etcd3

etcd1: 10.62.232.41 
etcd2: 10.62.232.42
etcd3: 10.62.232.43

1.2 环境配置

在每台主机上运行以下命令:

cat >> /etc/hosts <<EOF
# etcd hosts
10.62.232.41 etcd1
10.62.232.42 etcd2
10.62.232.43 etcd3
EOF

如果需要启动防火墙，请确保打开官方给出的etcd端口, 官方端口号查询

这里简单粗暴的禁用防火墙

systemctl stop firewalld
systemctl disable firewalld

禁用Selinux

setenforce 0
vi /etc/selinux/config
SELINUX=disabled

2. TLS 密钥和证书

使用TLS证书对通信进行加密，并开启即与CA根证书签名的双向认证

2.1 建立目录

每台主机都要确保以下目录存在

证书目录:
- /etc/ssl/etcd/ssl/
数据目录:
- /var/lib/etcd/

mkdir -p /etc/ssl/etcd/ssl/
mkdir -p /var/lib/etcd/

2.2 创建CA证书和私钥

有很多方式可以创建CA证书和私钥，其中比较流行的有两种

openssl
cfssl

这里使用openssl来生成私钥ca.key和证书ca.crt

登录etcd1主机，运行以下命令

mkdir -p /xiak/ssl
cd /xiak/ssl
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=xiak" -days 10000 -out ca.crt

2.3 创建etcd证书和私钥

接下来我们通过之前生成的CA证书和私钥生成etcd的证书和私钥，整个过程需要5步：

创建配置文件etcd-ca.conf

cd /xiak/ssl
cat > /xiak/ssl/etcd-ca.conf <<EOF
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
    
[ dn ]
C = CN
ST = Sichuan
L = Chengdu
O = etcd
OU = xiak
CN = etcd
    
[ req_ext ]
subjectAltName = @alt_names
    
[ alt_names ]
DNS.1 = localhost
DNS.2 = etcd1
DNS.3 = etcd2
DNS.4 = etcd3
IP.1 = 127.0.0.1
IP.2 = 10.62.232.41
IP.3 = 10.62.232.42
IP.4 = 10.62.232.43
    
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
EOF

生成密钥

openssl genrsa -out etcd.key 2048

生成证书签发请求(certificate signing request)

openssl req -new -key etcd.key -out etcd.csr -config etcd-ca.conf

生成证书

openssl x509 -req -in etcd.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out etcd.crt -days 10000 \
-extensions v3_ext -extfile etcd-ca.conf

验证证书

openssl verify -CAfile ca.crt etcd.crt

3 安装etcd

3.1 安装证书

将CA证书ca.crt, etcd证书etcd.crt和秘钥etcd.key, 拷贝到各节点的/etc/ssl/etcd/ssl/目录中。

3.2 下载etcd二进制包

在每个节点上下载etcd二进制包

mkdir -p /xiak/pkg
cd /xiak/pkg
wget https://github.com/coreos/etcd/releases/download/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz

3.3 安装etcd二进制包

解压缩etcd-v3.3.5-linux-amd64.tar.gz，将其中的etcd和etcdctl两个可执行文件复制到各节点的/usr/bin目录

cd /xiak/pkg
tar -zxvf etcd-v3.3.8-linux-amd64.tar.gz
cd ./etcd-v3.3.8-linux-amd64/
cp {etcd,etcdctl} /usr/local/bin

3.4 以systemd形式管理etcd服务

在每个节点上创建etcd的systemd unit文件/usr/lib/systemd/system/etcd.service

在etcd1主机上执行以下命令集
```bash
mkdir -p /etc/etcd/
cat > /etc/etcd/etcd.conf <<EOF
# [Member Flags]
# ETCD_ELECTION_TIMEOUT=1000
# ETCD_HEARTBEAT_INTERVAL=100
# 指定etcd的数据目录
ETCD_NAME=etcd1
ETCD_DATA_DIR=/var/lib/etcd/

# [Cluster Flags]
# ETCD_AUTO_COMPACTION_RETENTIO:N=0
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_ADVERTISE_CLIENT_URLS=https://10.62.232.41:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.62.232.41:2380
ETCD_LISTEN_CLIENT_URLS=https://10.62.232.41:2379,https://127.0.0.1:2379
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_LISTEN_PEER_URLS=https://10.62.232.41:2380
ETCD_INITIAL_CLUSTER=etcd1=https://10.62.232.41:2380,etcd2=https://10.62.232.42:2380,etcd3=https://10.62.232.43:2380

# [Proxy Flags]
ETCD_PROXY=off

# [Security flags]
# ETCD_CLIENT_CERT_AUTH=
# ETCD_PEER_CLIENT_CERT_AUTH=
# 指定etcd的公钥证书和私钥
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key
# 指定etcd的Peers通信的公钥证书和私钥
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key

# [Profiling flags]
# ETCD_METRICS={{ etcd_metrics }}
EOF
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
NotifyAccess=all
Restart=always
RestartSec=5s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
EOF

详细配置见etcd configuration

etcd2和etcd3主机需要修改

ETCD_NAME=etcd(2或3)
ETCD_ADVERTISE_CLIENT_URLS=https://10.62.232.(42或43):2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.62.232.(42或43):2380
ETCD_LISTEN_CLIENT_URLS=https://10.62.232.(42或43):2379,https://127.0.0.1:2379
ETCD_LISTEN_PEER_URLS=https://10.62.232.(42或43):2380

3.5 启动etcd:

在各个节点运行

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

只有一个节点启动etcd服务后，etcd会去访问其他两个节点的2380端口，如果一定时间内无法访问，则会服务启动失败。所以应当迅速启动至少两个节点，服务才会成功启动。

3.6 检查集群是否健康:

在任意节点运行

etcdctl \
  --ca-file=/etc/ssl/etcd/ssl/ca.crt \
  --cert-file=/etc/ssl/etcd/ssl/etcd.crt \
  --key-file=/etc/ssl/etcd/ssl/etcd.key \
  --endpoints=https://10.62.232.41:2379,https://10.62.232.42:2379,https://10.62.232.43:2379 \
  cluster-health

看到cluster is healthy的输出，说明etcd集群部署成功。