ETCD TLS 配置的坑-白红宇

ETCD TLS 配置的坑

阅读量：6250 次

发布时间：2019-06-22

本文共 4849 字，大约阅读时间需要 16 分钟。

一、环境准备

环境总共 3 台虚拟机，系统为centos7，1个 master，2 个 etcd 节点，master 同时也作为 node 负载 pod，在分发证书等阶段将在另外一台主机上执行，该主机对集群内所有节点配置了 ssh 秘钥登录

IP	节点
192.168.0.153	master、node、etcd
192.168.0.154	master、node、etcd
192.168.0.164	master、node、etcd

由于 Etcd 和 Kubernetes 全部采用 TLS 通讯，所以先要生成 TLS 证书，证书生成工具采用

二、证书相关处理

1、证书说明

证书名称	配置文件	用途
etcd-root-ca.pem	etcd-root-ca-csr.json	etcd 根 CA 证书
etcd.pem	etcd-gencert.json、etcd-csr.json	etcd 集群证书

2、CFSSL 工具安装

首先下载 cfssl，并给予可执行权限，然后扔到 PATH 目录下

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64chmod +x cfssl_linux-amd64 cfssljson_linux-amd64mv cfssl_linux-amd64 /usr/local/bin/cfsslmv cfssljson_linux-amd64 /usr/local/bin/cfssljson

Etcd 证书生成所需配置文件如下:

etcd-root-ca-csr.json

{   "key": {     "algo": "rsa",     "size": 4096   },   "names": [     {       "O": "etcd",       "OU": "etcd Security",       "L": "Beijing",       "ST": "Beijing",       "C": "CN"     }   ],   "CN": "etcd-root-ca"}

etcd-gencert.json

{  "signing": {    "default": {        "usages": [          "signing",          "key encipherment",          "server auth",          "client auth"        ],        "expiry": "87600h"    }  }}

etcd-csr.json

{  "key": {    "algo": "rsa",    "size": 4096  },  "names": [    {      "O": "etcd",      "OU": "etcd Security",      "L": "Beijing",      "ST": "Beijing",      "C": "CN"    }  ],  "CN": "etcd",  "hosts": [    "127.0.0.1",    "localhost",    "192.168.0.153",    "192.168.0.154",    "192.168.0.164",    "master",    "node1",    "node2"  ]}

最后生成 Etcd 证书

cfssl gencert --initca=true etcd-root-ca-csr.json | cfssljson --bare etcd-root-cacfssl gencert --ca etcd-root-ca.pem --ca-key etcd-root-ca-key.pem --config etcd-gencert.json etcd-csr.json | cfssljson --bare etcd

生成的证书列表如下

三、部署 HA ETCD

安装前准备

关闭 selinux, setenforce 0

关闭防火墙， systemctl stop firewalld; iptables -F

ntpdate 时间同步

ntpdate time1.aliyun.com

1、安装 Etcd

以下操作都是在master节点上操作

ETCD 直接采用 rpm 安装，RPM 可以从获取 spec 文件自己 build，或者直接从搜索

#下载 rpm包wget ftp://195.220.108.108/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/e/etcd-3.2.7-1.fc28.x86_64.rpm#分发并安装I="192.168.0.153 192.168.0.154 192.168.0.164"for IP in $I; do    scp etcd-3.2.7-1.fc28.x86_64.rpm root@$IP:~    ssh root@$IP rpm -ivh etcd-3.2.7-1.fc28.x86_64.rpmdone

2、分发证书

I="192.168.0.153 192.168.0.154 192.168.0.164"for IP in $I; do    ssh root@$IP mkdir /etc/etcd/ssl/    scp *.pem root@$IP:/etc/etcd/ssl/    ssh root@$IP chown -R etcd:etcd /etc/etcd/ssl/    ssh root@$IP chmod -R 755 /etc/etcd/done

3、修改配置

rpm 安装好以后直接修改 /etc/etcd/etcd.conf 配置文件即可，其中单个节点配置如下(其他节点只是名字和 IP 不同)

# [member]

ETCD_NAME=etcd0

ETCD_DATA_DIR="/var/lib/etcd/etcd0.etcd"

ETCD_WAL_DIR="/var/lib/etcd/wal"

ETCD_SNAPSHOT_COUNT="100"

ETCD_HEARTBEAT_INTERVAL="100"

ETCD_ELECTION_TIMEOUT="1000"

ETCD_LISTEN_PEER_URLS="https://192.168.0.153:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.0.153:2379,http://127.0.0.1:2379"

ETCD_MAX_SNAPSHOTS="5"

ETCD_MAX_WALS="5"

#ETCD_CORS=""

# [cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.153:2380"

# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."

ETCD_INITIAL_CLUSTER="etcd0=https://192.168.0.153:2380,etcd1=https://192.168.0.154:2380,etcd2=https://192.168.0.164:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.153:2379"

#ETCD_DISCOVERY=""

#ETCD_DISCOVERY_SRV=""

#ETCD_DISCOVERY_FALLBACK="proxy"

#ETCD_DISCOVERY_PROXY=""

#ETCD_STRICT_RECONFIG_CHECK="false"

#ETCD_AUTO_COMPACTION_RETENTION="0"

# [proxy]

#ETCD_PROXY="off"/

#ETCD_PROXY_FAILURE_WAIT="5000"

#ETCD_PROXY_REFRESH_INTERVAL="30000"

#ETCD_PROXY_DIAL_TIMEOUT="1000"

#ETCD_PROXY_WRITE_TIMEOUT="5000"

#ETCD_PROXY_READ_TIMEOUT="0"

# [security]

ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"

ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"

ETCD_CLIENT_CERT_AUTH="true"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"

ETCD_AUTO_TLS="true"

ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"

ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"

ETCD_PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"

ETCD_PEER_AUTO_TLS="true"

# [logging]

#ETCD_DEBUG="false"

# examples for -log-package-levels etcdserver=WARNING,security=DEBUG

#ETCD_LOG_PACKAGE_LEVELS=""

node 节点要修改的地方：

ETCD_NAMEETCD_LISTEN_PEER_URLS ETCD_LISTEN_CLIENT_URLSETCD_INITIAL_ADVERTISE_PEER_URLSETCD_ADVERTISE_CLIENT_URLS

4、启动验证

配置修改后在每个节点进行启动即可，注意，Etcd 各个节点间必须保证时钟同步，否则会造成启动失败等错误

systemctl daemon-reloadsystemctl start etcdsystemctl enable etcd

启动成功后验证节点状态

export ETCDCTL_API=3etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.0.153:2379,https://192.168.0.154:2379,https://192.168.0.164:2379 endpoint health