2022-06-10
Ansible playbook 自动化部署方案
1. 方案背景
当前Jlite非标部署比较复杂,需要部署的服务比较多,每个服务又有很多配置。一是部署时间比较长、二是部署不够标准(基础软件版本、日志文件配置等不统计)。为了使部署更加简单、方便、快捷,特提供本部署方案。
ansible playbook自动化部署技术目前比较成熟、文档齐全,固本次选择ansible playbook为自动化部署技术方案。
2. Ansible 简介
Ansible是一种常用的自动运维化工具,基于python开发,分布式,无需客户端,轻量级,配置语言采用YAML。
2.1. Ansible的特性:
- 模块化:调用特定的模块,完成特殊的任务。
- Paramiko(python对ssh的实现),PyYaml,jinja2(模块语言)三个关键模块。
- 支持自定义模块,可使用任何编程语言写模块。
- 基于python语言实现。
- 部署简单,基于python和SSH(默认已安装),agentless,无需代理不依赖KPI(无需SSL)。
- 安全,基于OpenSSH
- 幂等性:一个任务执行一次和执行n遍效果一样,不因重复执行带来意外情况。
- 支持playbook编排任务,YAML格式,编排任务,支持丰富的数据结构。
- 较强大的多层解决方案role。
2.2 Ansible的作用目标:
- 自动化部署APP
- 自动化管理配置项
- 自动化的持续交付
- 自动化的云服务管理
3. Playbook简介
playbook是ansible用于配置,部署和管理托管主机剧本,通过playbook的详细描述,执行其中一系列tasks,可以让远程主机达到预期状态。playbook是由一个或多个"play"组成的列表。
也可以说,playbook字面意思及剧本,现实中由演员按剧本表演,在ansible中由计算机进行安装,部署应用,提供对外服务,以及组织计算机处理各种各样的事情
3.1. Playbook 核心元素
-
Hosts
执行的远程主机列表(应用在哪些主机上)
-
Tasks
任务集
-
Variables
内置变量或自定义变量在playbook中调用
-
Templates
可替换模板文件中的变量并实现一些简单逻辑的文件
-
Handlers/Notify
两个为一个组合,由特定条件触发的操作,满足条件执行,不满足不执行。
-
Tags
指定某条任务执行,用于选择运行playbook中的部署代码。
4. 部署组件图
1654850242(1).png结合Ansible Playbook、Docker、Docker Compose、Docker Swarm集群、Nacos功能特点。可以使用如下组件图方案对Jlite实施自动化部署。
4.1. 本地nacos配置信息说明
因为Ansible Playbook并没有提供一个界面配置,所以我们使用Nacos作为所有部署时参数的配置中心。
- 以客户为 worksapce(工作空间),SAAS环境的话,每个SAAS环境为一个worksapce
- 每个服务为一个group
- 一个服务的不同配置为dataId
4.1.1 远程服务器资源
-
远程服务器资源配置信息 remote-hosts.yaml
--- remote_hosts: swarm: # docker swarm 集群的leader服务器ip地址 leader: - 10.10.205.115 # docker swarm 集群的manager服务器ip地址 manager: - 10.10.205.114 # docker swarm 集群的worker服务器ip地址 worker: - 10.10.205.95 - 10.10.205.94 ...
4.1.2 服务部署信息
-
需要部署的服务参数 service-docker-compose-parameter.yaml ,playbook用于生成最终的docker-compose.yaml文件。
--- # 根docker-compose.yaml文件类似 services: #jlite服务 jlite: #版本号 version: 3.7.11 #外发时的镜像文件名 image: osdfo3234234220220611 #端口号 ports: - "1081:1081" deploy: #节点数 replicas: 2 #redis服务 redis: ports: - "6397:6397" password: jsicp #lucene服务 lucene: ports: - "8108:8108" ...
4.1.3 每个服务依赖的配置
每个服务自已依赖的nacos配置不变,只需要要本地配置就好,自动化脚本自动下载后上传到远程部署服务器的nacos上。
5. 服务的 docker-compose.yaml
每个服务的docker-compose.yaml文件由研发提供,可以在里面可以使用参数占位符。参数由前面的service-docker-compose-parameter.yaml定义。
version: '3'
services:
jlite:
image: {{ services.jlite.image }}
container_name: jlite_node_noe
dns_search: .
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- /home/jht/logs/jlite:/home/jht/logs
environment:
- TZ=Asia/Shanghai
ports: {{ services.jlite.ports }}
networks:
- swarm_net
deploy:
mode: replicated
replicas: {{ services.jlite.deploy.replicas }}
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
update_config:
parallelism: 2
delay: 5s
depends_on:
- redis
- lucene
- emqx
- nginx
redis:
image: redis_docker_images_name
container_name: redis_container
environment:
- TZ=Asia/Shanghai
ports: {{ services.redis.ports }}
networks:
- swarm_net
deploy:
mode: replicated
replicas: {{ services.redis.deploy.replicas }}
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
update_config:
parallelism: 2
delay: 5s
6. 安装包
安装包由OA流程走完后,运维工具跑定时任务,定时打镜像到仓库中。
7. 日志文件
可以在docker swarm的集群服务器上共享挂载磁盘的方式,集中在一起。如下
services:
jlite:
volumes:
- /home/docker/logs/jlite:/home/jht/logs
8. 自动化部署流程图
image.png9. ansible playbook 环境安装配置
9.1 获取读取nacos配置文件 playbook脚本
---
- hosts: localhost
remote_user: jht
vars:
- workspace: 93549ceb-522d-4137-b02d-7aa05ab10852
tasks:
- name: login nacos
uri:
url: http://10.10.204.114:8848/nacos/v1/auth/users/login?username=nacos&password=nacos
method: post
return_content: yes
status_code: 200
register: login
- debug:
var: login.json.accessToken
- name: download config
uri:
url: http://10.10.204.114:8848/nacos/v1/cs/configs?show=all&dataId=serviceConfiguration&group=ansible&tenant={{ workspace }}&accessToken={{ login.json.accessToken }}&namespaceId={{ workspace }}
method: get
return_content: yes
status_code: 200
register: config
- debug:
var: config.json.content
- name: write file
file:
path: /home/jht/playbook/templates
state: directory
- name: create file
file:
path: /home/jht/playbook/templates/serviceConfiguration.yaml
state: touch
- name: write file content
blockinfile:
path: /home/jht/playbook/templates/serviceConfiguration.yaml
block: "{{ config.json.content }}"
9.2 安装docker
- name: install required packages
yum:
name: "{{ item }}"
state: present
with_items:
- yum-utils
- device-mapper-persistent-data
- lvm2
- name: add docker repo to /etc/yum.repos.d
shell: yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
register: result
- name: debug
debug:
var: not result.failed
- name: install docker-ce
yum:
name: docker-ce
state: present
when: not result.failed
- name: install docker-ce-cli
yum:
name: docker-ce-cli
state: present
when: not result.failed
- name: install containerd.io
yum:
name: containerd.io
state: present
when: not result.failed
- name: create /etc/docker
file:
path: "{{ docker_config_dir }}"
state: directory
- name: start docker service
systemd:
name: docker
state: started
enabled: true
- name: provide docker-ce configfile
template:
src: daemon.json.j2
dest: /etc/docker/daemon.json
notify: restart docker
9.3 安装docker compose
- name: copy docker-compose-Linux-x86_64 to /usr/local/bin
copy:
src: ../files/docker-compose-linux-x86_64
dest: "{{ docker_compose_dir }}/docker-compose-Linux-x86_64"
- name: install docker-compose
shell: cd "{{ docker_compose_dir }}" && mv docker-compose-Linux-x86_64 docker-compose && chmod +x docker-compose
9.4 安装docker swarm leader
---
- name: Create leader node of docker swarm cluster
hosts: leader
remote_user: jht
gather_facts: yes
become: yes
become_method: sudo
become_user: root
vars:
ip_addr: "{{ ansible_default_ipv4['address'] }}"
tasks:
- name: show ip_add
debug:
var: ip_addr
- name: Ensure docker is running
become: yes
service:
name: docker
state: started
- name: Get docker info
shell: docker info
register: docker_info
- name: show docker_info
debug:
var: docker_info
- name: Create docker swarm leader node
shell: docker swarm init --advertise-addr {{ ip_addr }}
when: "docker_info.stdout.find('Swarm: inactive') != -1"
- name: Get docker swarm manager token
shell: docker swarm join-token -q manager
register: manager_token
- name: Get docker swarm worker token
shell: docker swarm join-token -q worker
register: worker_token
- name: Output tokens
debug:
msg: "{{ manager_token.stdout }} ----------------- {{ worker_token.stdout }}"
- name: Add tokens to dummy host, to be shared between multiple hosts
add_host:
name: swarm_tokens_host
worker_token: "{{ worker_token.stdout }}"
manager_token: "{{ manager_token.stdout }}"
leader_ip_addr: "{{ ip_addr }}"
- name: Show nodes of docker swarm
shell: docker node ls
9.5 设置 docker swarm manager
---
- name: Create manager nodes of docker swarm cluster
hosts: managers
remote_user: jht
become: true
become_method: sudo
become_user: root
vars:
manager_token: "SWMTKN-1-0xwaprlwrnc1ep0839lpuwgl2muq10vxu09forqgh7ojo4hx89-2fx5aapfq3uwq65slca5ph99r"
leader_ip_addr: "10.10.205.115"
tasks:
- name: Ensure docker is running
service:
name: docker
state: started
- name: Get docker info
shell: docker info
register: docker_info
- name: Join as manager node
shell: docker swarm join --token {{ manager_token }} {{ leader_ip_addr }}:2377
when: "docker_info.stdout.find('Swarm: inactive') != -1"
retries: 3
delay: 20
...
9.6 设置 docker swarm workers
---
- name: Create docker swarm worker nodes
gather_facts: yes
hosts: workers
remote_user: jht
become: yes
become_method: sudo
become_user: root
vars:
worker_token: "SWMTKN-1-0xwaprlwrnc1ep0839lpuwgl2muq10vxu09forqgh7ojo4hx89-5yps35hmuo6eda9cncl2vra39"
leader_ip_addr: "10.10.205.115"
tasks:
- name: Ensure docker is running
become: yes
service:
name: docker
state: started
- name: Get docker info
shell: docker info
register: docker_info
- name: Join as worker node
shell: docker swarm join --token {{ worker_token }} {{ leader_ip_addr }}:2377
when: "docker_info.stdout.find('Swarm: inactive') != -1"
retries: 3
delay: 20
...
10. 部署实例
10.1 部署 nginx
-
docker-compose.yaml
version: '3.6' services: nginx: image: nginx:1.17.6-alpine ports: - "80:80" networks: - swarm_net deploy: mode: replicated replicas: 2 restart_policy: condition: on-failure delay: 5s max_attempts: 3 window: 120s update_config: parallelism: 2 delay: 5s networks: swarm_net: driver: overlay
- deploy-nginx-service.yml
---
- name: Deploy services to docker swarm cluster
gather_facts: no
hosts: leader
remote_user: jht
become: true
become_method: sudo
become_user: root
vars:
src_docker_compose: ./docker-compose.yml
dest_docker_compose: ~/docker-compose.yml
stack_name: first_stack
tasks:
- name: Upload docker compose file
template:
src: "{{ src_docker_compose }}"
dest: "{{ dest_docker_compose }}"
- name: Deploy services
shell: docker stack deploy --with-registry-auth --prune --compose-file {{ dest_docker_compose }} {{ stack_name }}
- name: Pause for 10 seconds to wait for services being running
pause:
seconds: 10
- name: Ensure services deployed
shell: docker service ls
register: service_output
- name: Output service list
debug:
msg: "{{ service_output.stdout_lines }}"
- name: Show stacks
shell: docker stack ls
...