一、背景

mongoDB主从数据同步
主库数据小于从库，从库回滚到跟主库同步状态

image.png

从上可知，假如原来主从数据同步出了问题，主库数据量远大于从库数据量，而你并没有察觉，某次主、从、仲裁进程全部挂了

如果先起原来从库和仲裁，将原来从库选举为现在PRIMARY，然后再启原来的主库成为现在的从库，那么就会导致原来的主库也就是现在的从库数据发生回滚

显然这不是你想看到的的情况，所以务必先启动原来的主库，让其也成为现在的主库，避免此类问题的发生！

二、mongoDB重启playbook

针对primary-secondary-arbiter架构，mongoDB的实例重启：
1.从arbiter日志中获取之前主节点IP
2.启动primary节点mongo进程
3.启动arbiter节点的mongo进程
4.启动secondary节点mongo进程

$ cat hosts

[primary_secondary]
xx.xx.xx.xx    ansible_ssh_host=xx.xx.xx.xx  ansible_ssh_pass=XXX
xx.xx.xx.xx    ansible_ssh_host=xx.xx.xx.xx  ansible_ssh_pass=XXX

[arbiter]
xx.xx.xx.xx    ansible_ssh_host=xx.xx.xx.xx  ansible_ssh_pass=XXX

[all:vars]
ansible_ssh_extra_args='-o StrictHostKeyChecking=no'
ansible_hosts_dir='/path/to/'
instance_name='XXX'
instance_port='XXX'

$ cat start_primary-secondary-arbiter.yml

# 从仲裁日志中获取主节点ip，如果获取值为空，退出本启动程序
- hosts: arbiter[0]
  gather_facts: false
  tasks:
    - name: "arbiter"
      block:
        - name: "print mongo instance name"
          debug:
            msg: "#################### {{ instance_name }} ####################"

        - name: "get mongo primary ip from arbiter log"
          shell: |
            mongo_primary_ip=`grep -w "now in state PRIMARY" /path/to/mongod.log | awk 'END {print}' | awk -F "Member" '{print $2}' |awk -F ":" '{print $1}'`
            if [ "${mongo_primary_ip}" == "" ]; then
              echo "无法从仲裁日志中获取主节点ip，请检查！"
              exit 1
            else
              echo "${mongo_primary_ip}"
            fi
          register: mongo_primary_ip

        - name: "print mongo primary ip"
          debug:
            msg: "{{ mongo_primary_ip.stdout_lines }}"


# 从本机远程调用主节点启动脚本，并判断是否启动成功，如果失败，退出本启动程序
- hosts: localhost
  gather_facts: false
  tasks:
    - name: "localhost"
      block:
        - name: "start primary"
          shell: ansible -i {{ ansible_hosts_dir }}/hosts "{{ hostvars[groups['arbiter'][0]].mongo_primary_ip.stdout_lines[0] }}" -m shell -a "sh /path/to/start_mongo.sh"

        - name: "check if primary is started"
          shell: |
            if ansible -i {{ ansible_hosts_dir }}/hosts "{{ hostvars[groups['arbiter'][0]].mongo_primary_ip.stdout_lines[0] }}" -m shell -a "ps aux | grep mongod | grep -w '{{ instance_port }}' | grep -v 'grep'"; then
              echo "主节点启动成功！"
            else
              echo "主节点启动失败，退出实例启动程序，请检查！"
              exit 1
            fi
          register: primary_start_result

        - name: "print primary start result"
          debug:
            msg: "{{ primary_start_result.stdout_lines }}"


# 启动仲裁节点
- hosts: arbiter
  gather_facts: false
  tasks:
    - name: "arbiter"
      block:
        - name: "start arbiter"
          shell: sh /path/to/start_mongo.sh


# 启动从节点
- hosts: primary_secondary
  gather_facts: false
  tasks:
    - name: "{{ instance_name }}"
      block:
        - name: "start secondary"
          shell: >
            if ! ps aux | grep "mongod" | grep -w "{{ instance_name }}" | grep -w "{{ instance_port }}" | grep -v "grep" > /dev/null 2>&1; then
              sh /path/to/start_mongo.sh  
            fi

$ ansible-playbook -i hosts start_primary-secondary-arbiter.yml

三、参考

mongoDB副本集 stateStr状态说明
https://www.jianshu.com/p/7f196c22af43