Graylog

Graylog集群环境搭建

2017-10-26  本文已影响206人  洮生

本次多节点环境部署示例是基于centos7.2进行,由于资源有限,将ES与Graylog和MongoDB部署在同一台server上。以下内容仅供参考,正式生产环境根据需要进行调整。


前置条件

准备三个节点,系统版本为centos7.2。

192.168.2.121      graylogNode1
192.168.2.122      graylogNode2
192.168.2.123      graylogNode3

可以直接使用vagrant创建三个节点,Vagrantfile如下:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|

  (1..3).each do |i|
    config.vm.define "graylogNode#{i}" do |s|
      s.vm.box = "bento/centos-7.2"
      s.vm.hostname = "graylogNode#{i}"
      n = 120 + i
      s.vm.provision :shell, inline: "sed 's/127\.0\.0\.1.*node.*/192\.168\.2\.#{n} node#{i}/' -i /etc/hosts"
      s.vm.network "private_network", ip: "192.168.2.#{n}"
      s.vm.provider "virtualbox" do |v|
        v.cpus = 1
        v.memory = 2048
        v.name = "graylog-node#{i}"
      end
    end
  end

end

启动节点:

$ vagrant up

在每个节点上安装好MongoDB、Elasticsearch、Graylog,具体安装步骤参考上一篇 Graylog安装使用


集群配置

集群配置包括三个部分:配置MongoDB副本集、配置Elasticsearch集群、Graylog多节点配置。

配置MongoDB副本集

(1)指定副本集名称。
方法一:通过配置文件指定
更改每个节点上的MongoDB配置文件/etc/mongod.conf,将bindIp所在的行前加上#进行注释,replication部分添加replSetName: rs0,更改后的配置文件如下:

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile

# network interfaces
net:
  port: 27017
#  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.


#security:

#operationProfiling:

#replication:
replication:
  replSetName: rs0

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

重启服务:

$ sudo systemctl restart mongod.service

方法二:使用mongo --replSet命令行指定
在每个节点上执行命令:

$ mongod --replSet "rs0"

(2)在集群中的一个节点上执行mongo命令行:

$ mongo

初始化副本集
使用本机hostname或IP加端口,如下

$ rs.initiate( {
    _id : "rs0",
    members: [ { _id : 0, host : "192.168.2.121:27017" } ]
 })

查看配置

rs.conf()

配置如下:

{
        "_id" : "rs0",
        "version" : 1,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.2.121:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("59ef0832a5da3378b1487f4e")
        }
}

向副本集中添加成员

rs0:PRIMARY> rs.add("192.168.2.122:27017")
{ "ok" : 1 }
rs0:PRIMARY> rs.add("192.168.2.123:27017")
{ "ok" : 1 }

添加完成之后的配置:

rs0:PRIMARY> rs.config()
{
        "_id" : "rs0",
        "version" : 4,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.168.2.121:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "192.168.2.122:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "192.168.2.123:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("59ef0832a5da3378b1487f4e")
        }
}

查看状态

rs0:PRIMARY> rs.status()

创建graylog数据库,添加graylog用户

rs0:PRIMARY> use graylog
switched to db graylog
rs0:PRIMARY> db.createUser( {
... user: "graylog",
... pwd: "75PN76Db66En",
... roles: [ { role: "readWrite", db: "graylog" } ]
... });
rs0:PRIMARY> db.grantRolesToUser( "graylog" , [ { role: "dbAdmin", db: "graylog" } ])
rs0:PRIMARY> show users
rs0:PRIMARY> db.auth("graylog","75PN76Db66En")

配置Elasticsearch集群

(1)修改每个节点的配置文件

$ sudo vim /etc/elasticsearch/elasticsearch.yml

需要更改的部分如下:

#es集群名称,每个节点中cluster.name要保持一致(建议名称为graylog)
cluster.name: graylog  
#节点名称
node.name: es-node-01
#当前节点IP
network.host: 192.168.2.121
#端口
http.port: 9200
#集群中的主机
discovery.zen.ping.unicast.hosts: ["192.168.2.121", "192.168.2.122", "192.168.2.123"]
#可发现的主节点
discovery.zen.minimum_master_nodes: 2

(2)重启服务

$ sudo systemctl restart elasticsearch.service

(3)查看集群状态

$ curl 'http://192.168.2.121:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 80,
  "active_shards" : 80,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

(4)查看集群中的节点

$ curl 'http://192.168.2.121:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.2.122            4          96   4    0.01    0.04     0.12 mdi       -      es-node-02
192.168.2.121            4          96   5    0.03    0.04     0.06 mdi       *      es-node-01
192.168.2.123            4          97   6    0.04    0.18     0.20 mdi       -      es-node-03

Graylog多节点配置

  1. 打开配置文件进行编辑:
    $ sudo vim /etc/graylog/server/server.conf
    
    (1)设置master节点
    本次环境是将192.168.2.121作为graylog主节点,因此在192.168.2.121上修改配置文件/etc/graylog/server/server.confis_master = true,其他两个节点的配置文件中设置为is_master = false
    (2)修改mongodb连接配置mongodb_uri
    mongodb_uri = mongodb://graylog:75PN76Db66En@192.168.2.121:27017,192.168.2.122:27017,192.168.2.123:27017/graylog?replicaSet=rs0
    
    (3)修改elasticsearch连接配置elasticsearch_hosts
    elasticsearch_hosts = http://192.168.2.121:9200,http://192.168.2.122:9200,http://192.168.2.123:9200
    
    (4)开启web界面web_enable
    web_enable = true
    
    (5)修改web_listen_uri
    #不同的节点不同的IP
    web_listen_uri = http://192.168.2.121:9000/
    
    (6)修改rest_listen_uri
    #不同的节点不同的IP
    rest_listen_uri = http://192.168.2.121:9000/api/
    
  2. 重启服务
    $ sudo systemctl restart graylog-server.service
    
  3. 创建负载均衡器,对graylog配置负载均衡
    本次使用nginx进行负载均衡,安装步骤如下:
    添加yum源:在/etc/yum.repos.d/目录下新建文件nginx.repo,添加如下内容:
    [nginx]
    name=nginx repo
    baseurl=http://nginx.org/packages/mainline/OS/OSRELEASE/$basearch/
    gpgcheck=0
    enabled=1
    
    补充OS换成你的系统,如:“centos”、“rhel”;OSRELEASE换成系统版本,本次的系统环境是centos7,所以写成如下的内容。
    [nginx]
    name=nginx repo
    baseurl=http://nginx.org/packages/mainline/centos/7/$basearch/
    gpgcheck=0
    enabled=1
    
    安装
    $ sudo yum -y install nginx
    
    启动服务$ sudo service nginx start
    配置Nginx
    更改Nginx的配置文件
       server {
       listen       80;
       listen [::]:80 default_server ipv6only=on;
       server_name graylog.example.org;
    
       location / {
           proxy_set_header Host $http_host;
           proxy_set_header X-Forwarded-Host $host;
           proxy_set_header X-Forwarded-Server $host;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Graylog-Server-URL http://$server_name/api;
           proxy_pass http://graylog-web-cluster;
       }
    }
        upstream graylog-web-cluster {
            server 192.168.2.121:9000 max_fails=3 fail_timeout=30s;
            server 192.168.2.122:9000 max_fails=3 fail_timeout=30s;
            server 192.168.2.123:9000 max_fails=3 fail_timeout=30s;
        }
    
    重启服务:sudo nginx -s reload
    此时可以通过访问http://graylog.example.org/查看graylog,可看到已经搭建好的节点信息:
graylog nodes view

Q&A


参考文章

上一篇下一篇

猜你喜欢

热点阅读