ElasticSearch

ElasticSearch-beats

2017-11-17  本文已影响204人  壹点零

下载

https://www.elastic.co/downloads/beats/

1、metricbeat

支持的服务列表

Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、Redis、System、Zookeeper

上传并解压缩metricbeat-5.4.0-linux-x86_64.tar.gz

tar -zxvf metricbeat-5.4.0-linux-x86_64.tar.gz

配置metricbeat，可以参考metricbeat.full.yml中的配置。

vim /usr/local/metricbeat-5.4.0-linux-x86_64/metricbeat.yml

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "123456"

name:"metricbeat-1-103"
tags: ["service-X", "web-tier"]
fields:
  env: dev

检查配置是否正确

cd /usr/local/metricbeat-5.4.0-linux-x86_64/
./metricbeat -c metricbeat.yml -configtest

创建mapping模板，因为metricbeat第一次启动时候，会把mapping自动建立在es上，所以不用手工创建了。如果直接输出到logstash，需要手工创建

curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.101:9200/_template/metricbeat' -d@/usr/local/metricbeat-5.4.0-linux-x86_64/metricbeat.template.json

启动

sudo /usr/local/metricbeat-5.4.0-linux-x86_64/metricbeat start

报错处理，必须文件所属于root

metricbeat2017/05/31 02:21:09.305375 beat.go:339: CRIT Exiting: error loading config file: config file ("metricbeat.yml") must be owned by the beat user (uid=0) or root
Exiting: error loading config file: config file ("metricbeat.yml") must be owned by the beat user (uid=0) or root

chown -R root /usr/local/metricbeat-5.4.0-linux-x86_64

导入kibana dashboard

cd /usr/local/metricbeat-5.4.0-linux-x86_64/

在线安装

./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456

离线安装

./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456

2、heartbeat

支持的协议

ICMP (v4 and v6) 、TCP、HTTP

上传并解压缩heartbeat-5.4.0-linux-x86_64.tar.gz

tar -zxvf heartbeat-5.4.0-linux-x86_64.tar.gz

配置metricbeat，可以参考heartbeat.full.yml中的配置。

vim /usr/local/heartbeat-5.4.0-linux-x86_64/heartbeat.yml

###################### Heartbeat #######################

# Configure monitors
heartbeat.monitors:
- type: http

  # List or urls to query
  urls: ["http://192.168.1.101"]

  # Configure task schedule
  schedule: '@every 10s'
  # 与java的 quartz cron表达式一致：秒 分 时 日 月 周 年
  #schedule: '*/10 * * * * * *'

  # Total test connection and data exchange timeout
  #timeout: 16s

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "123456"

检查配置是否正确

cd /usr/local/heartbeat-5.4.0-linux-x86_64/
./heartbeat -c heartbeat.yml -configtest

启动

/usr/local/heartbeat-5.4.0-linux-x86_64/heartbeat start

导入kibana dashboard

cd /usr/local/heartbeat-5.4.0-linux-x86_64/

在线安装

./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456

离线安装

./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456

进入kibana查看：

http://192.168.1.102:5601/

其他配置模块

heartbeat.monitors:
- type: icmp
  schedule: '*/5 * * * * * *'
  hosts: ["myhost"]
- type: tcp
  schedule: '@every 5s'
  hosts: ["myhost:7"]  # default TCP Echo Protocol
  check.send: "Check"
  check.receive: "Check"
- type: http
  schedule: '@every 5s'
  urls: ["http://localhost:80/service/status"]
  check.response.status: 200
heartbeat.scheduler:
  limit: 10

3、packetbeat

支持协议列表

ICMP (v4 and v6)、DNS、HTTP、AMQP 0.9.1、Cassandra、Mysql、PostgreSQL、Redis、Thrift-RPC、MongoDB、Memcache

上传并解压缩packetbeat-5.4.0-linux-x86_64.tar.gz

tar -zxvf packetbeat-5.4.0-linux-x86_64.tar.gz

配置packetbeat，可以参考packetbeat.full.yml中的配置。

vim /usr/local/packetbeat-5.4.0-linux-x86_64/packetbeat.yml

packetbeat.interfaces.device: any
packetbeat.protocols.http:
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002, 9100l]

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "123456"

检查配置是否正确

cd /usr/local/packetbeat-5.4.0-linux-x86_64/
chown -R root *
./packetbeat -c packetbeat.yml -configtest

启动

sudo /usr/local/packetbeat-5.4.0-linux-x86_64/packetbeat start

导入kibana dashboard

cd /usr/local/packetbeat-5.4.0-linux-x86_64/

在线安装

./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456

离线安装

./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456

4、filebeat

上传并解压缩filebeat-5.4.0-linux-x86_64.tar.gz

tar -zxvf filebeat-5.4.0-linux-x86_64.tar.gz

配置filebeat，可以参考filebeat.full.yml中的配置。

vim /usr/local/filebeat-5.4.0-linux-x86_64/filebeat.yml

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /usr/local/heartbeat-5.4.0-linux-x86_64/logs/heartbeat
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  //如果下一行没有匹配上，则属于前一行
  #multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

检查配置是否正确

cd /usr/local/filebeat-5.4.0-linux-x86_64/filebeat -c filebeat.yml -configtest

启动

/usr/local/filebeat-5.4.0-linux-x86_64/filebeat start

导入kibana dashboard

cd /usr/local/filebeat-5.4.0-linux-x86_64

在线安装

./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456

离线安装

./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456

上一篇 下一篇

猜你喜欢

热点阅读