ElasticSearch-beats
下载
https://www.elastic.co/downloads/beats/
1、metricbeat
支持的服务列表
Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、Redis、System、Zookeeper
上传并解压缩metricbeat-5.4.0-linux-x86_64.tar.gz
tar -zxvf metricbeat-5.4.0-linux-x86_64.tar.gz
配置metricbeat，可以参考metricbeat.full.yml中的配置。
vim /usr/local/metricbeat-5.4.0-linux-x86_64/metricbeat.yml
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "123456"
name:"metricbeat-1-103"
tags: ["service-X", "web-tier"]
fields:
env: dev
检查配置是否正确
cd /usr/local/metricbeat-5.4.0-linux-x86_64/
./metricbeat -c metricbeat.yml -configtest
创建mapping模板，因为metricbeat第一次启动时候，会把mapping自动建立在es上，所以不用手工创建了。如果直接输出到logstash，需要手工创建
curl -H 'Content-Type: application/json' -XPUT 'http://192.168.1.101:9200/_template/metricbeat' -d@/usr/local/metricbeat-5.4.0-linux-x86_64/metricbeat.template.json
启动
sudo /usr/local/metricbeat-5.4.0-linux-x86_64/metricbeat start
报错处理，必须文件所属于root
metricbeat2017/05/31 02:21:09.305375 beat.go:339: CRIT Exiting: error loading config file: config file ("metricbeat.yml") must be owned by the beat user (uid=0) or root
Exiting: error loading config file: config file ("metricbeat.yml") must be owned by the beat user (uid=0) or root
chown -R root /usr/local/metricbeat-5.4.0-linux-x86_64
导入kibana dashboard
cd /usr/local/metricbeat-5.4.0-linux-x86_64/
在线安装
./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456
离线安装
./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456
2、heartbeat
支持的协议
ICMP (v4 and v6) 、TCP、HTTP
上传并解压缩heartbeat-5.4.0-linux-x86_64.tar.gz
tar -zxvf heartbeat-5.4.0-linux-x86_64.tar.gz
配置metricbeat，可以参考heartbeat.full.yml中的配置。
vim /usr/local/heartbeat-5.4.0-linux-x86_64/heartbeat.yml
###################### Heartbeat #######################
# Configure monitors
heartbeat.monitors:
- type: http
# List or urls to query
urls: ["http://192.168.1.101"]
# Configure task schedule
schedule: '@every 10s'
# 与java的 quartz cron表达式一致：秒 分 时 日 月 周 年
#schedule: '*/10 * * * * * *'
# Total test connection and data exchange timeout
#timeout: 16s
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "123456"
检查配置是否正确
cd /usr/local/heartbeat-5.4.0-linux-x86_64/
./heartbeat -c heartbeat.yml -configtest
启动
/usr/local/heartbeat-5.4.0-linux-x86_64/heartbeat start
导入kibana dashboard
cd /usr/local/heartbeat-5.4.0-linux-x86_64/
在线安装
./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456
离线安装
./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456
进入kibana查看：
http://192.168.1.102:5601/
其他配置模块
heartbeat.monitors:
- type: icmp
schedule: '*/5 * * * * * *'
hosts: ["myhost"]
- type: tcp
schedule: '@every 5s'
hosts: ["myhost:7"] # default TCP Echo Protocol
check.send: "Check"
check.receive: "Check"
- type: http
schedule: '@every 5s'
urls: ["http://localhost:80/service/status"]
check.response.status: 200
heartbeat.scheduler:
limit: 10
3、packetbeat
支持协议列表
ICMP (v4 and v6)、DNS、HTTP、AMQP 0.9.1、Cassandra、Mysql、PostgreSQL、Redis、Thrift-RPC、MongoDB、Memcache
上传并解压缩packetbeat-5.4.0-linux-x86_64.tar.gz
tar -zxvf packetbeat-5.4.0-linux-x86_64.tar.gz
配置packetbeat，可以参考packetbeat.full.yml中的配置。
vim /usr/local/packetbeat-5.4.0-linux-x86_64/packetbeat.yml
packetbeat.interfaces.device: any
packetbeat.protocols.http:
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
ports: [80, 8080, 8000, 5000, 8002, 9100l]
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["192.168.1.101:9200","192.168.1.102:9200","192.168.1.103:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "123456"
检查配置是否正确
cd /usr/local/packetbeat-5.4.0-linux-x86_64/
chown -R root *
./packetbeat -c packetbeat.yml -configtest
启动
sudo /usr/local/packetbeat-5.4.0-linux-x86_64/packetbeat start
导入kibana dashboard
cd /usr/local/packetbeat-5.4.0-linux-x86_64/
在线安装
./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456
离线安装
./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456
4、filebeat
上传并解压缩filebeat-5.4.0-linux-x86_64.tar.gz
tar -zxvf filebeat-5.4.0-linux-x86_64.tar.gz
配置filebeat，可以参考filebeat.full.yml中的配置。
vim /usr/local/filebeat-5.4.0-linux-x86_64/filebeat.yml
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
- /usr/local/heartbeat-5.4.0-linux-x86_64/logs/heartbeat
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ["^DBG"]
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ["^ERR", "^WARN"]
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: [".gz$"]
# Optional additional fields. These field can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
//如果下一行没有匹配上，则属于前一行
#multiline.negate: true
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
检查配置是否正确
cd /usr/local/filebeat-5.4.0-linux-x86_64/filebeat -c filebeat.yml -configtest
启动
/usr/local/filebeat-5.4.0-linux-x86_64/filebeat start
导入kibana dashboard
cd /usr/local/filebeat-5.4.0-linux-x86_64
在线安装
./scripts/import_dashboards -es http://192.168.1.101:9200 -user elastic -pass 123456
离线安装
./scripts/import_dashboards -file beats-dashboards-5.4.0.zip -es http://192.168.1.101:9200 -user elastic -pass 123456
