搭建ELK日志分析系统
2020-03-09 本文已影响0人
chuan_bai
ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成。官方网站：https://www.elastic.co/products
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个完全开源的工具，他可以对你的日志进行收集、过滤，并将其存储供以后使用（如，搜索）。
Kibana 也是一个开源和免费的工具，它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。
接下来使用elasticsearch 5.6.4，kibana5.2.0 ，logstash5.6.3三个版本进行搭建
一、安装elasticsearch
- 修改配置文件elasticsearch.yml
cluster.name: my-elk
node.name: node-1
#允许外网访问
network.host: 0.0.0.0
#访问地址端口号，默认为8200
http.port: 9200
- 解压安装包后启动elasticsearch出现错误，提示说不能用root用户启动
oot@iZbp12jkgn61msxjvtrypsZ:/opt/elK/elasticsearch-5.6.4# ./bin/elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-03-09T02:55:35,488][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.4.jar:5.6.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.4.jar:5.6.4]
- 新增用户
#新增user用户组
groupadd user
#在user用户下新增bc用户
useradd -g user bc
#为bc用户配置密码
passwd bc
- 为新增的bc用户配置权限，打开/etc/sudoers配置文件，新增一条bc用户的信息
# User privilege specification
root ALL=(ALL:ALL) ALL
bc ALL=(ALL:ALL) ALL
- 改变elasticsearch文件夹所有者到bc
sudo chown -R bc:user elasticsearch-5.6.4
- 切换到bc用户，启动elasticsearch
- 若出现max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]错误
则打开vi /etc/security/limits.conf文件，新增两行配置
bc hard nofile 65536
bc soft nofile 65536
- 若出现max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]错误
则打开vi /etc/sysctl.conf ，新增配置
vm.max_map_count=655360
并执行命令
sysctl -p
- 接下来启动elasticsearch
./bin/elasticsearch -d
- 用curl命令测试后，表示elasticsearch已经成功安装
root@iZbp12jkgn61msxjvtrypsZ:/opt/elk/elasticsearch-5.6.4# curl 127.0.0.1:9200
{
"name" : "node-1",
"cluster_name" : "my-elk",
"cluster_uuid" : "hmggWjhpQJmlLcx3IrWsEQ",
"version" : {
"number" : "5.6.4",
"build_hash" : "8bbedf5",
"build_date" : "2017-10-31T18:55:38.105Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
二、安装logstash
- 在config文件夹下 新建logstash.conf文件，写入配置
input {
file {
type => "log"
path => "/opt/logs/*.log"
start_position => beginning
}
}
output {
stdout {
codec => rubydebug { }
}
elasticsearch {
hosts => "localhost:9200"
index => "log-%{+YYYY.MM.dd}"
}
}
iutput {
stdout {
codec => rubydebug { }
}
elasticsearch {
hosts => "localhost:9200"
index => "log-%{+YYYY.MM.dd}"
}
}
2.启动后报jvm错误,判断可能是服务器内存不够
[1] 29833
root@iZbp12jkgn61msxjvtrypsZ:/opt/elk/logstash-5.6.3# OpenJDK 64-Bit Server VM warning: Option UseParNewGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d4cc0000, 724828160, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/elk/logstash-5.6.3/hs_err_pid29833.log
- 打开conf下的jvm.options文件修改默认分配的jvm空间把1g改为512m
4.后台启动命令
./bin/logstash -f ./config/logstash.conf &
三、安装kibana
- 配置文件
#请求数据指向的elasticsearch服务器
elasticsearch.url: "http://localhost:9200"
#指定本机ip让外部能访问
server.host: 0.0.0.0
# 端口
server.port: 5601
2.启动
./kibana
