ELK(ElasticSearch、Logstash和Kiaba
2018-07-09 本文已影响0人
心疼你萌萌哒
=============基础环境安装(需要epel源)================
[on master 10.0.0.20 slave 10.0.0.21]
1）下载并安装GPG Key
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2）添加yum仓库
vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3）安装elasticsearch
yum install -y elasticsearch
4) 安装Redis nginx java
yum install -y redis nginx java
安装完java后，检测
java -version
=============[on master 10.0.0.20]====================
systemctl stop firewalld && setenforce 0
hostnamectl set-hostname master
cat >> /etc/hosts <<eof
10.0.0.20 master
10.0.0.21 slave
eof
1）配置修改配置文件
mkdir -p /data/es-data
vim /etc/elasticsearch/elasticsearch.yml //将里面内容清空，配置下面内容
cluster.name: huanqiu # 组名（同一个组，组名必须一致）
node.name: master # 节点名称，建议和主机名一致
path.data: /data/es-data # 数据存放的路径
path.logs: /var/log/elasticsearch/ # 日志存放的路径
bootstrap.mlockall: true # 锁住内存，不被使用到交换分区去
network.host: 0.0.0.0 # 网络设置
http.port: 9200 # 端口
2）启动并查看
chown -R elasticsearch.elasticsearch /data/
systemctl start elasticsearch
systemctl status elasticsearch
CGroup: /system.slice/elasticsearch.service
└─3005 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSI...
注意：上面可以看出elasticsearch设置的内存最小256m，最大1g
netstat -antlp |egrep "9200|9300"
tcp6 0 0 :::9200 :::* LISTEN 3005/java
tcp6 0 0 :::9300 :::* LISTEN 3005/java
a）安装head插件
[root@elk-node1 src]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
[root@elk-node1 src]# systemctl restart elasticsearch
b）安装kopf监控插件
[root@elk-node1 src]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
[root@elk-node1 src]# systemctl restart elasticsearch
=====================[on slave 10.0.0.21]===============
systemctl stop firewalld && setenforce 0
hostnamectl set-hostname slave
cat >> /etc/hosts <<eof
10.0.0.20 master
10.0.0.21 slave
eof
1）配置修改配置文件
mkdir -p /data/es-data
cat /etc/elasticsearch/elasticsearch.yml
cluster.name: huanqiu
node.name: slave
path.data: /data/es-data
path.logs: /var/log/elasticsearch/
bootstrap.mlockall: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.1.160", "192.168.1.161"]
2) 修改权限配置
chown -R elasticsearch.elasticsearch /data/
3) 启动服务
[root@slave src]# systemctl start elasticsearch
[root@slave src]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2016-09-28 16:49:41 CST; 1 weeks 3 days ago
Docs: http://www.elastic.co
Process: 17798 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 17800 (java)
CGroup: /system.slice/elasticsearch.service
└─17800 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFra...
Oct 09 13:42:22 slave elasticsearch[17800]: [2016-10-09 13:42:22,295][WARN
4) 查看端口
netstat -antlp|egrep "9200|9300"
tcp6 0 0 :::9200 :::* LISTEN 2928/java
tcp6 0 0 :::9300 :::* LISTEN 2928/java
tcp6 0 0 127.0.0.1:48200 127.0.0.1:9300 TIME_WAIT -
tcp6 0 0 ::1:41892 ::1:9300 TIME_WAIT -
a）安装head插件
[root@elk-node1 src]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
[root@elk-node1 src]# systemctl restart elasticsearch
b）安装kopf监控插件
[root@elk-node1 src]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
[root@elk-node1 src]# systemctl restart elasticsearch
=================访问两个插件：================
http://10.0.0.20:9200/_plugin/head/
http://10.0.0.20:9200/_plugin/kopf/#!/cluster
http://10.0.0.21:9200/_plugin/head/
http://10.0.0.21:9200/_plugin/kopf/#!/cluster
====Logstash安装配置（这个在客户机上是要安装的。[on master 10.0.0.20 slave 10.0.0.21]）===
基础环境安装（客户端安装logstash，收集到的数据写入到elasticsearch里，就可以登陆logstash界面查看到了）
1）下载并安装GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2）添加yum仓库
[root@hadoop-node1 ~]# vim /etc/yum.repos.d/logstash.repo
[logstash-2.1]
name=Logstash repository for 2.1.x packages
baseurl=http://packages.elastic.co/logstash/2.1/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3）安装logstash
[root@elk-node1 ~]# yum install -y logstash
4）logstash启动
[root@elk-node1 ~]# systemctl start elasticsearch
[root@elk-node1 ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2016-11-07 18:33:28 CST; 3 days ago
Docs: http://www.elastic.co
Main PID: 8275 (java)
CGroup: /system.slice/elasticsearch.service
└─8275 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFrac...
===================Kibana安装配置=====================
1）kibana的安装：
[root@master ~]# cd /usr/local/src
[root@master src]# wget https://download.elastic.co/kibana/kibana/kibana-4.3.1-linux-x64.tar.gz
[root@master src]# tar zxf kibana-4.3.1-linux-x64.tar.gz
[root@master src]# mv kibana-4.3.1-linux-x64 /usr/local/kibana
2）修改配置文件：
[root@slave config]# pwd
/usr/local/kibana/config
[root@master config]# cp kibana.yml kibana.yml.bak
[root@master config]# vim kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.0.0.20:9200"
kibana.index: ".kibana"
[root@master ~]# yum -y install screen
[root@master ~]# screen -S "zz" ＃这样就另开启了一个终端窗口
[root@master config]# /usr/local/kibana/bin/kibana [ctrl+Z 运行到后台]
log [04:44:50.393] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready
log [04:44:50.428] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [04:44:50.448] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [04:44:50.458] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready
在起一个终端
[root@maste终端2 ~]# screen -ls
There is a screen on:
15041.zz pts-0.elk-node1 (Detached)
1 Socket in /var/run/screen/S-root.
[root@tivf18 root]# screen -r zz
3）收集java日志，其中包含上面讲到的日志收集
[root@maste终端2 ~]# vim file.conf
input {
file {
path => "/var/log/messages"
type => "system"
start_position => "beginning"
}
}
input {
file {
path => "/var/log/elasticsearch/huanqiu.log"
type => "es-error"
start_position => "beginning" \\死的
}
}
output {
if [type] == "system"{
elasticsearch {
hosts => ["10.0.0.20:9200"]
index => "system-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error"{
elasticsearch {
hosts => ["10.0.0.20:9200"]
index => "es-error-%{+YYYY.MM.dd}"
}
}
}
[root@maste终端2 ~]# /opt/logstash/bin/logstash -f file.conf & \\放后台
访问kibana：http://10.0.0.20:5601/
在settings中填写 system-* (file.conf里定义的system-%{+YYYY.MM.dd})--》create
a)查看日志登陆，需要点击“Discover”-->"message",点击它后面的“add”
注意：
需要右边查看日志内容时带什么属性，就在左边点击相应属性后面的“add”
b)添加新的日志采集项，点击Settings->+Add New，比如添加
system系统日志。注意后面的*不要忘了。
c)如果打开kibana查看日志，发现没有日志内容，出现“No results found”，如下图所示，这说明要查看的日志在当前时间没有日志信息
输出，可以点击右上角的时间钟来调试日志信息的查看。
日志配置实例
修改nginx的配置文件，分别在nginx.conf的http和server配置区域添加下面内容：
http里追加下面:
log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"client":"$remote_addr",'
'"url":"$uri",'
'"status":"$status",'
'"domain":"$host",'
'"host":"$server_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"referer": "$http_referer",'
'"ua": "$http_user_agent"'
'}';
server追加下面:
access_log /var/log/nginx/access_json.log json;
启动nginx服务：
[root@master ~]# systemctl start nginx
[root@master ~]# systemctl status nginx
[root@master ~]# cat file.conf (基于端口要编辑 /etc/rsyslog.conf)
input {
file {
path => "/var/log/messages"
type => "system"
start_position => "beginning"
}
file {
path => "/var/log/elasticsearch/huanqiu.log"
type => "es-error"
start_position => "beginning"
codec => multiline {
pattern => "^\["
negate => true
what => "previous"
}
}
file {
path => "/var/log/nginx/access_json.log"
codec => json
start_position => "beginning"
type => "nginx-log"
}
syslog {
type => "system-syslog"
host => "10.0.0.20"
port => "514"
}
}
output {
if [type] == "system"{
elasticsearch {
hosts => ["10.0.0.20:9200"]
index => "system-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error"{
elasticsearch {
hosts => ["10.0.0.20:9200"]
index => "es-error-%{+YYYY.MM.dd}"
}
}
if [type] == "nginx-log"{
elasticsearch {
hosts => ["10.0.0.20:9200"]
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
if [type] == "system-syslog"{
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "system-syslog-%{+YYYY.MM.dd}"
}
}
}
[root@master ~]# /opt/logstash/bin/logstash -f file.conf --configtest
在开一个终端(screen 也可)
[root@master ~]# /usr/local/kibana/bin/kibana
基于端口收集系统日志
格式
[root@elk-node1 ~]# cat syslog.conf
input {
syslog {
type => "system-syslog"
host => "10.0.0.21"
port => "514"
}
}
output {
stdout {
codec => "rubydebug"
}
}
对上面的采集文件进行执行：
[root@elk-node1 ~]# /opt/logstash/bin/logstash -f syslog.conf
重新开启一个窗口，查看服务是否启动：
[root@master ~]# netstat -ntlp|grep 514
tcp6 0 0 192.168.1.160:514 :::* LISTEN 17842/java
[root@master ~]# vim /etc/rsyslog.conf
#*.* @@remote-host:514 【在此行下面添加如下内容】
*.* @192.168.1.160 客户端上的配置
[root@master ~]# systemctl restart rsyslog
回到原来的窗口(即上面采集文件的执行终端)，就会出现数据：
