6、filebeat->kafka->logstash->es

一、部署忽略
参考
https://www.jianshu.com/p/a6084c9450c6

二、开启system模块
filebeat modules enable system

三、配置filebeat

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log

- type: filestream
  enabled: false
  paths:
    - /var/log/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:
  host: "192.168.18.13:5601"
setup.template.overwrite: true
setup.template.enabled: true
setup.ilm.enabled: false

output.kafka:
  hosts: ["192.168.18.15:9092","192.168.18.16:9092"]
  topic: "linux"  

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

四、配置logsatsh
1、input.conf

input {
    kafka {
        bootstrap_servers => "192.168.18.15:9092,192.168.18.16:9092"
        topics => ["linux"]
        add_field => { type_name => "linux_system" }
        consumer_threads => 5
        codec => json               
    }
}

2、output.conf

output {
        elasticsearch {
            hosts => ["192.168.18.13:9200"]
            index => "test-os-linux-%{+YYYY.MM.dd}"
        }
}

五、在kibana中查看索引和数据

image.png
image.png

日志时间为logstash收到的时间，后面优化

六、在kafka中查看消息的命令

/etc/kafka/kafka_2.12-2.2.2/bin/kafka-console-consumer.sh --bootstrap-server 192.168.18.16:9092 --topic linux --from-beginning