ELK之Logstash

2022-09-15  本文已影响0人  漫画三毛

版本与Elasticsearch一致

**1、下载并安装适用于 Linux 的存档编辑
**Elasticsearch v8.4.1 的 Linux 存档可以按如下方式下载和安装：

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.6-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.6-linux-x86_64.tar.gz.sha512
shasum -a 512 -c logstash-7.17.6-linux-x86_64.tar.gz.sha512
tar -xzf logstash-7.17.6-linux-x86_64.tar.gz
cd logstash-7.17.6/

2、logstash.yml配置文件

#该值为true时，开启转义（避免写法格式错误导致浪费太多时间）
config.support_escapes: true
<meta name="wolai" content="bAHmkmE9rVVWdBhyjXgUDd1666174563784">

配合ES开启认证：

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system > xpack.monitoring.elasticsearch.password: *
xpack.monitoring.elasticsearch.hosts: ["http://ip:9200"]

如开启认证，pipeline output如使用es，需配置user/password（elastic/****）

3、pipeline配置

4、启动命令

sudo -u elasticsearch ./bin/logstash

# 指定文件运行
sudo -u elasticsearch ./bin/logstash -f aaa.config

# -t 指定文件运行,验证文件是否存在问题
sudo -u elasticsearch ./bin/logstash -f aaa.config -t

nohup sudo -u elasticsearch ./logstash/bin/logstash  > /www/logs/logstash.log &1 &

nohup sudo -u elasticsearch ./bin/logstash  > /www/logs/logstash.log &1 &

5、使用logstash过滤出特定格式的日志

日志内容：[2018-11-24 08:33:43,253][ERROR][http-nio-8080-exec-4][com.hh.test.logs.LogsApplication][code:200,msg:测试录入错误日志,param:{}]

filter {
  if "nova" in [tags]{
    grok {
      # 筛选过滤
      match => {
        "message" => "(?<date>\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\]\[(?<level>[A-Z]{4,5})\]\[(?<thread>[A-Za-z0-9/\-]{4,40})\]\[(?<class>[A-Za-z0-9/.]{4,40})\]\[(?<msg>.*)"
      }
    mutate {
      remove_field => [
        "message",
      ]
    }
    # 不匹配正则则删除，匹配正则用=~
    if [level] !~ "(ERROR|WARN|INFO)" {
      # 删除日志
      drop {}
    }
  }
}

正则表达式在线的调试库，以供参考： http://grokdebug.herokuapp.com/
需注意：”-“ 破折号，需要转移不然会出现解析异常

6、异常问题

📌参考文献:

http://www.noobyard.com/article/p-ucnfmacz-gk.html

https://www.elastic.co/guide/en/logstash/7.17/introduction.html

https://blog.51cto.com/u_15047490/4228036

样例：

input {
    file {
        path => "/www/log/java-web/log_info.log"
        type => "java-web-info"
        start_position => "beginning"
        # 多行输入
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
        }
    }

    file {
        path => "/www/log/java-web/log_error.log"
        type => "java-web-error"
        start_position => "beginning"
        # 多行输入
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
        }
    }
}

filter {

    if [type] == "java-web-info" {
    grok {
        match => {
            "message" => "(?<date>\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3})\s(?<level>[A-Z]{4,5})\s{1,2}(?<pid>\d{1,20})\s\-{1,5}\s\[(?<thread>[A-Za-z0-9./-]{4,40})\]\s(?<method>[A-Z
a-z0-9.\\(\\:/\-\\)]{0,200}):\s(?<msg>.*)"
        }
    }

    # 过滤debug
    if [level] !~ "(ERROR|WARN|INFO)" {
        drop {}
    }
    # 过滤指定内容
    if [thread] == "com.alibaba.nacos.naming.push.receiver" {
        drop {}
    }

    }
}

output {

    if [type] == "java-web-info" {
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "java-web-info-%{+YYYY.MM.dd}"
            #开启认证后，需配置
            #user => "elastic"
            #password => "******"
        }
    }
    
    if [type] == "java-web-error" {
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "java-web-error-%{+YYYY.MM.dd}"
            #开启认证后，需配置
            #user => "elastic"
            #password => "******"
        }
    }
}
上一篇 下一篇

猜你喜欢

热点阅读