日志聚合和分析服务平台的搭建

2016-07-15  本文已影响816人  Jeff

ELK + Redis

Window7
Oracle JDK8
Redis: Redis-x64-3.2.100 for windows
Elasticsearch: elasticsearch-2.3.4
Logstash: logstash-2.3.4
Kibana: kibana-4.5.2-windows

Paste_Image.png

安装

Redis

Elasticsearch

{
  "name" : "Dmitri Smerdyakov",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.4",
    "build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
    "build_timestamp" : "2016-06-30T11:24:31Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
# cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
# node.name: node-1
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
# path.data: /path/to/data
#
# Path to log files:
#
# path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.

Logstash

logstash-test-was.outlog.conf

input { 
    file {
        path => ["C:/IBM/ODM871/WAS/AppServer/profiles/ODMSample8711/logs/SamplesServer/SystemOut.log"]
        type => "websphere"
        start_position => beginning
    } 
} 
filter {
    # Combine lines that do not start with "[" with the previous message
    multiline {
        pattern => "^\["
        negate => true
        what => "previous"
    }
    if [message] =~ /.java:\d/ {
        mutate {
            add_tag => ["exception", "trace"]
        }
    }
}
output {
    redis {
        host => '9.110.74.130'
        port => '6379' 
        data_type => 'list'
        key => 'logstash:redis'
    }
}


input { 
    file {
        path => ["C:/IBM/ODM871/WAS/AppServer/profiles/ODMSample8711/logs/SamplesServer/SystemOut.log"]
        type => "websphere"
        start_position => beginning
    } 
} 

output {
    redis {
        host => '9.110.74.130'
        port => '6379' 
        data_type => 'list'
        key => 'logstash:redis'
    }
}

Kibana

# Kibana is served by a back end server. This controls which port to use.
 server.port: 5601

# The host to bind the server to.
 server.host: "9.110.74.130"

# If you are running kibana behind a proxy, and want to mount it at a path,
# specify that path here. The basePath can't end in a slash.
# server.basePath: ""

# The maximum payload size in bytes on incoming server requests.
# server.maxPayloadBytes: 1048576

# The Elasticsearch instance to use for all your queries.
 elasticsearch.url: "http://localhost:9200"

各种日志文件的配置

TBD

websphere

liberty

tomcat

input {
    file {
        path =>"/var/lib/tomcat7/logs/localhost_access_log.txt"
        start_position =>"beginning"
    }
}
filter {
    grok {
        match => { "message" => "%{COMMONAPACHELOG}" }
    }
    date    {
        match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
    }
    mutate {
        convert => ["response","integer"]
    convert => ["bytes","integer"]
    }
}
    output {
        elasticsearch {
        host => "localhost"
    }
}

运维

参考

Paste_Image.png
上一篇下一篇

猜你喜欢

热点阅读