Logstash搭建及配置
2019-11-08 本文已影响0人
王哲昭
-
JDK 环境检查
-
logstash下载
-
下载
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
-
添加文件
cd /etc/yum.repos.d
touch logstash.repo
vi logstash.repo
添加下面内容到logstash.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
-
执行
sudo yum install logstash
-
logstash 运行
cd /usr/share/logstash/bin
[root@ecs-s6-large-2-linux-20190906115149 bin]# systemctl stop logstash.service
[root@ecs-s6-large-2-linux-20190906115149 bin]# ./logstash -e 'input { stdin { } } output { stdout {} }'
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-09-12 16:07:04.466 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-09-12 16:07:04.477 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.3.1"}
[INFO ] 2019-09-12 16:07:05.512 [Converge PipelineAction::Create<main>] Reflections - Reflections took 38 ms to scan 1 urls, producing 19 keys and 39 values
[WARN ] 2019-09-12 16:07:06.271 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2019-09-12 16:07:06.274 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x131cf46e run>"}
[INFO ] 2019-09-12 16:07:06.393 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2019-09-12 16:07:06.547 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-09-12 16:07:06.959 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
控制台运行成功
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2019-09-12 16:14:28.766 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[ERROR] 2019-09-12 16:14:28.787 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[
控制台运行报错(这个我的解决方案是切换成root 就好了)
hello
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"@version" => "1",
"@timestamp" => 2019-09-12T08:07:16.767Z,
"message" => "hello",
"host" => "ecs-s6-large-2-linux-20190906115149"
}
测试成功了
-
logstash 后台运行
cd /etc/logstash/conf.d
touch test.conf
将如下内容写到 test.conf 里面
input {
tcp {
port => 9100
mode => "server"
tags => ["tags"]
codec => json_lines
}
}
output {
stdout { codec => "rubydebug" }
file {
path => "/var/log/logstash/%{+yyyy}/%{+MM}/%{+dd-HH}-%{host}.log"
codec => "rubydebug"
}
elasticsearch {
hosts => ["119.3.185.186:9200"] # 定义es服务器的ip
index => "system-syslog-%{+YYYY.MM}" # 定义索引
}
}
input 监听 服务器 9100端口的消息，
output stdout 输出到控制台
file 输出到文件
elasticsearch 输出到 elasticsearch
-
执行
在 /usr/share/logstash 添加
touch startUp.sh
chmod 777 startUp.sh
在startUp.sh 中添加
nohup bin/logstash -f /etc/logstash/conf.d/test.conf &
注意 /etc/logstash/conf.d/test.conf 为你的conf 配置位置
-
修改/etc/logstash/logstash.yml
http.host: "0.0.0.0"
注意： 为了外部可以远程连接到当前logstash 服务
-
运行
cd /usr/share/logstash
./startup.sh
tail -f nohup.out
看到
[INFO ] 2019-09-12 17:59:18.124 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2019-09-12 17:59:18.187 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:9100", :ssl_enable=>"false"}
[INFO ] 2019-09-12 17:59:18.378 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-09-12 17:59:18.986 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
就启动成功了
-
测试
[root@ecs-s6-large-2-linux-20190828090356 logstash]# netstat -lntp |grep 9100
tcp 0 0 :::9100 :::* LISTEN 10533/java
[root@ecs-s6-large-2-linux-20190828090356 logstash]# netstat -lntp |grep 9600
tcp 0 0 ::ffff:127.0.0.1:9600 :::* LISTEN 10533/java
就成功了
