Flume单机部署及测试agent
2019-08-02 本文已影响0人
cherishpf
Apache Flume 是一个分布式、高可靠、高可用的用来收集、聚合、转移不同来源的大量日志数据到中央数据仓库的工具。
1、Flume地址
Flume官网地址
http://flume.apache.org/
Flume安装包各版本下载地址
http://archive.apache.org/dist/flume/
Flume 1.9.0 User Guide文档
http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
Flume 1.8.0 User Guide中文翻译版
https://flume.liyifeng.org/#agent
2、在单机上安装Flume步骤
系统要求：
JAVA运行环境：Java 1.8或更高版本
安装步骤：
- 1、将apache-flume-1.9.0-bin.tar.gz下载到安装目录
[root@server ~]# cd /data/software
[root@server software]# wget http://archive.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
- 2、解压
[root@server software]# tar -zxvf apache-flume-1.9.0-bin.tar.gz
- 3、重命名
[root@server software]# mv apache-flume-1.9.0-bin flume
- 4、将flume/conf/下的flume-env.sh.template复制一份到flume-env.sh，并配置flume-env.sh
[root@server software]# cd flume/conf/
[root@server conf]# cp flume-env.sh.template flume-env.sh
[root@server conf]# echo $JAVA_HOME
/data/software/jdk
[root@server conf]# vim flume-env.sh
# Enviroment variables can be set here.
export JAVA_HOME=/data/software/jdk
- 5、将flume/conf/下的flume-conf.properties.template复制一份到flume-conf.properties，并配置flume-conf.properties
[root@server conf]# cp flume-conf.properties.template flume-conf.properties
[root@server conf]# vim flume-conf.properties
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
agent.sources = r1
agent.channels = c1
agent.sinks = k1
# For each one of the sources, the type is defined
agent.sources.r1.type = netcat
agent.sources.r1.bind = localhost
agent.sources.r1.port = 4444
# The channel can be defined as follows.
agent.sources.r1.channels = c1
# Each sink's type must be defined
agent.sinks.k1.type = logger
#Specify the channel the sink should use
agent.sinks.k1.channel = c1
# Each channel's type is defined.
agent.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.c1.capacity = 100
agent.channels.c1.transactionCapacity = 100
flume-conf.properties文件就是配置flume agent的地方，可以配置多个agent，上述文件配置了一个agent，实现的是：监控本机4444端口，将监听到的数据实时显示在控制台中。
- 6、启动flume agent
[root@server flume]# cd /data/software/flume/
[root@server flume]# ./bin/flume-ng agent --conf conf/ --name agent --conf-file conf/flume-conf.properties -Dflume.root.logger==INFO,console
- 7、测试flume agent
另开一个控制台窗口，通过telnet工具向本机的4444端口发送数据：
[root@server ~]# telnet localhost 4444
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
aaaaaaaaaaaaaa
OK
bbbbbbbbbbbbbb
OK
观察flume控制台窗口接收到的数据：
2019-08-02 11:41:38,441 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:131)] Checking file:conf/flume-conf.properties for changes
2019-08-02 11:41:40,492 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:316)] Starting connection handler
2019-08-02 11:41:44,611 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 16
2019-08-02 11:41:44,619 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1
2019-08-02 11:41:48,484 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 0D aaaaaaaaaaaaaa. }
2019-08-02 11:41:54,364 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 16
2019-08-02 11:41:54,364 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1
2019-08-02 11:41:54,365 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 62 62 62 62 62 62 62 62 62 62 62 62 62 62 0D bbbbbbbbbbbbbb. }
