Flume第二个Agent应用

本示例是把hive中的日志按照年月日、时分方式定时写入到hdfs中的文件夹中，并且文件夹根据时间自动创建。

准备jar包

进入到hadoop-2.5.0-cdh5.3.6\hadoop-2.5.0-cdh5.3.6\share\hadoop\mapreduce1\lib下copy4个jar包到flume的lib下

复制hadoop的core-site文件到flume的conf下

cp /opt/cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/core-site.xml  conf/

• 不复制core-site.xml会报错。

  
  
• 若不想复制core-site.xml文件的解决办法

a2.sinks.k2.hdfs.path = hdfs的全路径
hdfs://hadoop-senior.beifeng.com:/user/beifeng/flume/logs/%y-%m-%d/%H%M/%S

image
9000

agent的conf代码

# Name the components on this agent
a2.sources = r2
a2.sinks = k2
a2.channels = c2

# Describe/configure the source
a2.sources.r2.type = exec
a2.sources.r2.command = tail -F /opt/cdh5.3.6/hive-0.13.1-cdh5.3.6/logs/hive.log


# Describe the sink
a2.sinks.k2.type = hdfs
a2.sinks.k2.hdfs.path = /user/beifeng/flume/logs/%y-%m-%d/%H%M/%S
a2.sinks.k2.hdfs.filePrefix = events-
a2.sinks.k2.hdfs.useLocalTimeStamp =true
a2.sinks.k2.hdfs.writeFormat=Text
a2.sinks.k2.hdfs.round = true
a2.sinks.k2.hdfs.roundValue = 10
a2.sinks.k2.hdfs.roundUnit = minute

# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2

运行

bin/flume-ng agent  \
--conf conf   \
--conf-file conf/flume_tail.conf   \
--name a2   \
-Dflume.root.logger=INFO,console