07 flume多路复制

2020-06-22  本文已影响0人  张力的程序园

本小节将展示flume的多路复制。所谓多路复制,是指数据来自一个source,通过两个channel,进入到两个sink。

1、前提约束

2、操作步骤

# 在141中执行,拷贝141中的flume文件夹到142
scp -r /root/apache-flume-1.8.0-bin root@hadoop2:/root
# 在141中执行,拷贝141中的flume文件夹到143
scp -r /root/apache-flume-1.8.0-bin root@hadoop3:/root

# 定义一个服务名称为a1,source,channel,sink分别为r1, c1, k1
a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources.r1.selector.type = replicating 
# r1监听8888端口的网络状态
a1.sources.r1.type = netcat
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 8888
# sink 端的 avro 是一个数据发送者 
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop2
a1.sinks.k1.port = 4141 

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop3 
a1.sinks.k2.port = 4141 

# Describe the channel 
a1.channels.c1.type = memory 
a1.channels.c1.capacity = 1000 
a1.channels.c1.transactionCapacity = 100 

a1.channels.c2.type = memory 
a1.channels.c2.capacity = 1000 
a1.channels.c2.transactionCapacity = 100 

# Bind the source and sink to the channel 
a1.sources.r1.channels = c1 c2 
a1.sinks.k1.channel = c1 
a1.sinks.k2.channel = c2 
# 定义一个服务名称为a1,source,channel,sink分别为r1, c1, k1
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop2
a1.sources.r1.port = 4141 
# k1的输入时日志形式
a1.sinks.k1.type = logger
# c1类型为内存
a1.channels.c1.type = memory
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
# 定义一个服务名称为a1,source,channel,sink分别为r1, c1, k1
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop3
a1.sources.r1.port = 4141 
# k1的输入时日志形式
a1.sinks.k1.type = logger
# c1类型为内存
a1.channels.c1.type = memory
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
# 在141中执行以下语句
cd /root/apache-flume-1.8.0-bin
./flume-ng agent -c ../conf -f ../conf/flume-conf-avro.properties  -n a1
# 在142中执行以下语句
cd /root/apache-flume-1.8.0-bin
./flume-ng agent -c ../conf -f ../conf/flume-conf-avro.properties  -n a1
# 在143中执行以下语句
cd /root/apache-flume-1.8.0-bin
./flume-ng agent -c ../conf -f ../conf/flume-conf-avro.properties  -n a1
上一篇 下一篇

猜你喜欢

热点阅读