Flume pull方式和push方式整合

• Pull方式

Flume Agent 编写

# Name the components on this agent
simple-agent.sources = netcat-source
simple-agent.sinks = spark-sink
simple-agent.channels = memory-channel

# Describe/configure the source
simple-agent.sources.netcat-source.type = netcat
simple-agent.sources.netcat-source.bind = localhost
simple-agent.sources.netcat-source.port = 44444

# Describe the sink
simple-agent.sinks.spark-sink.type = org.apache.spark.streaming.flume.sink.SparkSink
simple-agent.sinks.spark-sink.hostname =localhost
simple-agent.sinks.spark-sink.port =41414
simple-agent.sinks.spark-sink.channel = memoryChannel

# Use a channel which buffers events in memory
simple-agent.channels.memory-channel.type = memory


# Bind the source and sink to the channel
simple-agent.sources.netcat-source.channels = memory-channel
simple-agent.sinks.spark-sink.channel = memory-channel

启动Flume

flume-ng agent \
--name simple-agent \
--conf conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/flume_pull_streaming.conf \
-Dflume.root.logger=INFO,console &

• Push方式

Flume Agent的编写

# Name the components on this agent
simple-agent.sources = netcat-source
simple-agent.sinks = avro-sink
simple-agent.channels = memory-channel

# Describe/configure the source
simple-agent.sources.netcat-source.type = netcat
simple-agent.sources.netcat-source.bind = localhost
simple-agent.sources.netcat-source.port = 44444

# Describe the sink
simple-agent.sinks.avro-sink.type = avro
simple-agent.sinks.avro-sink.hostname = localhost
simple-agent.sinks.avro-sink.port = 41414

# Use a channel which buffers events in memory
simple-agent.channels.memory-channel.type = memory


# Bind the source and sink to the channel
simple-agent.sources.netcat-source.channels = memory-channel
simple-agent.sinks.avro-sink.channel = memory-channel

启动flume

flume-ng agent \
--name simple-agent \
--conf conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/flume_push_streaming.conf \
-Dflume.root.logger=INFO,console &

==注意在本地和服务器上切换的时候需要修改flume的sink的hostname==

本地测试总结

1. 启动SparkStreaming作业
2. 启动flume agent
3. 通过telnet输入数据，观察IDEA控制台的输出

提交到服务器

spark-submit \
--class com.gwf.spark.FlumePushWordCount \
--master local[2] \
--packages org.apache.spark:spark-streaming-flume_2.11:2.2.0 \
/Users/gaowenfeng/Documents/IDE/newsell/spark-train/target/spark-train-1.0-SNAPSHOT.jar localhost 41414

spark-submit \
--class com.gwf.spark.FlumePushWordCount \
--master local[2] \
/Users/gaowenfeng/Documents/IDE/newsell/spark-train/target/spark-train-1.0-SNAPSHOT.jar localhost 41414