双层flume搭建过程中遇到的坑
2020-06-27 本文已影响0人
bigdata张凯翔
INFO - Creating /user/hive/warehouse/ods.db/origin_user_behavior/19-04-19/events.1555671386480.tmp
ERROR - Avro source r2: Unable to process event batch. Exception follows.
org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is
解决方法:
先将flume的JVM内存改为512M
capacity = 1000
transactionCapacity = 1000
keep-alive = 30
ERROR - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:389)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: hadoop104, port: 4545 }: Failed to send batch
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:314)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:373)
... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: hadoop104, port: 4545 }: Avro RPC call returned Status: FAILED
at org.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:390)
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:373)
at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:302)
... 4 more
image.png
请求超时,导致发送event失败
解决:
设置request-timeout长一点,默认20秒