Spark Streaming Config

2019-10-24 本文已影响0人 clive0x

spark.streaming.backpressure.enabled:false 开启流控

WriteAheadLogUtils.scala:

spark.streaming.receiver.writeAheadLog.enable：false 控制是否开户write ahead log

spark.streaming.receiver.writeAheadLog.class>>FileBasedWriteAheadLog OR BatchedWriteAheadLog

spark.streaming.receiver.writeAheadLog.rollingIntervalSecs:60S

spark.streaming.receiver.writeAheadLog.maxFailures:3

spark.streaming.receiver.writeAheadLog.closeFileAfterWrite:false,为True时，并行

FileBasedWriteAheadLog 每隔60S，往$checkpointdir/receivedData/$streamid目录写

文件名格式为log-starttime-stoptime

FileBasedWriteAheadLog启动时会从该目录文件恢复，内存结构为pastLogs。

另外receiveblock write ahead log地址：

$checkpointdir/receivedBlockMetadata/$streamid

Spark streaming block生成间隔　BlockGenerator.scala

spark.streaming.blockInterval:2000(ms)

spark.streaming.blockQueueSize:10(内存buf)