使用累积变量,记录处理条数
2017-11-21 本文已影响0人
DeepMine
val accum = sc.accumulator(0, "Records Accumulator")
hbaseRowsDF.repartition(repartition_num).map {
tuple => {
accum += 1
...
}
}
logger.info("记录数: " + accum.value)
val accum = sc.accumulator(0, "Records Accumulator")
hbaseRowsDF.repartition(repartition_num).map {
tuple => {
accum += 1
...
}
}
logger.info("记录数: " + accum.value)