flink的优点,exactly once语义(做快照和两阶段提

2018-12-19  本文已影响0人  Bitson

参考 

Flink Data Streaming Fault Tolerance

High-throughput, low-latency, and exactly-once stream processing with Apache Flink™

An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!)

true streaming的优点

- low latency

- flow control

- and true streaming programming model(session window,micro batch的处理方式需要窗口和interval对齐)

micro-batching的优点

- high throughput(批量处理效率更高)

- and exactly-once guarantees(一个小批次要不全部成功,要不全部失败)

flink集合了两种特性

For small state (e.g., counts or other statistical summaries), this backup overhead is usually negligible, while for large state, the checkpoint interval makes a tradeoff between throughput and recovery time. 

- 包含真流处理的优点

- 使用异步snapshot来实现exactly once,snapshot不会影响到数据流的处理

Other problems with Storm’s mechanism is low throughput and problems with flow control, as the acknowledgment mechanism often falsely classifies failures under backpressure.

上一篇下一篇

猜你喜欢

热点阅读