flink 内存分配计算

2020-05-14 本文已影响0人邵红晓

flink 1.10 taskmanager.memory.process.size: 1568m

2020-05-13 19:03:48,023 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 512 MiBytes
2020-05-13 19:03:48,023 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: /usr/java/jdk1.8.0_191
2020-05-13 19:03:48,025 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Hadoop version: 2.7.3.2.6.5.0-292
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options:
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx536870902  512m
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms536870902  512m
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=268435458 256m
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxMetaspaceSize=100663296    96m
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog.file=/export/home/xxx/test/xxx/flink/flink-1.10.0/log/flink-data-taskexecutor-0-shyt-hadoop-2441.log
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/export/home/xxx/test/xx.xxx/flink/flink-1.10.0/conf/log4j.properties
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/export/home/xxx/test/xxx/flink/flink-1.10.0/conf/logback.xml
2020-05-13 19:03:48,026 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments:
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /export/home/xxx/test/xxx/flink/flink-1.10.0/conf
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - taskmanager.memory.framework.off-heap.size=134217728b  128m
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - [taskmanager.memory.network.max=134217730b](http://taskmanager.memory.network.max=134217730b/)          128m  
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - [taskmanager.memory.network.min=134217730b](http://taskmanager.memory.network.min=134217730b/)           128m
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - taskmanager.memory.framework.heap.size=134217728b   128m
2020-05-13 19:03:48,027 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - taskmanager.memory.managed.size=536870920b           512m
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - taskmanager.cpu.cores=1.0
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - taskmanager.memory.task.heap.size=402653174b       384m
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -D
2020-05-13 19:03:48,028 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - taskmanager.memory.task.off-heap.size=0b

总结内存分配

jvm heap

jvm heap 512m{
taskmanager.memory.task.heap.size=402653174b 384m
taskmanager.memory.framework.heap.size=134217728b 128m
}

off-heap memory 1056m

managed memory = taskmanager.memory.managed.size=536870920b 512m (off-heap memory Unsafe#allocateMemory()，从 JVM 的 -XX:MaxDirectMemorySize 参数限制中分离出来，不受其管束，native memory 不受jvm管理，用于batch, rocks db，如果stream作业无状态，可以设置为0)
direct or native = {
taskmanager.memory.task.off-heap.size =0b
taskmanager.memory.framework.off-heap.size=134217728b 128m
taskmanager.memory.network.max=134217730b 128m
= -XX:MaxDirectMemorySize=268435458 256m jvm管理的
}
XX:MaxMetaspaceSize=100663296 96m
jvm-overhead 192m (默认大小，This includes native memory but not direct memory)

flink ui 显示说明

关于 JVM 的内存，堆内存（Heap Memory）的定义通常是比较清晰的，但堆外/非堆内存（Off-Heap/Non-Heap Memory）的定义却有很多不同的版本，这应该是导致你困惑的主要原因。让我们先抛开这些名词，本质上 Java 应用使用的内存（不包括 JVM 自身的开销）可以分为三类：

JVM 堆内存：Heap
不在 JVM 堆上但受到 JVM 管理的内存：Direct
Direct 内存是直接映射到 JVM 虚拟机外部的内存空间，但是其用量又受到 JVM 的管理和限制
完全不受 JVM 管理的内存：Native

MXBean的 Non-Heap 是不包括 Direct，而是由 Code Cache、Metaspace、Compressed Class Space 几个部分组成。
FLIP-102 讨论的是 metrics 如何在 WebUI 上展示，Flink metrics 是通过 MXBean 获取的

参考
http://apache-flink.147419.n8.nabble.com/Flink-td1869.html#a1885

flink 内存分配计算

总结内存分配

jvm heap

off-heap memory 1056m

flink ui 显示说明

猜你喜欢

热点阅读