【性能优化】Java 堆外内存的实战分析
2022-11-30 本文已影响0人
熊本极客
本文在 Kubernetes 部署场景下，分析容器内 Java进程的堆外内存
1.堆外内存的分析思路
采集 JVM 管理的内存区间和 Java 进程的所有内存区间，对比分析 Java 进程内非 JVM 管理的内存区间即堆外内存，接着分析这些内存的详情。首先利用 JVM 的 NMT（Native Memory Tracking）获取 JVM 管理的所有内存区间，然后通过 Linux 调试及运维工具 pmap 查看 Java 进程所有的内存映像信息，最后使用 GDB 分析非 JVM 管理的内存详情。
问题：什么是 JVM 的堆外内存？
答：堆外内存是指分配在 JVM 堆外的内存区域，即 JVM 不管理的内存区域。
注意：
① 尽管 JVM 通过-XX:MetaspaceSize、-XX:MaxMetaspaceSize设置 Metaspace 大小，但 Metaspace 也属于堆外内存
② 尽管 JVM 通过-XX:MaxDirectMemorySize设置 Direct Memory 大小，但 Direct Memory 也属于堆外内存
2.利用 NMT 获取 JVM 管理的所有内存区间
通过 JVM 的 NMT（Native Memory Tracking）查询 JVM 管理的所有内存区间
步骤 1：设置 Java 进程的启动参数 -XX:NativeMemoryTracking=detail
-XX:NativeMemoryTracking=detail
# off: 默认关闭
# summary: 只统计各个分类的内存使用情况.
# detail: Collect memory usage by individual call sites.
步骤 2：利用 jdk 命令，打印 JVM 管理的所有内存区间
#进入容器，查询java的pid
$kubectl get pod -ntest |grep application
application-job-xx-d54cc78b9-l5xtb 1/1 Running 0 5m
application-job-xx-taskmanager-1-1 1/1 Running 0 3m
$kubectl exec -ntest -it application-job-xx-d54cc78b9-l5xtb bash
$ps -ef |grep java |grep NativeMemoryTracking
paas 477 1 19 07:50 ? 00:00:35 /usr/local/jre1.8.0_322//bin/java -XX:NativeMemoryTracking=detail -Xmx400556031 -Xms400556031 -XX:MaxMetaspaceSize=125829120 ...省略
# 进入目录 */jdk/bin
$./jcmd 477 VM.native_memory detail scale=MB > native_memory.txt
# native_memory.txt的内容如下
477:
Native Memory Tracking:
Total: reserved=1959MB, committed=725MB
- Java Heap (reserved=382MB, committed=382MB)
(mmap: reserved=382MB, committed=382MB)
- Class (reserved=1116MB, committed=104MB)
(classes #18353)
(malloc=2MB #15599)
(mmap: reserved=1114MB, committed=103MB)
- Thread (reserved=96MB, committed=96MB)
(thread #96)
(stack: reserved=95MB, committed=95MB)
- Code (reserved=248MB, committed=25MB)
(malloc=4MB #8021)
(mmap: reserved=244MB, committed=21MB)
- GC (reserved=1MB, committed=1MB)
(mmap: reserved=1MB, committed=1MB)
- Internal (reserved=88MB, committed=88MB)
(malloc=88MB #23546)
- Symbol (reserved=24MB, committed=24MB)
(malloc=22MB #207856)
(arena=2MB #1)
- Native Memory Tracking (reserved=4MB, committed=4MB)
(tracking overhead=4MB)
Virtual memory map:
[0x00000000e8200000 - 0x0000000100000000] reserved 382MB for Java Heap from
[0x00007f5f03974a32] ReservedSpace::initialize(unsigned long, unsigned long, bool, char*, unsigned long, bool)+0xa2
[0x00007f5f039751c1] ReservedHeapSpace::ReservedHeapSpace(unsigned long, unsigned long, bool, char*)+0x61
[0x00007f5f0393f7a2] Universe::reserve_heap(unsigned long, unsigned long)+0x72
[0x00007f5f0345c508] GenCollectedHeap::allocate(unsigned long, unsigned long*, int*, ReservedSpace*)+0x198
[0x00000000f0150000 - 0x0000000100000000] committed 255MB from
[0x00007f5f0397585c] VirtualSpace::expand_by(unsigned long, bool)+0x13c
[0x00007f5f03472fc3] Generation::Generation(ReservedSpace, unsigned long, int)+0xc3
[0x00007f5f034733ff] CardGeneration::CardGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0x5f
[0x00007f5f03918865] TenuredGeneration::TenuredGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0x75
[0x00000000e8200000 - 0x00000000f0150000] committed 127MB from
[0x00007f5f0397585c] VirtualSpace::expand_by(unsigned long, bool)+0x13c
[0x00007f5f03472fc3] Generation::Generation(ReservedSpace, unsigned long, int)+0xc3
[0x00007f5f0336b529] DefNewGeneration::DefNewGeneration(ReservedSpace, unsigned long, int, char const*)+0x59
[0x00007f5f034748f3] GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0x423
[0x0000000100000000 - 0x0000000140000000] reserved 1024MB for Class from
[0x00007f5f03974a32] ReservedSpace::initialize(unsigned long, unsigned long, bool, char*, unsigned long, bool)+0xa2
[0x00007f5f03974c44] ReservedSpace::ReservedSpace(unsigned long, unsigned long, bool, char*, unsigned long)+0x24
[0x00007f5f0375011b] Metaspace::allocate_metaspace_compressed_klass_ptrs(char*, unsigned char*)+0x4b
[0x00007f5f037509e4] Metaspace::global_initialize()+0x4c4
[0x0000000100000000 - 0x0000000100dc0000] committed 14MB from
[0x00007f5f0397585c] VirtualSpace::expand_by(unsigned long, bool)+0x13c
[0x00007f5f03749ebf] VirtualSpaceList::expand_node_by(VirtualSpaceNode*, unsigned long, unsigned long)+0x4f
[0x00007f5f0374ec49] VirtualSpaceList::expand_by(unsigned long, unsigned long)+0x89
[0x00007f5f0374eda2] VirtualSpaceList::get_new_chunk(unsigned long, unsigned long)+0xa2
...省略
3.利用 PMAP 获取进程内的所有内存区间
通过 Linux 调试及运维工具 pmap - report memory map of a process，查看进程的内存映像信息，即需要打印进程内所有内存块地址区间
步骤 1：利用 pmap 打印进程内所有的内存地址区间
#进入容器，查询java的pid
$kubectl get pod -ntest |grep application
application-job-xx-d54cc78b9-l5xtb 1/1 Running 0 5m
application-job-xx-taskmanager-1-1 1/1 Running 0 3m
$kubectl exec -ntest -it application-job-xx-d54cc78b9-l5xtb bash
$ps -ef |grep java |grep NativeMemoryTracking
paas 477 1 19 07:50 ? 00:00:35 /usr/local/jre1.8.0_322//bin/java -XX:NativeMemoryTracking=detail -Xmx400556031 -Xms400556031 -XX:MaxMetaspaceSize=125829120 ...省略
#利用pmap
$pmap -x 477 | sort -n -k3 > process_memory_address.txt
# process_memory_address.txt的内容如下
---------------- ------- ------- -------
00007f5eaf6fe000 3064 2164 2164 rw--- [ anon ]
00007f5ec6e9d000 3064 2164 2164 rw--- [ anon ]
00007f5ea94a7000 3064 2208 2208 rw--- [ anon ]
00007f5eadde9000 3064 2208 2208 rw--- [ anon ]
00007f5f004e5000 2488 2488 2488 rw--- [ anon ]
00007f5ecc000000 2708 2708 2708 rw--- [ anon ]
00007f5ee8000000 3328 3328 3328 rw--- [ anon ]
00007f5ec0000000 3968 3840 3840 rw--- [ anon ]
00007f5eb4000000 4232 3928 3928 rw--- [ anon ]
000055a992a85000 4640 4544 4544 rw--- [ anon ]
00007f5eec0ac000 6520 6520 0 r--s- flink-dist_2.11-1.13.6.jar
00007f5ebc000000 6952 6928 6928 rw--- [ anon ]
00007f5f0302d000 9684 9036 0 r-x-- libjvm.so
00007f5eec70a000 9176 9176 0 r--s- WordCount.jar
00007f5f01b8f000 9188 9184 9184 rw--- [ anon ]
00007f5ed0000000 12748 12748 12748 rw--- [ anon ]
00007f5efc000000 22508 21820 21820 rw--- [ anon ]
00007f5edc000000 32016 26808 26808 rw--- [ anon ]
00007f5eed000000 27776 27748 27748 rwx-- [ anon ]
00000000e8200000 405248 217592 217592 rw--- [ anon ]
total kB 3019696 480300 445160
...省略
cat /proc/477/smaps 可以打印详细的内存地址，例如
$ cat /proc/477/smaps
e8200000-100dc0000 rw-p 00000000 00:00 0
Size: 405248 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 217588 kB
Pss: 217588 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 217588 kB
Referenced: 217420 kB
Anonymous: 217588 kB
LazyFree: 0 kB
AnonHugePages: 8192 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
THPeligible: 1
VmFlags: rd wr mr mw me ac sd
...省略
步骤 2：对比上述 2 中打印的全部 JVM 所管理的内存区间，确定比较大的非 JVM 管理的内存地址区间
# 从process_memory_address.txt确定内存地址 00007f5ea97b5000 比较大
00007f5ebc000000 6952 6928 6928 rw--- [ anon ]
# 获取连续的内存地址
$pmap -x 477 |grep -A 1 00007f5ea97b5000
00007f5ea97b5000 6952 6928 6928 rw--- [ anon ]
00007f5ea99b4000 2044 24 0 ----- [ anon ]
4.利用 GDB 分析非 JVM 管理的内存详情
GDB 全称“GNU symbolic debugger”，是 Linux 下常用的程序调试器。分析非 JVM 管理的内存详情，主要使用了 dump 内存的功能。利用工具 nsenter 和 容器 Pid 进入容器进程的 namespace，最后通过 gdb dump 内存和 hexdump 查看 bin 二进制文件
步骤 1：进入容器的命名空间
# 查询容器名称
$kubectl get pod -ntest |grep application
application-job-xx-d54cc78b9-l5xtb 1/1 Running 0 4h15m
application-job-xx-taskmanager-1-1 1/1 Running 0 5h13m
# 查看容器id
$docker ps |grep application-job-xx-d54cc78b9-l5xtb
5456099bd648 1bd14b7f018a "/opt/container/flink/scri…" 4 hours ago Up 4 hours k8s_flink-main-container_application-job-xx-d54cc78b9-l5xtb_flink_6ae62c3f-e5e9-4810-9ade-2dd908f2a37a_0
99d3788c5285 192.168.31.37:5000/pause:3.2 "/pause" 4 hours ago Up 4 hours k8s_POD_application-job-xx-d54cc78b9-l5xtb_flink_6ae62c3f-e5e9-4810-9ade-2dd908f2a37a_0
# 查看虚拟机上目标容器的Pid
$docker inspect 5456099bd648 |grep Pid
"Pid": 6789,
"PidMode": "",
"PidsLimit": 0,
# 进入目标Pid的命名空间
$nsenter -t 6789
步骤 2：利用 gdb dump 目标内存地址的详情
# 查看虚拟机上目标容器的进程id
$ps -ef |grep application-job-xx-d54cc78b9-l5xtb
service 8028 6789 2 02:55 ? 00:06:38 /usr/local/jre1.8.0_322//bin/java -XX:NativeMemoryTracking=detail -Xmx400556031 -Xms400556031 -XX:MaxMetaspaceSize=125829120 ...省略
root 125820 124355 0 07:25 pts/9 00:00:00 grep --color=auto application-job-xx-d54cc78b9-l5xtb
# 进入gdb dum
$gdb dump
GNU gdb (GDB) EulerOS 8.3.1-12.eulerosv2r9
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-openEuler-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
dump: No such file or directory.
# 绑定进程id
$(gdb) attach 8028
Attaching to process 8028
[New LWP 8031]
[New LWP 8032]
[New LWP 8033]
...省略
...省略
Missing separate debuginfo for target:/usr/lib64/libnss_myhostname.so.2
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f8/86f51fb3550c00854193e611c36670c74759d7.debug
warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable. Connect to gdbserver inside the container.
warning: Expected absolute pathname for libpthread in the inferior, but got target:/usr/lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
0x00007fdf244802dd in ?? () from target:/usr/lib64/libpthread.so.0
#dump目标内存地址的详情
(gdb) dump memory /tmp/dump.bin 0x00007f5ea97b5000 00007f5ea99b4000
利用 hexdump 查看 gdb dump 的 bin 文件
$hexdump -C /tmp/dump.bin
00000000 e0 aa ff ff 78 01 00 00 00 41 0e 10 86 02 45 0d |....x....A....E.|
00000010 06 45 8f 03 45 8e 04 4b 8d 05 49 8c 06 44 83 07 |.E..E..K..I..D..|
00000020 03 14 01 0a 0c 07 08 44 0b 00 00 00 00 00 00 00 |.......D........|
00000030 1c 00 00 00 94 16 00 00 28 ac ff ff 19 00 00 00 |........(.......|
00000040 00 41 0e 10 86 02 4e 0d 06 45 0c 07 08 00 00 00 |.A....N..E......|
00000050 2c 00 00 00 b4 16 00 00 28 ac ff ff 64 00 00 00 |,.......(...d...|
00000060 00 41 0e 10 86 02 46 0d 06 4b 8c 06 83 07 57 8f |.A....F..K....W.|
00000070 03 8e 04 8d 05 76 0c 07 08 00 00 00 00 00 00 00 |.....v..........|
00000080 34 00 00 00 e4 16 00 00 68 ac ff ff 34 01 00 00 |4.......h...4...|
00000090 00 41 0e 10 86 02 43 0d 06 4b 8c 04 83 05 52 8d |.A....C..K....R.|
000000a0 03 02 8d 0a 0c 07 08 42 0b 02 62 0a 0c 07 08 46 |.......B..b....F|
000000b0 0b 00 00 00 00 00 00 00 24 00 00 00 1c 17 00 00 |........$.......|
000000c0 70 ad ff ff 2b 00 00 00 00 41 0e 10 86 02 47 0d |p...+....A....G.|
000000d0 06 4f 0a 0c 07 08 49 0b 48 0c 07 08 00 00 00 00 |.O....I.H.......|
000000e0 2c 00 00 00 44 17 00 00 78 ad ff ff 48 00 00 00 |,...D...x...H...|
000000f0 00 43 0e 10 86 02 49 0d 06 58 0a 0c 07 08 44 0b |.C....I..X....D.|
00000100 48 0a 0c 07 08 48 0b 4f 0c 07 08 00 00 00 00 00 |H....H.O........|
00000110 2c 00 00 00 74 17 00 00 98 ad ff ff 61 01 00 00 |,...t.......a...|
00000120 00 41 0e 10 86 02 4d 0d 06 45 8f 03 49 8e 04 49 |.A....M..E..I..I|
00000130 8d 05 45 8c 06 7f 83 07 02 e1 0a 0c 07 08 41 0b |..E...........A.|
00000140 1c 00 00 00 a4 17 00 00 d8 ae ff ff 15 00 00 00 |................|
00000150 00 41 0e 10 86 02 4e 0d 06 41 0c 07 08 00 00 00 |.A....N..A......|
00000160 24 00 00 00 c4 17 00 00 d8 ae ff ff 58 00 00 00 |$...........X...|
00000170 00 41 0e 10 86 02 46 0d 06 4b 8c 05 83 06 55 8e |.A....F..K....U.|
00000180 03 8d 04 6c 0c 07 08 00 1c 00 00 00 ec 17 00 00 |...l............|
00000190 10 af ff ff 17 00 00 00 00 41 0e 10 86 02 43 0d |.........A....C.|
000001a0 06 52 0c 07 08 00 00 00 2c 00 00 00 0c 18 00 00 |.R......,.......|
000001b0 10 af ff ff 36 01 00 00 00 41 0e 10 86 02 43 0d |....6....A....C.|
000001c0 06 4b 8c 06 83 07 5c 8f 03 8e 04 8d 05 75 0a 0c |.K....\......u..|
000001d0 07 08 48 0b 00 00 00 00 2c 00 00 00 3c 18 00 00 |..H.....,...<...|
000001e0 20 b0 ff ff cd 00 00 00 00 41 0e 10 86 02 46 0d | ........A....F.|
000001f0 06 45 8f 03 4f 8d 05 8e 04 45 83 07 8c 06 02 96 |.E..O....E......|
00000200 0a 0c 07 08 4a 0b 00 00 2c 00 00 00 6c 18 00 00 |....J...,...l...|
00000210 c0 b0 ff ff dd 00 00 00 00 41 0e 10 86 02 46 0d |.........A....F.|
00000220 06 47 8e 04 8f 03 45 8d 05 45 8c 06 48 83 07 02 |.G....E..E..H...|
...省略
