docker中如何做jvm的gc、内存等问题排查

背景：docker内运行的Java服务不像宿主机直接运行那么方便查询问题，今天记录下，首先镜像内有些命令是不支持的，比如tomcat镜像没有ls、top等命令（jdk镜像默认有这些命令），不方便查找问题，tomcat的openjdk版本是有的，如tomcat:8.5.49-jdk8-openjdk

参考
碰到系统CPU飙高和频繁GC，你会怎么排查？
Linux使用jstat命令查看jvm的GC情况

小知识

1 根据容器名查看对应宿主机的进程id

docker inspect -f '{{.State.Pid}}' competent_lichterman

2 查看gc情况，9是进程id ，每5s自动刷新一次。

jstat -gc 9 5000

full gc的次数太多说明程序有问题，fgc 期间会导致服务不可用，后面有fgc的时间

图片.png

3 查看jvm线程的堆栈信息。查看进程id为1的线程信息，包含iot代码字样的附近30行内容，一般程序出问题，这里会定位到有问题的代码部分

jstack 1 | grep "iot" -A 30

导出线程信息到文本
jstack -l <pid> >jvm.txt

4 查看整个JVM内存状态
jmap -heap [pid]
查看JVM堆中对象详细占用情况
jmap -histo [pid]
导出整个JVM 中内存信息
jmap -dump:format=b,file=文件名 [pid]
jmap -dump:live,format=b,file=d:/20170417.dump PID

[root@01384fe8ffb3 /]# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 2092957696 (1996.0MB)
   NewSize                  = 44040192 (42.0MB)
   MaxNewSize               = 697303040 (665.0MB)
   OldSize                  = 88080384 (84.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 490733568 (468.0MB)
   used     = 24698304 (23.55413818359375MB)
   free     = 466035264 (444.44586181640625MB)
   5.032935509314904% used
From Space:
   capacity = 6815744 (6.5MB)
   used     = 6531296 (6.228729248046875MB)
   free     = 284448 (0.271270751953125MB)
   95.82660381610577% used
To Space:
   capacity = 14155776 (13.5MB)
   used     = 0 (0.0MB)
   free     = 14155776 (13.5MB)
   0.0% used
PS Old Generation
   capacity = 150470656 (143.5MB)
   used     = 50911384 (48.552879333496094MB)
   free     = 99559272 (94.9471206665039MB)
   33.83475911741888% used

35304 interned Strings occupying 4184288 bytes.
[root@01384fe8ffb3 /]# 

问题排查思路

1通过docker stats 查看服务cpu、内存占用情况，定位有问题的服务

图片.png

2 进入容器top命令查看cpu占用最多的进程id，一般jdk镜像java进程pid都是1

图片.png

3 查看进程中各线程占用情况
top -Hp 1

图片.png

4 jstack -F 线程id ：即可查看某线程正在运行情况(不加F会报错！！！)

[root@826d95fa3a92 /]# jstack -F 167
Attaching to process ID 167, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11
Deadlock Detection:

No deadlocks found.

Thread 436: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:482)
        at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
        at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:93)
        at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
        at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:86)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
        at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
        at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLo

5导出堆内存数据
jmap -dump:format=b,file=myjvm.dump 1