记一次服务被自动重启的线上事故

一、背景
生产环境的服务总共有5个，晚上7点30分左右，有其中一个发生了自动重启。在4月19日发生过一次，5月6日又再次发生。

二、思路
1、5个节点只有其中1个发生了自动重启，说明大概率不是程序代码的问题。
2、为什么被重启，是进程被操作系统杀死了？还是程序自身宕机了？
3、如果是操作系统层面的问题，需要查看机器的监控，可以从zabbix监控判断。
4、如果是程序自身宕机，需要查看jvm的监控，这块可以从prometheus监控判断。

5、上述都没有问题的话，还有一种排查方法，也就是程序打印的日志。

三、zabbix监控
1、内存不足导致进程被杀死？

内存.png

可以看出，在服务被杀掉的前后，内存释放4G左右。但是这里的空闲内存有15G多，并不是内存不足。

三、prometheus监控
1、难道是jvm进程自己宕掉的？
首先查看是否有导出堆内存快照。
因为我们在启动脚本有增加以下配置。

#出现 OOME 时生成堆 dump: 
-XX:+HeapDumpOnOutOfMemoryError
#生成堆文件地址：
-XX:HeapDumpPath=/home/opt/jvmlogs/

其次，通过grafana查看堆内存以及线程情况。

堆和栈.png

可以排查，并不是jvm进程自己挂掉的！！

四、ELK日志
1、到这一步，我们只能依赖于日志。

  o.s.b.a.e.ElasticsearchHealthIndicator   : Elasticsearch health check failed\n\norg.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.\n\tat org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71) ~[elasticsearch-5.6.8.jar!/:5.6.8]\n\tat org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:58) ~[elasticsearch-5.6.8.jar!/:5.6.8]\n\tat org.springframework.boot.actuate.elasticsearch.ElasticsearchHealthIndicator.doHealthCheck(ElasticsearchHealthIndicator.java:80) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) [spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:50) [spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat sun.reflect.GeneratedMethodAccessor1685.invoke(Unknown Source) ~[na:na]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]\n\tat org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223) [spring-core-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) [spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61) [spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:243) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:299) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat sun.reflect.GeneratedMethodAccessor1629.invoke(Unknown Source) ~[na:na]\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:870) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:776) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:870) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]\n\tat io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat com.xhtech.cloudwork.infrastructure.configurer.AbstractRequestLoggingFilter.doFilterInternal(AbstractRequestLoggingFilter.java:268) ~[classes!/:3.29.6]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat com.xhtech.cloudwork.config.spring.ContentEncodingFilter.doFilterInternal(ContentEncodingFilter.java:35) ~[classes!/:3.29.6]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:84) ~[spring-boot-actuator-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat com.xhtech.arch.ddd.jwt.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:78) ~[ddd-core-1.2.1.jar!/:na]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]\n\tat org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:162) ~[spring-cloud-sleuth-core-2.0.0.M7.jar!/:2.0.0.M7]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.server.handlers.GracefulShutdownHandler.handleRequest(GracefulShutdownHandler.java:69) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.server.Connectors.executeRootHandler(Connectors.java:336) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181]\n\tat java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\n\tat org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232) ~[elasticsearch-5.6.8.jar!/:5.6.8]\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67) ~[elasticsearch-5.6.8.jar!/:5.6.8]\n\tat org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69) ~[elasticsearch-5.6.8.jar!/:5.6.8]\n\t... 121 common frames omitted\n"

接下来，看到的日志就是consul的了。

  ConfigServletWebServerApplicationContext : Exception thrown from ApplicationListener handling ContextClosedEvent\n\ncom.ecwid.consul.v1.OperationException: OperationException(statusCode=403, statusMessage='Forbidden', statusContent='Permission denied')\n\tat com.ecwid.consul.v1.agent.AgentConsulClient.agentServiceDeregister(AgentConsulClient.java:296) ~[consul-api-1.3.0.jar!/:na]\n\tat com.ecwid.consul.v1.agent.AgentConsulClient.agentServiceDeregister(AgentConsulClient.java:284) ~[consul-api-1.3.0.jar!/:na]\n\tat com.ecwid.consul.v1.ConsulClient.agentServiceDeregister(ConsulClient.java:313) ~[consul-api-1.3.0.jar!/:na]\n\tat org.springframework.cloud.consul.serviceregistry.ConsulServiceRegistry.deregister(ConsulServiceRegistry.java:86) ~[spring-cloud-consul-discovery-2.0.0.M6.jar!/:2.0.0.M6]\n\tat org.springframework.cloud.consul.serviceregistry.ConsulServiceRegistry.deregister(ConsulServiceRegistry.java:41) ~[spring-cloud-consul-discovery-2.0.0.M6.jar!/:2.0.0.M6]\n\tat org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.deregister(AbstractAutoServiceRegistration.java:226) ~[spring-cloud-commons-2.0.0.M8.jar!/:2.0.0.M8]\n\tat org.springframework.cloud.consul.serviceregistry.ConsulAutoServiceRegistration.deregister(ConsulAutoServiceRegistration.java:100) ~[spring-cloud-consul-discovery-2.0.0.M6.jar!/:2.0.0.M6]\n\tat org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.stop(AbstractAutoServiceRegistration.java:241) ~[spring-cloud-commons-2.0.0.M8.jar!/:2.0.0.M8]\n\tat org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration$$FastClassBySpringCGLIB$$e099d3fe.invoke(\u003cgenerated\u003e) ~[spring-cloud-commons-2.0.0.M8.jar!/:2.0.0.M8]\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747) ~[spring-aop-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:156) ~[spring-retry-1.2.2.RELEASE.jar!/:na]\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.cloud.consul.serviceregistry.ConsulAutoServiceRegistration$$EnhancerBySpringCGLIB$$20ff647e.stop(\u003cgenerated\u003e) ~[spring-cloud-consul-discovery-2.0.0.M6.jar!/:2.0.0.M6]\n\tat com.xhtech.arch.ddd.lifecycle.GracefulShutdownConfiguration$1.deregister(GracefulShutdownConfiguration.java:33) ~[ddd-core-1.2.1.jar!/:na]\n\tat com.xhtech.arch.ddd.lifecycle.undertow.GracefulShutdown.onApplicationEvent(GracefulShutdown.java:35) ~[ddd-core-1.2.1.jar!/:na]\n\tat com.xhtech.arch.ddd.lifecycle.undertow.GracefulShutdown.onApplicationEvent(GracefulShutdown.java:18) ~[ddd-core-1.2.1.jar!/:na]\n\tat org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399) ~[spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:353) ~[spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:996) ~[spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n\tat org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:933) [spring-context-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]\n"

2、从上面两个日志，可以分析出以下结果。
a、spring boot在检测es的时候，出现了异常ElasticsearchTimeoutException。应该是存在超时的异常。
b、consul的报错信息里，说明这是在停机，尝试将服务从consul主动下线。
从GracefulShutdown字眼可以看出，这里的acl token应该是不对，导致了报错。
c、从第二结果中可以分析出，服务进程绝不是被操作系统杀死的，应该是人为停服。否则不太可能优雅停机，而是直接kill -9了。

d、引出一个假设，有没有可能是es不健康，然后被认为需要重启，最后触发了优雅停机呢？

五、总结
spring boot是有提供/hleath接口，以此判断应用是否健康，我们本次出现事故的原因就是在于此，服务被认为不健康，立马就被重启了。

建议，将一些不必要的健康开关禁用，省去一些可能由于网络抖动引起的超时检测不健康导致的失误。

# actuator
management:
  health:
    elasticsearch:
      enabled: false