用户侧反馈某个业务出问题了，找到相关工程，认为是老问题，选择先重启工程实例，再根据历史日志做相关排查工作。
意外的是，重启工程实例后，进程能起来，但是没有端口监听，看启动日志也没有明显指向，只好通过 jstack工具导出相关线程信息分析，发现有工程日志中没有打印的rabbitm相关字眼，高度怀疑是rabbitMQ服务端出了什么问题了。

最终通过重启rabbitmMQ服务端，再重启工程，才解决。
那么问题出在哪里呢？

Online Java Thread Dump Analyzer
http://spotify.github.io/threaddump-analyzer

上传 工程_jstsack.txt ，做在线解析

main
PRIORITY : 5

THREAD ID : 0X00007F622800C800

NATIVE ID : 0X1A7B4

NATIVE ID (DECIMAL) : 108468

STATE : WAITING


stackTrace:
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f87c8ea8> (a com.rabbitmq.utility.BlockingValueOrException)
at java.lang.Object.wait(Object.java:502)
at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:49)
- locked <0x00000000f87c8ea8> (a com.rabbitmq.utility.BlockingValueOrException)
at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:91)
- locked <0x00000000f87c8ea8> (a com.rabbitmq.utility.BlockingValueOrException)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:32)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:366)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:229)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:117)
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:948)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.queueDeclare(AutorecoveringChannel.java:333)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:980)
- locked <0x00000000f87c8f60> (a java.lang.Object)
at com.sun.proxy.$Proxy51.queueDeclare(Unknown Source)
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueues(RabbitAdmin.java:575)
at org.springframework.amqp.rabbit.core.RabbitAdmin.access$200(RabbitAdmin.java:66)
at org.springframework.amqp.rabbit.core.RabbitAdmin$3.doInRabbit(RabbitAdmin.java:210)
at org.springframework.amqp.rabbit.core.RabbitAdmin$3.doInRabbit(RabbitAdmin.java:207)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1455)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1411)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1387)
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueue(RabbitAdmin.java:207)
at com.kingdee.mobile.eclite.trd.mq.message.MessageTodoMQConsumer.todoMsgCenterServiceQueue(MessageTodoMQConsumer.java:80)
at com.kingdee.mobile.eclite.trd.mq.message.MessageTodoMQConsumer.declareQueues(MessageTodoMQConsumer.java:69)
at com.kingdee.mobile.eclite.trd.mq.message.MessageTodoMQConsumer.init(MessageTodoMQConsumer.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1622)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
- locked <0x000000008022b6d0> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
- locked <0x000000008022b928> (a java.lang.Object)
at com.kingdee.mobile.netty.server.App.start(App.java:83)
at com.kingdee.mobile.netty.server.Main.main(Main.java:80)

image.png image.png

rabbitMQ服务端日志

image.png

参考

locked <0x00000000f8cbd8c8> (a com.rabbitmq.utility.BlockingValueOrException) #264
https://github.com/rabbitmq/rabbitmq-java-client/issues/264

com.rabbitmq.utility.BlockingValueOrException
https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/utility/class-use/BlockingValueOrException.html

Universal GC Log Analyzer
https://gceasy.io