数据采集，分析及调用链追踪
2017-04-09 本文已影响2433人
Jeff
要解决的问题
对应用进行运行时数据收集，提供追踪和报告。
- 数据采集（埋点或者自动获取），存储和分析
- 生成调用链 并能在UI展示
- 自动分析并发现报告问题，例如：性能
- 和测试运维工具集成（？？？）
相应的组件有：
- model：采集模型
- agent/collector：agent提供埋点，采集和发送
- aggregator/storage：聚合，加工和存储数据
- analyzer：分析数据
- UI：展示调用链，dashboard和问题报告，告警设置和触发
考察如下的一些开源方案：
- Zipkin
- Pinpoint
- CAT
Zipkin
项目地址：https://github.com/openzipkin/zipkin
架构：http://zipkin.io/pages/architecture.html
以下是数据流图
Zipkin architecture
提供REST API支持，可以查询和推送数据：http://zipkin.io/zipkin-api
快速开始：下载源码后在根目录运行
./mvnw -DskipTests --also-make -pl zipkin-server clean install
编译UI模块可能遇到node-saas找不到的信息，解决办法：
- 把node_modules目录删除重新运行node install
- 运行npm rebuild node-sass
- 升级node和npm，可能编译好的node-sass和你本地的node版本不一致()，会报错：Error: Node Sass does not yet support your current environment: OS X 64-bit with Unsupported runtime (51)，建议用nvm管理node安装版本。
启动通过运行： java -jar ./zipkin-server/target/zipkin-server-*-exec.jar，然后访问: http://localhost:9411 (REST API: http://localhost:9411/api/v1/spans ...)
数据格式：
- traceId
- name
- id
- parentId
- timestamp
- duration
- annotations: [
endpoint: {
...
}
]
- binaryAnnotations: [
{
- key
- value
- endpoint: {
...
}
}
]
数据样例：
[
{
"traceId": "bd7a977555f6b982",
"name": "get",
"id": "bd7a977555f6b982",
"timestamp": 1458702548467000,
"duration": 386000,
"annotations": [
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548467000,
"value": "sr"
},
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548853000,
"value": "ss"
}
],
"binaryAnnotations": []
},
{
"traceId": "bd7a977555f6b982",
"name": "get-traces",
"id": "ebf33e1a81dc6f71",
"parentId": "bd7a977555f6b982",
"timestamp": 1458702548478000,
"duration": 354374,
"annotations": [],
"binaryAnnotations": [
{
"key": "lc",
"value": "JDBCSpanStore",
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
}
},
{
"key": "request",
"value": "QueryRequest{serviceName=zipkin-query, spanName=null, annotations=[], binaryAnnotations={}, minDuration=null, maxDuration=null, endTs=1458702548478, lookback=86400000, limit=1}",
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
}
}
]
},
{
"traceId": "bd7a977555f6b982",
"name": "query",
"id": "be2d01e33cc78d97",
"parentId": "ebf33e1a81dc6f71",
"timestamp": 1458702548786000,
"duration": 13000,
"annotations": [
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548786000,
"value": "cs"
},
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548799000,
"value": "cr"
}
],
"binaryAnnotations": [
{
"key": "jdbc.query",
"value": "select distinct `zipkin_spans`.`trace_id` from `zipkin_spans` join `zipkin_annotations` on (`zipkin_spans`.`trace_id` = `zipkin_annotations`.`trace_id` and `zipkin_spans`.`id` = `zipkin_annotations`.`span_id`) where (`zipkin_annotations`.`endpoint_service_name` = ? and `zipkin_spans`.`start_ts` between ? and ?) order by `zipkin_spans`.`start_ts` desc limit ?",
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
}
},
{
"key": "sa",
"value": true,
"endpoint": {
"serviceName": "spanstore-jdbc",
"ipv4": "127.0.0.1",
"port": 3306
}
}
]
},
{
"traceId": "bd7a977555f6b982",
"name": "query",
"id": "13038c5fee5a2f2e",
"parentId": "ebf33e1a81dc6f71",
"timestamp": 1458702548817000,
"duration": 1000,
"annotations": [
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548817000,
"value": "cs"
},
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548818000,
"value": "cr"
}
],
"binaryAnnotations": [
{
"key": "jdbc.query",
"value": "select `zipkin_spans`.`trace_id`, `zipkin_spans`.`id`, `zipkin_spans`.`name`, `zipkin_spans`.`parent_id`, `zipkin_spans`.`debug`, `zipkin_spans`.`start_ts`, `zipkin_spans`.`duration` from `zipkin_spans` where `zipkin_spans`.`trace_id` in (?)",
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
}
},
{
"key": "sa",
"value": true,
"endpoint": {
"serviceName": "spanstore-jdbc",
"ipv4": "127.0.0.1",
"port": 3306
}
}
]
},
{
"traceId": "bd7a977555f6b982",
"name": "query",
"id": "37ee55f3d3a94336",
"parentId": "ebf33e1a81dc6f71",
"timestamp": 1458702548827000,
"duration": 2000,
"annotations": [
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548827000,
"value": "cs"
},
{
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"timestamp": 1458702548829000,
"value": "cr"
}
],
"binaryAnnotations": [
{
"key": "jdbc.query",
"value": "select `zipkin_annotations`.`trace_id`, `zipkin_annotations`.`span_id`, `zipkin_annotations`.`a_key`, `zipkin_annotations`.`a_value`, `zipkin_annotations`.`a_type`, `zipkin_annotations`.`a_timestamp`, `zipkin_annotations`.`endpoint_ipv4`, `zipkin_annotations`.`endpoint_port`, `zipkin_annotations`.`endpoint_service_name` from `zipkin_annotations` where `zipkin_annotations`.`trace_id` in (?) order by `zipkin_annotations`.`a_timestamp` asc, `zipkin_annotations`.`a_key` asc",
"endpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
}
},
{
"key": "sa",
"value": true,
"endpoint": {
"serviceName": "spanstore-jdbc",
"ipv4": "127.0.0.1",
"port": 3306
}
}
]
}
]
在UI上的展示：
Pinpoint
https://github.com/naver/pinpoint
https://github.com/naver/pinpoint/wiki/Technical-Overview-Of-Pinpoint
https://sconts.com/11
- 对代码无侵入，分析request／response
- 插件式
- 支持的代码文件有限
美团CAT
http://tech.meituan.com/CAT_in_Depth_Java_Application_Monitoring.html
http://unidal.org/cat/r/home?op=view&docName=develop
https://github.com/dianping/cat
- 埋点收集
