可观测性三大支柱

• Traces：分布式链路跟踪，提供了一个请求从接收到处理完成整个生命周期的跟踪路径
• Metrics：提供cpu、请求延迟、用户访问数等Counter、Gauge、Histogram指标
• Logs：传统的日志，提供精确的系统记录

很长一段时间内，这三者是独立存在的
随着时间的推移，发现这三者是相互关联，相辅相成，组合在一起看，更容易发现和定位问题

OpenTelemetry

OpenTelemetry is split into two parts:

• an API to instrument code with 提供了API接口定义
• and SDKs that implement the API. 以及一套SDK实现

---

以OpenTelemetry-Go为例，关注OpenTelemetry的链路分析

链路最基本的能力：采集+上报
要实现采集上报的能力，先熟悉OpenTelemetry几个概念：

- Tracer   ---|
- Exporter ---|-->TracerProvider
- Resource ---|

Tracer

Tracer 负责采集链路数据(instrument the method chain)
说白了，Tracer的职责就是在链路上你想监测的位置埋点收集数据

The OpenTelemetry Tracing API provides a Tracer to create traces. These Tracers are designed to be associated with one instrumentation library.

what is trace

a trace is a type of telemetry that represents work being done by a service. A trace is a record of the connection(s) between participants processing a transaction, often through client/server requests processing and other forms of communication. Each part of the work that a service performs is represented in the trace by a span

trace = Spans的有向无环图DAG
span = description, timestamp and key-value tag

单个 Trace 中的 Spans 之间的因果关系

        [Span A]  ←←←(the root span)
            |
     +------+------+
     |             |
 [Span B]      [Span C] ←←←(Span C is a `child` of Span A)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F]

单个 Trace 中的 Spans 之间的时间关系

––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–> time

 [Span A···················································]
   [Span B··············································]
      [Span D··········································]
    [Span C········································]
         [Span E·······]        [Span F··]

代码中Tracer的使用

// golang
// Run starts this app
func (a *App) Run(ctx context.Context) error {
    for {
        // Each execution of the run loop, we should get a new "root" span and context.
        // otel.GetTracerProvider().Tracer(name) loads a named Tracer, Start generates a new span
        newCtx, span := otel.GetTracerProvider().Tracer(name).Start(ctx, "Run")

        n, err := a.Poll(newCtx)
        if err != nil {
            span.End()
            return err
        }

        a.Write(newCtx, n)
        // span ends here
        span.End()
    }
}

The above code creates a span for every iteration of the for loop. The span is created using a Tracer from the global TracerProvider. (more about TracerProvider is in the later section)

Exporter

Exporters allow telemetry data to be emitted somewhere - either to the console, or to a remote system or collector for further analysis and/or enrichment.

OpenTelemetry supports a variety of exporters through its ecosystem including popular open source tools like Jaeger, Zipkin, and Prometheus.

说白了，Exporter就是导出数据用的

// newExporter returns a console exporter.
func newExporter(w io.Writer) (trace.SpanExporter, error) {
    return stdouttrace.New(
        stdouttrace.WithWriter(w),
        // Use human-readable output.
        stdouttrace.WithPrettyPrint(),
        // Do not print timestamps for the demo.
        stdouttrace.WithoutTimestamps(),
    )
}

Resource

OpenTelemetry uses a Resource to represent the entity producing telemetry.
Telemetry data can be crucial to solving issues with a service. The catch is, you need a way to identify what service, or even what service instance, that data is coming from.

说白了，Resource是用来标识telemetry数据的，是telemetry数据的身份证

// newResource returns a resource describing this application.
func newResource() *resource.Resource {
    r, _ := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("fib"),
            semconv.ServiceVersionKey.String("v0.1.0"),
            attribute.String("environment", "demo"),
        ),
    )
    return r
}

Tracer Provider

You have your application instrumented to produce telemetry data and you have an exporter to send that data to the console, but how are they connected? This is where the TracerProvider is used.
It is a centralized point where instrumentation will get a Tracer from and funnels the telemetry data from these Tracers to export pipelines.

TracerProvider串联了 Tracer、Resource、Exporter，内部自动实现了Tracer data with referenced Resource will be sent to Exporter automaticly

只要注册了TracerProvider，trace链中所有埋点采集的数据就与Exporter打通了

usage example

func main() {
    l := log.New(os.Stdout, "", 0)

    // Write telemetry data to a file.
    f, err := os.Create("traces.txt")
    if err != nil {
        l.Fatal(err)
    }
    defer f.Close()

    // exporter
    exp, err := newExporter(f)
    if err != nil {
        l.Fatal(err)
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exp),
        trace.WithResource(newResource()),
    )
    defer func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            l.Fatal(err)
        }
    }()

    // SetTracerProvider registers `tp` as the global trace provider.
    otel.SetTracerProvider(tp)

    /* … */
}

more for https://opentelemetry.io/docs/instrumentation/go/getting-started/#sdk-installation