系统监控-Metric

Metrics is a Java library which gives you unparalleled insight into what your code does in production.
Metrics provides a powerful toolkit of ways to measure the behavior of critical components in your production environment.

简单的说Metrics是一个监控系统性能指标的一个Java库。Cassandra使用它进行指标统计。

---

在Maven的pom.xml中引入Metrics：

<dependencies>
    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-core</artifactId>
        <version>${metrics.version}</version>
    </dependency>
</dependencies>

---

Metrics使用非常简单，首先是Metrics的核心类

MetricRegistry类

The centerpiece of Metrics is the MetricRegistry class, which is the container for all your application’s metrics. Go ahead and create a new one:

我们一般直接new出Registry对象就可以完成Registry的创建。

final MetricRegistry metrics = new MetricRegistry();

MetricRegistryclass中维护了一个Metric集合和一个MetricRegistryListener列表

    private final ConcurrentMap<String, Metric> metrics;
    private final List<MetricRegistryListener> listeners;
    /**
     * Creates a new {@link MetricRegistry}.
     */
    public MetricRegistry() {
        this.metrics = buildMap();
        this.listeners = new CopyOnWriteArrayList<MetricRegistryListener>();
    }

Metric的注册、快速创建、获取等等的基本功能，都会维护Metric集合。

// 注册
    public <T extends Metric> T register(String name, T metric) throws IllegalArgumentException {
        if (metric instanceof MetricSet) {
            registerAll(name, (MetricSet) metric);
        } else {
            final Metric existing = metrics.putIfAbsent(name, metric);
            if (existing == null) {
                onMetricAdded(name, metric);
            } else {
                throw new IllegalArgumentException("A metric named " + name + " already exists");
            }
        }
        return metric;
    }
// 创建+注册 or 获取
    public Meter meter(String name) {
        return getOrAdd(name, MetricBuilder.METERS);
    }
    public Timer timer(String name) {
        return getOrAdd(name, MetricBuilder.TIMERS);
    }
// 获取
    public SortedMap<String, Gauge> getGauges() {
        return getGauges(MetricFilter.ALL);
    }
// etc

JmxReporterstrat的时候会向MetricRegistry注册一个JmxListenerclass，MetricRegistry会将它维护到MetricRegistryListener列表中。 JmxReporter会在后面介绍到，它是统计结果的输出控制类。MetricRegistry会在Metricadd or remove时将Metric传递给所有的JmxListener。JmxListener是Metric的真正使用者。
JmxReporter:

    private final JmxListener listener;

    private JmxReporter(MBeanServer mBeanServer,
                        String domain,
                        MetricRegistry registry,
                        MetricFilter filter,
                        MetricTimeUnits timeUnits, 
                        ObjectNameFactory objectNameFactory) {
        this.registry = registry;
        this.listener = new JmxListener(mBeanServer, domain, filter, timeUnits, objectNameFactory);
    }
    public void start() {
        registry.addListener(listener);
    }

MetricRegistry:

    private void onMetricAdded(String name, Metric metric) {
        for (MetricRegistryListener listener : listeners) {
            notifyListenerOfAddedMetric(listener, metric, name);
        }
    }

---

Reporter接口

它是Metrics输出控制类的统一接口，它将采集的数据展现到不同的位置。Metrics提供两个实现类JmxReporter和ScheduledReporter。
JmxReporter会通过JMX报告指标，它需要依赖metrics-jmx包

<dependency>
    <groupId>io.dropwizard.metrics</groupId>
    <artifactId>metrics-jmx</artifactId>
    <version>${metrics.version}</version>
</dependency>

final JmxReporter reporter = JmxReporter.forRegistry(registry).build();
reporter.start();

ScheduledReporter顾名思义，会定时的输出报表。ConsoleReporter、Slf4jReporter等等Metric自带的所以Reporter都是继承于它，一般我们自定义的Reporter也会继承它。

A Console Reporter is exactly what it sounds like - report to the console. This reporter will print every second.

ConsoleReporter reporter = ConsoleReporter.forRegistry(metrics)
       .convertRatesTo(TimeUnit.SECONDS)
       .convertDurationsTo(TimeUnit.MILLISECONDS)
       .build();
   reporter.start(1, TimeUnit.SECONDS);

---

Metrics

最后来看下Metrics提供了五个基本的Metrics ：

1. Meters（TPS）

A meter measures the rate of events over time (e.g., “requests per second”). In addition to the mean rate, meters also track 1-, 5-, and 15-minute moving averages.

使用起来非常简单，这里暂不多介绍，只从官网复制了些实例代码展示下如何使用：

private final MetricRegistry metrics = new MetricRegistry();
private final Meter requests = metrics.meter("requests");

public void handleRequest(Request request, Response response) {
    requests.mark();
    // etc
}

结果：

19-6-11 10:09:32 ===============================================================

-- Meters ----------------------------------------------------------------------
meter
             count = 56
         mean rate = 1.00 events/second
     1-minute rate = 1.00 events/second
     5-minute rate = 1.00 events/second
    15-minute rate = 1.00 events/second

还有一些常用方法：

    @Override
    public long getCount() {
        return count.sum();
    }
    @Override
    public double getOneMinuteRate() {
        tickIfNecessary();
        return m1Rate.getRate(TimeUnit.SECONDS);
    }
// etc

2. Gauges
   如果说上面Meters是一个动态的统计的话，Gauges就是一个静态的统计。它能统计出关注对象当时的状态。

A gauge is an instantaneous measurement of a value. For example, we may want to measure the number of pending jobs in a queue:

public class QueueManager {
    private final Queue queue;

    public QueueManager(MetricRegistry metrics, String name) {
        this.queue = new Queue();
        metrics.register(MetricRegistry.name(QueueManager.class, name, "size"),
                         new Gauge<Integer>() {
                             @Override
                             public Integer getValue() {
                                 return queue.size();
                             }
                         });
    }
}

MetricRegistry.name(QueueManager.class, "jobs", "size");

它的效率受时间复杂度影响

For most queue and queue-like structures, you won’t want to simply return queue.size(). Most of java.util and java.util.concurrent have implementations of #size() which are O(n), which means your gauge will be slow (potentially while holding a lock).

3. Counters
   类似于一个AtomicLong。

private final Counter pendingJobs = metrics.counter(name(QueueManager.class, "pending-jobs"));

public void addJob(Job job) {
    pendingJobs.inc();
    queue.offer(job);
}

public Job takeJob() {
    pendingJobs.dec();
    return queue.take();
}

官方推荐使用#counter(String)代替#register(String, Metric)创建。

As you can see, the API for counters is slightly different: #counter(String) instead of #register(String, Metric). While you can use register and create your own Counter instance, #counter(String) does all the work for you, and allows you to reuse metrics with the same name.
Also, we’ve statically imported MetricRegistry’s name method in this scope to reduce clutter.

4. Histograms（直方图）
   直方图是一种用于统计数据的图表，Histogram能统计数据的最小值、最大值、平均值等，还有测量中位数、75位、90位、95位、98位、99位和99.9。

A histogram measures the statistical distribution of values in a stream of data. In addition to minimum, maximum, mean, etc., it also measures median, 75th, 90th, 95th, 98th, 99th, and 99.9th percentiles.

private final Histogram responseSizes = metrics.histogram(name(RequestHandler.class, "response-sizes"));

public void handleRequest(Request request, Response response) {
    // etc
    responseSizes.update(response.getContent().length);
}

有一点在官网上特别提醒：历史数据的大小之间影响到了Histogram的响应速度，所以Histogram都需要设定了默认大小。

This histogram will measure the size of responses in bytes.

Histogram基本功能都是通过内部的Reservoir来实现的。使用#histogram(String)来创建会默认使用ExponentiallyDecayingReservoir，并且他的默认大小是1028。

        MetricBuilder<Histogram> HISTOGRAMS = new MetricBuilder<Histogram>() {
            @Override
            public Histogram newMetric() {
                return new Histogram(new ExponentiallyDecayingReservoir());
            }

            @Override
            public boolean isInstance(Metric metric) {
                return Histogram.class.isInstance(metric);
            }
        };

5. Timers
   Timer与 Meter类似，但它的功能比Meter强大很多。Meter只统计次数，Timer会统计过程的耗时，并提供更丰富的输出统计。

private final Timer responses = metrics.timer(name(RequestHandler.class, "responses"));

public String handleRequest(Request request, Response response) {
    final Timer.Context context = responses.time();
    try {
        // etc;
        return "OK";
    } finally {
        context.stop();
    }
}

在使用上，我们需在计时开始处创建Context，出口处调用#context.stop()。
简单看下源码，我们能发现Timer内部其实就是使用 Meter和Histogram来实现的。

    /**
     * Creates a new {@link Timer} that uses the given {@link Reservoir} and {@link Clock}.
     *
     * @param reservoir the {@link Reservoir} implementation the timer should use
     * @param clock  the {@link Clock} implementation the timer should use
     */
    public Timer(Reservoir reservoir, Clock clock) {
        this.meter = new Meter(clock);
        this.clock = clock;
        this.histogram = new Histogram(reservoir);
    }

    private void update(long duration) {
        if (duration >= 0) {
            histogram.update(duration);
            meter.mark();
        }
    }

---

Health Checks

Metrics还提供了服务的集中健康检查。
这需要在Maven的pom.xml中引入新的Metrics包：

<dependencies>
    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-healthchecks</artifactId>
        <version>${metrics.version}</version>
    </dependency>
</dependencies>

基本使用流程：

1. 创建各种健康检查的HealthCheck。

public class DatabaseHealthCheck extends HealthCheck {
    private final Database database;

    public DatabaseHealthCheck(Database database) {
        this.database = database;
    }

    @Override
    public HealthCheck.Result check() throws Exception {
        if (database.isConnected()) {
            return HealthCheck.Result.healthy();
        } else {
            return HealthCheck.Result.unhealthy("Cannot connect to " + database.getUrl());
        }
    }
}

2. 将HealthCheck注册到HealthCheckRegistry。

final HealthCheckRegistry healthChecks = new HealthCheckRegistry();

healthChecks.register("postgres", new DatabaseHealthCheck(database));

3. 运行HealthCheckRegistry:#runHealthChecks()。

final Map<String, HealthCheck.Result> results = healthChecks.runHealthChecks();

4. 通过遍历返回值判断健康状态。

for (Entry<String, HealthCheck.Result> entry : results.entrySet()) {
    if (entry.getValue().isHealthy()) {
        System.out.println(entry.getKey() + " is healthy");
    } else {
        System.err.println(entry.getKey() + " is UNHEALTHY: " + entry.getValue().getMessage());
        final Throwable e = entry.getValue().getError();
        if (e != null) {
            e.printStackTrace();
        }
    }
}

---

JVM Instrumentation

这是Metrics中对于jvm的监控。

The metrics-jvm module contains a number of reusable gauges and metric sets which allow you to easily instrument JVM internals.

它需要引入对于Jar包：

<dependencies>
    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-jvm</artifactId>
        <version>${metrics.version}</version>
    </dependency>
</dependencies>

Metrics提供了各种jvm的指标metric和MetricSet子类，在子类对ManagementFactory各种获取属性进行了分装，来实现对jvm的监控。

    /**
     * Creates a new set of gauges using the default MXBeans.
     * Caches the information for the given interval and time unit.
     *
     * @param interval cache interval
     * @param unit     cache interval time unit
     */
    public CachedThreadStatesGaugeSet(long interval, TimeUnit unit) {
        this(ManagementFactory.getThreadMXBean(), new ThreadDeadlockDetector(), interval, unit);
    }

    public ClassLoadingGaugeSet() {
        this(ManagementFactory.getClassLoadingMXBean());
    }

    /**
     * Creates a new gauge using the platform OS bean.
     */
    public FileDescriptorRatioGauge() {
        this(ManagementFactory.getOperatingSystemMXBean());
    }

// etc

它支持的指标包括：

• 运行所有支持的垃圾收集器的计数和已用时间
• 所有内存池的内存使用情况，包括堆外内存
• 线程状态的细分，包括死锁
• 文件描述符用法
• 缓冲池大小和利用率
  BufferPoolMetricSet：JVM的直接缓冲池和映射缓冲池的数量统计。
  ThreadStatesGaugeSet：检测处于不同状态的线程数和死锁检测的Gauge。
  CachedThreadStatesGaugeSet：同上，多创建了一个CachedGauge。
  ClassLoadingGaugeSet：JVM类加载器的Gauge。
  FileDescriptorRatioGauge：是对于操作系统FileDescriptor的一个统计，可以用来监控IO流的释放等等。
  GarbageCollectorMetricSet：可以统计GC的次数和使用时间。
  JmxAttributeGauge：主要统计JVM的runtime，还在Gauge获取了JVM的名称、供应商信息。
  MemoryUsageGaugeSet：对于内存的监控（包含堆内内存和堆外内存）。

---

---

Metric官方Document