Prometheus监控nginx
2019-12-11 本文已影响0人
baiyongjie
配置nginx-module-vts模块
收集nginx指标的较多采用的有nginx-vts模块，prometheus-lua两种方式进行采集，本文采用nginx-vts模块方式进行数据收集。
nginx配置
查看是否安装了nginx-vts模块
$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.0
...
configure arguments: --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
安装nginx-vts模块
$ cd /root/
$ git clone https://github.com/vozlt/nginx-module-vts
$ cd /usr/local/src/nginx-1.10.0
$ ./configure --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
$ make
# 使用新编译的nginx执行文件替换掉旧的
$ mv /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx.bak.20191211
$ cp objs/nginx /usr/local/nginx/sbin/
# 生成新的nginx.pid文件
$ kill -USR2 `cat /usr/local/nginx/logs/nginx.pid`
# 退出旧版本低nginx
$ /usr/local/nginx/sbin/nginx.bak.20191211 -s stop
# 查看最新的编译参数
$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.0
...
configure arguments: --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module
添加nginx配置文件,暴露监控指标
$ /usr/local/nginx/conf/nginx.conf
http {
...
vhost_traffic_status_zone;
...
}
$ vim /usr/local/nginx/conf/vhost/nginx-vts-status.conf
server {
listen 8088;
location /status {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
}
#需要重启模块才能生效
$ /usr/local/nginx/sbin/nginx -s stop
$ /usr/local/nginx/sbin/nginx
然后访问ip:8088/status,可以看到下面的数据
image
指标的数据类型转换
要将获取到的数据接入prometheus还需要将数据转为metrics类型的数据，vts模块中直接提供了/status/format/prometheus 接口，访问IP:8088/status/format/prometheus 即可看到转换后的数据。
image
部署nginx-vts-exporter
默认端口为 :9913/metrics
$ wget http://download.baiyongjie.com/linux/prometheus/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
$ tar zxvf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
$ cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter /usr/local/node_exporter/
$ mv nginx-vts-exporter-0.10.3.linux-amd64/ /usr/local/src/
$ cd /tmp/; nohup /usr/local/node_exporter/nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1:8088/status/format/json &> /dev/null &
$ netstat -nptl|grep 9913
tcp6 0 0 :::9913 :::* LISTEN 27435/nginx-vts-exp
$ curl -s 127.0.0.1:9913/metrics|tail
nginx_upstream_responseMsec{backend="100.115.173.4:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.5:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.6:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.7:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.8:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="100.115.173.9:80",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="127.0.0.1:9000",upstream="::nogroups"} 0
# HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.
# TYPE nginx_vts_exporter_build_info gauge
nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.10",revision="8aa2881c7050d9b28f2312d7ce99d93458611d04",version="0.10.3"} 1
配置prometheus
$ vim /usr/local/prometheus/prometheus.yml
...
- job_name: 'nginx-vts'
static_configs:
- targets:
- 172.31.217.169:9913
$ kill -hup `ps -ef |grep prometheus|grep -v grep|awk '{print $2}'`
查看Prometheus是否采集到的数据
image
配置grafana
- 导入模板: 2949
image
安装GeoIP模块
$ yum -y install epel-release geoip-devel
$ cd /usr/local/src/nginx-1.10.0
$ ./configure --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_geoip_module
$ make
...平滑升级参考上面安装nginx-vts模块
# 查看最新的编译参数
$ /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.10.0
...
configure arguments: --add-module=/root/nginx-module-vts --prefix=/usr/local/nginx --user=nginx --group=nginx --with-stream --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_geoip_module
$ ldd /usr/local/nginx/sbin/nginx |grep libGeoIP
libGeoIP.so.1 => /lib64/libGeoIP.so.1 (0x00007f49d503d000)
# 引用地址库
$ vim /usr/local/nginx/conf/nginx.conf
http {
...
vhost_traffic_status_zone;
geoip_country /usr/share/GeoIP/GeoIP.dat;
...
}
# 重启nginx
$ /usr/local/nginx/sbin/nginx -s stop
$ /usr/local/nginx/sbin/nginx
通过VTS自定义key
通过以上部署只能拿到默认的指标，生产中可能还会需要监控uri的请求量，监控IP访问情况（同一个IP出现大量访问时可能被攻击），获取不同agent请求量用于分析等，通过vts模块的vhost_traffic_status_filter_by_set_key功能可以自定义需要获取的指标。此处的指标需要加到对应的server配置中
# 添加自定义配置
$ vim /usr/local/nginx/conf/vhost/baiyongjie.com.conf
server {
listen 443 ssl;
server_name baiyongjie.com blog.baiyongjie.com;
...
vhost_traffic_status_filter_by_set_key $uri uri::$server_name; #每个uri访问量
vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name; #不同国家/区域请求量
vhost_traffic_status_filter_by_set_key $status $server_name; #http code统计
vhost_traffic_status_filter_by_set_key $upstream_addr upstream::backend; #后端转发统计
vhost_traffic_status_filter_by_set_key $remote_port client::ports::$server_name; #请求端口统计
vhost_traffic_status_filter_by_set_key $remote_addr client::addr::$server_name; #请求IP统计
location ~ ^/storage/(.+)/.*$ {
set $volume $1;
vhost_traffic_status_filter_by_set_key $volume storage::$server_name; #请求路径统计
}
...
}
再次访问status接口, 可以看到多了很多自定义的监控项
image
到Prometheus查询,可以看到新的字段已经可以查询到了
image
参考:
