nginx+logstash+mysql 实时存储数据

实现思路:
1.硬件端日志访问nginx附带json格式日志
2.后通过logstash实时读取nginx日志存储于mysql中

1.nginx安装和配置

1.1 point-record为自定义nginx服务名

docker run  -p 80:80 --name=point-record -v /opt/point-record/conf.d:/etc/nginx/conf.d -v /opt/point-record/web:/etc/nginx/web -v  /opt/nginx-log/point-record/:/var/log/nginx/ -v /etc/timezone:/etc/timezone -v /etc/localtime:/etc/localtime -d nginx

1.2 在/opt/point-record/conf.d创建配置文件point.conf

cd /opt/point-record/conf.d

nano point.conf

point.conf:

log_format json escape=json '{"timestamp":"$time_iso8601",'
                           '"version":"1",'
                           '"client":"$remote_addr",'
                           '"url":"$uri",'
                           '"status":"$status",'
                           '"domain":"$host",'
                           '"host":"$server_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"referer":"$http_referer",'
                            #消息体打印
                           '"body":$request_body,'
                           '"ua": "$http_user_agent"'
               '}';
               
               
server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /etc/nginx/web;
        location / {
            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_redirect off;
            #必须经过一次转发,否则nginx不会读取消息体,打印的request_body为空
            proxy_pass $scheme://127.0.0.1:$server_port/success;
            access_log  /var/log/nginx/access_json.log  json;
           
        }
        location /success {
            return 200;
        }
        error_page 404 /404.html;
            location = /40x.html {
        }
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

2.logstash安装与配置

2.1 官网下载 logstash-7.5.1.tar.gz 上传服务器并解压

tar -xzvf logstash-7.5.1.tar.gz -C /usr/local/
                                                                                               
cd /usr/local/

#修改文件夹名称 ("logstash7" 是为了区分版本,可以同时安装多个版本)
mv logstash-7.5.1 logstash7

cd  logstash7

目录下文件:

image.png

每次send请求模拟一次硬件端的日志请求