Linux下td-agent（fluentd）的安装和配置

2018-08-09 本文已影响189人 peerless_1024

1.Fluentd安装之前的准备工作

1.在节点上设置NTP守护程序,以获得准确的当前时间戳。

安装ntp服务

//ubuntu
sudo apt install ntp
//centos
yum install ntp -y

编辑 /etc/ntp.conf

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

#新增:日志目录.
logfile /var/log/ntpd.log

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#新增:时间服务器列表.
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst

注意:ubuntu下服务叫做ntp,centos下叫做ntpd
修改配置文件之后，你需要重新加载 ntpd：

systemctl enable ntp
systemctl start ntp

或者

/etc/init.d/ntp stop
/etc/init.d/ntp start

ubuntu主机下例子:

root@ubuntu:~# systemctl status ntp
● ntp.service - LSB: Start NTP daemon
   Loaded: loaded (/etc/init.d/ntp; bad; vendor preset: enabled)
   Active: active (running) since 四 2018-10-11 14:15:51 CST; 1 months 8 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 2
   Memory: 2.2M
      CPU: 3min 42.282s
   CGroup: /system.slice/ntp.service
           └─1506 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 123:132

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@ubuntu:~# systemctl is-enabled ntp
ntp.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install is-enabled ntp
enabled

centos主机下例子:

[root@master ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
[root@master ~]# systemctl enable ntpd
[root@master ~]# systemctl is-enabled ntpd
enabled
[root@master ~]# systemctl start ntpd     
[root@master ~]# systemctl status ntpd    
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-11-19 16:55:01 CST; 5s ago
  Process: 7170 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 7172 (ntpd)
    Tasks: 1
   Memory: 1.4M
   CGroup: /system.slice/ntpd.service
           └─7172 /usr/sbin/ntpd -u ntp:ntp -g

Nov 19 16:55:01 master.novalocal systemd[1]: Starting Network Time Service...
Nov 19 16:55:01 master.novalocal ntpd[7172]: proto: precision = 0.089 usec
Nov 19 16:55:01 master.novalocal ntpd[7172]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Nov 19 16:55:01 master.novalocal systemd[1]: Started Network Time Service.

查看节点同步状态

root@ubuntu:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
+ntp.hkg10.hk.le 130.133.1.10     2 u   38   64  377  121.432  -10.265   0.534
*45.125.1.20 (45 101.231.167.217  2 u   30   64  377  353.921   12.721   0.429
+118.140.184.98  223.255.185.2    2 u   34   64  157    9.668   13.966  40.916
-alphyn.canonica 192.53.103.108   2 u   37   64   73  341.456   41.184  37.807
-chilipepper.can 17.253.34.253    2 u   96   64   76  299.193  -17.247  29.960
-golem.canonical 145.238.203.14   2 u   36   64   37  274.246  -28.804  28.527
root@ubuntu:~# ntpstat
synchronised to NTP server (45.125.1.20) at stratum 3 
   time correct to within 236 ms
   polling server every 64 s

2.增加最大文件描述符数

使用ulimit -n命令检查当前号码。

$ ulimit -n
1024

如果控制台显示1024，是不够的。在/etc/security/limits.conf文件中添加以下行并重新启动计算机。

root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536

3.优化网络内核参数。

对于由许多Fluentd实例组成的高负载环境，请将这些参数添加到您的/etc/sysctl.conf文件中。

net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535

并输入sysctl -p或重新启动节点以使更改生效。

sysctl -p

2. 安装Fluentd的客户端td-agent

参考官网安装教程
对配置文件进行配置

//根据版本下载相应客户端，我的是16
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
//CentOS Linux release 7.4.1708 (Core)
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
//开始
systemctl start td-agent.service
//查看状态
systemctl status td-agent.service
//停止
systemctl stop td-agent.service
//打开配置文件
vim /etc/td-agent/td-agent.conf

配置文件的语法可以参考官网给出的，或者参考博客1和博客2

3.简明使用

Fluentd 安装使用可以参考该文章。
参考该博客，有比较详尽的fluentd工具使用经验，日志收集工具Fluentd使用总结

fluentd配置文件所在目录：/etc/td-agent/td-agent.conf

fluentd日志文件所在目录：/var/log/td-agent/td-agent.log

fluentd二进制文件所在目录：/opt/td-agent/embedded/bin/

安装插件的方法：td-agent-gem install [插件名称]

4.监听python和HTTP请求的配置实例

配置文件的配置如下：

<source>
  @type forward
  port 24224
</source>

#匹配项：监听python的配置
<match fluentd.test.**>
  @type stdout
</match>

#匹配项：监听curl命令产生的HTTP的get和post请求
# listening for HTTP Requests
<source>
  @type http
  port 8888
  bind 0.0.0.0
</source>

# print the data arrived on each incoming request to standard output
<match test.cycle>
  @type stdout
</match>

<match debug.test>
  @type stdout
</match>

开一个终端用于跟踪Fluentd收集到的应用日志使用以下命令：

tail -f /var/log/td-agent/td-agent.log

打开另外一个终端用于测试和监听

curl命令操作

curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

curl -i -X POST -d 'json={"action":"login","user":2}' http://localhost:8888/test.cycle

python文件执行操作
创建两个文件如下图：

simple.py和test.py

# test.py
from fluent import sender
from fluent import event
sender.setup('fluentd.test', host='localhost', port=24224)
event.Event('follow', {
  'from': 'userA',
  'to':   'userB'
})

# simple.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import itertools
import json
from fluent import sender
from fluent import event

sender.setup('fluentd.test', host='localhost', port=24224)

class cartesian(object):
    def __init__(self):
        self._data_list=[]

    def add_data(self,data=[]): #添加生成笛卡尔积的数据列表
        self._data_list.append(data)

    def build(self): #计算笛卡尔积
        for item in itertools.product(*self._data_list):
            print(item)

if __name__=="__main__":
    car=cartesian()
    car.add_data([1,2,3,4])
    car.add_data([5,6,7,8])
    car.add_data([9,10,11,12])
    car.build()
    event.Event('Cartesian', {
       'Set': '[1,2,3,4],[5,6,7,8],[9,10,11,12]',
       'Cartesian product': 'results are stdout,not here'
    })

分别执行两个文件

python test.py
python simple.py

执行终端显示结果如下

执行终端显示

日志跟踪终端显示如下：

root@ubuntu:/var/log/td-agent# tail -f td-agent.log
2018-09-06 10:35:15 +0800 [info]: #0 starting fluentd worker pid=8670 ppid=8665 worker=0
2018-09-06 10:35:15 +0800 [info]: #0 listening port port=24224 bind="0.0.0.0"
2018-09-06 10:35:15 +0800 [info]: #0 fluentd worker is now running worker=0
……
2018-09-06 15:34:56.333452308 +0800 debug.test: {"json":"message"}
2018-09-06 15:35:08.899919433 +0800 test.cycle: {"action":"login","user":2}
2018-09-06 15:35:23.000000000 +0800 fluentd.test.follow: {"to":"userB","from":"userA"}
2018-09-06 15:35:33.000000000 +0800 fluentd.test.Cartesian: {"Cartesian product":"results are stdout,not here","Set":"[1,2,3,4],[5,6,7,8],[9,10,11,12]"}

5.监听docker容器的配置实例

Docker版本需要在17.05以上才可以使用日志驱动插件(见官网)。
使用参见官网说明：https://docs.docker.com/config/containers/logging/fluentd/

修改 /etc/passwd 文件

vim /etc/passwd

找到如下行，把td-agent用户ID修改为 0 ，如下所示：

td-agent:x:0:133::/var/lib/td-agent:/bin/false

修改配置文件（/etc/td-agent/td-agent.conf ）如下：

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match docker.*>
  @type stdout
</match>

td-agent.conf

重启fluentd

systemctl restart td-agent.service

运行容器

docker run  -d -p 8081:80 --name nginx_bashlog -v /work/DOCKER/ubuntu-fluentd-test01:/root --log-driver=fluentd --log-opt tag=docker.nginx_test --log-opt fluentd-async-connect nginx

查看fluentd运行状态

查看收集到的日志

cd /var/log/td-agent
tail -f td-agent.log

收集到的日志如下：

2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"source":"stdout","log":"172.17.0.1 - - [10/Sep/2018:08:01:41 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog"}
2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"log":"2018/09/10 08:01:41 [error] 10#10: *1 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"localhost:8081\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog","source":"stderr"}
2018-09-10 16:01:41.000000000 +0800 docker.nginx_test: {"container_name":"/nginx_bashlog","source":"stdout","log":"172.17.0.1 - - [10/Sep/2018:08:01:41 +0000] \"GET /favicon.ico HTTP/1.1\" 404 169 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6"}

td-agent.log