以C-S模式使用fluentd来收集docker容器日志
2018-09-14 本文已影响53人
peerless_1024
-
1.安装服务器端
环境：CentOS Linux release 7.4.1708 (Core)
安装td-agent:
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
下载服务器端所需插件：
td-agent-gem install fluent-plugin-file-alternative
td-agent-gem install fluent-plugin-forest
-
2.安装客户端及所需插件
环境：Ubuntu 16.04.3 LTS
安装td-agent:
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
# 客户端需要安装的插件
td-agent-gem install fluent-plugin-rewrite-tag-filter
td-agent-gem install fluent-plugin-grep
td-agent-gem install fluent-plugin-record-reformer
- 3.赋予td-agent用户root权限：
修改 /etc/passwd 文件
vim /etc/passwd
找到如下行，把td-agent用户ID修改为 0 ，如下所示：
td-agent:x:0:133::/var/lib/td-agent:/bin/false
-
4.修改客户端和服务器端的配置文件：
客户端/etc/td-agent/td-agent.conf：
root@ubuntu:/etc/td-agent# vim td-agent.conf
# 客户端配置文件
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
# To get the informations of plugins http://localhost:24220/api/plugins.json
<source>
@type monitor_agent
bind 0.0.0.0
port 24220
</source>
<match docker.*>
@type forward
<server>
host 192.16.1.148
port 24224
</server>
<buffer>
flush_interval 10s
</buffer>
</match>
服务器端/etc/td-agent/td-agent.conf：
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match docker.*>
@type stdout
</match>
- 5.关闭客户端和服务器端的防火墙
- 先测试能否通信：（ctrl+] 然后在telnet 命令行输入quit 就可以退出telnet）
#客户机端
root@ubuntu:~# telnet 192.16.1.148 24224
Trying 192.16.1.148...
Connected to 192.16.1.148.
#服务器端
[root@worker1 td-agent]# telnet 192.16.1.157 24224
Trying 192.16.1.157...
Connected to 192.16.1.157.
- 若不能通信，可能是打开了防火墙所致，粗暴的解决方式：关闭防火墙
service firewalld status
service firewalld stop
- 更精准的解决方式是添加防火墙规则,例如：
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.16.1.157" port protocol="tcp" port="24224" accept"
//在server端也就是centos（ip=192.16.1.148）上
[root@worker1 td-agent]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.16.1.157" port protocol="tcp" port="24224" accept"
success
[root@worker1 td-agent]# systemctl restart firewalld.service
[root@worker1 td-agent]# firewall-cmd --list-all
public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: dhcpv6-client ssh
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule family="ipv4" source address="192.16.1.157" port port="24224" protocol="tcp" accept
[root@worker1 td-agent]#
- 6.查看服务器端收集到的来着客户端的日志
[root@worker1 td-agent]# tail -f td-agent.log
2018-09-14 19:57:46 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=10
2018-09-14 19:57:46 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=13
2018-09-14 19:57:46 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=10
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=17
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=8
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=93
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=13
2018-09-14 19:58:17 +0800 [warn]: #0 incoming chunk is broken: host="192.16.1.157" msg=10
2018-09-14 20:01:50.000000000 +0800 docker.nginx_test: {"source":"stdout","log":"172.17.0.1 - - [14/Sep/2018:12:01:50 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\"","container_id":"a3ea9a89fbdd94d6650e6e11e8bc7ce2a87e3e3b38d349720ad470222c65c0f6","container_name":"/nginx_bashlog"}
2018-09-14 20:02:03.000000000 +0800 docker.test_of_nginx01: {"container_id":"2c27be33f81ba79f10c56183b0f7f71c833b021d59333d9876b80e66fda756c9","container_name":"/test_of_nginx01","source":"stdout","log":"172.17.0.1 - - [14/Sep/2018:12:02:03 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0\" \"-\""}
成功效果图
