在containerd中使用fluentbit收集日志
2021-07-16 本文已影响0人
文茶君
https://github.com/fluent/fluent-bit-kubernetes-logging
$ kubectl create namespace logging
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml
If the cluster uses a CRI runtime, like containerd or CRI-O, change the Parser described in input-kubernetes.conf from docker to cri.
配置如下
[Service]
Parsers_File parsers.conf
[Input]
Name tail
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_kube-system_*.log , /var/log/containers/_fluent-bit-*.log
Refresh_Interval 10
Skip_Long_Lines true
Mem_Buf_Limit 15MB
Parser cri
Tag kube.*
[Output]
Name stdout
Match *
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
至于加了一个logtag F
它的issue在这https://github.com/cri-o/cri-o/issues/3181
这里这个issue主要说了他发现CRI-O的输出格式与docker的json输出格式不兼容，导致fluentbit无法采集这种非结构体的数据。希望维护者能修改，维护者说如果改的话会破坏一致性，欢迎提PR，但不修改，而且docker已经要被启用，全转成crio，建议其他监控软件适配crio格式。然后issue的提出者就说他找到办法了，就是把Parse docker换成parser cri。
输出如下：
[2021/07/15 09:37:33] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1816045 watch_fd=1 name=/var/log/containers/fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log
[0] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341845.370295867, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[1] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341846.370299957, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[2] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341847.370446809, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[3] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341848.370738978, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[4] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341849.370786338, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[5] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341850.370946823, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[6] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341851.371154644, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[7] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341852.371360475, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[8] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341853.371534931, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
[9] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-fab8bdb7cee68f6d72f424cc3ae3dcda0906f83d040b59e84281970c6f7a6542.log: [1626341854.371558540, {"stream"=>"stdout", "logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100"}]
此时我删除kubesphere中所有过滤配置，在docker下是这样的，收集到的数据是这样的
[0] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404307.787585128, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:27.787585128Z"}]
[1] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404308.787787980, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:28.78778798Z"}]
[2] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404309.788042268, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:29.788042268Z"}]
[3] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404310.787986287, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:30.787986287Z"}]
[4] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404311.788826620, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:31.78882662Z"}]
[5] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404312.789448843, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:32.789448843Z"}]
[6] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404313.789184546, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:33.789184546Z"}]
[7] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404314.789205103, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:34.789205103Z"}]
[8] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404315.789437595, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:35.789437595Z"}]
[9] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-8f418b692f09a77635c729529dfa3690e035fa20c90b8c157409dd3488111a27.log: [1626404316.789575205, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "stream"=>"stdout", "time"=>"2021-07-16T02:58:36.789575205Z"}]
而kubesphere中加了配置时数据是这样的
[1] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-db3293f5251348024d939c86f1243f395b0fabb1beb42ff6c92a08e7c2c7de0d.log: [1626343069.999612573, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "time"=>"2021-07-15T09:57:49.999612573Z", "kubernetes"=>{"pod_name"=>"fb-test-pod", "namespace_name"=>"fb-test", "container_name"=>"fb-test01", "docker_id"=>"db3293f5251348024d939c86f1243f395b0fabb1beb42ff6c92a08e7c2c7de0d", "container_image"=>"wenchajun/testnoserial:v0.5"}}]
[2] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-db3293f5251348024d939c86f1243f395b0fabb1beb42ff6c92a08e7c2c7de0d.log: [1626343070.999768607, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "time"=>"2021-07-15T09:57:50.999768607Z", "kubernetes"=>{"pod_name"=>"fb-test-pod", "namespace_name"=>"fb-test", "container_name"=>"fb-test01", "docker_id"=>"db3293f5251348024d939c86f1243f395b0fabb1beb42ff6c92a08e7c2c7de0d", "container_image"=>"wenchajun/testnoserial:v0.5"}}]
[3] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-db3293f5251348024d939c86f1243f395b0fabb1beb42ff6c92a08e7c2c7de0d.log: [1626343071.999982164, {"log"=>"name:eloncheng||fluent-bit-test|The current number is100
", "time"=>"2021-07-15T09:57:51.999982164Z", "kubernetes"=>{"pod_name"=>"fb-test-pod", "namespace_name"=>"fb-test", "container_name"=>"fb-test01", "docker_id"=>"db3293f5251348024d939c86f1243f395b0fabb1beb42ff6c92a08e7c2c7de0d", "container_image"=>"wenchajun/testnoserial:v0.5"}}]
在这里我觉得都转换为json格式了，就安装kubesphere-logging-system来检测是否成功
echo DOCKER_ROOT_DIR=$(docker info -f '{{.DockerRootDir}}')
Command 'docker' not found, but can be installed with:
sudo snap install docker # version 19.03.13, or
sudo apt install docker.io # version 19.03.8-0ubuntu1.20.04.1
See 'snap info docker' for additional versions.
DOCKER_ROOT_DIR=
这里居然安装成功了，我也有点惊奇，于是观察一下配置。在这里因为这个命令导致在containerd下的fluentbit下的fluentbit不一致
因为
echo DOCKER_ROOT_DIR=$(docker info -f '{{.DockerRootDir}}')
DOCKER_ROOT_DIR=/var/lib/docker
所以：
| docker | containerd |
|---|---|
| 类型: HostPath 主机路径：/containers | 类型: HostPath 主机路径：/var/lib/docker/containers |
| 主机路径：/var/log/journal | 无 |
其他并无差别。fluentbit-operator的版本为0.2.0 高版本有一些错误（parser crd不识别，导致整体安装不了）后面经过检查，发现是版本过低
fluentbit的版本升级至1.7.3
[8] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-7585d7e07d31978d71309fc7d07a48294b2d516461c80bc85b03f2a853041b57.log: [1626605537.429795357, {"logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100", "kubernetes"=>{"pod_name"=>"fb-test-pod", "namespace_name"=>"fb-test", "container_name"=>"fb-test01", "docker_id"=>"7585d7e07d31978d71309fc7d07a48294b2d516461c80bc85b03f2a853041b57", "container_image"=>"docker.io/wenchajun/testnoserial:v0.5"}}]
[9] kube.var.log.containers.fb-test-pod_fb-test_fb-test01-7585d7e07d31978d71309fc7d07a48294b2d516461c80bc85b03f2a853041b57.log: [1626605538.430361348, {"logtag"=>"F", "message"=>"name:eloncheng||fluent-bit-test|The current number is100", "kubernetes"=>{"pod_name"=>"fb-test-pod", "namespace_name"=>"fb-test", "container_name"=>"fb-test01", "docker_id"=>"7585d7e07d31978d71309fc7d07a48294b2d516461c80bc85b03f2a853041b57", "container_image"=>"docker.io/wenchajun/testnoserial:v0.5"}}]
https://github.com/microsoft/fluentbit-containerd-cri-o-json-log/blob/main/config.yaml
service.kubelet log信息
service.kubelet: [1626714010.001564000, {"_TRANSPORT"=>"stdout", "PRIORITY"=>"6", "SYSLOG_FACILITY"=>"3", "_UID"=>"0", "_GID"=>"0", "_CAP_EFFECTIVE"=>"3fffffffff", "_SELINUX_CONTEXT"=>"unconfined
", "_SYSTEMD_SLICE"=>"system.slice", "_BOOT_ID"=>"5e5cd8e18abe4d119ed680d5a0cf02b8", "_MACHINE_ID"=>"5bca3f64c15b39e5ba4ca3420f9cea70", "_HOSTNAME"=>"node1", "_STREAM_ID"=>"75d1ff21b2184153a50d60ad10f47c08", "SYSLOG_IDENTIFIER"=>"kubelet", "_PID"=>"6098", "_COMM"=>"kubelet", "_EXE"=>"/usr/local/bin/kubelet", "_CMDLINE"=>"/usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=192.168.1.3 --hostname-override=node1", "_SYSTEMD_CGROUP"=>"/system.slice/kubelet.service", "_SYSTEMD_UNIT"=>"kubelet.service", "_SYSTEMD_INVOCATION_ID"=>"ef73d93aca054534b9e928f10b1be717", "MESSAGE"=>"W0720 01:00:10.001517 6098 pod_container_deletor.go:79] Container "cfd2fe39b590a873bd78de78bf01dfc8b2ff92b0d8d5ee28890925ea6d5e7346" not found in pod's containers"}]
