收集 Kubernetes event事件
2019-10-09 本文已影响0人
baiyongjie
简介
Event是什么?
Event作为kubernetes的一个对象资源，记录了集群运行所遇到的各种大事件,有助于排错，但大量的事件如果都存储在etcd中，会带来较大的性能与容量压力，所以etcd中默认只保存最近1小时的。
查看Event
[root@T01 elasticsearch]# kubectl get event
LAST SEEN TYPE REASON OBJECT MESSAGE
5m16s Normal Pulled pod/nginxtest-bbccd685f-gtf9x Container image "nginx:1.10" already present on machine
5m15s Normal Created pod/nginxtest-bbccd685f-gtf9x Created container nginxtest
5m15s Normal Started pod/nginxtest-bbccd685f-gtf9x Started container nginxtest
[root@T01 elasticsearch]# kubectl get event -o wide
LAST SEEN TYPE REASON OBJECT SUBOBJECT SOURCE MESSAGE FIRST SEEN COUNT NAME
5m22s Normal Pulled pod/nginxtest-bbccd685f-gtf9x spec.containers{nginxtest} kubelet, t01 Container image "nginx:1.10" already present on machine 5h40m 5 nginxtest-bbccd685f-gtf9x.15c919914460c103
5m21s Normal Created pod/nginxtest-bbccd685f-gtf9x spec.containers{nginxtest} kubelet, t01 Created container nginxtest 5h40m 5 nginxtest-bbccd685f-gtf9x.15c9199145e21995
5m21s Normal Started pod/nginxtest-bbccd685f-gtf9x spec.containers{nginxtest} kubelet, t01 Started container nginxtest 5h40m 5 nginxtest-bbccd685f-gtf9x.15c919914bd75bfe
收集event的方案
- 使用开源项目eventrouter进行收集
- 项目地址: https://github.com/heptiolabs/eventrouter
再容器内部收集,直接然后发送到es
大概流程
- 启动eventrouter容器,挂载/data/log/eventrouter目录
- 启动filebeat容器,挂载/data/log/eventrouter目录
- filebeat收集/data/log/eventrouter目录下的日志
- filebeat数据发送到elasticsearch
- kibana添加索引,并展示数据
| IP | 角色 |
|---|---|
| 192.168.109.128 | Kubernetes |
| 192.168.109.128 | kibana |
| 192.168.109.128 | elasticsearch |
es,kibana准备
$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
## elasticsearch
$ yum -y install java
$ yum -y install elasticsearch-6.3.2
$ systemctl start elasticsearch
$ systemctl enable elasticsearch
## kibana
$ yum -y install kibana-6.3.2
$ chown kibana. /var/log/kibana/
$ vim /etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.109.128"
elasticsearch.url: "http://192.168.109.128:9200"
kibana.defaultAppId: "discover"
elasticsearch.pingTimeout: 3000
elasticsearch.shardTimeout: 0
elasticsearch.startupTimeout: 9000
pid.file: /tmp/kibana.pid
logging.dest: /var/log/kibana/kibana.log
logging.verbose: false
ops.interval: 5000
$ systemctl start kibana
$ systemctl enable kibana
$ systemctl status kibana
yaml文件
$ cat eventrouter-infilebeat.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: eventrouter
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: eventrouter
rules:
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: eventrouter
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: eventrouter
subjects:
- kind: ServiceAccount
name: eventrouter
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: eventrouter-cm
namespace: kube-system
data:
config.json: |-
{
"sink": "glog"
}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
data:
filebeat.yml: |-
filebeat.prospectors:
- input_type: log
paths:
- "/data/log/eventrouter/*"
output.elasticsearch:
hosts: ["192.168.109.128:9200"]
index: "filebeat-k8s-pre-event-%{+yyyy.MM.dd}"
setup.template.name: "filebeat-k8s-pre-event"
setup.template.pattern: "filebeat-k8s-pre-event-"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: eventrouter
namespace: kube-system
labels:
app: eventrouter
spec:
replicas: 1
selector:
matchLabels:
app: eventrouter
template:
metadata:
labels:
app: eventrouter
tier: control-plane-addons
spec:
containers:
- name: kube-eventrouter
image: baiyongjie/eventrouter:v0.2
command:
- "/bin/sh"
args:
- "-c"
- "/eventrouter -v 3 -log_dir /data/log/eventrouter"
volumeMounts:
- name: eventrouter-cm
mountPath: /etc/eventrouter
- name: log-path
mountPath: /data/log/eventrouter
- name: filebeat
image: docker.elastic.co/beats/filebeat:6.3.2
command:
- "/bin/sh"
args:
- "-c"
- "filebeat -c /etc/filebeat/filebeat.yml"
volumeMounts:
- name: filebeat-config
mountPath: /etc/filebeat/
- name: log-path
mountPath: /data/log/eventrouter
serviceAccount: eventrouter
volumes:
- name: eventrouter-cm
configMap:
name: eventrouter-cm
- name: filebeat-config
configMap:
name: filebeat-config
- name: log-path
emptyDir: {}
$ kubectl apply -f eventrouter-infilebeat.yaml
serviceaccount/eventrouter created
clusterrole.rbac.authorization.k8s.io/eventrouter created
clusterrolebinding.rbac.authorization.k8s.io/eventrouter created
configmap/eventrouter-cm created
configmap/filebeat-config created
deployment.apps/eventrouter created
$ kubectl get pods -n kube-system |grep event
eventrouter-7bb898ff4b-2jp4r 2/2 Running 0 29s
查看es索引
$ curl http://192.168.109.128:9200/_cat/indices
yellow open filebeat-k8s-pre-event-2019.09.30 GL1lIT6VRp-qvI-reyjiNA 5 1 134 0 32kb 32kb
在kibana添加索引并查看
添加索引.png
kibana展示.png
模拟nginx pod重启
$ kubectl exec -it nginxtest-bbccd685f-gtf9x -- /bin/bash
root@nginxtest-bbccd685f-gtf9x:/# nginx -s stop
2019/09/30 09:02:46 [notice] 18#18: signal process started
root@nginxtest-bbccd685f-gtf9x:/# command terminated with exit code 137
$ kubectl describe pods nginxtest-bbccd685f-gtf9x | grep -A 20 Events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 83s (x5 over 5h36m) kubelet, t01 Container image "nginx:1.10" already present on machine
Normal Created 82s (x5 over 5h36m) kubelet, t01 Created container nginxtest
Normal Started 82s (x5 over 5h36m) kubelet, t01 Started container nginxtest
kibana查看.png
(轻易科技ops部)
