kubernets学习kuberne...

kube-apiserver的listwatche机制

2022-08-17  本文已影响0人  zoux

Table of Contents

1. 背景

client-go实际只是一个客户端，list-watch我们经常听到。但实际上是apisever的实现。在apisever注册资源对象的create, update, delete等等hanlder时，也注册了List-watch的实现。

所以在研究client-go是如果处理list watch之前，先了解一个apiserver的list watch机制

2. list watch机制

List-watch是K8S统一的异步消息处理机制，保证了消息的实时性，可靠性，顺序性，性能等等，为声明式风格的API奠定了良好的基础，它是优雅的通信方式，是K8S 架构的精髓。

2.1 如何实现实时性

一般客户端和服务器端的同步，无非就是两种大类：一种是客户端轮训服务器端。第二种就是服务器端主动发起通知。

k8s采用的是第二种，主动发起通知。这里具体就是使用了watch机制。

list, watch其实都是特殊的get接口。

get default命名空间所有的pod url如下： curl http://xxx:port/api/v1/namespaces/default/pods

watch default命名空间所有的pod url如下: curl http://XXX/api/v1/namespaces/default/pods?watch=true 就是多了一个watch=true的参数

root:/# curl http://XXX/api/v1/namespaces/default/pods?watch=true
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-xv5zw","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-xv5zw","uid":"639944b7-3495-4fbb-a21d-cbc7f4d6f7a5","resourceVersion":"157197390","creationTimestamp":"2021-11-12T10:59:39Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"Pod resources updated by hamster-vpa: container 0: memory request, cpu request, memory limit, cpu limit; container 1: cpu request, memory request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"7.34.19.14","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"}],"hostIP":"7.34.19.14","podIP":"7.34.19.14","podIPs":[{"ip":"7.34.19.14"}],"startTime":"2021-11-12T10:59:39Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:59:46Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:45Z","finishedAt":"2021-11-14T02:59:45Z","containerID":"docker://87a70d2061b7fb37b0f97be3a4f9d44b345fbd54be3dcc4d8a61879dd5c6a127"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://bc586f53f363e9afb08c7a214eef06c8c1202f72439fc972d4c7d6177cfb8e63","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:59:47Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:46Z","finishedAt":"2021-11-14T02:59:46Z","containerID":"docker://37d8dd54be6d27ed9f055049e700f12fa4aa30ec29f2fd16fd5176218b2acce9"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://9ccb7968bee2c155c472e03d56a5987c9cf7e6833a4cb125084ceb19158474ed","started":true}],"qosClass":"Guaranteed"}}}
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-mw6mr","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-mw6mr","uid":"4e7f3a44-7483-434d-a917-52b37c0eae33","resourceVersion":"157192079","creationTimestamp":"2021-11-12T10:52:37Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"Pod resources updated by hamster-vpa: container 0: cpu request, memory request, cpu limit, memory limit; container 1: memory request, cpu request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"10.90.67.175","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:52:37Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:52:54Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:52:54Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:52:37Z"}],"hostIP":"10.90.67.175","podIP":"10.90.67.175","podIPs":[{"ip":"10.90.67.175"}],"startTime":"2021-11-12T10:52:37Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:52:50Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:52:47Z","finishedAt":"2021-11-14T02:52:47Z","containerID":"docker://ddf625ee9c90ba70ba5f1d27caa4d61ded938143a724dccbfada898271ac7fd0"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://854091543fc0ba88d3dc4a839f8014d21ecbaecf4e40221d2ce9d6a343ddbe29","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:52:53Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:52:50Z","finishedAt":"2021-11-14T02:52:50Z","containerID":"docker://0241d05357e1f6b8eec73810341bddf14479a168aaa7958ee580855eb2f4300f"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://ad0d09158c0db8b10d2c282f2749c1c1e97fdb707e10392b9033aa619b162450","started":true}],"qosClass":"Guaranteed"}}}





// 一旦有对象改变就会收到事件。type有三种，modifyed, added ,deleted
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-xv5zw","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-xv5zw","uid":"639944b7-3495-4fbb-a21d-cbc7f4d6f7a5","resourceVersion":"157401529","creationTimestamp":"2021-11-12T10:59:39Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"11111111111Pod resources updated by hamster-vpa: container 0: memory request, cpu request, memory limit, cpu limit; container 1: cpu request, memory request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"7.34.19.14","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"}],"hostIP":"7.34.19.14","podIP":"7.34.19.14","podIPs":[{"ip":"7.34.19.14"}],"startTime":"2021-11-12T10:59:39Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:59:46Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:45Z","finishedAt":"2021-11-14T02:59:45Z","containerID":"docker://87a70d2061b7fb37b0f97be3a4f9d44b345fbd54be3dcc4d8a61879dd5c6a127"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://bc586f53f363e9afb08c7a214eef06c8c1202f72439fc972d4c7d6177cfb8e63","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:59:47Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:46Z","finishedAt":"2021-11-14T02:59:46Z","containerID":"docker://37d8dd54be6d27ed9f055049e700f12fa4aa30ec29f2fd16fd5176218b2acce9"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://9ccb7968bee2c155c472e03d56a5987c9cf7e6833a4cb125084ceb19158474ed","started":true}],"qosClass":"Guaranteed"}}}



//删除一个pod，发现会进入
MODIFIED （设置deletiontimestamp）-> 
ADDED  "status":{"phase":"Pending","qosClass":"Guaranteed"}}} 新pod pending
MODIFIED podScheduled
MODIFIED ContainerCreating
MODIFIED.. 到pod running
DELETED  删除旧Pod


 curl http://7.34.19.44:58201/api/v1/watch/namespaces/default/pods
 看起来也是一样的效果

通过上面的实践可以发现：

（1）watch其实就是一种特殊的get

（2）可以看到删除操作后，对象的整个变化过程

（3）watch每次都会返回type，和完整的对象信息

2.2 如何实现顺序性

K8S在每个资源的事件中都带一个resourceVersion的标签，这个标签是递增的数字，所以当客户端并发处理同一个资源的事件时，它就可以对比resourceVersion来保证最终的状态和最新的事件所期望的状态保持一致。

2.3 如何实现消息可靠性

list和watch一起保证了消息的可靠性，避免因消息丢失而造成状态不一致场景。具体而言，list API可以查询当前的资源及其对应的状态(即期望的状态)，客户端通过拿期望的状态和实际的状态进行对比，纠正状态不一致的资源。Watch API和apiserver保持一个长链接，接收资源的状态变更事件并做相应处理。如果仅调用watch API，若某个时间点连接中断，就有可能导致消息丢失，所以需要通过list API解决消息丢失的问题。从另一个角度出发，我们可以认为list API获取全量数据，watch API获取增量数据。虽然仅仅通过轮询list API，也能达到同步资源状态的效果，但是存在开销大，实时性不足的问题。

2.4 如何解决性能问题

（1）list-watch机制的结合就已经在apiserver做了性能优化。（是不是可以watch的时候，只传递更新了的字段，而不是全量数据）

（2）client-go的 tool.cache做了客户端的性能优化问题

3.总结

本节主要从apiserver端探究了以下list-watch。接下来从client-go端源码看看具体是如何实现的

上一篇下一篇

猜你喜欢

热点阅读