kubebuilder实战之六:构建部署运行

2021-06-30  本文已影响0人  程序员欣宸

欢迎访问我的GitHub

https://github.com/zq2599/blog_demos

内容:所有原创文章分类汇总及配套源码,涉及Java、Docker、Kubernetes、DevOPS等;

系列文章链接

  1. kubebuilder实战之一:准备工作
  2. kubebuilder实战之二:初次体验kubebuilder
  3. kubebuilder实战之三:基础知识速览
  4. kubebuilder实战之四:operator需求说明和设计
  5. kubebuilder实战之五:operator编码
  6. kubebuilder实战之六:构建部署运行
  7. kubebuilder实战之七:webhook
  8. kubebuilder实战之八:知识点小记

本篇概览

  1. 部署CRD
  2. 本地运行Controller
  3. 通过yaml文件新建elasticweb资源对象
  4. 通过日志和kubectl命令验证elasticweb功能是否正常
  5. 浏览器访问web,验证业务服务是否正常
  6. 修改singlePodQPS,看elasticweb是否自动调整pod数量
  7. 修改totalQPS,看elasticweb是否自动调整pod数
  8. 删除elasticweb,看相关的service和deployment被自动删除
  9. 构建Controller镜像,在kubernetes运行此Controller,验证上述功能是否正常

部署CRD

zhaoqin@zhaoqindeMBP-2 elasticweb % make install
/Users/zhaoqin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/elasticwebs.elasticweb.com.bolingcavalry configured
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl api-versions|grep elasticweb
elasticweb.com.bolingcavalry/v1

本地运行Controller

在这里插入图片描述
zhaoqin@zhaoqindeMBP-2 elasticweb % pwd
/Users/zhaoqin/github/blog_demos/kubebuilder/elasticweb
zhaoqin@zhaoqindeMBP-2 elasticweb % make run
/Users/zhaoqin/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/zhaoqin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2021-02-20T20:46:16.774+0800    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2021-02-20T20:46:16.774+0800    INFO    setup   starting manager
2021-02-20T20:46:16.775+0800    INFO    controller-runtime.controller   Starting EventSource    {"controller": "elasticweb", "source": "kind source: /, Kind="}
2021-02-20T20:46:16.776+0800    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2021-02-20T20:46:16.881+0800    INFO    controller-runtime.controller   Starting Controller     {"controller": "elasticweb"}
2021-02-20T20:46:16.881+0800    INFO    controller-runtime.controller   Starting workers        {"controller": "elasticweb", "worker count": 1}

新建elasticweb资源对象

apiVersion: v1
kind: Namespace
metadata:
  name: dev
  labels:
    name: dev
---
apiVersion: elasticweb.com.bolingcavalry/v1
kind: ElasticWeb
metadata:
  namespace: dev
  name: elasticweb-sample
spec:
  # Add fields here
  image: tomcat:8.0.18-jre8
  port: 30003
  singlePodQPS: 500
  totalQPS: 600
  1. 使用的namespace为<font color="blue">dev</font>
  2. 本次测试部署的应用为tomcat
  3. service使用宿主机的<font color="blue">30003</font>端口暴露tomcat的服务
  4. 假设单个pod能支撑500QPS,外部请求的QPS为600
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl apply -f config/samples/elasticweb_v1_elasticweb.yaml
namespace/dev created
elasticweb.elasticweb.com.bolingcavalry/elasticweb-sample created
2021-02-21T10:03:57.108+0800    INFO    controllers.ElasticWeb  1. start reconcile logic        {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.108+0800    INFO    controllers.ElasticWeb  3. instance : Image [tomcat:8.0.18-jre8], Port [30003], SinglePodQPS [500], TotalQPS [600], RealQPS [nil]       {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.210+0800    INFO    controllers.ElasticWeb  4. deployment not exists        {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.313+0800    INFO    controllers.ElasticWeb  set reference   {"func": "createService"}
2021-02-21T10:03:57.313+0800    INFO    controllers.ElasticWeb  start create service    {"func": "createService"}
2021-02-21T10:03:57.364+0800    INFO    controllers.ElasticWeb  create service success  {"func": "createService"}
2021-02-21T10:03:57.365+0800    INFO    controllers.ElasticWeb  expectReplicas [2]      {"func": "createDeployment"}
2021-02-21T10:03:57.365+0800    INFO    controllers.ElasticWeb  set reference   {"func": "createDeployment"}
2021-02-21T10:03:57.365+0800    INFO    controllers.ElasticWeb  start create deployment {"func": "createDeployment"}
2021-02-21T10:03:57.382+0800    INFO    controllers.ElasticWeb  create deployment success       {"func": "createDeployment"}
2021-02-21T10:03:57.382+0800    INFO    controllers.ElasticWeb  singlePodQPS [500], replicas [2], realQPS[1000] {"func": "updateStatus"}
2021-02-21T10:03:57.407+0800    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "elasticweb", "request": "dev/elasticweb-sample"}
2021-02-21T10:03:57.407+0800    INFO    controllers.ElasticWeb  1. start reconcile logic        {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.407+0800    INFO    controllers.ElasticWeb  3. instance : Image [tomcat:8.0.18-jre8], Port [30003], SinglePodQPS [500], TotalQPS [600], RealQPS [1000]      {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.407+0800    INFO    controllers.ElasticWeb  9. expectReplicas [2], realReplicas [2] {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.407+0800    INFO    controllers.ElasticWeb  10. return now  {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T10:03:57.407+0800    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "elasticweb", "request": "dev/elasticweb-sample"}
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl apply -f config/samples/elasticweb_v1_elasticweb.yaml
namespace/dev created
elasticweb.elasticweb.com.bolingcavalry/elasticweb-sample created
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get elasticweb -n dev                                 
NAME                AGE
elasticweb-sample   35s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get service -n dev                                    
NAME                TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
elasticweb-sample   NodePort   10.107.177.158   <none>        8080:30003/TCP   41s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get deployment -n dev                                 
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
elasticweb-sample   2/2     2            2           46s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev                                        
NAME                                 READY   STATUS    RESTARTS   AGE
elasticweb-sample-56fc5848b7-l5thk   1/1     Running   0          50s
elasticweb-sample-56fc5848b7-lqjk5   1/1     Running   0          50s

浏览器验证业务功能

在这里插入图片描述

修改单个Pod的QPS

spec:
  singlePodQPS: 800
kubectl patch elasticweb elasticweb-sample \
-n dev \
--type merge \
--patch "$(cat config/samples/update_single_pod_qps.yaml)"
在这里插入图片描述
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev                                                                                       
NAME                                 READY   STATUS    RESTARTS   AGE
elasticweb-sample-56fc5848b7-l5thk   1/1     Running   0          30m

修改总QPS

spec:
  totalQPS: 2600
kubectl patch elasticweb elasticweb-sample \
-n dev \
--type merge \
--patch "$(cat config/samples/update_total_qps.yaml)"
在这里插入图片描述
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev
NAME                                 READY   STATUS    RESTARTS   AGE
elasticweb-sample-56fc5848b7-8n7tq   1/1     Running   0          8m22s
elasticweb-sample-56fc5848b7-f2lpb   1/1     Running   0          8m22s
elasticweb-sample-56fc5848b7-l5thk   1/1     Running   0          48m
elasticweb-sample-56fc5848b7-q8p5f   1/1     Running   0          8m22s

删除验证

在这里插入图片描述
kubectl delete elasticweb elasticweb-sample -n dev
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl delete elasticweb elasticweb-sample -n dev
elasticweb.elasticweb.com.bolingcavalry "elasticweb-sample" deleted
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev                            
NAME                                 READY   STATUS        RESTARTS   AGE
elasticweb-sample-56fc5848b7-9lcww   1/1     Terminating   0          45s
elasticweb-sample-56fc5848b7-n7p7f   1/1     Terminating   0          45s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev
NAME                                 READY   STATUS        RESTARTS   AGE
elasticweb-sample-56fc5848b7-n7p7f   0/1     Terminating   0          73s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev
No resources found in dev namespace.
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get deployment -n dev
No resources found in dev namespace.
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get service -n dev   
No resources found in dev namespace.
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get namespace dev 
NAME   STATUS   AGE
dev    Active   97s

构建镜像

  1. 前面咱们在开发环境将controller运行起来尝试了所有功能,在实际生产环境中,controller并非这样独立于kubernetes之外,而是以pod的状态运行在kubernetes之中,接下来咱们尝试将controller代码编译构建成docker镜像,再在kubernetes上运行起来;
  2. 要做的第一件事,就是在前面的controller控制台上执行<font color="blue">Ctrl+C</font>,把那个controller停掉;
  3. 这里有个要求,就是您要有个kubernetes可以访问的镜像仓库,例如局域网内的Harbor,或者公共的hub.docker.com,我这为了操作方便选择了hub.docker.com,使用它的前提是拥有hub.docker.com的注册帐号;
  4. 在kubebuilder电脑上,打开一个控制台,执行<font color="blue">docker login</font>命令登录,根据提示输入hub.docker.com的帐号和密码,这样就可以在当前控制台上执行docker push命令将镜像推送到hub.docker.com上了(这个网站的网络很差,可能要登录好几次才能成功);
  5. 执行以下命令构建docker镜像并推送到hub.docker.com,镜像名为<font color="blue">bolingcavalry/elasticweb:002</font>:
make docker-build docker-push IMG=bolingcavalry/elasticweb:002
  1. hub.docker.com的网络状况不是一般的差,kubebuilder电脑上的docker一定要设置镜像加速,上述命令如果遭遇超时失败,请重试几次,此外,构建过程中还会下载诸多go模块的依赖,也需要您耐心等待,也很容易遇到网络问题,需要多次重试,所以,最好是使用局域网内搭建的Habor服务;
  2. 最终,命令执行成功后输出如下:
zhaoqin@zhaoqindeMBP-2 elasticweb % make docker-build docker-push IMG=bolingcavalry/elasticweb:002
/Users/zhaoqin/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/Users/zhaoqin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
?       elasticweb      [no test files]
?       elasticweb/api/v1       [no test files]
ok      elasticweb/controllers  8.287s  coverage: 0.0% of statements
docker build . -t bolingcavalry/elasticweb:002
[+] Building 146.8s (17/17) FINISHED                                                                                                                                                                                                  
 => [internal] load build definition from Dockerfile                                                                                                                                                                             0.1s
 => => transferring dockerfile: 37B                                                                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                                  0.0s
 => [internal] load metadata for gcr.io/distroless/static:nonroot                                                                                                                                                                1.8s
 => [internal] load metadata for docker.io/library/golang:1.13                                                                                                                                                                   0.7s
 => [builder 1/9] FROM docker.io/library/golang:1.13@sha256:8ebb6d5a48deef738381b56b1d4cd33d99a5d608e0d03c5fe8dfa3f68d41a1f8                                                                                                     0.0s
 => [stage-1 1/3] FROM gcr.io/distroless/static:nonroot@sha256:b89b98ea1f5bc6e0b48c8be6803a155b2a3532ac6f1e9508a8bcbf99885a9152                                                                                                  0.0s
 => [internal] load build context                                                                                                                                                                                                0.0s
 => => transferring context: 14.51kB                                                                                                                                                                                             0.0s
 => CACHED [builder 2/9] WORKDIR /workspace                                                                                                                                                                                      0.0s
 => CACHED [builder 3/9] COPY go.mod go.mod                                                                                                                                                                                      0.0s
 => CACHED [builder 4/9] COPY go.sum go.sum                                                                                                                                                                                      0.0s
 => CACHED [builder 5/9] RUN go mod download                                                                                                                                                                                     0.0s
 => CACHED [builder 6/9] COPY main.go main.go                                                                                                                                                                                    0.0s
 => CACHED [builder 7/9] COPY api/ api/                                                                                                                                                                                          0.0s
 => [builder 8/9] COPY controllers/ controllers/                                                                                                                                                                                 0.1s
 => [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go                                                                                                                      144.5s
 => CACHED [stage-1 2/3] COPY --from=builder /workspace/manager .                                                                                                                                                                0.0s
 => exporting to image                                                                                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                                                                                          0.0s
 => => writing image sha256:622d30aa44c77d93db4093b005fce86b39d5ba5c6cd29f1fb2accb7e7f9b23b8                                                                                                                                     0.0s
 => => naming to docker.io/bolingcavalry/elasticweb:002                                                                                                                                                                          0.0s
docker push bolingcavalry/elasticweb:002
The push refers to repository [docker.io/bolingcavalry/elasticweb]
eea77d209b68: Layer already exists 
8651333b21e7: Layer already exists 
002: digest: sha256:c09ab87f6fce3d85f1fda0ffe75ead9db302a47729aefd3ef07967f2b99273c5 size: 739
  1. 去hub.docker.com网站看看,如下图,新镜像已经上传,这样只要任何机器只要能上网就能pull此镜像到本地使用了:
在这里插入图片描述
  1. 镜像准备好之后,执行以下命令即可在kubernetes环境部署controller:
make deploy IMG=bolingcavalry/elasticweb:002
  1. 接下来像之前那样创建elasticweb资源对象,验证所有资源是否创建成功:
zhaoqin@zhaoqindeMBP-2 elasticweb % make deploy IMG=bolingcavalry/elasticweb:002
/Users/zhaoqin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=bolingcavalry/elasticweb:002
kustomize build config/default | kubectl apply -f -
namespace/elasticweb-system created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/elasticwebs.elasticweb.com.bolingcavalry configured
role.rbac.authorization.k8s.io/elasticweb-leader-election-role created
clusterrole.rbac.authorization.k8s.io/elasticweb-manager-role created
clusterrole.rbac.authorization.k8s.io/elasticweb-proxy-role created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/elasticweb-metrics-reader created
rolebinding.rbac.authorization.k8s.io/elasticweb-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/elasticweb-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/elasticweb-proxy-rolebinding created
service/elasticweb-controller-manager-metrics-service created
deployment.apps/elasticweb-controller-manager created
zhaoqin@zhaoqindeMBP-2 elasticweb % 
zhaoqin@zhaoqindeMBP-2 elasticweb % 
zhaoqin@zhaoqindeMBP-2 elasticweb % 
zhaoqin@zhaoqindeMBP-2 elasticweb % 
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl apply -f config/samples/elasticweb_v1_elasticweb.yaml 
namespace/dev created
elasticweb.elasticweb.com.bolingcavalry/elasticweb-sample created
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get service -n dev  
NAME                TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
elasticweb-sample   NodePort   10.96.234.7   <none>        8080:30003/TCP   13s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get deployment -n dev
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
elasticweb-sample   2/2     2            2           18s
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pod -n dev     
NAME                                 READY   STATUS    RESTARTS   AGE
elasticweb-sample-56fc5848b7-559lw   1/1     Running   0          22s
elasticweb-sample-56fc5848b7-hp4wv   1/1     Running   0          22s
  1. 这还不够!还有个重要的信息需要咱们检查---controller的日志,先看有哪些pod:
zhaoqin@zhaoqindeMBP-2 elasticweb % kubectl get pods --all-namespaces
NAMESPACE           NAME                                             READY   STATUS    RESTARTS   AGE
dev                 elasticweb-sample-56fc5848b7-559lw               1/1     Running   0          68s
dev                 elasticweb-sample-56fc5848b7-hp4wv               1/1     Running   0          68s
elasticweb-system   elasticweb-controller-manager-5795d4d98d-t6jvc   2/2     Running   0          98s
kube-system         coredns-7f89b7bc75-5pdwc                         1/1     Running   15         20d
kube-system         coredns-7f89b7bc75-nvbvm                         1/1     Running   15         20d
kube-system         etcd-hedy                                        1/1     Running   15         20d
kube-system         kube-apiserver-hedy                              1/1     Running   15         20d
kube-system         kube-controller-manager-hedy                     1/1     Running   16         20d
kube-system         kube-flannel-ds-v84vc                            1/1     Running   22         20d
kube-system         kube-proxy-hlppx                                 1/1     Running   15         20d
kube-system         kube-scheduler-hedy                              1/1     Running   16         20d
test-clientset      client-test-deployment-7677cc9669-kd7l7          1/1     Running   9          9d
test-clientset      client-test-deployment-7677cc9669-kt5rv          1/1     Running   9          9d
  1. 可见controller的pod名称为<font color="blue">elasticweb-controller-manager-5795d4d98d-t6jvc</font>,执行以下命令可以查看日志,多了<font color="blue">-c manager</font>参数是因为这个pod里面有两个容器,需要指定正确的容器才能看到日志:
kubectl logs -f \
elasticweb-controller-manager-5795d4d98d-t6jvc \
-c manager \
-n elasticweb-system
  1. 再次看到了熟悉的业务日志:
2021-02-21T08:52:27.064Z        INFO    controllers.ElasticWeb  1. start reconcile logic        {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.064Z        INFO    controllers.ElasticWeb  3. instance : Image [tomcat:8.0.18-jre8], Port [30003], SinglePodQPS [500], TotalQPS [600], RealQPS [nil]       {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.064Z        INFO    controllers.ElasticWeb  4. deployment not exists        {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.064Z        INFO    controllers.ElasticWeb  set reference   {"func": "createService"}
2021-02-21T08:52:27.064Z        INFO    controllers.ElasticWeb  start create service    {"func": "createService"}
2021-02-21T08:52:27.107Z        INFO    controllers.ElasticWeb  create service success  {"func": "createService"}
2021-02-21T08:52:27.107Z        INFO    controllers.ElasticWeb  expectReplicas [2]      {"func": "createDeployment"}
2021-02-21T08:52:27.107Z        INFO    controllers.ElasticWeb  set reference   {"func": "createDeployment"}
2021-02-21T08:52:27.107Z        INFO    controllers.ElasticWeb  start create deployment {"func": "createDeployment"}
2021-02-21T08:52:27.119Z        INFO    controllers.ElasticWeb  create deployment success       {"func": "createDeployment"}
2021-02-21T08:52:27.119Z        INFO    controllers.ElasticWeb  singlePodQPS [500], replicas [2], realQPS[1000] {"func": "updateStatus"}
2021-02-21T08:52:27.198Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "elasticweb", "request": "dev/elasticweb-sample"}
2021-02-21T08:52:27.198Z        INFO    controllers.ElasticWeb  1. start reconcile logic        {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.198Z        INFO    controllers.ElasticWeb  3. instance : Image [tomcat:8.0.18-jre8], Port [30003], SinglePodQPS [500], TotalQPS [600], RealQPS [1000]      {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.198Z        INFO    controllers.ElasticWeb  9. expectReplicas [2], realReplicas [2] {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.198Z        INFO    controllers.ElasticWeb  10. return now  {"elasticweb": "dev/elasticweb-sample"}
2021-02-21T08:52:27.198Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "elasticweb", "request": "dev/elasticweb-sample"}
  1. 再用浏览器验证tomcat已经启动成功;

卸载和清理

make uninstall

你不孤单,欣宸原创一路相伴

  1. Java系列
  2. Spring系列
  3. Docker系列
  4. kubernetes系列
  5. 数据库+中间件系列
  6. DevOps系列

欢迎关注公众号:程序员欣宸

微信搜索「程序员欣宸」,我是欣宸,期待与您一同畅游Java世界...
https://github.com/zq2599/blog_demos

上一篇下一篇

猜你喜欢

热点阅读