Mac Docker - Prometheus QPS Hpa
top
0x00 TLDR;
在本机K8s环境验证基于QPS实施 hpa 的功能 . 顺带看一下qps 指标如何使用.
k8s 版本: v1.22.4
前提: 有一个干净的Mac Docker Desktop, 启用 Kubernetes.
# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcd69978-77tph 1/1 Running 0 91s
kube-system coredns-78fcd69978-x7gf8 1/1 Running 0 91s
kube-system etcd-docker-desktop 1/1 Running 0 97s
kube-system kube-apiserver-docker-desktop 1/1 Running 0 96s
kube-system kube-controller-manager-docker-desktop 1/1 Running 0 95s
kube-system kube-proxy-j2fgt 1/1 Running 0 92s
kube-system kube-scheduler-docker-desktop 1/1 Running 0 94s
kube-system storage-provisioner 1/1 Running 0 65s
kube-system vpnkit-controller 1/1 Running 0 65s
0x01 安装 ingress nginx
参考: https://kubernetes.github.io/ingress-nginx/deploy/#docker-for-mac
helm upgrade --install ingress-nginx ingress-nginx \
--repo https://kubernetes.github.io/ingress-nginx \
--namespace ingress-nginx --create-namespace
- 查看部署结果
k get po -A | grep nginx
ingress-nginx ingress-nginx-controller-cb87575f5-sxjgs 1/1 Running 0 9m20s
注意: 默认的系统端口 80, 是无法访问到集群里面的. 需要转到系统其他端口.
k get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 18m
ingress-nginx ingress-nginx-controller LoadBalancer 10.108.140.103 localhost 80:30725/TCP,443:32615/TCP 15m
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.107.145.159 <none> 443/TCP 15m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 18m
# 转发到主机 8000端口
k port-forward service/ingress-nginx-controller -n ingress-nginx 8000:80
- 请求默认 nginx-ingress 响应: 404, 表示成功.
curl localhost:8000
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
0x02 关于 Prometheus 在K8s中使用
为了实现自定义metrics提取功能 , 需要三个条件:
- 安装 kube-prometheus : 在K8s中部署promtheus 全家桶
- pod里的服务支持 /metrics 接口
- 安装 prometheus-adapter: 实现 k8s 监控格式向 prometheus 格式转换
0x03 部署 kube-prometheus
git clone https://github.com/coreos/kube-prometheus.git --depth=1
cd kube-prometheus
kubectl create -f manifests/setup
kubectl create -f manifests
这里有个小坑: node-expoter-pod 运行异常. 这个问题不是本篇主要方向, 直接砍掉.
有缘以后再回来补上.
k delete -f manifests/nodeExporter-daemonset.yaml
查看Pod状态:
# k get po -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 2 (2m25s ago) 9m29s
alertmanager-main-1 2/2 Running 2 (2m25s ago) 9m29s
alertmanager-main-2 2/2 Running 2 (2m25s ago) 9m29s
blackbox-exporter-69894767d5-5r684 3/3 Running 3 (2m25s ago) 13m
grafana-7bb5967c6-5n289 1/1 Running 1 (2m25s ago) 13m
kube-state-metrics-5d6885d89-kdvcm 3/3 Running 3 (2m25s ago) 13m
prometheus-adapter-6cf5d8bfcf-56rbl 1/1 Running 1 (2m25s ago) 13m
prometheus-adapter-6cf5d8bfcf-9tq8d 1/1 Running 2 (2m4s ago) 13m
prometheus-k8s-0 2/2 Running 0 9m28s
prometheus-k8s-1 2/2 Running 0 9m28s
prometheus-operator-7f58778b57-5xbdb 2/2 Running 2 (2m25s ago) 13m
暴露一下 grafana 和 prometheus-server的主机访问
k port-forward service/grafana -n monitoring 3000
k port-forward service/prometheus-k8s -n monitoring 9090
网页打开:
grafana : http://localhost:3000, admin/admin
prometheus: http://localhost:9090
验证一下 top po 指令:
k top po -A
NAMESPACE NAME CPU(cores) MEMORY(bytes)
ingress-nginx ingress-nginx-controller-cb87575f5-sxjgs 2m 156Mi
kube-system coredns-78fcd69978-77tph 2m 18Mi
kube-system coredns-78fcd69978-x7gf8 3m 19Mi
kube-system etcd-docker-desktop 20m 75Mi
kube-system kube-apiserver-docker-desktop 49m 330Mi
kube-system kube-controller-manager-docker-desktop 22m 51Mi
kube-system kube-proxy-j2fgt 0m 15Mi
kube-system kube-scheduler-docker-desktop 2m 19Mi
kube-system storage-provisioner 1m 6Mi
kube-system vpnkit-controller 0m 8Mi
monitoring alertmanager-main-0 2m 32Mi
monitoring alertmanager-main-1 2m 33Mi
monitoring alertmanager-main-2 4m 29Mi
monitoring blackbox-exporter-69894767d5-5r684 0m 21Mi
monitoring grafana-7bb5967c6-5n289 10m 58Mi
monitoring kube-state-metrics-5d6885d89-kdvcm 3m 33Mi
monitoring prometheus-adapter-6cf5d8bfcf-56rbl 2m 25Mi
monitoring prometheus-adapter-6cf5d8bfcf-9tq8d 4m 23Mi
monitoring prometheus-k8s-0 21m 180Mi
monitoring prometheus-k8s-1 18m 189Mi
monitoring prometheus-operator-7f58778b57-5xbdb 0m 35Mi
0x04 部署测试程序
参考: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/walkthrough.md
部署应用
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app
labels:
app: sample-app
spec:
replicas: 1
selector:
matchLabels:
app: sample-app
template:
metadata:
labels:
app: sample-app
spec:
containers:
- image: luxas/autoscale-demo:v0.1.2
name: metrics-provider
ports:
- name: http
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
labels:
app: sample-app
name: sample-app
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app: sample-app
type: ClusterIP
暴露访问接口
k get po
NAME READY STATUS RESTARTS AGE
sample-app-7cfb596f98-hxwqj 1/1 Running 0 4m25s
# 直接把Pod端口暴露出来
k port-forward sample-app-7cfb596f98-hxwqj 8010:8080
# 访问 验证
curl localhost:8010/metrics
# HELP http_requests_total The amount of requests served by the server in total
# TYPE http_requests_total counter
http_requests_total 2
此时, 测试项目已支持 /metrics接口.
部署HPA:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: sample-app
spec:
scaleTargetRef:
# point the HPA at the sample application
# you created above
apiVersion: apps/v1
kind: Deployment
name: sample-app
# autoscale between 1 and 10 replicas
minReplicas: 1
maxReplicas: 10
metrics:
# use a "Pods" metric, which takes the average of the
# given metric across all pods controlled by the autoscaling target
- type: Pods
pods:
# use the metric that you used above: pods/http_requests
metricName: http_requests
# target 500 milli-requests per second,
# which is 1 request every two seconds
targetAverageValue: 500m
此时查看 日志, 发现 会有错误:
k logs -f kube-controller-manager-docker-desktop -n kube-system
I0315 09:47:07.652833 1 event.go:291] "Event occurred" object="default/sample-app" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric http_requests: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered"
意思是, hpa 找不到 http_requests 这个指标.
这就扯出下一个话题: 部署 adapter
0x05 部署 prometheus-adapter
参考: https://www.cnblogs.com/dudu/p/12146344.html
git clone https://github.com/DirectXMan12/k8s-prometheus-adapter.git --depth=1
cd k8s-prometheus-adapter
# 创建证书
k create ns custom-metrics
export PURPOSE=serving
openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout ${PURPOSE}.key -out ${PURPOSE}.crt -subj "/CN=ca"
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "${PURPOSE}-ca-config.json"
kubectl -n custom-metrics create secret generic cm-adapter-serving-certs --from-file=./serving.crt --from-file=./serving.key
# 部署
cd deploy/manifests
# 更新 custom-metrics-apiservice.yaml:
# apiregistration.k8s.io/v1beta1 --> apiregistration.k8s.io/v1
# 更新 custom-metrics-apiserver-deployment.yaml
# 镜像修改: image: directxman12/k8s-prometheus-adapter:v0.8.4
# URL修改: --prometheus-url=http://prometheus.prom.svc:9090/ --> --prometheus-url=http://prometheus-k8s.monitoring.svc:9090/
k apply -f .
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
deployment.apps/custom-metrics-apiserver created
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
serviceaccount/custom-metrics-apiserver created
service/custom-metrics-apiserver created
apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
apiservice.apiregistration.k8s.io/v1beta2.custom.metrics.k8s.io created
apiservice.apiregistration.k8s.io/v1beta1.external.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
configmap/adapter-config created
clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created
安装 monitor, 这一步才是自定义指标的关键
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
name: sample-app
labels:
app: sample-app
spec:
selector:
matchLabels:
app: sample-app
endpoints:
- port: http
0x05 验证HPA
前面几步如果正常进行的话, 可以先通过api 进行确认:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests?selector=app=sample-app" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "sample-app-7cfb596f98-87hms",
"apiVersion": "/v1"
},
"metricName": "http_requests",
"timestamp": "2022-03-16T01:09:49Z",
"value": "66m",
"selector": null
}
...
}
看到上面的输出, 表示可以adapter可以正常输出 http_requests 指标.
接下来, 多请求几次接口: http://localhost:8010/metrics
查看 hpa 变化
k get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
sample-app Deployment/sample-app 66m/500m 1 10 4 15h
看到po 增加到4个.
停止请求, 等待一段时间 , 查看 hpa, 回落到 1个副本
k get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
sample-app Deployment/sample-app 66m/500m 1 10 1 15h
到此, 基于QPS的 HPA 实验完毕
0x06 自定义规则: http_requests_total 和 http_requests 的关系
我们来查看一下, /metrics 接口只暴露了 http_requests_total, 那么这个 http_requets是怎么来的.
先查看一下下面的结果 :
k get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "sample-app-7cfb596f98-hxwqj",
"apiVersion": "/v1"
},
"metricName": "http_requests",
"timestamp": "2022-03-16T02:44:08Z",
"value": "66m",
"selector": null
}
]
}
表示测试 pod 支持 "metricName": "http_requests"
可是这货从哪来的 ?
进一步跟一下 : https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config-walkthrough.md
配置adapter规则大致分四步:
The adapter considers metrics in the following ways:
First, it discovers the metrics available (Discovery)
Then, it figures out which Kubernetes resources each metric is associated with (Association)
Then, it figures out how it should expose them to the custom metrics API (Naming)
Finally, it figures out how it should query Prometheus to get the actual numbers (Querying)
参考一段例子: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/sample-config.yaml
# this rule matches cumulative non-cAdvisor metrics
- seriesQuery: '{namespace!="",__name__!="^container_.*"}'
name: {matches: "^(.*)_total$"}
resources:
# specify an a generic mapping between resources and labels. This
# is a template, like the `metricsQuery` template, except with the `.Group`
# and `.Resource` strings available. It will also be used to match labels,
# so avoid using template functions which truncate the group or resource.
# Group will be converted to a form acceptible for use as a label automatically.
template: "<<.Resource>>"
# if we wanted to, we could also specify overrides here
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])) by (<<.GroupBy>>)"
{matches: "^(.*)_total$"} 这里仿佛解释了 http_requests_total 和 http_requests 的关系了.
相当于对(.*)_total 这类计数型指标, 统一转换成名称为 \1 , 值为 概要数据.
0x07 Spring工程自定义指标
大致过程:
- pod 已经有/metrics 输出
- 通过 prometheus 进行表达式验证
官方指引: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config.md
Spring Demo 参考: https://tanzu.vmware.com/developer/guides/spring-prometheus/
编写测试用Spring项目:
- pom.xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
- application.yaml
spring:
application:
name: java-demo
management:
server:
port: 8611
endpoints:
web:
exposure:
include: "*"
base-path: /metrics
metrics:
tags:
application: ${spring.application.name}
server:
port: 8888
- 测试类
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
@RestController
@SpringBootApplication
public class QpsDemoApplication {
public static void main(String[] args) {
SpringApplication.run(QpsDemoApplication.class, args);
}
Counter visitCounter;
public QpsDemoApplication(MeterRegistry registry) {
visitCounter = Counter.builder("mvc_access").description("http access count").register(registry);
}
@GetMapping({"/", "/index"})
public String index() {
visitCounter.increment();
return "running count .. " + visitCounter.count();
}
}
本机运行检测指标接口:
curl -s http://localhost:8611/metrics/prometheus | grep access
# HELP mvc_access_total http access count
# TYPE mvc_access_total counter
mvc_access_total 0.0
运行业务接口:
curl -s http://localhost:8888/
running count .. 2.0
注意 mvc_access_total 这个名称的来自 Java 代码
编译镜像
- Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /tmp
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
COPY target/*.jar app.jar
ENTRYPOINT [ "java", "-jar", "/app.jar" ]
- 编译
# package
mvn clean package -DskipTests -U
# 编译镜像,存到本机 Docker 仓库
docker build . -t mvc_access:0.1
部署应用
---
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mvc-demo
namespace: default
labels:
app: mvc-demo
spec:
replicas: 1
template:
metadata:
name: mvc-demo
labels:
app: mvc-demo
spec:
containers:
- name: mvc-demo
image: mvc_access:0.1
imagePullPolicy: IfNotPresent
args:
- --server.port=8080
restartPolicy: Always
selector:
matchLabels:
app: mvc-demo
---
apiVersion: v1
kind: Service
metadata:
name: mvc-demo
namespace: default
labels:
app: mvc-demo
spec:
selector:
app: mvc-demo
ports:
- port: 8080
targetPort: 8080
name: http
- port: 8611
targetPort: 8611
name: probe
type: ClusterIP
暴露并验证两种接口
k port-forward mvc-demo-58d6b877b4-n6245 8088:8080
k port-forward mvc-demo-58d6b877b4-n6245 8611
# 业务接口
curl -s http://localhost:8088/index
running count .. 1.0
# 指标接口
curl -s http://localhost:8611/metrics/prometheus | grep access
# HELP mvc_access_total http access count
# TYPE mvc_access_total counter
mvc_access_total 1.0
查看 api 指标收集:
# 目前没有 mvc 相关指标
k get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq . | grep mvc
为了让prometheus发现这些目标, 来部署一下 ServiceMonitor
kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
name: mvc-demo
labels:
app: mvc-demo
spec:
selector:
matchLabels:
app: mvc-demo
endpoints:
- port: probe
path: "/metrics/prometheus"
接下来打开PrometheusUI http://localhost:9090/targets , 查看serviceMonitor状态.
这里有个小坑, target active 一直没出来, 查看k8s日志, 查明是权限问题.
k logs -f prometheus-k8s-0 -n monitoring
ts=2022-03-17T01:19:52.747Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.22.5/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
参考: https://github.com/prometheus-operator/kube-prometheus/issues/483#issuecomment-610427646
更新Role角色权限:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.32.1
name: prometheus-k8s
namespace: monitoring
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
- services
- pods
- endpoints
verbs:
- get
- list
- watch
- nonResourceURLs:
- /metrics
- /metrics/prometheus
verbs:
- get
其实, 这一步应该在安装 kube-protmethes时就准备好.
注意: - nonResourceURLs: 定义了默认提取信息的路径, 可以和客户端约定好, 就少写配置了.
然后, 再来刷新一下 PrometheusUI: http://localhost:9090/targets
service monitor targets
Nice, 说明Pod原始的数据, 已经上报给Promtheus了, 我们通过PromQL来确认一下: http://localhost:9090/graph
输入我们的metrics名称: mvc_access
PromQL
Cool ~, 到这里, 我们在代码里定义的指标名称 和 说明, 已经上报到Promtheus数据库, 而且支持查询.
这里就有两个后续了, 一种常规思路就是通过 Grafana 编写 PromQL 去显示这些自定义指标数据.
但我们的目标是, 基于这些指标, 去创建HPA 能识别的规则.
0x08 编写 adapter rule 文件
上面聊到HPA 基于自定义指标(QPS) 来伸缩, 但是上一步的指标是 Prometheus的数值, hpa 是k8s的核心功能 ,它们协作的话, 需要 adapter rule .
官方例子:
rules:
- seriesQuery: '{__name__=~"^container_.*",container!="POD",namespace!="",pod!=""}'
resources:
overrides:
namespace: {resource: "namespace"},
pod: {resource: "pod"},
name:
matches: "^container_(.*)_seconds_total$"
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])) by (<<.GroupBy>>)"
写rule 之前呢, 先了解一下 PromQL的入门用法.
首先, 给Pod改成 两个实例.
指标查询
对比一下 例子的第一行, 不难发现, seriesQuery 对标的就是 Promtheus指标的 tag .
我们可以试一下效果:
输入 {application="mvc-demo"}, 会把关联tag 的所有指标输出来.
query
参考 https://prometheus.io/docs/prometheus/latest/querying/functions/#rate
解释一下
rate(): rate用来计算两个 间隔时间内发生的变化率（一段时间内平均每秒的增量）。
rate
rate(mvc_access_total[1m]) 返回每个Pod, 一分钟内每秒请求次数
所以, 接下来再来一个sum 就是表示一个业务类型的合计QPS:
sum
直接更新rule 配置文件, 参考: https://github.com/ntman4real/nginxmets#to-configure-the-custom-metrics-rule-and-cusotmizong-the-name-to-myrequests
# k edit cm adapter-config -n custom-metrics
apiVersion: v1
data:
config.yaml: |
rules:
- seriesQuery: 'mvc_access_total'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: ""
as: "mvc_qps"
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])) by (<<.GroupBy>>)
重启 metrics-server, 使其读取rule configmap 配置
# k get po -A | grep metrics-api
custom-metrics custom-metrics-apiserver-54bdbb4ff8-tx6bb 1/1 Running 0 7m20s
k get po custom-metrics-apiserver-54bdbb4ff8-tx6bb -n custom-metrics -o yaml | k replace --force -f -
提交后, 验证一下 custom api
#k get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq . | grep qps
"name": "namespaces/mvc_qps",
"name": "pods/mvc_qps",
看到上面的信息, 表示已经为我们的Pod添加了 mvc_qps 指标.
创建 hpa, 1000m 表示: service每秒请求数为1时, 就先触发伸缩.
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: mvc-demo
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: mvc-demo
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: mvc_qps
targetAverageValue: 1000m
多请求几次外部接口
查看 hpa 状态
k get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
mvc-demo Deployment/mvc-demo 1189m/1 2 10 2 100s
sample-app Deployment/sample-app 66m/500m 1 10 1 45h
mvc-demo Deployment/mvc-demo 274m/1 2 10 3 2m
mvc-demo Deployment/mvc-demo 318m/1 2 10 3 3m
mvc-demo Deployment/mvc-demo 240m/1 2 10 3 4m
mvc-demo Deployment/mvc-demo 18m/1 2 10 3 5m
mvc-demo Deployment/mvc-demo 0/1 2 10 3 6m
mvc-demo Deployment/mvc-demo 0/1 2 10 2 7m
看到 replicas 副本数配置生效.
到这里, 淦了3天 自定义 qps HPA任务终于达成.
继续学习:
https://github.com/kubernetes-sigs/prometheus-adapter
https://www.cnblogs.com/dudu/p/12146344.html
https://www.cnblogs.com/charlieroro/p/11898521.html
参考:
http://www.xuyasong.com/?p=1520
What's Next
上面的实现方式代码侵入性太强, 不适合生产.
找时间查找其他方式.
