Cron HPA动态扩缩容实践
2022-07-02 本文已影响0人
Xiao_Yang
Kubernetes 默认 HPA 功能可实现基于Metric CPU 与 MEM 的使用率来进行动态扩缩容，在实现工作中针对如果已知的突峰流量的场景（如指定时间段的业务推广等）可能存在扩容置后不及时的问题，为了实现此类场景可以通过类似 Crontab 定时机制来动态扩缩容。本次实践是使用了一个开源的 FinOps 项目Crane ，项目在 k8s 默认的 HPA基础上封装了更智能的 ehpa、 Analytics、tsp 等功能，如果有兴趣可以进一步了解。今天主要实践了基于 cron 方式的ehpa 功能。
一、 安装
k8s 集群内安装 Crane (推荐采用自定义安装方式)
- clone 最新代码并切到最新分支内容
git clone https://github.com/gocrane/crane.git
cd crane
CRANE_LATEST_VERSION=$(curl -s https://api.github.com/repos/gocrane/crane/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")')
git checkout $CRANE_LATEST_VERSION
- 应用 CRD 资源对象的创建
kubectl apply -f deploy/manifests
- 修改 craned 连接的 prometheus server 的地址配置
vi deploy/craned/deployment.yaml
command:
- /craned
- --prometheus-address=http://<ipaddr>:<port>
- 应用核心服务部署
kubectl apply -f deploy/craned
kubectl apply -f deploy/metric-adapter
二、 配置
定义定时扩缩调度策略配置
#配置实例，创建 cron-scale.yaml
apiVersion: autoscaling.crane.io/v1alpha1
kind: EffectiveHorizontalPodAutoscaler
metadata:
name: ehpa-cron-test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx #指定scale deployment对象名称
minReplicas: 1 #最大复本数
maxReplicas: 5 #最小复本数
scaleStrategy: Auto
crons:
- name: "biz-push-cron-1"
timezone: "Asia/Shanghai" #时区定义
description: "scale up" #扩容至3个POD复本
start: "36 17 ? * *" ##17:36~17:37 执行scale up复本数为3
end: "37 17 ? * *"
targetReplicas: 3
- name: "biz-push-cron-2" #缩容至1个POD复本
timezone: "Asia/Shanghai"
description: "scale down"
start: "37 17 ? * *"
end: "45 17 ? * *" ##scale down复本数为1，注意start-end须>5m
targetReplicas: 1
# 应用策略
kubectl apply -f cron-scale.yaml
三、测试验证
- 查看应用策略状态
[crane]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ehpa-ehpa-cron-test Deployment/nginx 1/1 (avg) 1 5 1 3m
[root@itg-k8s-master01 crane]# kubectl get ehpa
NAME STRATEGY MINPODS MAXPODS SPECIFICPODS REPLICAS AGE
ehpa-cron-test Auto 1 5 1 3m3s
- 时间到达触发后状态查看，扩容到达三个
[crane]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ehpa-ehpa-cron-test Deployment/nginx 1/1 (avg) 1 5 3 5m29s
[crane]# kubectl get ehpa
NAME STRATEGY MINPODS MAXPODS SPECIFICPODS REPLICAS AGE
ehpa-cron-test Auto 1 5 3 5m58s
- 查看 pod craned 日志
I0624 1 effective_hpa_controller.go:40] Got ehpa default/ehpa-cron-test
I0624 1 effective_hpa_controller.go:139] Update EffectiveHorizontalPodAutoscaler status successful, ehpa default/ehpa-cron-test
- 时间到达触发后查看 ehpa 实例的状态信息
# 第一次cron调度(biz-push-cron-1)
# kubectl get ehpa ehpa-cron-test -oyaml
Status:
Conditions:
Last Transition Time: 2022-06-29T09:36:07Z
Message: Effective HPA is ready
Reason: EffectiveHorizontalPodAutoscalerReady
Status: True
Type: Ready
Last Transition Time: 2022-06-29T09:36:07Z
Message: the HPA controller was able to update the target scale to 3
Reason: SucceededRescale
Status: True
Type: AbleToScale
Last Transition Time: 2022-06-29T09:36:07Z
Message: the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
Reason: ValidMetricFound
Status: True
Type: ScalingActive
Last Transition Time: 2022-06-29T09:36:07Z
Message: the desired count is within the acceptable range
Reason: DesiredWithinRange
Status: False
Type: ScalingLimited
Current Replicas: 1
Expect Replicas: 3
---------------
Status:
Conditions:
Last Transition Time: 2022-06-29T09:37:07Z
Message: Effective HPA is ready
Reason: EffectiveHorizontalPodAutoscalerReady
Status: True
Type: Ready
Last Transition Time: 2022-06-29T09:37:07Z
Message: recent recommendations were higher than current one, applying the highest recent recommendation
Reason: ScaleDownStabilized
Status: True
Type: AbleToScale
Last Transition Time: 2022-06-29T09:37:07Z
Message: the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
Reason: ValidMetricFound
Status: True
Type: ScalingActive
Last Transition Time: 2022-06-29T09:37:07Z
Message: the desired count is within the acceptable range
Reason: DesiredWithinRange
Status: False
Type: ScalingLimited
Current Replicas: 3
Expect Replicas: 3
# (上次结束时间+5分钟)第二次 cron 调度 (biz-push-cron-2)
Status:
Conditions:
Last Transition Time: 2022-06-29T09:41:53Z
Message: Effective HPA is ready
Reason: EffectiveHorizontalPodAutoscalerReady
Status: True
Type: Ready
Last Transition Time: 2022-06-29T09:41:53Z
Message: the HPA controller was able to update the target scale to 1
Reason: SucceededRescale
Status: True
Type: AbleToScale
Last Transition Time: 2022-06-29T09:41:53Z
Message: the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
Reason: ValidMetricFound
Status: True
Type: ScalingActive
Last Transition Time: 2022-06-29T09:41:53Z
Message: the desired count is within the acceptable range
Reason: DesiredWithinRange
Status: False
Type: ScalingLimited
Current Replicas: 3
Expect Replicas: 1 #触发期望复本数
---------------
Status:
Conditions:
Last Transition Time: 2022-06-29T09:42:08Z
Message: Effective HPA is ready
Reason: EffectiveHorizontalPodAutoscalerReady
Status: True
Type: Ready
Last Transition Time: 2022-06-29T09:42:08Z
Message: recommended size matches current size
Reason: ReadyForNewScale
Status: True
Type: AbleToScale
Last Transition Time: 2022-06-29T09:42:08Z
Message: the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
Reason: ValidMetricFound
Status: True
Type: ScalingActive
Last Transition Time: 2022-06-29T09:42:08Z
Message: the desired count is within the acceptable range
Reason: DesiredWithinRange
Status: False
Type: ScalingLimited
Current Replicas: 1 # HPA 达成所期望复本数
Expect Replicas: 1
～～～ Finish ～～～
