cilium 节点断电后无法启动
2023-04-10 本文已影响0人
微凉哇
启动日志
level=error msg="Unable to contact k8s api-server" error="Get \"https://10.233.0.1:443/api/v1/namespaces/kube-system\": dial tcp 10.233.0.1:443: i/o timeout" ipAddr="https://10.233.0.1:443" subsys=k8s-client
level=error msg="Start hook failed" error="Get \"https://10.233.0.1:443/api/v1/namespaces/kube-system\": dial tcp 10.233.0.1:443: i/o timeout" function="client.(*compositeClientset).onStart" subsys=hive
level=debug msg="Executing stop hook" function="gops.registerGopsHooks.func2 (cell.go:51)" subsys=hive
level=info msg="Stopped gops server" address="127.0.0.1:9891" subsys=gops
level=info msg="Stop hook executed" duration="111.536µs" function="gops.registerGopsHooks.func2 (cell.go:51)" subsys=hive
level=fatal msg="failed to start: Get \"https://10.233.0.1:443/api/v1/namespaces/kube-system\": dial tcp 10.233.0.1:443: i/o timeout" subsys=cilium-operator-generic
异常节点执行
$ curl 10.233.0.1:443
curl: (7) Failed connect to 10.233.0.1:443; Connection refused
显然 kube-apiserver 服务无法通过 svc 地址进行访问,看了眼异常节点 kube-ipvs0 网桥不存在,且ipvs规则为空。
手动创建 ipvs 规则
ipvsadm -A -t 10.233.0.1:443 -s rr
ipvsadm -a -t 10.233.0.1:443 -r 10.9.20.41:6443 -m -w 1
手动创建 kube-ipvs0 网桥
ip li add kube-ipvs0 type dummy
ip addr add 10.233.0.1/32 dev kube-ipvs0
手动创建完毕后,cilium agent 异常节点启动正常,原因未知。