锻骨境-第9层 k8s自建Ingress Controlle
2019-10-30 本文已影响0人
一笑醉红颜zh
Nginx Ingress
这里使用Nginx Ingress 提供对外的服务。
ingress 的配置是针对郁域名方式的
正常情况下 ，我们用的域名都是带TLS证书 ，防止篡改的。
这里自定义一个域名叫 k8s.local . 在自己的电脑上配置下window host. 192.168.10.133 *.k8s.local
使用cfssl给自己的集群加个域名证书
- 编辑k8s-local.json
{
"CN": "k8s.local",
"hosts": [
"127.0.0.1",
"*.k8s.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shenzhen",
"L": "shenzhen",
"O": "k8s",
"OU": "System"
}
]
}
- 生成两个 证书，*.k8s.local这个域名的,分别在kube-system 和 kubernets-dashboard 命名空间下，注意和自己的 服务 在一个空间下。请注意本层dashboard 使用的是自己颁发的证书：
必须创建secret 名为： kubernetes-dashboard-certs 。 如果没有，dashboard 会使用自己的证书。
[root@k8s-node1 etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes k8s-local.json | cfssljson -bare k8s-local
2019/11/03 15:03:39 [INFO] generate received request
2019/11/03 15:03:39 [INFO] received CSR
2019/11/03 15:03:39 [INFO] generating key: rsa-2048
2019/11/03 15:03:39 [INFO] encoded CSR
2019/11/03 15:03:39 [INFO] signed certificate with serial number 65727990797100765876932917522552114529933202146
2019/11/03 15:03:39 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]# ls
ca-config.json ca-csr.json ca.pem etcd-2.etcd etcd.yaml k8s-csr.json k8s-local.json k8s-local.pem kubernetes-key.pem nohup.out
ca.csr ca-key.pem etcd etcdctl install k8s-local.csr k8s-local-key.pem kubernetes.csr kubernetes.pem ssl
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]# kubectl -n kubernetes-dashboard create secret tls kubernetes-dashboard-certs --key=k8s-local-key.pem --cert=k8s-local.pem
secret/default-tls-cert created
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]#
[root@k8s-node1 etcd]# kubectl -n kube-system create secret tls default-tls-cert --key=k8s-local-key.pem --cert=k8s-local.pem
secret/default-tls-cert created
[root@k8s-node1 etcd]#
开始搭建Nginx Ingress Controller
官方文档：https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
下载下来之后，对于他的文档，我 修改了部署到的命名空间为kube-system 并去除了nodeSelector 节点选择器。
- 我自己的ingress 部署文件
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: kube-system
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
---
如果不修改镜像，会pull 失败 ,用describe命令查看pod ，此时，要么更换国内被人的镜像，要么FQ。 这里我使用代理后下载官网的镜像。
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m34s default-scheduler Successfully assigned kube-system/nginx-ingress-controller-5c64dcfbf9-4gdj7 to k8s-master
Warning Failed 3m14s kubelet, k8s-master Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 3m14s kubelet, k8s-master Error: ErrImagePull
Normal BackOff 3m13s kubelet, k8s-master Back-off pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"
Warning Failed 3m13s kubelet, k8s-master Error: ImagePullBackOff
Normal Pulling 2m58s (x2 over 3m29s) kubelet, k8s-master pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1"
[root@k8s-master k8s]#
- pull 镜像，其实k8s 也会自己pull （可选，请自己科学上网）
[root@k8s-master etcd]# docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
0.26.1: Pulling from kubernetes-ingress-controller/nginx-ingress-controller
c8775c51b291: Already exists
4fc647720de5: Downloading [============================> ] 9.911MB/17.21MB
4bee7d3b55eb: Downloading [==================================> ] 538.5kB/771.2kB
85ee1a272ac8: Downloading [=> ] 3.242MB/121.9MB
6a37290ece43: Waiting
cd42756652d1: Waiting
ba0a46163f53: Waiting
b15c3bdebd38: Waiting
69a3d60cbd64: Waiting
7b1f35e5645a: Waiting
8c7b9cd15ac7: Waiting
9021fc6169bc: Waiting
- 镜像好了之后，部署service，nginx-service.yaml 如下：
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx-ingress-controller
name: kube-system
name: ingress-nginx
spec:
externalTrafficPolicy: Local
ports:
- name: cce-service-0
nodePort: 31304
port: 80
protocol: TCP
targetPort: 80
- name: cce-service-1
nodePort: 31381
port: 443
protocol: TCP
targetPort: 443
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer: {}
- 查看结果：
[root@k8s-master k8s]# kubectl apply -f nginx-ingress-svc.yaml -n kube-system
service/nginx-lb-svc created
[root@k8s-master k8s]#
[root@k8s-master k8s]#
[root@k8s-master k8s]#
[root@k8s-master k8s]#
[root@k8s-master k8s]#
[root@k8s-master k8s]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 23h
nginx-lb-svc LoadBalancer 10.98.182.170 192.168.10.134 80:31304/TCP,443:31381/TCP 19s
配置dashboard 域名访问：
配置ClusterIP 类型的svc
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-internal
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
type: ClusterIP
- 配置ingress 路由规则：
注意： nginx.ingress.kubernetes.io/secure-backends: "true" 在0.18 之后失效。
- 重点是这几个注解：
nginx.ingress.kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
- Ingress 配置规则：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
generation: 1
name: dashboard-ingress
namespace: kubernetes-dashboard
spec:
tls:
- hosts:
- admin.k8s.local
secretName: default-tls-cert
rules:
- host: admin.k8s.local
http:
paths:
- backend:
serviceName: kubernetes-dashboard-internal
servicePort: 443
path: /
- 此时可以用Chrome 查看页面：
请现在hosts 里配置自己的域名解析
image.png
至此，完成了用Ingress Controller 的路由规则访问dashboard,且使用https 的方式。
至于自定义的域名证书怎么被浏览器认为是安全的，见：
锻骨境-号外1 k8s 里定义证书，本地计算机使用
- 其他相关知识
- k8s https 管理 (https://my.oschina.net/u/2306127/blog/1929905)
- 客户端管理集群管理工具： https://kubernetic.com/
