k8s动态存储nfs部署
2020-08-24 本文已影响0人
Firetheworld
k8s中存储分为静态存储以及动态存储。k8s提供了一套可以自动创建PV的机制,即:Dynamic Provisioning.而这个机制的核心在于:StorageClass这个API对象.
用户在部署应用。通过指定对应的动态存储卷，自动创建pvc。格式如下：
1.自动创建的 PV 以${namespace}-${pvcName}-${pvName}这样的命名格式创建在 NFS 服务器上的共享数据目录中
2.而当这个 PV 被回收后会以archieved-${namespace}-${pvcName}-${pvName}这样的命名格式存在 NFS 服务器上
StorageClass对象会定义下面两部分内容:
1,PV的属性.比如,存储类型,Volume的大小等.
2,创建这种PV需要用到的存储插件
有了这两个信息之后,Kubernetes就能够根据用户提交的PVC,找到一个对应的StorageClass,之后Kubernetes就会调用该StorageClass声明的存储插件,进而创建出需要的PV.
过程如下：
1）k8s运维人员先创建好storageclass；
2）用户创建使用存储类的持久化存储声明(PVC：PersistentVolumeClaim)；
3）存储持久化声明通知系统，它需要一个持久化存储(PV: PersistentVolume)；
4）系统读取存储类的信息；
5）系统基于存储类的信息，在后台自动创建PVC需要的PV；
6）用户创建一个使用PVC的Pod；
7）Pod中的应用通过PVC进行数据的持久化；
8）而PVC使用PV进行数据的最终持久化处理。
一、 我们进行动态存储卷的部署
首先需要准备一台nfs服务器、k8s集群v1.153。
动态存储卷由三部分组成：①权限控制创建②创建NFS provisioner③创建NFS资源的StorageClass。动态存储的yaml文件如'pub-nfs-sc.yaml：
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: storageclass #根据实际环境设定namespace,下面类同
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
namespace: storageclass #根据实际环境设定namespace,下面类同
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
namespace: storageclass #根据实际环境设定namespace,下面类同
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: storageclass
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: storageclass
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
namespace: storageclass #根据实际环境设定namespace,下面类同
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: storageclass
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: storageclass #与RBAC文件中的namespace保持一致
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: docker.in.zwxict.com/official/kubesphere/nfs-client-provisioner:v3.1.0-k8s1.11
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: pub-nfs-storage #provisioner名称,请确保该名称与 nfs-StorageClass.yaml文件中的provisioner名称保持一致
- name: NFS_SERVER
value: 10.192.52.140 #NFS Server IP地址
- name: NFS_PATH
value: /mnt/nfs/pub-nfs-sc #NFS挂载卷
volumes:
- name: nfs-client-root
nfs:
server: 10.192.52.140 #NFS Server IP地址
path: /mnt/nfs/pub-nfs-sc #NFS 挂载卷
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: pub-nfs-sc
provisioner: pub-nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致
reclaimPolicy: Retain # 回收策略，pv以及pvc删除后，nfs服务器文件保留
部署：
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl apply -f pub-nfs-sc.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
deployment.apps/nfs-client-provisioner created
storageclass.storage.k8s.io/pub-nfs-sc created
查看创建的storageclass
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl get sc
NAME PROVISIONER AGE
pub-nfs-sc pub-nfs-storage 6m1s
二、验证
创建pvctest-claim.yaml，并挂载到pod。
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: pub-nfs-sc # 指定创建的sc名称
resources:
requests:
storage: 1Mi
查看pvc的状态是否为Bound：
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl apply -f test-claim.yaml
persistentvolumeclaim/test-claim created
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim Bound pvc-b06ba929-bb98-42cf-93f8-465cb9c64520 1Mi RWX pub-nfs-sc 7s
test-po.yaml使用指定创建的pvc：
kind: Pod
apiVersion: v1
metadata:
name: test-pod
spec:
containers:
- name: test-pod
image: docker.in.zwxict.com/tools/busybox:latest
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1" #创建一个SUCCESS文件后退出
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: test-claim #与PVC名称保持一致
到nfs路径中查看是否正常创建SUCCESS文件：
[root@nfs-140 pub-nfs-sc]# ls default-test-claim-pvc-b06ba929-bb98-42cf-93f8-465cb9c64520/ #文件规则是按照${namespace}-${pvcName}-${pvName}创建的
SUCCESS
ps: 创建pvc的时候发现pvc的状态为Pending，查看nfs创建的日志确认原因，大部分是由于角色绑定对应的权限没有生效导致的，确保角色与权限以及对应的deployment在同一命名下空间。
创建一个nginx应用test-pod.yaml，把web页面挂载出来，指定动态存储卷：
---
apiVersion: v1
kind: Service
metadata:
name: nginx-headless
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None #注意此处的值,None表示无头服务
selector:
app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: web
spec:
serviceName: "nginx"
replicas: 2 #两个副本
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: docker.in.zwxict.com/official/nginx:1.19.2
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
annotations:
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: pub-nfs-sc
resources:
requests:
storage: 1Mi
结果如下：
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl apply -f nginx.yaml
service/nginx-headless created
statefulset.apps/web created
[root@k8s-10-192-52-123 pub-nfs-sc]# cat nginx.yaml
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl get pvc # 查看pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-web-0 Bound pvc-4a12ddef-8c37-4c9d-ba35-e200595e1d16 1Mi RWO pub-nfs-sc 92s
www-web-1 Bound pvc-f7a17f18-21b7-44da-bf23-48671bf8a0f2 1Mi RWO pub-nfs-sc 88s
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl get pv|grep pub-nfs-sc # 查看pv
pvc-4a12ddef-8c37-4c9d-ba35-e200595e1d16 1Mi RWO Retain Bound default/www-web-0 pub-nfs-sc 2m58s
pvc-b06ba929-bb98-42cf-93f8-465cb9c64520 1Mi RWX Retain Bound default/test-claim pub-nfs-sc 12m
pvc-f7a17f18-21b7-44da-bf23-48671bf8a0f2 1Mi RWO Retain Bound default/www-web-1 pub-nfs-sc 2m54s
nfs服务器查看自动创建的结果如下：
default-test-claim-pvc-b06ba929-bb98-42cf-93f8-465cb9c64520
[root@nfs-140 pub-nfs-sc]# ll
total 0
drwxrwxrwx 2 root root 6 Aug 25 14:31 default-www-web-0-pvc-4a12ddef-8c37-4c9d-ba35-e200595e1d16
drwxrwxrwx 2 root root 6 Aug 25 14:31 default-www-web-1-pvc-f7a17f18-21b7-44da-bf23-48671bf8a0f2
到nfs服务器上给两个nginx添加不同内容,并进入pod中进行访问：
[root@nfs-140 pub-nfs-sc]# echo "web-00" > default-www-web-0-pvc-4a12ddef-8c37-4c9d-ba35-e200595e1d16/index.html
[root@nfs-140 pub-nfs-sc]# echo "web-11" > default-www-web-1-pvc-f7a17f18-21b7-44da-bf23-48671bf8a0f2/index.html
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl exec -it web-0 -- /bin/sh
/ # nslookup nginx-headless
Server: 192.168.0.10
Address: 192.168.0.10:53
Name: nginx-headless.default.svc.cluster.local
Address: 192.168.58.207
Name: nginx-headless.default.svc.cluster.local
Address: 192.168.90.42
/ # curl 192.168.58.207
web-00
/ # curl 192.168.90.42
web-11
设置默认的storageclass
[root@k8s-master-155-221 classStorage]# kubectl patch storageclass pub-nfs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' #设置pub-nfs-sc为默认后端存储
storageclass.storage.k8s.io/managed-nfs-storage patched
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl get sc
NAME PROVISIONER AGE
pub-nfs-sc (default) pub-nfs-storage 30s
[root@k8s-master-155-221 deploy]# kubectl patch storageclass pub-nfs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' #取消默认存储后端
storageclass.storage.k8s.io/managed-nfs-storage patched
[root@k8s-10-192-52-123 pub-nfs-sc]# kubectl get sc
NAME PROVISIONER AGE
pub-nfs-sc pub-nfs-storage 2m44s
