第9章 项目部署到kubernetes平台
2019-06-13 本文已影响0人
六弦极品
一. 准备工作与注意事项
1. 部署的项目情况
(1) 业务架构及服务(dubbo, spring cloud)
(2) 第三方服务，例如mysql, redis, zookeeper, eruke, mq
(3) 服务之间怎么通信
(4) 资源消耗：硬件资源，带宽
2. 项目部署涉及相关k8s资源
(1) 使用namespace 进行不同项目隔离，或者隔离不同环境( test, prod, dev)
(2) 无状态应用(deployment)
(3) 有状态应用(statefulset, pv, pvc)
(4) 发布暴露外部访问( Service, ingress)
(5) 存储一些数据(secret, configmap)
3. 项目的基础镜像
4. 编排部署(镜像为交付物)
https://github.com/lizhenliang/tomcat-java-demo
https://github.com/lizhenliang/php-demo
代码仓库下都会有一个Dockerfile 镜像构建文件
(1). 持续集成与交付
项目构建(java)，CI/CD 环境这个阶段自动完成（代码拉取->代码编译构建->镜像打包 -> 推送到镜像仓库）
(2). 编写yaml文件
编写yaml文件， 使用这个镜像
5. 持续部署
kubectl -> yaml -> 镜像仓库拉取镜像 -> Service(集群内部访问)/Ingress暴露外部用户访问
二. 部署Java项目
1. NFS网络存储准备
安装NFS服务,每个node节点都安装nfs客户端：
# yum install nfs-utils -y
# cat /etc/exports
/ifs/kubernetes *(rw,no_root_squash)
# mkdir /ifs/kubernetes -p
# systemctl start nfs
2. 配置NFS动态卷供给PV
下载三个yaml文件：
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client/deploy
class.yaml 动态创建PV
deployment.yaml 创建控制及pod等资源
rbac.yaml deployment.yaml部署的pod时，需要访问api-server权限请求k8s的PV&PVC资源
注意： deployment.yaml中定义连接NFS的地址及目录
3. 部署私有镜像仓库harbor
https://www.jianshu.com/p/7ca6c59f9882
注意：私有镜像仓库-harbor 使用http访问必须在每个node做信任,并重启docker
# cat /etc/docker/daemon.json
{
"registry-mirrors": ["http://bc437cce.m.daocloud.io"],
"insecure-registries":["http://10.40.6.165"]
}
4. java代码拉取
用一个node节点clone java代码：
# yum install git -y
# git clone https://github.com/lizhenliang/tomcat-java-demo.git
# cd tomcat-java-demo/
# ll
total 24
drwxr-xr-x 2 root root 34 Jun 15 11:40 db
-rw-r--r-- 1 root root 148 Jun 15 11:40 Dockerfile
-rw-r--r-- 1 root root 11357 Jun 15 11:40 LICENSE
-rw-r--r-- 1 root root 1930 Jun 15 11:40 pom.xml
-rw-r--r-- 1 root root 89 Jun 15 11:40 README.md
drwxr-xr-x 3 root root 18 Jun 15 11:40 src
要用容器化部署自己的项目，一般在你项目代码创建一个Dockerfile文件，使用Dockerfile构建项目镜像。
# cat Dockerfile
FROM lizhenliang/tomcat
LABEL maintainer www.ctnrs.com
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
5. maven和java环境准备
java是动态语言，需要构建war包或jar包，这构建需要maven和java环境
# tar xvf apache-maven-3.6.1-bin.tar.gz
# tar xvf jdk-8u181-linux-x64.tar.gz
# mv apache-maven-3.6.1 /usr/local/services/
# mv jdk1.8.0_181 /usr/local/services/
# cat /etc/profile.d/java.sh
JAVA_HOME=/usr/local/services/jdk1.8.0_181
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH
# source /etc/profile
# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
# ln -sv /usr/local/services/apache-maven-3.6.1/bin/mvn /usr/bin/mvn
6. 项目代码编译
编译完成后会生产一个target 目录，此目录会有一个war包。
# cd tomcat-java-demo/
# mvn clean package
# ll target/
total 17840
drwxr-xr-x 5 root root 95 Jun 15 14:45 classes
drwxr-xr-x 3 root root 25 Jun 15 14:45 generated-sources
drwxr-xr-x 4 root root 37 Jun 15 14:46 ly-simple-tomcat-0.0.1-SNAPSHOT
-rw-r--r-- 1 root root 18265402 Jun 15 14:46 ly-simple-tomcat-0.0.1-SNAPSHOT.war
drwxr-xr-x 2 root root 28 Jun 15 14:46 maven-archiver
drwxr-xr-x 3 root root 35 Jun 15 14:45 maven-status
7. 构建项目镜像并推送至镜像仓库
通过Dockefile构建项目镜像
# cat Dockerfile
FROM lizhenliang/tomcat
LABEL maintainer www.ctnrs.com
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
# docker build -t 10.40.6.165/project/java-demo:v1 . ##docker build -t 镜像仓库推送地址
# docker login 10.40.6.165
# docker push 10.40.6.165/project/java-demo:v1 ##推到到镜像仓库
8. 编写部署项目yaml文件
yaml部署资源流程:
tomcat：
deployment
service
ingress
mysql：
statefuleset
healess service
pv,pvc(storageclass PV 自动供给 )
yaml文件：
namespace.yaml
deployment.yaml
ingress.yaml
service.yaml
mysql.yaml
registry-pull-secret.yaml
这些yaml文件可以统一写到一个yaml, 使用'---'分隔即可。
9. 创建项目资源
(1). 创建命名空间
# cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: test
#kubectl create -f namespace.yaml
# kubectl get ns -n test
(2). 创建镜像仓库Secret用户验证
拉取镜像需要用户验证，先创建kubernetes 登录镜像仓库的Secret用户验证，注意拉取镜像的yaml配置文件中Secret name字段，要与此创建的一致
# kubectl create secret --help
# kubectl create secret docker-registry --help
....
Usage:
kubectl create secret docker-registry NAME --docker-username=user --docker-password=password --docker-email=email
指定Secret创建到 test命名空间，因为我们的项目创建在test命名空间
# kubectl create secret docker-registry registry-pull-secret --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@test.com --docker-server=10.40.6.165 -n test
# kubectl get secret -n test
NAME TYPE DATA AGE
default-token-vqllw kubernetes.io/service-account-token 3 30m
registry-pull-secret kubernetes.io/dockerconfigjson 1 69s
(3). 创建项目pod
# cat deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: tomcat-java-demo
namespace: test
spec:
replicas: 3
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: tomcat
image: 10.40.6.165/project/java-demo:v1
imagePullPolicy: Always
ports:
- containerPort: 8080
name: web
protocol: TCP
resources:
requests:
cpu: 0.5
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
# kubectl create -f deployment.yaml
# kubectl get pod -n test
(4). 创建 service
# cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: tomcat-java-demo
namespace: test
spec:
selector:
project: www
app: java-demo
ports:
- name: web
port: 80
targetPort: 8080
# kubectl create -f service.yaml
# kubectl get svc -n test
(5). 创建ingress规则
# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tomcat-java-demo
namespace: test
spec:
rules:
- host: java.ctnrs.com
http:
paths:
- path: /
backend:
serviceName: tomcat-java-demo
servicePort: 80
# kubectl create -f ingress.yaml
# kubectl get ingress -n test
# kubectl get pods,svc,ing -n test
10. 测试访问
# kubectl get pods -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-ingress-controller-74cbc544f6-9v5qv 1/1 Running 0 2d3h 172.17.59.5 10.40.6.213 <none>
nginx-ingress-controller-74cbc544f6-qklwh 1/1 Running 0 2d3h 172.17.31.5 10.40.6.210 <none>
# kubectl get ing -n test
NAME HOSTS ADDRESS PORTS AGE
tomcat-java-demo java.ctnrs.com 80 10m
绑定域名到ingress-nginx的节点IP上访问：
10.40.6.210 java.ctnrs.com
http://java.ctnrs.com
11. 创建数据库service、pod、PV和PVC
# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: java-demo-mysql
namespace: test
labels:
project: java-demo
app: mysql
spec:
ports:
- port: 3306
name: java-demo-mysql
clusterIP: None
selector:
project: java-demo
app: mysql
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: db
namespace: test
spec:
selector:
matchLabels:
project: java-demo
app: mysql
serviceName: "java-demo-mysql"
template:
metadata:
labels:
project: java-demo
app: mysql
spec:
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
ports:
volumeMounts:
- mountPath: "/var/lib/mysql"
name: mysql-data
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: ["ReadWriteMany"]
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 2Gi
# kubectl create -f mysql.yaml
# kubectl get svc,pods -n test
# kubectl get pv,pvc -n test
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-03054e78-8f53-11e9-8287-005056b614b3 2Gi RWX Delete Bound test/mysql-data-db-0 managed-nfs-storage 14m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-data-db-0 Bound pvc-03054e78-8f53-11e9-8287-005056b614b3 2Gi RWX managed-nfs-storage 14m
12. 数据库配置
将tables_ly_tomcat.sql sql文件导入进去，并修改数据库连接
# scp db/tables_ly_tomcat.sql 10.40.6.201:~ ##将文件拷贝到master节点
# kubectl cp /root/tables_ly_tomcat.sql db-0:/ -n test ## 将sql文件cp到db-0容器中
# kubectl exec -it db-0 -n test bash
root@db-0:/# mysql -uroot -p123456
mysql> source /tables_ly_tomcat.sql
Query OK, 1 row affected (0.01 sec)
Database changed
Query OK, 0 rows affected (0.02 sec)
Query OK, 0 rows affected (0.00 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.01 sec)
mysql> use test
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user |
+----------------+
1 row in set (0.00 sec)
mysql> desc user;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(100) | NO | | NULL | |
| age | int(3) | NO | | NULL | |
| sex | char(1) | YES | | NULL | |
+-------+--------------+------+-----+---------+----------------+
4 rows in set (0.02 sec)
登录项目容器测试mysql容器域名：
域名格式：pod_name.service_name.namespace
# kubectl exec -it tomcat-java-demo-5f87cd895d-4zszj -n test bash
[root@tomcat-java-demo-5f87cd895d-4zszj tomcat]# ping -c 1 db-0.java-demo-mysql.test
PING db-0.java-demo-mysql.test.svc.cluster.local (172.17.59.6) 56(84) bytes of data.
# kubectl get pod -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
db-0 1/1 Running 0 20m 172.17.59.6 10.40.6.213 <none>
修改项目连接数据库配置文件 src/main/resources/application.yml
# cat src/main/resources/application.yml
server:
port: 8080
spring:
datasource:
url: jdbc:mysql://db-0.java-demo-mysql.test:3306/test?characterEncoding=utf-8
username: root
password: 12345
driver-class-name: com.mysql.jdbc.Driver
freemarker:
allow-request-override: false
cache: true
check-template-location: true
charset: UTF-8
content-type: text/html; charset=utf-8
expose-request-attributes: false
expose-session-attributes: false
expose-spring-macro-helpers: false
suffix: .ftl
template-loader-path:
- classpath:/templates/
更新代码后更新镜像，并推送到镜像仓库
# docker build -t 10.40.6.165/project/java-demo:v2 .
# docker push 10.40.6.165/project/java-demo:v2
修改deployment.yaml 项目镜像版本为：10.40.6.165/project/java-demo:v2
滚动更新项目：kubectl apply -f deployment.yaml
然后访问站点添加美女.
添加美女.png
今晚翻盘哪个美女.png
三. 部署PHP
部署跟java基本一致，只是代码不需要编译构建
clone代码 -> 制作代码镜像 -> 代码镜像推送至镜像仓库->编写yaml配置文件 ->创建资源对象
1. clone代码
git clone https://github.com/lizhenliang/php-demo.git
2. 构建项目镜像并推送至镜像仓库
这里的数据库就用上边java项目创建的数据库，修改php代码连接数据库的配置文件：
php代码连接数据库的配置文件
# cat php-demo/wp-config.php
....
/** MySQL主机 */
define('DB_HOST', 'db-0.java-demo-mysql.test');
....
代码目录php-demo中的Dockfile文件：
# cat Dockerfile
FROM lizhenliang/nginx-php:latest
MAINTAINER www.ctnrs.com
ADD . /usr/local/nginx/html
构建项目镜像,并推送到镜像仓库
# docker build -t 10.40.6.165/project/php-demo:1.0 .
# docker push 10.40.6.165/project/php-demo:1.0
3. 编写yaml配置文件并创建资源对象
命名空间test 在java 项目已经创建，这里不再创建
(1). 创建pod
# cat deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: php-demo
namespace: test
spec:
replicas: 2
selector:
matchLabels:
project: www
app: php-demo
template:
metadata:
labels:
project: www
app: php-demo
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: nginx
image: 10.40.6.165/project/php-demo:1.0
imagePullPolicy: Always
ports:
- containerPort: 80
name: web
protocol: TCP
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
livenessProbe:
httpGet:
path: /index.php
port: 80
initialDelaySeconds: 6
timeoutSeconds: 20
# kubectl create -f deployment.yaml
# kubectl get pod -n test
(2). 创建service
# cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: php-demo
namespace: test
spec:
selector:
project: www
app: php-demo
ports:
- name: web
port: 80
targetPort: 80
# kubectl create -f service.yaml
# kubectl get svc -n test
(3). 创建ingress规则
# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: php-demo
namespace: test
spec:
rules:
- host: php.ctnrs.com
http:
paths:
- path: /
backend:
serviceName: php-demo
servicePort: 80
# kubectl create -f ingress.yaml
# kubectl get ing -n test
(4). 配置数据库
php项目与java项目同用一个数据库test, test库中有同用一个user表，这里将之前java项目到user表删除掉
# kubectl exec -it db-0 bash -n test
root@db-0:/# mysql -uroot -p123456
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql> use test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user |
+----------------+
1 row in set (0.00 sec)
mysql> drop table user;
Query OK, 0 rows affected (0.01 sec)
(5). 验证
绑定域名，浏览器访问
# kubectl get ing -n test
NAME HOSTS ADDRESS PORTS AGE
php-demo php.ctnrs.com 80 14m
tomcat-java-demo java.ctnrs.com 80 8h
本地绑定hosts:
10.40.6.210 php.ctnrs.com
访问: http://php.ctnrs.com
