Mac 配置 Dataturks，非 docker

2019-10-21 本文已影响0人谢小帅

配置全过程主要根据：

Dataturks - Github
Dockerfile: Dataturks/hope/docker/Dockerfile

1. 根据 Dockerfile 配置 mysql 数据库

mac 安装mysql详细教程
注意根据教程修改 mysql 登录密码

Dockerfile 中设置 mysql 过程，跳过 COPY，专注 RUN

COPY $LOCAL_DOCKER_DIR/mysqlInit.sql $BASE_DIR/mysqlInit.sql
COPY $LOCAL_DOCKER_DIR/init.sh $BASE_DIR/init.sh
RUN chmod +x ./init.sh && \
    ./init.sh

根据 init.sh 创建 hope 数据库，用户为 dataturks/12345

#!/bin/bash
/usr/bin/mysqld_safe  &
sleep 5

mysql -u root -e "CREATE DATABASE hope"
mysql -u root -e "CREATE USER dataturks@'localhost' IDENTIFIED BY '12345';"
mysql -u root -e "GRANT SELECT, INSERT, UPDATE, DELETE ON hope.* TO dataturks@localhost;FLUSH PRIVILEGES;"

# exe sql to build tables
mysql -u root hope < /home/dataturks/mysqlInit.sql

执行完毕后，本机 hope 数据库下将出现以下表：

mysql> use hope;
Database changed
mysql> show tables;
+----------------------+
| Tables_in_hope       |
+----------------------+
| configs              |
| d_api_keys           |
| d_hits               |
| d_hits_result        |
| d_org_users          |
| d_orgs               |
| d_project_invites    |
| d_project_users      |
| d_projects           |
| d_subscription_plans |
| d_subscriptions      |
| d_users              |
| keyValues            |
+----------------------+
13 rows in set (0.00 sec)

查看已存的 demo 数据集信息

mysql> select id, name, taskType from d_projects;
+----------------------------------+-----------------------------------+-------------------------------+
| id                               | name                              | taskType                      |
+----------------------------------+-----------------------------------+-------------------------------+
| 2c9fafb062282fd801622ddbac320024 | Demo POS Tagging Project          | POS_TAGGING                   |
| 2c9fafb062282fd801622de773dc0027 | Demo Text Classification Project  | TEXT_CLASSIFICATION           |
| 2c9fafb062ca71c00162cdd45c8e0008 | Demo Document Annotations         | DOCUMENT_ANNOTATION           |
| 2c9fafb063a1ebf30163a7e3002b0035 | Demo Image Classification Project | IMAGE_CLASSIFICATION          |
| 2c9fafb0656f461b0165843201f200d2 | Demo Image Bounding Box Project   | IMAGE_POLYGON_BOUNDING_BOX_V2 |
| 2c9fafb0656f461b0165843c734c00d3 | Demo Image Polygon Bounding Box   | IMAGE_POLYGON_BOUNDING_BOX_V2 |
| 2c9fafb0656f461b016589a1ccdf0125 | Demo Video Classification         | IMAGE_CLASSIFICATION          |
+----------------------------------+-----------------------------------+-------------------------------+
7 rows in set (0.01 sec)

2. 根据 Dataturks 安装前后端依赖库

React 前端

brew install node@8
brew link node@8
npm install  # node@8 根据 package.json 安装依赖包，放在 node_modules 目录

Java-mysql 后端

mvn package -DskipTests  # 跳过 test，打 dataturks-1.0-SNAPSHOT.jar 包

关于 mvn

mvn clean：移除所有上一次构建生成的文件
mvn pacakge: 将 pom.xml 中所有 dependencies 和项目一同打成 jar 包
mvn 执行命令时，之前的命令都会顺序执行，除非加 skip 参数

mvn 生命周期

3. 本机 onprem 模式运行，修改一些 js 配置文件

3.1 前端

① 修改 DataTurks/bazaar/src/config.js，适应 onprem 模式

module.exports = Object.assign({
  host: process.env.HOST || 'localhost',
  port: process.env.PORT,
  apiHost: process.env.APIHOST || 'localhost',
  apiPort: process.env.APIPORT,
  // apiURL: process.env.BASE_URL || 'https://dataturks.com/dataturks/',
  apiURL: process.env.BASE_URL || 'http://localhost:9090/dataturks/',
  // servingEnv: process.env.NODE_DEST || 'online',
  servingEnv: process.env.NODE_DEST || 'onpremise',

npm run dev 会指定一些参数，定义在 pacakge.json

host: 未指定，使用 localhost
port: 3000，"start-dev", "start-dev-api" 指定
apiHost: 未指定，使用 localhost
apiPort: 3030，"start-dev-api" 指定
apiURL: 默认给定的是 online 模式的，修改为本地
servingEnv: 虽然 "watch-client" 有给出，但没什么用，修改

② 修改 DataTurks/bazaar/src/helpers/dthelper.js，'online' -> 'onpremise'

// export，希望外界可以读取此变量
export const BASE_URL = (config.servingEnv === 'onpremise' ? config.apiURL : window.location.protocol + "//" + window.location.hostname + '/dataturks/');
console.log('servingEnv:', config.servingEnv);
console.log('BASE_URL:', BASE_URL); // 可在此输出 BASE_URL

③ npm run dev，等到 webpack built ... in xxxxx ms 就成功了，注意 💻 下面的两行与 ② 添加的两行 console.log 对应

[0] ==> 🚧  Webpack development server listening on port 3001
[1] servver bin  undefined
[2] Mon, 21 Oct 2019 04:33:13 GMT body-parser deprecated undefined extended: provide extended option at api/api.js:73:30
[2] ----
[2] ==> 🌎  API is running on port 3030
[2] ==> 💻  Send requests to http://localhost:3030
[1] servingEnv: onpremise
[1] BASE_URL: http://localhost:9090/dataturks/
[1] Rangy is not supported in this environment. Reason: Rangy can only run in a browser
[1] target url is  http://localhost:3030
[1] ----
[1] ==> ✅  AI + X 数据智能处理引擎 is running, talking to API server on 3030.
[1] ==> 💻  Open http://localhost:3000 in a browser to view the app.
···
···
[0] webpack built cccc3bdd9671f6610ffc in 54473ms

3.2 后台

① 命令行监控 logs 输出
注释 DataTurks/hope/onprem.yml 部分配置
方便 development 模式，production 模式保留 logs 更好

32-36 行，./logs/root.log
49-56 行，./logs/access.log

② Intellij IEDA 配置后台运行，logs 也会直接输出
注意 Main class 的 java 类是程序运行起点，有 main 函数

参考 dropwizard 框架，执行逻辑：https://my.oschina.net/wangbaofeng/blog/669139
参考 IDEA 配置：https://blog.csdn.net/lisheng19870305/article/details/94627317

4. 生成 docker 镜像

① 复制 .dockerignore 到 Dataturks 根目录

# Ignore everything
**

# Allow files and directories
# 如果 ignore 重复，以后面匹配为准，以下路径开头保留
!Dockerfile
!.dockerignore
!./bazaar/**
!./hope/docker/**
!./hope/target/dataturks-1.0-SNAPSHOT.jar
!./hope/onprem.yml

# Ignore unnecessary files inside allowed directories
# This should go after the allowed directories
# 排除所有缓存和日志
**/*~
**/*.log
**/.DS_Store
**/Thumbs.db
**/.idea
./bazaar/node_modules/**

② 执行 docker build

docker build -t vipa/vipaturks:1.0 -f hope/docker/Dockerfile .

生成日志

Sending build context to Docker daemon  107.9MB
Step 1/54 : FROM openjdk:8-jdk
 ---> 57c2c2d2643d
Step 2/54 : ENV DEBIAN_FRONTEND noninteractive
 ---> Running in 5f465ce7165c
Removing intermediate container 5f465ce7165c
···
···
Step 52/52 : CMD ./startup.sh > ./startup_log.log
 ---> Running in a8600e16b062
Removing intermediate container a8600e16b062
 ---> 4f75fba25e14
Successfully built 4f75fba25e14
Successfully tagged vipa/vipaturks:1.0

③ 测试打包好的 vipa/vipaturks:1.0

参考：Docker run 命令参数及使用

$ docker run --name=vipa_turks -d \
-p 80:80 \
-v /nfs/xs/retail/uploads:/home/dataturks/bazaar/uploads \
vipa/vipaturks:1.0

--name=vipa_turks：指定 container 名字，注意放置位置
-d：后台运行，Run container in background and print container ID
-p 80:80：本机 80 端口映射到 container 80，注意 mac apache 配置要先关闭监听 80，不然无法使用本机 80，也可使用其他端口
-v Source:Target：指定主机和容器路径映射，保护上传到容器的数据（默认情况下 Dataturks 上传数据到 uploads 目录，通过映射将数据保存到本地）可设置多个路径映射，根据 container_id 配置 config.v2.json, hostconfig.json
vipa/vipaturks:1.0：镜像名称 <用户名>/<软件名><TAG>

可查看本机运行镜像

$ docker ps -a    
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                                         NAMES
34d3b511c8d3        vipa/vipaturks:1.0   "/bin/sh -c './start…"   19 seconds ago      Up 17 seconds       3000-3001/tcp, 0.0.0.0:80->80/tcp, 9090/tcp   vipa_turks

命令行访问 localhost 就可以用自己打包的镜像玩耍了 😛

④ push 到 docker hub

# 字段 2 定义 私有仓库镜像名称
$ docker tag vipa/vipaturks:1.0 shuaix/vipaturks:1.0
$ docker push shuaix/vipaturks:1.0