Centos7搭建azkaban3
准备工作
部署在208服务器,mysql用211服务器
1. 下载压缩包
mkdir -p /home/usersoft/azkaban
cd azkaban
curl -O https://github.com/azkaban/azkaban/archive/3.81.0.tar.gz
curl -O https://services.gradle.org/distributions/gradle-4.6-all.zip
解压azkaban-3.81.0.tar.gz
tar -zxvf azkaban-3.81.0.tar.gz
拷贝gradle到 azkaban 下的 gradle/wrapper 目录
mv gradle-4.6-all.zip azkaban-3.81.0/gradle/wrapper/
编辑gradle-wrapper.properties, 修改 distributionUrl=gradle-4.6-all.zip
2. 执行编译安装命令
cd /home/usrsoft/azkaban/azkaban-3.81.0
修改 build.gradle ,将几处 mavenCentral() 、 mavenLocal() 注释掉,改用阿里云镜像 http://maven.aliyun.com/nexus/content/groups/public/ :
repositories {
//mavenCentral()
maven {
//url 'https://plugins.gradle.org/m2/'
url 'http://maven.aliyun.com/nexus/content/groups/public/'
}
}
......
repositories {
//mavenCentral()
//mavenLocal()
maven {
url 'http://maven.aliyun.com/nexus/content/groups/public/'
}
}
然后执行安装命令
./gradlew installDist -x test
安装成功后,检查azkaban-web-server/build/install/azkaban-web-server和 azkaban-exec-server/build/install/azkaban-exec-server 下的目录
bin启动脚本目录conf配置文件目录(没有的话从solo-server的目录中拷贝过来)lib依赖jar包目录extlib附加jar包目录(没有的话手动创建)plugins插件安装目录 (没有的话从solo-server目录中拷贝过来)web资源文件logs日志文件(手动创建)
执行
mkdir /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/extlib
mkdir /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/logs
mkdir /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/extlib
mkdir /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/logs
mkdir /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/web
cp -r /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-solo-server/build/install/azkaban-solo-server/plugins /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/
3. 前往211mysql数据库建表
mysql -u bigdataAll -p
create database azkaban;
use azkaban;
create user 'azkaban' identified by 'azkaban';
grant all on azkaban.* to 'azkaban';
在211执行
source /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-db/build/sql/create-all-sql-0.1.0-SNAPSHOT.sql
4. 编辑azkaban-web-server中的 azkaban.properties、 azkaban-exec-server 中的 azkaban.properties
(1) azkaban-web-server
vim azkaban-web-server/build/install/azkaban-web-server/conf/azkaban.properties
修改下面的参数(修改为绝对路径)
web.resource.dir=/home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/web/
user.manager.xml.file=/home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/conf/azkaban-users.xml
executor.global.properties=/home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/conf/global.properties
时区改为Asia/Shanghai
(2) azkaban-exec-server
vim azkaban-exec-server/build/install/azkaban-exec-server/conf/azkaban.properties
修改下面的参数(修改为绝对路径)
executor.global.properties=/home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/conf/global.properties
azkaban.jobtype.plugin.dir=/home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/plugins/jobtypes
5. 配置keystore
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
密码为123456
将生成的keystone文件拷贝到web-server的安装目录下,和conf等目录同级
6. 先启动exec-server、然后启动 web-server , jps 检查进程
cd /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/bin
./start-exec.sh
cd /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/bin
./start-web.sh
7. 访问
vim azkaban-users.xml 增加管理员用户
访问端口在 azkaban-web-server/conf/azkaban.properties 配置文件中:
jetty.port=8443
8.问题解决
1.任务调起一直为preparing状态,修改web-server下azkaban.properties
把MinimumFreeMemory去掉
2. 任务执行时直接被kill
错误日志:
ERROR [ExecutorServlet] [Azkaban] executor became inactive before setting up the flow 1
azkaban.executor.ExecutorManagerException: executor became inactive before setting up the flow 1
需要手动去激活一下executor,方式如下:
执行curlhttp://192.168.16.208:12321/executor?action=activate
为了彻底解决,直接把这句话加入azkaban-exec-server的 start-exec.sh
9. 修改azkaban执行用户
启动进程用户即为azkaban执行用户:
chown -R hive:hive azkaban-3.81.0/
su hive
cd /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/bin
./start-exec.sh
cd /home/usrsoft/azkaban/azkaban-3.81.0/azkaban-web-server/build/install/azkaban-web-server/bin
./start-web.sh