大数据玩转大数据程序员

Ranger-Yarn插件安装

2018-04-26  本文已影响205人  木木与呆呆

Ranger-Yarn插件ranger-0.6.0-yarn-plugin安装到Yarn的所有ResourceManager节点,
其他的NodeManager节点不需要安装。

登陆hdfs安装的用户,garrison/zdh1234(用户组hadoop),获取安装包解压安装
scp /home/backup/ranger/ranger-0.6.0-yarn-plugin.tar.gz .
tar –zxvf ranger-0.6.0-yarn-plugin.tar.gz
vi install.properties
修改的参数如下:

POLICY_MGR_URL=http://10.43.159.245:6080
SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
REPOSITORY_NAME=yarndev
CUSTOM_USER=garrison
CUSTOM_GROUP=hadoop

安装Ranger Yarn Plugin,注意: ./enable-yarn-plugin.sh script should be run as root.:
./enable-yarn-plugin.sh
创建完成后,需要重新启动yarn.

将zdh-245的包考到zdh-240的garrison里面
scp -r garrison@zdh-245:/home/garrison/ranger-0.6.0-yarn-plugin .
用root执行安装脚本,并且重新启动yarn.

Ranger-Admin里注册hdfs plugin的服务
YARN新建Service,修改如下

Service Name = yarnpdev
UserName = garrison
Password = zdh1234
YARN REST URL = http://10.43.159.240:8188

然后点击TestConnection,成功即可保存。

关闭all-queue策略,
新建root.default策略,给mysql用户提交队列的权限。
使用mysql用户执行mapreduce任务,给mysql访问hdfs相应目录的权限:

export JAVA_HOME=/usr/share/java/jdk1.7.0_80
/home/garrison/hadoop-2.7.1/bin/hadoop jar /home/garrison/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount -Dmapreduce.job.queuename=default hdfs://gagcluster/usr/wordcout.txt /usr/wordresult_002
/home/garrison/hadoop-2.7.1/bin/hadoop fs -text /usr/wordresult_002/part-r-00000

没有权限提交作业会报错:
User usersync cannot submit applications to queue root.default

Yarn队列权限支持通过capacity schedule queues实现
yarn-site.xml配置文件中加入配置项:

<property>
    <name>yarn.resourcemanager.scheduler.class</name>    
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

capacity-scheduler.xml配置如下,只允许garrison用户提交作业:

<property>
  <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
   <value>garrison</value>
   <description>
     The ACL of who can submit jobs to the root queue.
   </description>
 </property>
 <property>
  <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
  <value>garrison</value>
  <description>
    The ACL of who can administer jobs on the default queue.
  </description>
</property>
<property>
  <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
  <value>garrison</value>
  <description>
    The ACL of who can submit jobs to the default queue.
  </description>
</property>
<property>
  <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
  <value>garrison</value>
  <description>
    The ACL of who can administer jobs on the default queue.
  </description>
</property>
上一篇下一篇

猜你喜欢

热点阅读