第6章 MapReduce应用开发-管理配置
2019-05-28 本文已影响0人
主君_05c4
1、多环境配置切换
开发时,如果需要经常在本地运行与集群运行间切换,可以编写多个hadoop配置文件,每个文件包含每个环境的连接设置,运行时指定使用哪一个配置文件,可以把这些文件集中放置在安装目录树之外。假设有如下三个配置:
<!-- hadoop-local.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>file:///</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>local</value>
</property>
</configuration>
<!-- hadoop-localhost.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>localhost:8032</value>
</property>
</configuration>
<!-- hadoop-cluster.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://namenode:9000</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>resourcenode:8032</value>
</property>
</configuration>
即可通过如下命令进行切换配置:
hadoop jar xxx.jar -conf hadoop-cluster.xml
hadoop jar xxx.jar -conf hadoop-local.xml
...
本地调试,可以传入运行参数java -jar xxx.jar -conf hadoop-local.xml
public class MultiConfig extends Configured implements Tool {
@Override
public int run(String[] args) throws Exception {
Configuration conf = getConf();
String configOfFS = conf.get("fs.defaultFS");
String configOfFramework = conf.get("mapreduce.framework.name");
String configOfYarnRN = conf.get("yarn.resourcemanager.address");
System.out.println("configOfFS = " + configOfFS);
System.out.println("configOfFramework = " + configOfFramework);
System.out.println("configOfYarnRN = " + configOfYarnRN);
return 0;
}
public static void main(String[] args) throws Exception {
System.out.println(Arrays.toString(args));
ToolRunner.run(new MultiConfig(), args);
}
}
运行结果:
[-conf, D:\hadoop-localhost.xml]
configOfFS = hdfs://localhost
configOfFramework = yarn
configOfYarnRN = localhost:8032
也可以通过配置环境HADOOP_CONF_DIR省略每次运行指定-conf参数。
2、打印配置
可通过调用configuration.forEach
循环所有属性;
对于运行中守护进程,可以通过/conf页面查看配置;
有些属性无法在客户端设置,如 yarn.nodemanager.resource.memory-mb
,只能在yarn-site.xml设置,可通过属性名获知该属性在哪配置,如上述以yarn.nodemanager开头,但这不是硬性的,有些需要尝试或者查看源代码。