第6章 MapReduce应用开发-管理配置

2019-05-28  本文已影响0人  主君_05c4
1、多环境配置切换

开发时,如果需要经常在本地运行与集群运行间切换,可以编写多个hadoop配置文件,每个文件包含每个环境的连接设置,运行时指定使用哪一个配置文件,可以把这些文件集中放置在安装目录树之外。假设有如下三个配置:

<!-- hadoop-local.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>file:///</value>
    </property>

    <property>
        <name>mapreduce.framework.name</name>
        <value>local</value>
    </property>
</configuration>

<!-- hadoop-localhost.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost</value>
    </property>

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

    <property>
        <name>yarn.resourcemanager.address</name>
        <value>localhost:8032</value>
    </property>

</configuration>

<!-- hadoop-cluster.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://namenode:9000</value>
    </property>

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

    <property>
        <name>yarn.resourcemanager.address</name>
        <value>resourcenode:8032</value>
    </property>
</configuration>

即可通过如下命令进行切换配置:
hadoop jar xxx.jar -conf hadoop-cluster.xml
hadoop jar xxx.jar -conf hadoop-local.xml...
本地调试,可以传入运行参数java -jar xxx.jar -conf hadoop-local.xml

public class MultiConfig extends Configured implements Tool {

    @Override
    public int run(String[] args) throws Exception {
        Configuration conf = getConf();
        String configOfFS = conf.get("fs.defaultFS");
        String configOfFramework = conf.get("mapreduce.framework.name");
        String configOfYarnRN = conf.get("yarn.resourcemanager.address");

        System.out.println("configOfFS = " + configOfFS);
        System.out.println("configOfFramework = " + configOfFramework);
        System.out.println("configOfYarnRN = " + configOfYarnRN);

        return 0;
    }

    public static void main(String[] args) throws Exception {
        System.out.println(Arrays.toString(args));
        ToolRunner.run(new MultiConfig(), args);
    }
}

运行结果:

[-conf, D:\hadoop-localhost.xml]
configOfFS = hdfs://localhost
configOfFramework = yarn
configOfYarnRN = localhost:8032

也可以通过配置环境HADOOP_CONF_DIR省略每次运行指定-conf参数。

2、打印配置

可通过调用configuration.forEach循环所有属性;
对于运行中守护进程,可以通过/conf页面查看配置;
有些属性无法在客户端设置,如 yarn.nodemanager.resource.memory-mb,只能在yarn-site.xml设置,可通过属性名获知该属性在哪配置,如上述以yarn.nodemanager开头,但这不是硬性的,有些需要尝试或者查看源代码。

上一篇 下一篇

猜你喜欢

热点阅读