运行一个mapReduce用例

执行一个计算圆周率的任务

cd /usr/local/hadoop/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-3.3.0.jar pi 2 10 #2表示线程数，10表示采样点数

问题

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your <HADOOP_HOME>/etc/hadoop/mapred-site.xml contains the below configuration:
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>

错误信息指出我没有配置以上三个变量，所以在mapred-site.xml文件中追加这三个配置。

<property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_COMMON_HOME}</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_COMMON_HOME}</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_COMMON_HOME}</value>
    </property>

重新运行命令，计算得出结果

root@ALBB:/usr/local/hadoop/share/hadoop/mapreduce# hadoop jar hadoop-mapreduce-examples-3.3.0.jar pi 2 10
Number of Maps  = 2
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Starting Job
2021-05-11 01:23:10,774 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2021-05-11 01:23:11,450 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1620665185590_0002
2021-05-11 01:23:11,660 INFO input.FileInputFormat: Total input files to process : 2
2021-05-11 01:23:11,753 INFO mapreduce.JobSubmitter: number of splits:2
2021-05-11 01:23:12,241 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1620665185590_0002
2021-05-11 01:23:12,241 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-05-11 01:23:12,618 INFO conf.Configuration: resource-types.xml not found
2021-05-11 01:23:12,619 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-05-11 01:23:12,726 INFO impl.YarnClientImpl: Submitted application application_1620665185590_0002
2021-05-11 01:23:12,779 INFO mapreduce.Job: The url to track the job: http://ALBB:8088/proxy/application_1620665185590_0002/
2021-05-11 01:23:12,780 INFO mapreduce.Job: Running job: job_1620665185590_0002
2021-05-11 01:23:25,132 INFO mapreduce.Job: Job job_1620665185590_0002 running in uber mode : false
2021-05-11 01:23:25,133 INFO mapreduce.Job:  map 0% reduce 0%
2021-05-11 01:23:36,275 INFO mapreduce.Job:  map 50% reduce 0%
2021-05-11 01:23:37,305 INFO mapreduce.Job:  map 100% reduce 0%
2021-05-11 01:23:43,376 INFO mapreduce.Job:  map 100% reduce 100%
2021-05-11 01:23:43,386 INFO mapreduce.Job: Job job_1620665185590_0002 completed successfully
2021-05-11 01:23:43,551 INFO mapreduce.Job: Counters: 54
    File System Counters
        FILE: Number of bytes read=50
        FILE: Number of bytes written=793758
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=526
        HDFS: Number of bytes written=215
        HDFS: Number of read operations=13
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=3
        HDFS: Number of bytes read erasure-coded=0
    Job Counters 
        Launched map tasks=2
        Launched reduce tasks=1
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=19023
        Total time spent by all reduces in occupied slots (ms)=4483
        Total time spent by all map tasks (ms)=19023
        Total time spent by all reduce tasks (ms)=4483
        Total vcore-milliseconds taken by all map tasks=19023
        Total vcore-milliseconds taken by all reduce tasks=4483
        Total megabyte-milliseconds taken by all map tasks=19479552
        Total megabyte-milliseconds taken by all reduce tasks=4590592
    Map-Reduce Framework
        Map input records=2
        Map output records=4
        Map output bytes=36
        Map output materialized bytes=56
        Input split bytes=290
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=56
        Reduce input records=4
        Reduce output records=0
        Spilled Records=8
        Shuffled Maps =2
        Failed Shuffles=0
        Merged Map outputs=2
        GC time elapsed (ms)=377
        CPU time spent (ms)=1360
        Physical memory (bytes) snapshot=599326720
        Virtual memory (bytes) snapshot=7957811200
        Total committed heap usage (bytes)=332320768
        Peak Map Physical memory (bytes)=233218048
        Peak Map Virtual memory (bytes)=2649092096
        Peak Reduce Physical memory (bytes)=133976064
        Peak Reduce Virtual memory (bytes)=2659897344
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=236
    File Output Format Counters 
        Bytes Written=97
Job Finished in 33.008 seconds
Estimated value of Pi is 3.80000000000000000000