Distributed Cache在mapreduce中读取小文

title: Distributed Cache在mapreduce中读取小文件
date: 2016/11/26 22:48:13
tags: MapReduce
categories: 大数据

Distributed Cache 在 MapReduce 任务中应用很广， 它可以大大提高一些被频繁读取文件的访问速度。被添加到 Distributed Cache 的文件会被拷贝到 Mapper 和 Reducer 的运行目录中。

**在job添加如下方法 **

remoteReGamePath为hdfs文件路径字符串
job.addCacheFile(new Path(remoteReGamePath).toUri());

以下例子为在map中读取此文件并存入集合

private Set<String> recommendGame = new HashSet<String>();
/**
         * 读取推荐游戏文件
         * 
         * @param uri
         */
        private void readReGame(URI uri) {
            try {
                Path patternsPath = new Path(uri.getPath());
                String patternsFileName = patternsPath.getName().toString();
                BufferedReader reader = new BufferedReader(new FileReader(
                        patternsFileName));
                String line;
                while ((line = reader.readLine()) != null) {
                    // TODO: your code here
                    //
                    recommendGame.add(line.split(",")[0]);
                }
                reader.close();

            } catch (FileNotFoundException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

        }

        @Override
        protected void setup(
                Mapper<LongWritable, Text, Text, Text>.Context context)
                throws IOException, InterruptedException {
            // TODO Auto-generated method stub
            super.setup(context);
            //获取cache  uri
            URI[] uri = context.getCacheFiles();

            readReGame(uri[0]);

        }