当前位置: 首页>>代码示例>>Java>>正文


Java TestMRFieldSelection类代码示例

本文整理汇总了Java中org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection的典型用法代码示例。如果您正苦于以下问题:Java TestMRFieldSelection类的具体用法?Java TestMRFieldSelection怎么用?Java TestMRFieldSelection使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。


TestMRFieldSelection类属于org.apache.hadoop.mapreduce.lib.fieldsel包,在下文中一共展示了TestMRFieldSelection类的2个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: launch

import org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection; //导入依赖的package包/类
public static void launch() throws Exception {
  JobConf conf = new JobConf(TestFieldSelection.class);
  FileSystem fs = FileSystem.get(conf);
  int numOfInputLines = 10;

  Path OUTPUT_DIR = new Path("build/test/output_for_field_selection_test");
  Path INPUT_DIR = new Path("build/test/input_for_field_selection_test");
  String inputFile = "input.txt";
  fs.delete(INPUT_DIR, true);
  fs.mkdirs(INPUT_DIR);
  fs.delete(OUTPUT_DIR, true);

  StringBuffer inputData = new StringBuffer();
  StringBuffer expectedOutput = new StringBuffer();

  TestMRFieldSelection.constructInputOutputData(inputData,
    expectedOutput, numOfInputLines);
  FSDataOutputStream fileOut = fs.create(new Path(INPUT_DIR, inputFile));
  fileOut.write(inputData.toString().getBytes("utf-8"));
  fileOut.close();

  System.out.println("inputData:");
  System.out.println(inputData.toString());
  JobConf job = new JobConf(conf, TestFieldSelection.class);
  FileInputFormat.setInputPaths(job, INPUT_DIR);
  job.setInputFormat(TextInputFormat.class);
  job.setMapperClass(FieldSelectionMapReduce.class);
  job.setReducerClass(FieldSelectionMapReduce.class);

  FileOutputFormat.setOutputPath(job, OUTPUT_DIR);
  job.setOutputKeyClass(Text.class);
  job.setOutputValueClass(Text.class);
  job.setOutputFormat(TextOutputFormat.class);
  job.setNumReduceTasks(1);

  job.set(FieldSelectionHelper.DATA_FIELD_SEPERATOR, "-");
  job.set(FieldSelectionHelper.MAP_OUTPUT_KEY_VALUE_SPEC, "6,5,1-3:0-");
  job.set(FieldSelectionHelper.REDUCE_OUTPUT_KEY_VALUE_SPEC, ":4,3,2,1,0,0-");

  JobClient.runJob(job);

  //
  // Finally, we compare the reconstructed answer key with the
  // original one.  Remember, we need to ignore zero-count items
  // in the original key.
  //
  boolean success = true;
  Path outPath = new Path(OUTPUT_DIR, "part-00000");
  String outdata = MapReduceTestUtil.readOutput(outPath,job);

  assertEquals(expectedOutput.toString(),outdata);
  fs.delete(OUTPUT_DIR, true);
  fs.delete(INPUT_DIR, true);
}
 
开发者ID:naver,项目名称:hadoop,代码行数:55,代码来源:TestFieldSelection.java

示例2: launch

import org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection; //导入依赖的package包/类
public static void launch() throws Exception {
  JobConf conf = new JobConf(TestFieldSelection.class);
  FileSystem fs = FileSystem.get(conf);
  int numOfInputLines = 10;

  String baseDir = System.getProperty("test.build.data", "build/test/data");
  Path OUTPUT_DIR = new Path(baseDir + "/output_for_field_selection_test");
  Path INPUT_DIR = new Path(baseDir + "/input_for_field_selection_test");
  String inputFile = "input.txt";
  fs.delete(INPUT_DIR, true);
  fs.mkdirs(INPUT_DIR);
  fs.delete(OUTPUT_DIR, true);

  StringBuffer inputData = new StringBuffer();
  StringBuffer expectedOutput = new StringBuffer();

  TestMRFieldSelection.constructInputOutputData(inputData,
    expectedOutput, numOfInputLines);
  FSDataOutputStream fileOut = fs.create(new Path(INPUT_DIR, inputFile));
  fileOut.write(inputData.toString().getBytes("utf-8"));
  fileOut.close();

  System.out.println("inputData:");
  System.out.println(inputData.toString());
  JobConf job = new JobConf(conf, TestFieldSelection.class);
  FileInputFormat.setInputPaths(job, INPUT_DIR);
  job.setInputFormat(TextInputFormat.class);
  job.setMapperClass(FieldSelectionMapReduce.class);
  job.setReducerClass(FieldSelectionMapReduce.class);

  FileOutputFormat.setOutputPath(job, OUTPUT_DIR);
  job.setOutputKeyClass(Text.class);
  job.setOutputValueClass(Text.class);
  job.setOutputFormat(TextOutputFormat.class);
  job.setNumReduceTasks(1);

  job.set(FieldSelectionHelper.DATA_FIELD_SEPERATOR, "-");
  job.set(FieldSelectionHelper.MAP_OUTPUT_KEY_VALUE_SPEC, "6,5,1-3:0-");
  job.set(FieldSelectionHelper.REDUCE_OUTPUT_KEY_VALUE_SPEC, ":4,3,2,1,0,0-");

  JobClient.runJob(job);

  //
  // Finally, we compare the reconstructed answer key with the
  // original one.  Remember, we need to ignore zero-count items
  // in the original key.
  //
  boolean success = true;
  Path outPath = new Path(OUTPUT_DIR, "part-00000");
  String outdata = MapReduceTestUtil.readOutput(outPath,job);

  assertEquals(expectedOutput.toString(),outdata);
  fs.delete(OUTPUT_DIR, true);
  fs.delete(INPUT_DIR, true);
}
 
开发者ID:hopshadoop,项目名称:hops,代码行数:56,代码来源:TestFieldSelection.java


注:本文中的org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection类示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。