本文整理匯總了Java中org.apache.spark.SparkConf.getInt方法的典型用法代碼示例。如果您正苦於以下問題:Java SparkConf.getInt方法的具體用法?Java SparkConf.getInt怎麽用?Java SparkConf.getInt使用的例子?那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類org.apache.spark.SparkConf
的用法示例。
在下文中一共展示了SparkConf.getInt方法的2個代碼示例,這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚,您的評價將有助於係統推薦出更棒的Java代碼示例。
示例1: main
import org.apache.spark.SparkConf; //導入方法依賴的package包/類
public static void main(String[] args) throws Exception {
if (args.length != 1) {
System.err.println("Usage: JavaSleep <seconds>");
System.exit(1);
}
SparkConf sparkConf = new SparkConf().setAppName("JavaSleep");
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
Integer parallel = sparkConf.getInt("spark.default.parallelism", ctx.defaultParallelism());
Integer seconds = Integer.parseInt(args[0]);
Integer[] init_val = new Integer[parallel];
Arrays.fill(init_val, seconds);
JavaRDD<Integer> workload = ctx.parallelize(Arrays.asList(init_val), parallel).map(new Function<Integer, Integer>() {
@Override
public Integer call(Integer s) throws InterruptedException {
Thread.sleep(s * 1000);
return 0;
}
});
List<Integer> output = workload.collect();
ctx.stop();
}
示例2: main
import org.apache.spark.SparkConf; //導入方法依賴的package包/類
public static void main(String[] args) throws Exception {
if (args.length < 2) {
System.err.println("Usage: JavaTeraSort <HDFS_INPUT> <HDFS_OUTPUT>");
System.exit(1);
}
SparkConf sparkConf = new SparkConf().setAppName("JavaTeraSort");
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
JavaRDD<String> lines = ctx.textFile(args[0], 1);
Integer parallel = sparkConf.getInt("spark.default.parallelism", ctx.defaultParallelism());
Integer reducer = Integer.parseInt(IOCommon.getProperty("hibench.default.shuffle.parallelism").get());
JavaPairRDD<String, String> words = lines.mapToPair(new PairFunction<String, String, String>() {
@Override
public Tuple2<String, String> call(String s) throws Exception {
return new Tuple2<String, String>(s.substring(0, 10), s.substring(10));
}
});
JavaPairRDD<String, String> sorted = words.sortByKey(true, reducer);
JavaRDD<String> result = sorted.map(new Function<Tuple2<String, String>, String>() {
@Override
public String call(Tuple2<String, String> e) throws Exception {
return e._1() + e._2();
}
});
result.saveAsTextFile(args[1]);
ctx.stop();
}