本文整理汇总了Java中org.apache.spark.api.java.JavaSparkContext.hadoopConfiguration方法的典型用法代码示例。如果您正苦于以下问题:Java JavaSparkContext.hadoopConfiguration方法的具体用法?Java JavaSparkContext.hadoopConfiguration怎么用?Java JavaSparkContext.hadoopConfiguration使用的例子?那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类org.apache.spark.api.java.JavaSparkContext
的用法示例。
在下文中一共展示了JavaSparkContext.hadoopConfiguration方法的1个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的Java代码示例。
示例1: start
import org.apache.spark.api.java.JavaSparkContext; //导入方法依赖的package包/类
public synchronized void start() { // 加锁,单线程执行
String id = getID();
if (id != null) {
log.info("Starting Batch Layer {}", id);
}
streamingContext = buildStreamingContext();
JavaSparkContext sparkContext = streamingContext.sparkContext();//saprk初始化方法
Configuration hadoopConf = sparkContext.hadoopConfiguration();
//设置路径
Path checkpointPath = new Path(new Path(modelDirString), ".checkpoint");
log.info("Setting checkpoint dir to {}", checkpointPath);
sparkContext.setCheckpointDir(checkpointPath.toString());
//spark 读取kafka的topic
log.info("Creating message stream from topic");
JavaInputDStream<ConsumerRecord<K,M>> kafkaDStream = buildInputDStream(streamingContext);
JavaPairDStream<K,M> pairDStream =
kafkaDStream.mapToPair(mAndM -> new Tuple2<>(mAndM.key(), mAndM.value()));
Class<K> keyClass = getKeyClass();
Class<M> messageClass = getMessageClass();
//对每条spark里读取的kafka信息做处理
pairDStream.foreachRDD(
new BatchUpdateFunction<>(getConfig(),
keyClass,
messageClass,
keyWritableClass,
messageWritableClass,
dataDirString,
modelDirString,
loadUpdateInstance(),
streamingContext));
// "Inline" saveAsNewAPIHadoopFiles to be able to skip saving empty RDDs
// spark读取kafka数据,写入到hdfs上,每条数据进行处理
pairDStream.foreachRDD(new SaveToHDFSFunction<>(
dataDirString + "/oryx",
"data",
keyClass,
messageClass,
keyWritableClass,
messageWritableClass,
hadoopConf));
// Must use the raw Kafka stream to get offsets
kafkaDStream.foreachRDD(new UpdateOffsetsFn<>(getGroupID(), getInputTopicLockMaster()));
if (maxDataAgeHours != NO_MAX_AGE) {
pairDStream.foreachRDD(new DeleteOldDataFn<>(hadoopConf,
dataDirString,
Pattern.compile("-(\\d+)\\."),
maxDataAgeHours));
}
if (maxModelAgeHours != NO_MAX_AGE) {
pairDStream.foreachRDD(new DeleteOldDataFn<>(hadoopConf,
modelDirString,
Pattern.compile("(\\d+)"),
maxModelAgeHours));
}
log.info("Starting Spark Streaming");
streamingContext.start();
}