本文整理汇总了Java中org.apache.spark.api.java.JavaPairRDD.foreachPartition方法的典型用法代码示例。如果您正苦于以下问题：Java JavaPairRDD.foreachPartition方法的具体用法？Java JavaPairRDD.foreachPartition怎么用？Java JavaPairRDD.foreachPartition使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类org.apache.spark.api.java.JavaPairRDD的用法示例。
在下文中一共展示了JavaPairRDD.foreachPartition方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。
示例1: publishAdditionalModelData
import org.apache.spark.api.java.JavaPairRDD; //导入方法依赖的package包/类
@Override
public void publishAdditionalModelData(JavaSparkContext sparkContext,
PMML pmml,
JavaRDD<String> newData,
JavaRDD<String> pastData,
Path modelParentPath,
TopicProducer<String, String> modelUpdateTopic) {
// Send item updates first, before users. That way, user-based endpoints like /recommend
// may take longer to not return 404, but when they do, the result will be more complete.
log.info("Sending item / Y data as model updates");
String yPathString = AppPMMLUtils.getExtensionValue(pmml, "Y");
JavaPairRDD<String,float[]> productRDD = readFeaturesRDD(sparkContext, new Path(modelParentPath, yPathString));
String updateBroker = modelUpdateTopic.getUpdateBroker();
String topic = modelUpdateTopic.getTopic();
// For now, there is no use in sending known users for each item
productRDD.foreachPartition(new EnqueueFeatureVecsFn("Y", updateBroker, topic));
log.info("Sending user / X data as model updates");
String xPathString = AppPMMLUtils.getExtensionValue(pmml, "X");
JavaPairRDD<String,float[]> userRDD = readFeaturesRDD(sparkContext, new Path(modelParentPath, xPathString));
if (noKnownItems) {
userRDD.foreachPartition(new EnqueueFeatureVecsFn("X", updateBroker, topic));
} else {
log.info("Sending known item data with model updates");
JavaRDD<String[]> allData =
(pastData == null ? newData : newData.union(pastData)).map(MLFunctions.PARSE_FN);
JavaPairRDD<String,Collection<String>> knownItems = knownsRDD(allData, true);
userRDD.join(knownItems).foreachPartition(
new EnqueueFeatureVecsAndKnownItemsFn("X", updateBroker, topic));
}
}