本文整理汇总了Java中org.apache.spark.api.java.JavaRDD.getNumPartitions方法的典型用法代码示例。如果您正苦于以下问题：Java JavaRDD.getNumPartitions方法的具体用法？Java JavaRDD.getNumPartitions怎么用？Java JavaRDD.getNumPartitions使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类org.apache.spark.api.java.JavaRDD的用法示例。
在下文中一共展示了JavaRDD.getNumPartitions方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。
示例1: partition
import org.apache.spark.api.java.JavaRDD; //导入方法依赖的package包/类
/**
* Partition instances by the specified partitioning (e.g. by instance type)
*
* @param instances RDD of instances to partition
* @return partitioned RDD if requested, original RDD if no partitioning is specified
*/
public JavaRDD<Instance> partition(JavaRDD<Instance> instances) {
if (!config.isRepartitionByType()) {
return instances;
}
log.info("Getting counts by type hash");
Map<Integer, Long> typeCounts = getApproximateTypeHashCounts(instances);
int numPartitions = instances.getNumPartitions();
long totalInstances = instances.count();
long instancesPerPartition = totalInstances / numPartitions + 1;
JavaPairRDD<Integer, Instance> instanceWithPartitions = instances.mapToPair(instance -> {
int typeHash = getTypeHash(instance);
int splitIncrement = getSplitIncrement(instance.getId(), typeCounts.get(typeHash), instancesPerPartition);
return new Tuple2<>(typeHash + splitIncrement, instance);
});
log.info("Partitioning instances by type");
return instanceWithPartitions
.partitionBy(new HashPartitioner(numPartitions))
.values();
}