本文整理汇总了Java中org.apache.crunch.types.avro.Avros.generics方法的典型用法代码示例。如果您正苦于以下问题：Java Avros.generics方法的具体用法？Java Avros.generics怎么用？Java Avros.generics使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类org.apache.crunch.types.avro.Avros的用法示例。
在下文中一共展示了Avros.generics方法的2个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。
示例1: asSource
import org.apache.crunch.types.avro.Avros; //导入方法依赖的package包/类
/**
* Expose the given {@link Dataset} as a Crunch {@link ReadableSource}.
*
* Only the FileSystem {@code Dataset} implementation is supported and the
* file format must be {@code Formats.PARQUET} or {@code Formats.AVRO}.
*
* @param dataset the dataset to read from
* @param type the Java type of the entities in the dataset
* @param <E> the type of entity produced by the source
* @return the {@link ReadableSource}, or <code>null</code> if the dataset is not
* filesystem-based.
*/
@SuppressWarnings("unchecked")
public static <E> ReadableSource<E> asSource(Dataset<E> dataset, Class<E> type) {
Path directory = Accessor.getDefault().getDirectory(dataset);
if (directory != null) {
List<Path> paths = Lists.newArrayList(
Accessor.getDefault().getPathIterator(dataset));
AvroType<E> avroType;
if (type.isAssignableFrom(GenericData.Record.class)) {
avroType = (AvroType<E>) Avros.generics(dataset.getDescriptor().getSchema());
} else {
avroType = Avros.records(type);
}
final Format format = dataset.getDescriptor().getFormat();
if (Formats.PARQUET.equals(format)) {
return new AvroParquetFileSource<E>(paths, avroType);
} else if (Formats.AVRO.equals(format)) {
return new AvroFileSource<E>(paths, avroType);
} else {
throw new UnsupportedOperationException(
"Not a supported format: " + format);
}
}
return null;
}
示例2: run
import org.apache.crunch.types.avro.Avros; //导入方法依赖的package包/类
@Override
public int run(String... args) throws Exception {
if (args.length != 3) {
System.err.println("Usage: " + CombinedLogFormatConverter.class.getSimpleName() +
" <input> <dataset_uri> <dataset name>");
return 1;
}
String input = args[0];
String datasetUri = args[1];
String datasetName = args[2];
Schema schema = new Schema.Parser().parse(
Resources.getResource("combined_log_format.avsc").openStream());
// Create the dataset
DatasetRepository repo = DatasetRepositories.open(datasetUri);
DatasetDescriptor datasetDescriptor = new DatasetDescriptor.Builder()
.schema(schema).build();
Dataset<Object> dataset = repo.create(datasetName, datasetDescriptor);
// Run the job
final String schemaString = schema.toString();
AvroType<GenericData.Record> outputType = Avros.generics(schema);
PCollection<String> lines = readTextFile(input);
PCollection<GenericData.Record> records = lines.parallelDo(
new ConvertFn(schemaString), outputType);
getPipeline().write(records, CrunchDatasets.asTarget(dataset),
Target.WriteMode.APPEND);
run();
return 0;
}