当前位置: 首页>>代码示例>>Java>>正文

Java Exec类代码示例

本文整理汇总了Java中org.embulk.spi.Exec类的典型用法代码示例。如果您正苦于以下问题：Java Exec类的具体用法？Java Exec怎么用？Java Exec使用的例子？那么, 这里精选的类代码示例或许可以为您提供帮助。


Exec类属于org.embulk.spi包，在下文中一共展示了Exec类的15个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: run

import org.embulk.spi.Exec; //导入依赖的package包/类
@Override
public TaskReport run(TaskSource taskSource,
        Schema schema, int taskIndex,
        PageOutput output)
{
    PluginTask task = taskSource.loadTask(PluginTask.class);
    Integer rows = task.getRows();
    final HashMap<Column, Map<String, Integer>> columnOptions = getColumnOptions(task);
    try (PageBuilder pagebuilder =
            new PageBuilder(Exec.getBufferAllocator(), schema, output)) {
        IntStream.rangeClosed(
                taskIndex * rows + 1,
                taskIndex * rows + rows
        ).boxed().forEach(rowNumber -> {
            RandomjColumnVisitor visitor = new RandomjColumnVisitor(pagebuilder, task, rowNumber, columnOptions);
            schema.visitColumns(visitor);
            pagebuilder.addRecord();
        });
        pagebuilder.finish();
    }

    TaskReport taskReport = Exec.newTaskReport();
    taskReport.set("columns", schema.size());
    taskReport.set("rows", rows);
    return taskReport;
}
开发者ID:yuokada，项目名称:embulk-input-randomj，代码行数:27，代码来源:RandomjInputPlugin.java

示例2: downloadBulkExtract

import org.embulk.spi.Exec; //导入依赖的package包/类
private File downloadBulkExtract(Function<BulkExtractRangeHeader, InputStream> getBulkExtractfunction)
{
    final File tempFile = Exec.getTempFileSpace().createTempFile(DEFAULT_FILE_FORMAT);
    long startByte = 0;
    int resumeTime = 0;
    while (resumeTime < MAX_RESUME_TIME) {
        BulkExtractRangeHeader bulkExtractRangeHeader = new BulkExtractRangeHeader(startByte);
        InputStream bulkExtractResult = getBulkExtractfunction.apply(bulkExtractRangeHeader);
        try {
            saveExtractedFile(bulkExtractResult, tempFile);
            return tempFile;
        }
        catch (DownloadBulkExtractException e) {
            startByte = startByte + e.getByteWritten();
            LOGGER.warn("will resume activity bulk extract at byte [{}]", startByte);
        }
        resumeTime = resumeTime + 1;
    }
    //Too many resume we still can't get the file
    throw new DataException("Can't down load bulk extract");
}
开发者ID:treasure-data，项目名称:embulk-input-marketo，代码行数:22，代码来源:MarketoServiceImpl.java

示例3: Icu4jPageOutput

import org.embulk.spi.Exec; //导入依赖的package包/类
public Icu4jPageOutput(TaskSource taskSource, Schema inputSchema, Schema outputSchema, PageOutput output) {
    this.task = taskSource.loadTask(PluginTask.class);
    this.keyNameColumns = Lists.newArrayList();
    this.transliteratorsList = Lists.newArrayList();
    this.inputSchema = inputSchema;
    this.outputSchema = outputSchema;

    for (String keyName : task.getKeyNames()) {
        keyNameColumns.add(inputSchema.lookupColumn(keyName));
    }
    for (Map<String, String> setting : task.getSettings()) {
        List<Transliterator> tokenizers = Lists.newArrayList();
        for (String convertType : setting.get("transliterators").split(",")) {
            Transliterator transliterator = Transliterator.getInstance(convertType);
            tokenizers.add(transliterator);
        }
        transliteratorsList.add(tokenizers);
    }
    reader = new PageReader(inputSchema);
    builder = new PageBuilder(Exec.getBufferAllocator(), outputSchema, output);
}
开发者ID:toyama0919，项目名称:embulk-filter-icu4j，代码行数:22，代码来源:Icu4jPageOutput.java

示例4: WootheePageOutput

import org.embulk.spi.Exec; //导入依赖的package包/类
public WootheePageOutput(TaskSource taskSource, Schema inputSchema, Schema outputSchema, PageOutput output) {
    this.task = taskSource.loadTask(PluginTask.class);
    this.outputColumns = outputSchema.getColumns();
    this.inputColumns = inputSchema.getColumns();
    Map<String, Column> inputColumnMap = Maps.newHashMap();
    this.wootheeColumnMap = Maps.newHashMap();
    for (Column column : outputColumns) {
        if (!inputColumns.contains(column)) {
            wootheeColumnMap.put(column.getName(), column);
        } else {
            inputColumnMap.put(column.getName(), column);
        }
    }
    this.reader = new PageReader(inputSchema);
    this.builder = new PageBuilder(Exec.getBufferAllocator(), outputSchema, output);
    this.keyNameColumn = inputColumnMap.get(task.getKeyName());
}
开发者ID:toyama0919，项目名称:embulk-filter-woothee，代码行数:18，代码来源:WootheePageOutput.java

示例5: checkDefaultValuesP12keyNull

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test(expected = ConfigException.class)
public void checkDefaultValuesP12keyNull()
{
    ConfigSource config = Exec.newConfigSource()
            .set("in", inputConfig())
            .set("parser", parserConfig(schemaConfig()))
            .set("type", "gcs")
            .set("bucket", GCP_BUCKET)
            .set("path_prefix", "my-prefix")
            .set("file_ext", ".csv")
            .set("auth_method", "private_key")
            .set("service_account_email", GCP_EMAIL)
            .set("p12_keyfile", null)
            .set("formatter", formatterConfig());

    Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();

    runner.transaction(config, schema, 0, new Control());
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:20，代码来源:TestGcsOutputPlugin.java

示例6: checkDefaultValuesConflictSetting

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test(expected = ConfigException.class)
public void checkDefaultValuesConflictSetting()
{
    ConfigSource config = Exec.newConfigSource()
            .set("in", inputConfig())
            .set("parser", parserConfig(schemaConfig()))
            .set("type", "gcs")
            .set("bucket", GCP_BUCKET)
            .set("path_prefix", "my-prefix")
            .set("file_ext", ".csv")
            .set("auth_method", "private_key")
            .set("service_account_email", GCP_EMAIL)
            .set("p12_keyfile", GCP_P12_KEYFILE)
            .set("p12_keyfile_path", GCP_P12_KEYFILE)
            .set("formatter", formatterConfig());

    Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();

    runner.transaction(config, schema, 0, new Control());
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:21，代码来源:TestGcsOutputPlugin.java

示例7: checkDefaultValuesInvalidPrivateKey

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test(expected = ConfigException.class)
public void checkDefaultValuesInvalidPrivateKey()
{
    ConfigSource config = Exec.newConfigSource()
            .set("in", inputConfig())
            .set("parser", parserConfig(schemaConfig()))
            .set("type", "gcs")
            .set("bucket", GCP_BUCKET)
            .set("path_prefix", "my-prefix")
            .set("file_ext", ".csv")
            .set("auth_method", "private_key")
            .set("service_account_email", GCP_EMAIL)
            .set("p12_keyfile", "invalid-key.p12")
            .set("formatter", formatterConfig());

    Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();

    runner.transaction(config, schema, 0, new Control());
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:20，代码来源:TestGcsOutputPlugin.java

示例8: checkDefaultValuesJsonKeyfileNull

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test(expected = ConfigException.class)
public void checkDefaultValuesJsonKeyfileNull()
{
    ConfigSource config = Exec.newConfigSource()
            .set("in", inputConfig())
            .set("parser", parserConfig(schemaConfig()))
            .set("type", "gcs")
            .set("bucket", GCP_BUCKET)
            .set("path_prefix", "my-prefix")
            .set("file_ext", ".csv")
            .set("auth_method", "json_key")
            .set("service_account_email", GCP_EMAIL)
            .set("json_keyfile", null)
            .set("formatter", formatterConfig());

    Schema schema = config.getNested("parser").loadConfig(CsvParserPlugin.PluginTask.class).getSchemaConfig().toSchema();

    runner.transaction(config, schema, 0, new Control());
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:20，代码来源:TestGcsOutputPlugin.java

示例9: config

import org.embulk.spi.Exec; //导入依赖的package包/类
public ConfigSource config()
{
    return Exec.newConfigSource()
            .set("in", inputConfig())
            .set("parser", parserConfig(schemaConfig()))
            .set("type", "gcs")
            .set("bucket", GCP_BUCKET)
            .set("path_prefix", GCP_PATH_PREFIX)
            .set("last_path", "")
            .set("file_ext", ".csv")
            .set("auth_method", "private_key")
            .set("service_account_email", GCP_EMAIL)
            .set("p12_keyfile", GCP_P12_KEYFILE)
            .set("json_keyfile", GCP_JSON_KEYFILE)
            .set("application_name", GCP_APPLICATION_NAME)
            .set("formatter", formatterConfig());
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:18，代码来源:TestGcsOutputPlugin.java

示例10: getBaseConfigSource

import org.embulk.spi.Exec; //导入依赖的package包/类
private ConfigSource getBaseConfigSource()
{
    return Exec.newConfigSource()
            .set("type", "hdfs")
            .set("path_prefix", pathPrefix)
            .set("file_ext", "csv")
            .setNested("formatter", Exec.newConfigSource()
                    .set("type", "csv")
                    .set("newline", "CRLF")
                    .set("newline_in_field", "LF")
                    .set("header_line", true)
                    .set("charset", "UTF-8")
                    .set("quote_policy", "NONE")
                    .set("quote", "\"")
                    .set("escape", "\\")
                    .set("null_string", "")
                    .set("default_timezone", "UTC"));
}
开发者ID:civitaspo，项目名称:embulk-output-hdfs，代码行数:19，代码来源:TestHdfsFileOutputPlugin.java

示例11: checkDefaultValues

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test
public void checkDefaultValues()
{
    ConfigSource config = Exec.newConfigSource()
            .set("uri", MONGO_URI)
            .set("collection", MONGO_COLLECTION);

    PluginTask task = config.loadConfig(PluginTask.class);
    assertEquals("{}", task.getQuery());
    assertEquals("{}", task.getSort());
    assertEquals(Optional.<Integer>absent(), task.getLimit());
    assertEquals(Optional.<Integer>absent(), task.getSkip());
    assertEquals((long) 10000, (long) task.getBatchSize());
    assertEquals("record", task.getJsonColumnName());
    assertEquals(Optional.absent(), task.getIncrementalField());
    assertEquals(Optional.absent(), task.getLastRecord());
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:18，代码来源:TestMongodbInputPlugin.java

示例12: testRunWithLimit

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test
public void testRunWithLimit() throws Exception
{
    ConfigSource config = Exec.newConfigSource()
            .set("uri", MONGO_URI)
            .set("collection", MONGO_COLLECTION)
            .set("limit", 1);
    PluginTask task = config.loadConfig(PluginTask.class);

    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);
    insertDocument(task, createValidDocuments());

    plugin.transaction(config, new Control());
    assertValidRecords(getFieldSchema(), output, 1, 0);
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:17，代码来源:TestMongodbInputPlugin.java

示例13: testRunWithLimitSkip

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test
public void testRunWithLimitSkip() throws Exception
{
    ConfigSource config = Exec.newConfigSource()
            .set("uri", MONGO_URI)
            .set("collection", MONGO_COLLECTION)
            .set("limit", 3)
            .set("skip", 1);
    PluginTask task = config.loadConfig(PluginTask.class);

    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);
    insertDocument(task, createValidDocuments());

    plugin.transaction(config, new Control());
    assertValidRecords(getFieldSchema(), output, 3, 1);
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:18，代码来源:TestMongodbInputPlugin.java

示例14: testRunWithConnectionParams

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test
public void testRunWithConnectionParams() throws Exception
{
    MongoClientURI uri = new MongoClientURI(MONGO_URI);
    String host = uri.getHosts().get(0);
    Integer port = (host.split(":")[1] != null) ? Integer.valueOf(host.split(":")[1]) : 27017;
    ConfigSource config = Exec.newConfigSource()
            .set("hosts", Arrays.asList(ImmutableMap.of("host", host.split(":")[0], "port", port)))
            .set("user", uri.getUsername())
            .set("password", uri.getPassword())
            .set("database", uri.getDatabase())
            .set("collection", MONGO_COLLECTION);
    PluginTask task = config.loadConfig(PluginTask.class);

    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);
    insertDocument(task, createValidDocuments());

    plugin.transaction(config, new Control());
    assertValidRecords(getFieldSchema(), output);
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:22，代码来源:TestMongodbInputPlugin.java

示例15: testRunWithIncrementalLoad

import org.embulk.spi.Exec; //导入依赖的package包/类
@Test
public void testRunWithIncrementalLoad() throws Exception
{
    ConfigSource config = Exec.newConfigSource()
            .set("uri", MONGO_URI)
            .set("collection", MONGO_COLLECTION)
            .set("incremental_field", Optional.of(Arrays.asList("int32_field")));
    PluginTask task = config.loadConfig(PluginTask.class);

    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);
    insertDocument(task, createValidDocuments());

    ConfigDiff diff = plugin.transaction(config, new Control());
    ConfigDiff lastRecord = diff.getNested("last_record");

    assertEquals("5", lastRecord.get(String.class, "int32_field"));
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:19，代码来源:TestMongodbInputPlugin.java


注：本文中的org.embulk.spi.Exec类示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。