当前位置: 首页>>代码示例>>Java>>正文

Java ConfigSource.loadConfig方法代码示例

本文整理汇总了Java中org.embulk.config.ConfigSource.loadConfig方法的典型用法代码示例。如果您正苦于以下问题：Java ConfigSource.loadConfig方法的具体用法？Java ConfigSource.loadConfig怎么用？Java ConfigSource.loadConfig使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类org.embulk.config.ConfigSource的用法示例。


在下文中一共展示了ConfigSource.loadConfig方法的15个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema schema,
        FormatterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    // validate avsc option
    try {
        File avsc = task.getAvsc().getFile();
        new org.apache.avro.Schema.Parser().parse(avsc);
    } catch (IOException e) {
        throw new ConfigException("avsc file is not found");
    }


    // validate column_options
    for (String columnName : task.getColumnOptions().keySet()) {
        schema.lookupColumn(columnName);  // throws SchemaConfigException
    }

    control.run(task.dump());
}
开发者ID:joker1007，项目名称:embulk-formatter-avro，代码行数:23，代码来源:AvroFormatterPlugin.java

示例2: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public ConfigDiff transaction(ConfigSource config,
        InputPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);
    configureParquetLogger(task);

    Path rootPath = new Path(task.getPath());

    try (PluginClassLoaderScope ignored = new PluginClassLoaderScope()) {
        Configuration conf = ConfigurationFactory.create(task);

        FileSystem fs = FileSystem.get(rootPath.toUri(), conf);
        List<FileStatus> statusList = listFileStatuses(fs, rootPath);
        if (statusList.isEmpty()) {
            throw new PathNotFoundException(rootPath.toString());
        }

        for (FileStatus status : statusList) {
            logger.debug("embulk-input-parquet_hadoop: Loading paths: {}, length: {}",
                    status.getPath(), status.getLen());
        }

        List<String> files = Lists.transform(statusList, new Function<FileStatus, String>() {
                @Nullable
                @Override
                public String apply(@Nullable FileStatus input)
                {
                    return input.getPath().toString();
                }
        });
        task.setFiles(files);
    }
    catch (IOException e) {
        throw Throwables.propagate(e);
    }

    Schema schema = newSchema();
    int taskCount = task.getFiles().size();

    return resume(task.dump(), schema, taskCount, control);
}
开发者ID:CyberAgent，项目名称:embulk-input-parquet_hadoop，代码行数:43，代码来源:ParquetHadoopInputPlugin.java

示例3: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public void transaction(ConfigSource config, ParserPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    File avsc = task.getAvsc().getFile();
    org.apache.avro.Schema avroSchema;
    try {
        avroSchema = new org.apache.avro.Schema.Parser().parse(avsc);
    } catch (IOException e) {
        throw new ConfigException("avsc file is not found");
    }

    Schema schema = buildSchema(task.getColumns(), avroSchema);

    control.run(task.dump(), schema);
}
开发者ID:joker1007，项目名称:embulk-parser-avro，代码行数:18，代码来源:AvroParserPlugin.java

示例4: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema inputSchema, FilterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);
    String columnName = task.getQueryStringColumnName();

    Schema.Builder builder = Schema.builder();
    for (Column inputColumn : inputSchema.getColumns()) {
        if (columnName.equals(inputColumn.getName())) {
            insertColumns(builder, task.getExpandedColumns());
        }
        else {
            builder.add(inputColumn.getName(), inputColumn.getType());
        }
    }
    control.run(task.dump(), builder.build());
}
开发者ID:mwed，项目名称:embulk-filter-query_string，代码行数:18，代码来源:QueryStringFilterPlugin.java

示例5: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema inputSchema,
        FilterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    String json_template = task.getJsonColumn().getTemplate();
    BuildJsonConfigChecker checker = new BuildJsonConfigChecker(json_template,inputSchema);
    checker.validateJSON();

    Schema outputSchema = buildOutputSchema(task, inputSchema);
    for (Column column : outputSchema.getColumns()) {
        logger.debug("OutputSchema: {}", column);
    }

    control.run(task.dump(), outputSchema);
}
开发者ID:hiroyuki-sato，项目名称:embulk-filter-build_json，代码行数:18，代码来源:BuildJsonFilterPlugin.java

示例6: testBuildIncrementalConditionFieldRequired

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Test
public void testBuildIncrementalConditionFieldRequired() throws Exception
{
    Map<String, Object> lastRecord = new HashMap<>();
    lastRecord.put("double_field", "0");

    ConfigSource config = Exec.newConfigSource()
            .set("uri", MONGO_URI)
            .set("collection", MONGO_COLLECTION)
            .set("incremental_field", Optional.of(Arrays.asList("invalid_field")))
            .set("last_record", Optional.of(lastRecord));
    PluginTask task = config.loadConfig(PluginTask.class);
    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);

    Method method = MongodbInputPlugin.class.getDeclaredMethod("buildIncrementalCondition", PluginTask.class);
    method.setAccessible(true);
    try {
        method.invoke(plugin, task); // field declaration was not set at incremental_field
    }
    catch (Exception ex) {
        assertEquals(ConfigException.class, ex.getCause().getClass());
    }
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:25，代码来源:TestMongodbInputPlugin.java

示例7: checkExtraConfigurations

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Test
public void checkExtraConfigurations()
        throws NoSuchMethodException, InvocationTargetException, IllegalAccessException
{
    ConfigSource map = Exec.newConfigSource()
            .set("foo", "bar");

    ConfigSource config = Exec.newConfigSource()
            .set("path_prefix", "test")
            .setNested("extra_configurations", map);

    ParquetOutputPlugin.PluginTask task = config.loadConfig(ParquetOutputPlugin.PluginTask.class);

    Map<String, String> extra = task.getExtraConfigurations();
    assertTrue(extra.containsKey("foo"));
    assertEquals("bar", extra.get("foo"));

    ParquetOutputPlugin plugin = new ParquetOutputPlugin();
    Method method = ParquetOutputPlugin.class.getDeclaredMethod("createConfiguration", Map.class);
    method.setAccessible(true);
    Configuration conf = (Configuration) method.invoke(plugin, extra);
    assertEquals("bar", conf.get("foo"));
}
开发者ID:choplin，项目名称:embulk-output-parquet，代码行数:24，代码来源:ParquetOutputPluginTest.java

示例8: testGenerateRemotePath

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Test
public void testGenerateRemotePath() throws Exception
{
    ConfigSource configSource = config();
    PluginTask task = configSource.loadConfig(PluginTask.class);
    Method method = GcsOutputPlugin.class.getDeclaredMethod("generateRemotePath", String.class, String.class, int.class, int.class, String.class);
    method.setAccessible(true);
    assertEquals("sample.000.01.csv", method.invoke(plugin, "/sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("sample.000.01.csv", method.invoke(plugin, "./sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("sample.000.01.csv", method.invoke(plugin, "../sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("sample.000.01.csv", method.invoke(plugin, "//sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("path/to/sample.000.01.csv", method.invoke(plugin, "/path/to/sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("path/to/./sample.000.01.csv", method.invoke(plugin, "path/to/./sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("path/to/../sample.000.01.csv", method.invoke(plugin, "path/to/../sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("sample.000.01.csv", method.invoke(plugin, "....../sample", task.getSequenceFormat(), 0, 1, ".csv"));
    assertEquals("sample.000.01.csv", method.invoke(plugin, "......///sample", task.getSequenceFormat(), 0, 1, ".csv"));
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:18，代码来源:TestGcsOutputPlugin.java

示例9: getFileContentsFromGcs

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
private ImmutableList<List<String>> getFileContentsFromGcs(String path) throws Exception
{
    ConfigSource config = config();

    PluginTask task = config.loadConfig(PluginTask.class);

    Method method = GcsOutputPlugin.class.getDeclaredMethod("createClient", PluginTask.class);
    method.setAccessible(true);
    Storage client = (Storage) method.invoke(plugin, task);
    Storage.Objects.Get getObject = client.objects().get(GCP_BUCKET, path);

    ImmutableList.Builder<List<String>> builder = new ImmutableList.Builder<>();

    InputStream is =  getObject.executeMediaAsInputStream();
    BufferedReader reader = new BufferedReader(new InputStreamReader(is));
    String line;
    while ((line = reader.readLine()) != null) {
        List<String> records = Arrays.asList(line.split(",", 0));

        builder.add(records);
    }
    return builder.build();
}
开发者ID:embulk，项目名称:embulk-output-gcs，代码行数:24，代码来源:TestGcsOutputPlugin.java

示例10: testDefaultValue

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Test
public void testDefaultValue()
{
    String configYaml = "" +
            "type: expand_json\n" +
            "json_column_name: _c0\n" +
            "expanded_columns:\n" +
            "  - {name: _j1, type: boolean}\n" +
            "  - {name: _j2, type: long}\n" +
            "  - {name: _j3, type: timestamp}\n" +
            "  - {name: _j4, type: double}\n" +
            "  - {name: _j5, type: string}\n" +
            "  - {name: _j6, type: json}\n";

    ConfigSource config = getConfigFromYaml(configYaml);
    PluginTask task = config.loadConfig(PluginTask.class);

    assertEquals("$.", task.getRoot());
    assertEquals("UTC", task.getDefaultTimeZone().getID());
    assertEquals("%Y-%m-%d %H:%M:%S.%N %z", task.getDefaultTimestampFormat());
    assertEquals(false, task.getStopOnInvalidRecord());
    assertEquals(false, task.getKeepExpandingJsonColumn());
}
开发者ID:civitaspo，项目名称:embulk-filter-expand_json，代码行数:24，代码来源:TestExpandJsonFilterPlugin.java

示例11: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema inputSchema,
                        FilterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    List<Column> distinctColumns = convertNameToColumn(inputSchema, task.getDistinctColumnNames());
    task.setDistinctColumns(distinctColumns);

    if (task.getDistinctColumns().isEmpty()) {
        throw new ConfigException(
                "inputSchema does not have any columns you configured.");
    }
    else {
        logger.debug("distinct columns: {}", task.getDistinctColumns());
    }

    Schema outputSchema = inputSchema;
    control.run(task.dump(), outputSchema);
}
开发者ID:civitaspo，项目名称:embulk-filter-distinct，代码行数:21，代码来源:DistinctFilterPlugin.java

示例12: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public ConfigDiff transaction(ConfigSource config, int taskCount,
        final FileOutputPlugin.Control control)
{
    final PluginTask task = config.loadConfig(PluginTask.class);
    compat(task);
    avoidDatabindError(task);

    Tx tx = task.getMode().newTx();
    return tx.transaction(task, new ControlRun()
    {
        @Override
        public List<TaskReport> run()
        {
            return control.run(task.dump());
        }
    });
}
开发者ID:civitaspo，项目名称:embulk-output-hdfs，代码行数:19，代码来源:HdfsFileOutputPlugin.java

示例13: testRunWithLimitSkip

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Test
public void testRunWithLimitSkip() throws Exception
{
    ConfigSource config = Exec.newConfigSource()
            .set("uri", MONGO_URI)
            .set("collection", MONGO_COLLECTION)
            .set("limit", 3)
            .set("skip", 1);
    PluginTask task = config.loadConfig(PluginTask.class);

    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);
    insertDocument(task, createValidDocuments());

    plugin.transaction(config, new Control());
    assertValidRecords(getFieldSchema(), output, 3, 1);
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:18，代码来源:TestMongodbInputPlugin.java

示例14: testRunWithConnectionParams

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Test
public void testRunWithConnectionParams() throws Exception
{
    MongoClientURI uri = new MongoClientURI(MONGO_URI);
    String host = uri.getHosts().get(0);
    Integer port = (host.split(":")[1] != null) ? Integer.valueOf(host.split(":")[1]) : 27017;
    ConfigSource config = Exec.newConfigSource()
            .set("hosts", Arrays.asList(ImmutableMap.of("host", host.split(":")[0], "port", port)))
            .set("user", uri.getUsername())
            .set("password", uri.getPassword())
            .set("database", uri.getDatabase())
            .set("collection", MONGO_COLLECTION);
    PluginTask task = config.loadConfig(PluginTask.class);

    dropCollection(task, MONGO_COLLECTION);
    createCollection(task, MONGO_COLLECTION);
    insertDocument(task, createValidDocuments());

    plugin.transaction(config, new Control());
    assertValidRecords(getFieldSchema(), output);
}
开发者ID:hakobera，项目名称:embulk-input-mongodb，代码行数:22，代码来源:TestMongodbInputPlugin.java

示例15: transaction

import org.embulk.config.ConfigSource; //导入方法依赖的package包/类
@Override
public void transaction(ConfigSource configSource, Control control)
{
    PluginTask task = configSource.loadConfig(PluginTask.class);

    if (! task.getColumnOptions().isEmpty()) {
        log.warn("embulk-parser-jsonl: \"column_options\" option is deprecated, specify type directly to \"columns\" option with typecast: true (default: true).");
    }

    SchemaConfig schemaConfig = getSchemaConfig(task);
    ImmutableList.Builder<Column> columns = ImmutableList.builder();
    for (int i = 0; i < schemaConfig.getColumnCount(); i++) {
        ColumnConfig columnConfig = schemaConfig.getColumn(i);
        Type type = getType(task, columnConfig);
        columns.add(new Column(i, columnConfig.getName(), type));
    }
    control.run(task.dump(), new Schema(columns.build()));
}
开发者ID:shun0102，项目名称:embulk-parser-jsonl，代码行数:19，代码来源:JsonlParserPlugin.java


注：本文中的org.embulk.config.ConfigSource.loadConfig方法示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。