当前位置: 首页>>代码示例>>Java>>正文

Java ConfigSource类代码示例

本文整理汇总了Java中org.embulk.config.ConfigSource类的典型用法代码示例。如果您正苦于以下问题：Java ConfigSource类的具体用法？Java ConfigSource怎么用？Java ConfigSource使用的例子？那么, 这里精选的类代码示例或许可以为您提供帮助。


ConfigSource类属于org.embulk.config包，在下文中一共展示了ConfigSource类的15个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: transaction

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema schema,
        FormatterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    // validate avsc option
    try {
        File avsc = task.getAvsc().getFile();
        new org.apache.avro.Schema.Parser().parse(avsc);
    } catch (IOException e) {
        throw new ConfigException("avsc file is not found");
    }


    // validate column_options
    for (String columnName : task.getColumnOptions().keySet()) {
        schema.lookupColumn(columnName);  // throws SchemaConfigException
    }

    control.run(task.dump());
}
开发者ID:joker1007，项目名称:embulk-formatter-avro，代码行数:23，代码来源:AvroFormatterPlugin.java

示例2: transaction

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Override
public ConfigDiff transaction(ConfigSource config,
        InputPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);
    configureParquetLogger(task);

    Path rootPath = new Path(task.getPath());

    try (PluginClassLoaderScope ignored = new PluginClassLoaderScope()) {
        Configuration conf = ConfigurationFactory.create(task);

        FileSystem fs = FileSystem.get(rootPath.toUri(), conf);
        List<FileStatus> statusList = listFileStatuses(fs, rootPath);
        if (statusList.isEmpty()) {
            throw new PathNotFoundException(rootPath.toString());
        }

        for (FileStatus status : statusList) {
            logger.debug("embulk-input-parquet_hadoop: Loading paths: {}, length: {}",
                    status.getPath(), status.getLen());
        }

        List<String> files = Lists.transform(statusList, new Function<FileStatus, String>() {
                @Nullable
                @Override
                public String apply(@Nullable FileStatus input)
                {
                    return input.getPath().toString();
                }
        });
        task.setFiles(files);
    }
    catch (IOException e) {
        throw Throwables.propagate(e);
    }

    Schema schema = newSchema();
    int taskCount = task.getFiles().size();

    return resume(task.dump(), schema, taskCount, control);
}
开发者ID:CyberAgent，项目名称:embulk-input-parquet_hadoop，代码行数:43，代码来源:ParquetHadoopInputPlugin.java

示例3: assertRecordsByResource

import org.embulk.config.ConfigSource; //导入依赖的package包/类
static void assertRecordsByResource(TestingEmbulk embulk,
                                    String inConfigYamlResourceName,
                                    String sourceResourceName, String resultCsvResourceName)
        throws Exception
{
    Path outputPath = embulk.createTempFile("csv");

    // in: config
    String inputPath = Resources.getResource(RESOURCE_NAME_PREFIX + sourceResourceName).toURI().toString();
    ConfigSource inConfig = embulk.loadYamlResource(RESOURCE_NAME_PREFIX + inConfigYamlResourceName)
            .set("path", inputPath);

    TestingEmbulk.RunResult result = embulk.inputBuilder()
            .in(inConfig)
            .outputPath(outputPath)
            .run();

    assertThat(readFile(outputPath), is(readResource(RESOURCE_NAME_PREFIX + resultCsvResourceName)));
}
开发者ID:CyberAgent，项目名称:embulk-input-parquet_hadoop，代码行数:20，代码来源:TestParquetHadoopInputPlugin.java

示例4: transaction

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Override
public void transaction(ConfigSource config, ParserPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    File avsc = task.getAvsc().getFile();
    org.apache.avro.Schema avroSchema;
    try {
        avroSchema = new org.apache.avro.Schema.Parser().parse(avsc);
    } catch (IOException e) {
        throw new ConfigException("avsc file is not found");
    }

    Schema schema = buildSchema(task.getColumns(), avroSchema);

    control.run(task.dump(), schema);
}
开发者ID:joker1007，项目名称:embulk-parser-avro，代码行数:18，代码来源:AvroParserPlugin.java

示例5: skipRecords

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void skipRecords()
        throws Exception
{
    SchemaConfig schema = schema(
            column("_c0", BOOLEAN), column("_c1", LONG), column("_c2", DOUBLE),
            column("_c3", STRING), column("_c4", TIMESTAMP), column("_c5", JSON));
    ConfigSource config = this.config.deepCopy().set("columns", schema);

    transaction(config, fileInput(
            "[",
            "[]",
            "\"embulk\"",
            "10",
            "true",
            "false",
            "null",
            " ",
            "]"
    ));

    List<Object[]> records = Pages.toObjects(schema.toSchema(), output.pages);
    assertEquals(0, records.size());
}
开发者ID:hiroyuki-sato，项目名称:embulk-parser-jsonpath，代码行数:25，代码来源:TestJsonpathParserPlugin.java

示例6: stopOnBrokenJson

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void stopOnBrokenJson()
        throws Exception
{
    SchemaConfig schema = schema(
            column("_c0", BOOLEAN), column("_c1", LONG), column("_c2", DOUBLE),
            column("_c3", STRING), column("_c4", TIMESTAMP), column("_c5", JSON));
    ConfigSource config = this.config.deepCopy().set("columns", schema).
            set("stop_on_invalid_record", true);

    try {
        transaction(config, fileInput("BROKEN"));
        fail();
    }
    catch (Throwable t) {
        assertTrue(t instanceof DataException);
    }
}
开发者ID:hiroyuki-sato，项目名称:embulk-parser-jsonpath，代码行数:19，代码来源:TestJsonpathParserPlugin.java

示例7: booleanStrings

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void booleanStrings()
        throws Exception
{
    SchemaConfig schema = schema(column("_c1", BOOLEAN), column("_c2", BOOLEAN),
            column("_c3", BOOLEAN), column("_c4", BOOLEAN), column("_c5", BOOLEAN),
            column("_c6", BOOLEAN), column("_c7", BOOLEAN), column("_c8", BOOLEAN),
            column("_c9", BOOLEAN), column("_c10", BOOLEAN), column("_c11", BOOLEAN),
            column("_c12", BOOLEAN));
    ConfigSource config = this.config.deepCopy().set("columns", schema);

    transaction(config, fileInput("[{\"_c1\" : \"yes\", \"_c2\" : \"true\", \"_c3\" : \"1\",",
            "\"_c4\" : \"on\", \"_c5\" : \"y\", \"_c6\" : \"t\",",
            "\"_c7\" : \"no\", \"_c8\" : \"false\", \"_c9\" : \"0\"," ,
            "\"_c10\" : \"off\", \"_c11\" : \"n\", \"_c12\" : \"f\"}]"));
    List<Object[]> records = Pages.toObjects(schema.toSchema(), output.pages);
    assertEquals(1, records.size());

    Object[] record = records.get(0);
    for (int i = 0; i < 5; i++) {
        assertTrue((boolean) record[i]);
    }
    for (int i = 6; i < 11; i++) {
        assertFalse((boolean) record[i]);
    }
}
开发者ID:hiroyuki-sato，项目名称:embulk-parser-jsonpath，代码行数:27，代码来源:TestJsonpathParserPlugin.java

示例8: invalidBooleanString

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void invalidBooleanString()
        throws Exception
{
    SchemaConfig schema = schema(column("_c1", BOOLEAN));
    ConfigSource config = this.config.deepCopy().set("columns", schema).
            set("stop_on_invalid_record", true);

    try {
        transaction(config,
                fileInput("[{\"_c1\" : \"INVALID\"}]"));
        fail();
    }
    catch (Throwable t) {
        assertTrue(t instanceof DataException);
    }
}
开发者ID:hiroyuki-sato，项目名称:embulk-parser-jsonpath，代码行数:18，代码来源:TestJsonpathParserPlugin.java

示例9: stopOnBrokenColumn

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void stopOnBrokenColumn()
        throws Exception
{
    SchemaConfig schema = schema(column("_c1", TIMESTAMP));
    ConfigSource config = this.config.deepCopy().set("columns", schema).
            set("stop_on_invalid_record", true);

    try {
        transaction(config, fileInput("{\"_c1\" : \"INVALID\"}"));
        fail();
    }
    catch (Throwable t) {
        assertTrue(t instanceof DataException);
    }
}
开发者ID:hiroyuki-sato，项目名称:embulk-parser-jsonpath，代码行数:17，代码来源:TestJsonpathParserPlugin.java

示例10: throwDataException

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void throwDataException()
        throws Exception
{
    SchemaConfig schema = schema(
            column("_c0", BOOLEAN), column("_c1", LONG), column("_c2", DOUBLE),
            column("_c3", STRING), column("_c4", TIMESTAMP), column("_c5", JSON));
    ConfigSource config = this.config.deepCopy().set("columns", schema).
            set("stop_on_invalid_record", true);

    try {
        transaction(config, fileInput(
                "\"not_map_value\""
        ));
        fail();
    }
    catch (Throwable t) {
        assertTrue(t instanceof DataException);
    }
}
开发者ID:hiroyuki-sato，项目名称:embulk-parser-jsonpath，代码行数:21，代码来源:TestJsonpathParserPlugin.java

示例11: transaction

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema inputSchema, FilterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);
    String columnName = task.getQueryStringColumnName();

    Schema.Builder builder = Schema.builder();
    for (Column inputColumn : inputSchema.getColumns()) {
        if (columnName.equals(inputColumn.getName())) {
            insertColumns(builder, task.getExpandedColumns());
        }
        else {
            builder.add(inputColumn.getName(), inputColumn.getType());
        }
    }
    control.run(task.dump(), builder.build());
}
开发者ID:mwed，项目名称:embulk-filter-query_string，代码行数:18，代码来源:QueryStringFilterPlugin.java

示例12: testOpenSuccessfully

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void testOpenSuccessfully()
{
    ConfigSource configSource = loadConfigSource("testOpen.yml");
    final Schema inputSchema = Schema.builder()
            .add("qb", STRING)
            .add("qs", STRING)
            .add("qa", STRING)
            .build();
    testQueryString(configSource, inputSchema, "/path?q1=1&q2=2#fragment", new AssertionWithPage()
    {
        @Override
        public void run(PageReader pageReader, TestPageBuilderReader.MockPageOutput pageOutput)
        {
            for (Page page : pageOutput.pages) {
                pageReader.setPage(page);

                assertThat(pageReader.getString(0), is("before"));
                assertThat(pageReader.getString(1), is("1"));
                assertEquals(2L, pageReader.getLong(2));
                assertThat(pageReader.getString(3), is("after"));
            }
        }
    });
}
开发者ID:mwed，项目名称:embulk-filter-query_string，代码行数:26，代码来源:TestQueryStringFilterPlugin.java

示例13: testOpenSuccessfullyWithHashbang

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Test
public void testOpenSuccessfullyWithHashbang()
{
    ConfigSource configSource = loadConfigSource("testOpen.yml");
    final Schema inputSchema = Schema.builder()
            .add("qb", STRING)
            .add("qs", STRING)
            .add("qa", STRING)
            .build();

    testQueryString(configSource, inputSchema, "/#!/path?q1=one&q2=2#fragment", new AssertionWithPage()
    {
        @Override
        public void run(PageReader pageReader, TestPageBuilderReader.MockPageOutput pageOutput)
        {
            for (Page page : pageOutput.pages) {
                pageReader.setPage(page);

                assertThat(pageReader.getString(0), is("before"));
                assertThat(pageReader.getString(1), is("one"));
                assertEquals(2L, pageReader.getLong(2));
                assertThat(pageReader.getString(3), is("after"));
            }
        }
    });
}
开发者ID:mwed，项目名称:embulk-filter-query_string，代码行数:27，代码来源:TestQueryStringFilterPlugin.java

示例14: testQueryString

import org.embulk.config.ConfigSource; //导入依赖的package包/类
private void testQueryString(ConfigSource configSource, final Schema inputSchema, final String path, final AssertionWithPage assertion)
{
    final QueryStringFilterPlugin plugin = new QueryStringFilterPlugin();
    plugin.transaction(configSource, inputSchema, new FilterPlugin.Control()
    {
        @Override
        public void run(TaskSource taskSource, Schema outputSchema)
        {
            TestPageBuilderReader.MockPageOutput mockPageOutput = new TestPageBuilderReader.MockPageOutput();
            PageOutput pageOutput = plugin.open(taskSource, inputSchema, outputSchema, mockPageOutput);

            List<Page> pages = PageTestUtils.buildPage(runtime.getBufferAllocator(), inputSchema, "before", path, "after");
            for (Page page : pages) {
                pageOutput.add(page);
            }

            pageOutput.finish();
            pageOutput.close();

            PageReader pageReader = new PageReader(outputSchema);
            assertion.run(pageReader, mockPageOutput);
            ;
        }
    });
}
开发者ID:mwed，项目名称:embulk-filter-query_string，代码行数:26，代码来源:TestQueryStringFilterPlugin.java

示例15: transaction

import org.embulk.config.ConfigSource; //导入依赖的package包/类
@Override
public void transaction(ConfigSource config, Schema inputSchema,
        FilterPlugin.Control control)
{
    PluginTask task = config.loadConfig(PluginTask.class);

    String json_template = task.getJsonColumn().getTemplate();
    BuildJsonConfigChecker checker = new BuildJsonConfigChecker(json_template,inputSchema);
    checker.validateJSON();

    Schema outputSchema = buildOutputSchema(task, inputSchema);
    for (Column column : outputSchema.getColumns()) {
        logger.debug("OutputSchema: {}", column);
    }

    control.run(task.dump(), outputSchema);
}
开发者ID:hiroyuki-sato，项目名称:embulk-filter-build_json，代码行数:18，代码来源:BuildJsonFilterPlugin.java


注：本文中的org.embulk.config.ConfigSource类示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。