当前位置: 首页>>代码示例>>Java>>正文


Java Coder.verifyDeterministic方法代码示例

本文整理汇总了Java中org.apache.beam.sdk.coders.Coder.verifyDeterministic方法的典型用法代码示例。如果您正苦于以下问题:Java Coder.verifyDeterministic方法的具体用法?Java Coder.verifyDeterministic怎么用?Java Coder.verifyDeterministic使用的例子?那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在org.apache.beam.sdk.coders.Coder的用法示例。


在下文中一共展示了Coder.verifyDeterministic方法的7个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于系统推荐出更棒的Java代码示例。

示例1: expand

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
@Override
public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
  applicableTo(input);

  // Verify that the input Coder<KV<K, V>> is a KvCoder<K, V>, and that
  // the key coder is deterministic.
  Coder<K> keyCoder = getKeyCoder(input.getCoder());
  try {
    keyCoder.verifyDeterministic();
  } catch (NonDeterministicException e) {
    throw new IllegalStateException(
        "the keyCoder of a GroupByKey must be deterministic", e);
  }

  // This primitive operation groups by the combination of key and window,
  // merging windows as needed, using the windows assigned to the
  // key/value input elements and the window merge operation of the
  // window function associated with the input PCollection.
  return PCollection.createPrimitiveOutputInternal(
      input.getPipeline(),
      updateWindowingStrategy(input.getWindowingStrategy()),
      input.isBounded(),
      getOutputKvCoder(input.getCoder()));
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:GroupByKey.java

示例2: create

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
/**
 * Returns an {@link ApproximateDistinctFn} combiner with the given input coder.
 *
 * @param coder the coder that encodes the elements' type
 */
public static <InputT> ApproximateDistinctFn<InputT> create(Coder<InputT> coder) {
  try {
    coder.verifyDeterministic();
  } catch (Coder.NonDeterministicException e) {
    throw new IllegalArgumentException("Coder is not deterministic ! " + e.getMessage(), e);
  }
  return new ApproximateDistinctFn<>(12, 0, coder);
}
 
开发者ID:apache,项目名称:beam,代码行数:14,代码来源:ApproximateDistinct.java

示例3: testNonDeterministicCoder

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
@Test
public void testNonDeterministicCoder() throws NonDeterministicException {
  thrown.expect(NonDeterministicException.class);
  thrown.expectMessage(MessageWithMap.class.getName() + " transitively includes Map field");

  Coder<MessageWithMap> coder = ProtoCoder.of(MessageWithMap.class);
  coder.verifyDeterministic();
}
 
开发者ID:apache,项目名称:beam,代码行数:9,代码来源:ProtoCoderTest.java

示例4: coderDeterministicInContext

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
/**
 * Verifies that for the given {@code Coder<T>}, {@code Coder.Context}, and values of
 * type {@code T}, if the values are equal then the encoded bytes are equal.
 */
public static <T> void coderDeterministicInContext(
    Coder<T> coder, Coder.Context context, T value1, T value2)
    throws Exception {
  try {
    coder.verifyDeterministic();
  } catch (NonDeterministicException e) {
    fail("Expected that the coder is deterministic");
  }
  assertThat("Expected that the passed in values are equal()", value1, equalTo(value2));
  assertThat(
      encode(coder, context, value1),
      equalTo(encode(coder, context, value2)));
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:CoderProperties.java

示例5: validateCoderIsCompatible

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
/**
 * Validates that the key portion of the given coder is deterministic.
 */
public static void validateCoderIsCompatible(IsmRecordCoder<?> coder) {
  for (Coder<?> keyComponentCoder : coder.getKeyComponentCoders()) {
    try {
        keyComponentCoder.verifyDeterministic();
    } catch (NonDeterministicException e) {
      throw new IllegalArgumentException(
          String.format("Key component coder %s is expected to be deterministic.",
              keyComponentCoder), e);
    }
  }
}
 
开发者ID:apache,项目名称:beam,代码行数:15,代码来源:IsmFormat.java

示例6: testDeterministicCoder

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
@Test
public void testDeterministicCoder() throws NonDeterministicException {
  Coder<MessageA> coder = ProtoCoder.of(MessageA.class);
  coder.verifyDeterministic();
}
 
开发者ID:apache,项目名称:beam,代码行数:6,代码来源:ProtoCoderTest.java

示例7: expand

import org.apache.beam.sdk.coders.Coder; //导入方法依赖的package包/类
@Override
public WriteFilesResult<DestinationT> expand(PCollection<UserT> input) {
  if (input.isBounded() == IsBounded.UNBOUNDED) {
    checkArgument(
        getWindowedWrites(),
        "Must use windowed writes when applying %s to an unbounded PCollection",
        WriteFiles.class.getSimpleName());
    // The reason for this is https://issues.apache.org/jira/browse/BEAM-1438
    // and similar behavior in other runners.
    checkArgument(
        getComputeNumShards() != null || getNumShardsProvider() != null,
        "When applying %s to an unbounded PCollection, "
            + "must specify number of output shards explicitly",
        WriteFiles.class.getSimpleName());
  }
  this.writeOperation = getSink().createWriteOperation();
  this.writeOperation.setWindowedWrites(getWindowedWrites());

  if (!getWindowedWrites()) {
    // Re-window the data into the global window and remove any existing triggers.
    input =
        input.apply(
            "RewindowIntoGlobal",
            Window.<UserT>into(new GlobalWindows())
                .triggering(DefaultTrigger.of())
                .discardingFiredPanes());
  }

  Coder<DestinationT> destinationCoder;
  try {
    destinationCoder =
        getDynamicDestinations()
            .getDestinationCoderWithDefault(input.getPipeline().getCoderRegistry());
    destinationCoder.verifyDeterministic();
  } catch (CannotProvideCoderException | NonDeterministicException e) {
    throw new RuntimeException(e);
  }
  @SuppressWarnings("unchecked")
  Coder<BoundedWindow> windowCoder =
      (Coder<BoundedWindow>) input.getWindowingStrategy().getWindowFn().windowCoder();
  FileResultCoder<DestinationT> fileResultCoder =
      FileResultCoder.of(windowCoder, destinationCoder);

  PCollectionView<Integer> numShardsView =
      (getComputeNumShards() == null) ? null : input.apply(getComputeNumShards());

  PCollection<FileResult<DestinationT>> tempFileResults =
      (getComputeNumShards() == null && getNumShardsProvider() == null)
          ? input.apply(
              "WriteUnshardedBundlesToTempFiles",
              new WriteUnshardedBundlesToTempFiles(destinationCoder, fileResultCoder))
          : input.apply(
              "WriteShardedBundlesToTempFiles",
              new WriteShardedBundlesToTempFiles(
                  destinationCoder, fileResultCoder, numShardsView));

  return tempFileResults
      .apply("GatherTempFileResults", new GatherResults<>(fileResultCoder))
      .apply(
          "FinalizeTempFileBundles",
          new FinalizeTempFileBundles(numShardsView, destinationCoder));
}
 
开发者ID:apache,项目名称:beam,代码行数:63,代码来源:WriteFiles.java


注:本文中的org.apache.beam.sdk.coders.Coder.verifyDeterministic方法示例由纯净天空整理自Github/MSDocs等开源代码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。