本文整理汇总了Python中tensor2tensor.data_generators.text_encoder.PAD属性的典型用法代码示例。如果您正苦于以下问题：Python text_encoder.PAD属性的具体用法？Python text_encoder.PAD怎么用？Python text_encoder.PAD使用的例子？那么恭喜您, 这里精选的属性代码示例或许可以为您提供帮助。您也可以进一步了解该属性所在类tensor2tensor.data_generators.text_encoder的用法示例。
在下文中一共展示了text_encoder.PAD属性的2个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: build_vocab_list
# 需要导入模块: from tensor2tensor.data_generators import text_encoder [as 别名]
# 或者: from tensor2tensor.data_generators.text_encoder import PAD [as 别名]
def build_vocab_list(data_path):
"""Reads a file to build a vocabulary with letters and phonemes.
Args:
data_path: data file to read list of words from.
Returns:
vocab_list: vocabulary list with both graphemes and phonemes."""
vocab = {}
with tf.gfile.GFile(data_path, "r") as data_file:
for line in data_file:
items = line.strip().split()
vocab.update({char:1 for char in list(items[0])})
vocab.update({phoneme:1 for phoneme in items[1:]})
vocab_list = [PAD, EOS]
for key in sorted(vocab.keys()):
vocab_list.append(key)
return vocab_list 示例2: test_reserved_tokens_in_corpus
# 需要导入模块: from tensor2tensor.data_generators import text_encoder [as 别名]
# 或者: from tensor2tensor.data_generators.text_encoder import PAD [as 别名]
def test_reserved_tokens_in_corpus(self):
"""Test that we handle reserved tokens appearing in the corpus."""
corpus = "A B {} D E F {} G {}".format(text_encoder.EOS,
text_encoder.EOS,
text_encoder.PAD)
encoder = text_encoder.TokenTextEncoder(None, vocab_list=corpus.split())
all_tokens = encoder._id_to_token.values()
# If reserved tokens are removed correctly, then the set of tokens will
# be unique.
self.assertEqual(len(all_tokens), len(set(all_tokens)))