本文整理匯總了Python中tensor2tensor.data_generators.text_encoder.PAD屬性的典型用法代碼示例。如果您正苦於以下問題：Python text_encoder.PAD屬性的具體用法？Python text_encoder.PAD怎麽用？Python text_encoder.PAD使用的例子？那麽, 這裏精選的屬性代碼示例或許可以為您提供幫助。您也可以進一步了解該屬性所在類tensor2tensor.data_generators.text_encoder的用法示例。
在下文中一共展示了text_encoder.PAD屬性的2個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: build_vocab_list
# 需要導入模塊: from tensor2tensor.data_generators import text_encoder [as 別名]
# 或者: from tensor2tensor.data_generators.text_encoder import PAD [as 別名]
def build_vocab_list(data_path):
"""Reads a file to build a vocabulary with letters and phonemes.
Args:
data_path: data file to read list of words from.
Returns:
vocab_list: vocabulary list with both graphemes and phonemes."""
vocab = {}
with tf.gfile.GFile(data_path, "r") as data_file:
for line in data_file:
items = line.strip().split()
vocab.update({char:1 for char in list(items[0])})
vocab.update({phoneme:1 for phoneme in items[1:]})
vocab_list = [PAD, EOS]
for key in sorted(vocab.keys()):
vocab_list.append(key)
return vocab_list 示例2: test_reserved_tokens_in_corpus
# 需要導入模塊: from tensor2tensor.data_generators import text_encoder [as 別名]
# 或者: from tensor2tensor.data_generators.text_encoder import PAD [as 別名]
def test_reserved_tokens_in_corpus(self):
"""Test that we handle reserved tokens appearing in the corpus."""
corpus = "A B {} D E F {} G {}".format(text_encoder.EOS,
text_encoder.EOS,
text_encoder.PAD)
encoder = text_encoder.TokenTextEncoder(None, vocab_list=corpus.split())
all_tokens = encoder._id_to_token.values()
# If reserved tokens are removed correctly, then the set of tokens will
# be unique.
self.assertEqual(len(all_tokens), len(set(all_tokens)))