本文整理汇总了Python中tensorflow.models.rnn.translate.data_utils.prepare_wmt_data方法的典型用法代码示例。如果您正苦于以下问题：Python data_utils.prepare_wmt_data方法的具体用法？Python data_utils.prepare_wmt_data怎么用？Python data_utils.prepare_wmt_data使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类tensorflow.models.rnn.translate.data_utils的用法示例。
在下文中一共展示了data_utils.prepare_wmt_data方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: prepare_date
# 需要导入模块: from tensorflow.models.rnn.translate import data_utils [as 别名]
# 或者: from tensorflow.models.rnn.translate.data_utils import prepare_wmt_data [as 别名]
def prepare_date(data_dir, vocab_size, sample=True):
"""
Downloads english/fench pair and returns the data set as an instace of LanguagePairDataSet
"""
from dataset import LanguageDataSet
pathes = data_utils.prepare_wmt_data(data_dir, vocab_size, vocab_size)
en2_path, fr2_path, en2013_path, fr2013_path, en_vocab_path, fr_vocab_path = pathes
en_index, en_vocab = read_vocab(en_vocab_path)
fr_index, fr_vocab = read_vocab(fr_vocab_path)
#FIXME: some non-ascii charachters
en_vocab_size = len(en_vocab) + 1
fr_vocab_size = len(fr_vocab) + 1
if sample:
en_ids = read_data(en2013_path)
fr_ids = read_data(fr2013_path)
else:
print("reading the full dataset")
en_ids = read_data(en2_path)
fr_ids = read_data(fr2_path)
#Make it the same length (= the max length of the sentences) with zeros for shorter sentences
return LanguageDataSet(en_ids, en_vocab, en_index), LanguageDataSet(fr_ids, fr_vocab, fr_index)