本文整理匯總了Python中tensorflow.models.rnn.translate.data_utils.prepare_wmt_data方法的典型用法代碼示例。如果您正苦於以下問題：Python data_utils.prepare_wmt_data方法的具體用法？Python data_utils.prepare_wmt_data怎麽用？Python data_utils.prepare_wmt_data使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類tensorflow.models.rnn.translate.data_utils的用法示例。
在下文中一共展示了data_utils.prepare_wmt_data方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: prepare_date
# 需要導入模塊: from tensorflow.models.rnn.translate import data_utils [as 別名]
# 或者: from tensorflow.models.rnn.translate.data_utils import prepare_wmt_data [as 別名]
def prepare_date(data_dir, vocab_size, sample=True):
"""
Downloads english/fench pair and returns the data set as an instace of LanguagePairDataSet
"""
from dataset import LanguageDataSet
pathes = data_utils.prepare_wmt_data(data_dir, vocab_size, vocab_size)
en2_path, fr2_path, en2013_path, fr2013_path, en_vocab_path, fr_vocab_path = pathes
en_index, en_vocab = read_vocab(en_vocab_path)
fr_index, fr_vocab = read_vocab(fr_vocab_path)
#FIXME: some non-ascii charachters
en_vocab_size = len(en_vocab) + 1
fr_vocab_size = len(fr_vocab) + 1
if sample:
en_ids = read_data(en2013_path)
fr_ids = read_data(fr2013_path)
else:
print("reading the full dataset")
en_ids = read_data(en2_path)
fr_ids = read_data(fr2_path)
#Make it the same length (= the max length of the sentences) with zeros for shorter sentences
return LanguageDataSet(en_ids, en_vocab, en_index), LanguageDataSet(fr_ids, fr_vocab, fr_index)