本文整理汇总了Python中gensim.corpora.WikiCorpus.metadata方法的典型用法代码示例。如果您正苦于以下问题：Python WikiCorpus.metadata方法的具体用法？Python WikiCorpus.metadata怎么用？Python WikiCorpus.metadata使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类gensim.corpora.WikiCorpus的用法示例。
在下文中一共展示了WikiCorpus.metadata方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: save_to_batches
# 需要导入模块: from gensim.corpora import WikiCorpus [as 别名]
# 或者: from gensim.corpora.WikiCorpus import metadata [as 别名]
def save_to_batches(input, doc_set=set(), batch_path='.', batch_size=1000, lang='@body'):
if not doc_set: # is empty
return
wiki = WikiCorpus(input, lemmatize=False, dictionary='empty dictionary')
wiki.metadata = True # request to extract page_id and title
num_docs_found = 0
batch_dict = {}
NNZ = 0
batch = artm.messages_pb2.Batch()
for (text, page_id_and_title) in wiki.get_texts():
page_id = page_id_and_title[0]
title = page_id_and_title[1]
if page_id in doc_set:
num_docs_found += 1
print num_docs_found, page_id, title
# get tokens tf in the text
text_tf = Counter(text)
for token in text:
# update batch dictionary
if token not in batch_dict:
batch.token.append(unicode(token, 'utf-8'))
batch_dict[token] = len(batch.token) - 1
# add item to batch
item = batch.item.add()
item.id = int(page_id)
item.title = title
field = item.field.add()
field.name = lang
for token in text_tf:
field.token_id.append(batch_dict[token])
field.token_count.append(text_tf[token])
NNZ += text_tf[token]
if len(batch.item) == batch_size:
artm.library.Library().SaveBatch(batch, batch_path)
print 'Batch done, |W| = ' + str(len(batch.token)) + ", NNZ = " + str(NNZ)
batch = artm.messages_pb2.Batch()
batch_dict = {}
NNZ = 0
if len(batch.item) > 0:
artm.library.Library().SaveBatch(batch, batch_path)
print 'Last batch done, |W| = ' + str(len(batch.token)) + ", NNZ = " + str(NNZ)