本文整理汇总了Python中storage.Storage.add_word_occurrence方法的典型用法代码示例。如果您正苦于以下问题：Python Storage.add_word_occurrence方法的具体用法？Python Storage.add_word_occurrence怎么用？Python Storage.add_word_occurrence使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类storage.Storage的用法示例。
在下文中一共展示了Storage.add_word_occurrence方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: crawl_web
# 需要导入模块: from storage import Storage [as 别名]
# 或者: from storage.Storage import add_word_occurrence [as 别名]
def crawl_web(seed):
tocrawl = set([seed])
crawled = []
database = Storage()
while tocrawl:
url = tocrawl.pop()
if url not in crawled:
print url
soup = get_page(url)
text = soup.get_text()
text = re.sub(r"[-/']", ' ', text)
text = re.sub(ur'[\u0932\u094b\u0917\u092a\u0930\u093f\u0926\u0943\u0936\u094d\u092f\u0938\u094d\u0925\u093e\u0928\u0915\u0930\u094d\u092e\u091a\u093e\u0930\u0940\u201c\u2013\u2019\u092b\u201d]', ' ', text)
outlinks = get_all_links(soup)
url_title = str(soup.title.string)
database.add_title(url, url_title)
for outlink in outlinks:
database.add_link(url, outlink)
punctuation = ".?;,!()|:\""
for word in text.split():
word = word.lstrip(punctuation).rstrip(punctuation).lower()
search = re.search(r"[^a-z0-9]", word)
if not search and word is not "":
database.add_word_occurrence(url, word)
tocrawl.update(outlinks)
crawled.append(url)
return database