本文整理汇总了Python中fetcher.Fetcher.fetch_soup_attributes方法的典型用法代码示例。如果您正苦于以下问题：Python Fetcher.fetch_soup_attributes方法的具体用法？Python Fetcher.fetch_soup_attributes怎么用？Python Fetcher.fetch_soup_attributes使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类fetcher.Fetcher的用法示例。
在下文中一共展示了Fetcher.fetch_soup_attributes方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: begin
# 需要导入模块: from fetcher import Fetcher [as 别名]
# 或者: from fetcher.Fetcher import fetch_soup_attributes [as 别名]
def begin(self, url_list, within_domain=False, download_location=False, depth=False):
self.unvisited_urls = url_list
self.visited_urls = url_list
self.configure()
cleaned_urls = []
fetcher = Fetcher()
while True:
print '#unvisited urls ', len(self.unvisited_urls)
for url in self.unvisited_urls:
print 'crawling ', url
domain_url = self.domain(url)
html_content = self.fetch_content(url)
soup = self.make_clean_soup(html_content)
out_links = self.get_all_links(soup)
if download_location:
fetcher.fetch_soup_attributes(download_location, url, soup)
cleaned_urls.extend(self.clean_urls(out_links, domain_url, within_domain))
self.unvisited_urls = self.not_visited_urls(cleaned_urls)
self.visited_urls.append(self.unvisited_urls)
if depth:
if depth==1:
break
else:
depth -= 1