本文整理匯總了Python中fetcher.Fetcher.fetch_soup_attributes方法的典型用法代碼示例。如果您正苦於以下問題：Python Fetcher.fetch_soup_attributes方法的具體用法？Python Fetcher.fetch_soup_attributes怎麽用？Python Fetcher.fetch_soup_attributes使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類fetcher.Fetcher的用法示例。
在下文中一共展示了Fetcher.fetch_soup_attributes方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: begin
# 需要導入模塊: from fetcher import Fetcher [as 別名]
# 或者: from fetcher.Fetcher import fetch_soup_attributes [as 別名]
def begin(self, url_list, within_domain=False, download_location=False, depth=False):
self.unvisited_urls = url_list
self.visited_urls = url_list
self.configure()
cleaned_urls = []
fetcher = Fetcher()
while True:
print '#unvisited urls ', len(self.unvisited_urls)
for url in self.unvisited_urls:
print 'crawling ', url
domain_url = self.domain(url)
html_content = self.fetch_content(url)
soup = self.make_clean_soup(html_content)
out_links = self.get_all_links(soup)
if download_location:
fetcher.fetch_soup_attributes(download_location, url, soup)
cleaned_urls.extend(self.clean_urls(out_links, domain_url, within_domain))
self.unvisited_urls = self.not_visited_urls(cleaned_urls)
self.visited_urls.append(self.unvisited_urls)
if depth:
if depth==1:
break
else:
depth -= 1