本文整理匯總了Python中fetcher.Fetcher.request方法的典型用法代碼示例。如果您正苦於以下問題：Python Fetcher.request方法的具體用法？Python Fetcher.request怎麽用？Python Fetcher.request使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類fetcher.Fetcher的用法示例。
在下文中一共展示了Fetcher.request方法的2個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: start
# 需要導入模塊: from fetcher import Fetcher [as 別名]
# 或者: from fetcher.Fetcher import request [as 別名]
def start(self):
url_queue = Queue.Queue()
url_queue.put((self.root_request_info.url, 0))
request_info = RequestInfo('', None, self.root_request_info.headers)
fetcher = Fetcher()
while not url_queue.empty():
curr_url, depth = url_queue.get()
#print 'url=%s, depth=%d' % (curr_url, depth)
print curr_url
if depth > self.depth_limit:
continue
depth += 1
request_info.url = curr_url
page_content = fetcher.request(request_info)
## parse page
## Content.parse(page_content)
url_list = HtmlParser.extract_url(curr_url, page_content)
if url_list:
for url in url_list:
url_queue.put((url, depth))示例2: RequestInfo
# 需要導入模塊: from fetcher import Fetcher [as 別名]
# 或者: from fetcher.Fetcher import request [as 別名]
#!/usr/bin/env python
from fetcher import RequestInfo
from fetcher import Fetcher
from html_parser import HttpParser
if __name__ == '__main__':
init_url = 'http://zhihu.com'
request = RequestInfo(init_url)
request.add_header('User-Agent', 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)')
fetcher = Fetcher()
response = fetcher.request(request)
#h = HttpParser()
if response:
url_list = HttpParser.extract_url(response)
if url_list:
for url in url_list:
print url