本文整理汇总了Python中fetcher.Fetcher.request方法的典型用法代码示例。如果您正苦于以下问题：Python Fetcher.request方法的具体用法？Python Fetcher.request怎么用？Python Fetcher.request使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类fetcher.Fetcher的用法示例。
在下文中一共展示了Fetcher.request方法的2个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: start
# 需要导入模块: from fetcher import Fetcher [as 别名]
# 或者: from fetcher.Fetcher import request [as 别名]
def start(self):
url_queue = Queue.Queue()
url_queue.put((self.root_request_info.url, 0))
request_info = RequestInfo('', None, self.root_request_info.headers)
fetcher = Fetcher()
while not url_queue.empty():
curr_url, depth = url_queue.get()
#print 'url=%s, depth=%d' % (curr_url, depth)
print curr_url
if depth > self.depth_limit:
continue
depth += 1
request_info.url = curr_url
page_content = fetcher.request(request_info)
## parse page
## Content.parse(page_content)
url_list = HtmlParser.extract_url(curr_url, page_content)
if url_list:
for url in url_list:
url_queue.put((url, depth))示例2: RequestInfo
# 需要导入模块: from fetcher import Fetcher [as 别名]
# 或者: from fetcher.Fetcher import request [as 别名]
#!/usr/bin/env python
from fetcher import RequestInfo
from fetcher import Fetcher
from html_parser import HttpParser
if __name__ == '__main__':
init_url = 'http://zhihu.com'
request = RequestInfo(init_url)
request.add_header('User-Agent', 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)')
fetcher = Fetcher()
response = fetcher.request(request)
#h = HttpParser()
if response:
url_list = HttpParser.extract_url(response)
if url_list:
for url in url_list:
print url