本文整理汇总了Python中proxy.Proxy.make_proxy_url方法的典型用法代码示例。如果您正苦于以下问题：Python Proxy.make_proxy_url方法的具体用法？Python Proxy.make_proxy_url怎么用？Python Proxy.make_proxy_url使用的例子？那么, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类proxy.Proxy的用法示例。
在下文中一共展示了Proxy.make_proxy_url方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: spider_page
# 需要导入模块: from proxy import Proxy [as 别名]
# 或者: from proxy.Proxy import make_proxy_url [as 别名]
def spider_page(page_url, parse):
headers = {'user-agent': utility.random_ua()}
try:
r = requests.get(page_url, headers=headers, timeout = 30)
except:
logger.traceback()
return
if r.status_code != 200:
logger.error("get list page failed, url = %s, status_code = %d", page_url, r.status_code)
return
client = ProxyPoolClient()
try:
client.open()
logger.debug_fun("connect ok")
except Thrift.TException:
logger.traceback()
logger.debug_fun("connect failed, quit")
return
logger.debug_fun("get list page ok, url = %s", page_url)
candidate_proxies = parse(r.content)
for ip, port in candidate_proxies:
logger.debug_fun("check proxy, ip = %s, port = %d", ip, port)
proxy_url = Proxy.make_proxy_url(ip, port)
ret, resp_second = validate_proxy(proxy_url)
if not ret or resp_second > settings.PROXY_MAX_DELAY:
logger.debug_fun("check proxy failed, proxy_url = %s", proxy_url)
continue
logger.debug_fun("check proxy ok, proxy_url = %s", proxy_url)
try:
proxy_exists = client.has_proxy(proxy_url)
except:
logger.traceback()
logger.debug_fun("check proxy exists failed, proxy_url = %s", proxy_url)
break
if proxy_exists:
logger.debug_fun("proxy exists, proxy_url = %s", proxy_url)
else:
try:
country = utility.get_ip_country(ip)
except:
logger.traceback()
logger.debug_fun("get country failed, proxy_url = %s", proxy_url)
break
try:
client.spot_proxy(ip, port, country)
except:
logger.traceback()
logger.debug_fun("spot new proxy failed, ip = %s, port = %d, country = %s", ip, port, country)
break
logger.debug_fun("spot new proxy, ip = %s, port = %d, country = %s", ip, port, country)