本文整理匯總了Python中robotparser.RobotFileParser.disallow_all方法的典型用法代碼示例。如果您正苦於以下問題：Python RobotFileParser.disallow_all方法的具體用法？Python RobotFileParser.disallow_all怎麽用？Python RobotFileParser.disallow_all使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類robotparser.RobotFileParser的用法示例。
在下文中一共展示了RobotFileParser.disallow_all方法的2個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: check_robots
# 需要導入模塊: from robotparser import RobotFileParser [as 別名]
# 或者: from robotparser.RobotFileParser import disallow_all [as 別名]
def check_robots(self, url):
'''check the robots.txt in this url's domain'''
hostname = urlparse(url).netloc
if hostname not in self.domain_list.keys(): # no records in domain_list
rp = RobotFileParser('http://%s/robots.txt' % hostname)
print("%s: fetching %s" % (url, rp.url))
try:
rp.read() # get new robots.txt
except IOError, e: # url's server not available(connection timeout)
log.error(str(e))
rp.disallow_all = True # reject all request
self.domain_list[hostname] = rp # add domain entry into domain_list示例2: print
# 需要導入模塊: from robotparser import RobotFileParser [as 別名]
# 或者: from robotparser.RobotFileParser import disallow_all [as 別名]
# update time is old, we update it
if rp.mtime() < (time.time() - settings.ROBOTS_TXT_CACHE):
self.logger.info("Refresh %s/robots.txt cache" % hostname)
try:
rp.read()
except Exception, e:
print(e)
self.logger.info("Unable to get or parse %s/robots.txt" % hostname)
rp.disallow_all = False
rp.allow_all = True
else:
self.logger.debug("Retrieve cached %s/robots.txt" % hostname)
else:
# First (or very long) time we see this domain, create a new
# RobotFileParser and read it once
self.logger.info("First hit on %s/robots.txt" % hostname)
rp = RobotFileParser(url="%s://%s/robots.txt" % (scheme, hostname))
try:
rp.read()
except Exception, e:
print(e)
self.logger.info("Unable to get or parse %s/robots.txt" % hostname)
rp.disallow_all = False
rp.allow_all = True
# In any case, we update the last robotstxt fetched time
rp.modified()
self.robotstxt[hostname] = rp
return rp