本文整理匯總了Python中robotparser.RobotFileParser.allow_all方法的典型用法代碼示例。如果您正苦於以下問題：Python RobotFileParser.allow_all方法的具體用法？Python RobotFileParser.allow_all怎麽用？Python RobotFileParser.allow_all使用的例子？那麽, 這裏精選的方法代碼示例或許可以為您提供幫助。您也可以進一步了解該方法所在類robotparser.RobotFileParser的用法示例。
在下文中一共展示了RobotFileParser.allow_all方法的1個代碼示例，這些例子默認根據受歡迎程度排序。您可以為喜歡或者感覺有用的代碼點讚，您的評價將有助於係統推薦出更棒的Python代碼示例。
示例1: print
# 需要導入模塊: from robotparser import RobotFileParser [as 別名]
# 或者: from robotparser.RobotFileParser import allow_all [as 別名]
# update time is old, we update it
if rp.mtime() < (time.time() - settings.ROBOTS_TXT_CACHE):
self.logger.info("Refresh %s/robots.txt cache" % hostname)
try:
rp.read()
except Exception, e:
print(e)
self.logger.info("Unable to get or parse %s/robots.txt" % hostname)
rp.disallow_all = False
rp.allow_all = True
else:
self.logger.debug("Retrieve cached %s/robots.txt" % hostname)
else:
# First (or very long) time we see this domain, create a new
# RobotFileParser and read it once
self.logger.info("First hit on %s/robots.txt" % hostname)
rp = RobotFileParser(url="%s://%s/robots.txt" % (scheme, hostname))
try:
rp.read()
except Exception, e:
print(e)
self.logger.info("Unable to get or parse %s/robots.txt" % hostname)
rp.disallow_all = False
rp.allow_all = True
# In any case, we update the last robotstxt fetched time
rp.modified()
self.robotstxt[hostname] = rp
return rp