本文整理汇总了Python中robotparser.RobotFileParser.allow_all方法的典型用法代码示例。如果您正苦于以下问题：Python RobotFileParser.allow_all方法的具体用法？Python RobotFileParser.allow_all怎么用？Python RobotFileParser.allow_all使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类robotparser.RobotFileParser的用法示例。
在下文中一共展示了RobotFileParser.allow_all方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: print
# 需要导入模块: from robotparser import RobotFileParser [as 别名]
# 或者: from robotparser.RobotFileParser import allow_all [as 别名]
# update time is old, we update it
if rp.mtime() < (time.time() - settings.ROBOTS_TXT_CACHE):
self.logger.info("Refresh %s/robots.txt cache" % hostname)
try:
rp.read()
except Exception, e:
print(e)
self.logger.info("Unable to get or parse %s/robots.txt" % hostname)
rp.disallow_all = False
rp.allow_all = True
else:
self.logger.debug("Retrieve cached %s/robots.txt" % hostname)
else:
# First (or very long) time we see this domain, create a new
# RobotFileParser and read it once
self.logger.info("First hit on %s/robots.txt" % hostname)
rp = RobotFileParser(url="%s://%s/robots.txt" % (scheme, hostname))
try:
rp.read()
except Exception, e:
print(e)
self.logger.info("Unable to get or parse %s/robots.txt" % hostname)
rp.disallow_all = False
rp.allow_all = True
# In any case, we update the last robotstxt fetched time
rp.modified()
self.robotstxt[hostname] = rp
return rp