本文整理汇总了Python中scrapy.crawler.CrawlerRunner._create_crawler方法的典型用法代码示例。如果您正苦于以下问题：Python CrawlerRunner._create_crawler方法的具体用法？Python CrawlerRunner._create_crawler怎么用？Python CrawlerRunner._create_crawler使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类scrapy.crawler.CrawlerRunner的用法示例。
在下文中一共展示了CrawlerRunner._create_crawler方法的2个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: get_crawler
# 需要导入模块: from scrapy.crawler import CrawlerRunner [as 别名]
# 或者: from scrapy.crawler.CrawlerRunner import _create_crawler [as 别名]
def get_crawler(spidercls=None, settings_dict=None):
"""Return an unconfigured Crawler object. If settings_dict is given, it
will be used to populate the crawler settings with a project level
priority.
"""
from scrapy.crawler import CrawlerRunner
from scrapy.settings import Settings
from scrapy.spider import Spider
runner = CrawlerRunner(Settings(settings_dict))
return runner._create_crawler(spidercls or Spider)示例2: my_CachingThreadedResolver
# 需要导入模块: from scrapy.crawler import CrawlerRunner [as 别名]
# 或者: from scrapy.crawler.CrawlerRunner import _create_crawler [as 别名]
cache_size = self.settings.getint('DNSCACHE_SIZE')
else:
cache_size = 0
print "wwj debug return my_cachingThreadedResolver"
return my_CachingThreadedResolver(
reactor=reactor,
cache_size=cache_size,
timeout=self.settings.getfloat('DNS_TIMEOUT')
)
settings = get_project_settings()
my_process = my_CrawlerProcess(settings)
runner = CrawlerRunner(settings)
#### one runner, more spiders
spidercls = runner.spider_loader.load('scrapy_spider')
my_crawler = runner._create_crawler(spidercls)
my_crawler.spider = my_crawler._create_spider('scrapy_spider')
my_crawler.engine = my_crawler._create_engine()
start_requests = iter(my_crawler.spider.start_requests())
close_if_idle = False
my_crawler.engine.open_spider(my_crawler.spider, start_requests, close_if_idle)
my_crawler.engine.start()
#process.crawl('scrapy_spider')
stop_after_crawl = False
my_process.start(stop_after_crawl)