本文整理汇总了Python中scrapy.crawler.CrawlerRunner.settings['ITEM_PIPELINES']方法的典型用法代码示例。如果您正苦于以下问题：Python CrawlerRunner.settings['ITEM_PIPELINES']方法的具体用法？Python CrawlerRunner.settings['ITEM_PIPELINES']怎么用？Python CrawlerRunner.settings['ITEM_PIPELINES']使用的例子？那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在类scrapy.crawler.CrawlerRunner的用法示例。
在下文中一共展示了CrawlerRunner.settings['ITEM_PIPELINES']方法的1个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于系统推荐出更棒的Python代码示例。
示例1: CrawlerRunner
# 需要导入模块: from scrapy.crawler import CrawlerRunner [as 别名]
# 或者: from scrapy.crawler.CrawlerRunner import settings['ITEM_PIPELINES'] [as 别名]
# logging setting
# more on http://doc.scrapy.org/en/latest/topics/logging.html
logging.basicConfig(
level = logging.DEBUG,
format = '%(asctime)s [%(name)s] %(levelname)s: %(message)s',
datefmt = '%m-%d %H:%M:%S',
filename = 'crawl.log',
filemode = 'w'
)
# add spiders
runner = CrawlerRunner()
runner.settings['ITEM_PIPELINES'] = {
'dirbot.pipelines.FilterWordsPipeline': 1,
#'dirbot.pipelines.MongoDBPipeline':1,
'dirbot.pipelines.MySQLPipeline':1,
}
#runner.crawl(DmozSpider.DmozSpider())
#runner.crawl(StackOverflowSpider.StackOverflowSpider())
#runner.crawl(CNStockSpider.CNStockSpider())
#runner.crawl(SinaSpider.SinaSpider())
#runner.crawl(IfengSpider.IfengSpider())
#runner.crawl(SZKXSpider.SZKXSpider())
runner.crawl(GeneralSpider.GeneralSpider())
#runner.crawl(BlogSinaSpider.BlogSinaSpider())
d = runner.join()
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until all crawling jobs are finished