scrapy-redis
2018-03-27 本文已影响0人
兔头咖啡
settings配置redis：
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_HOST = '127.0.0.1'
REDIS_PORT = 6379
爬虫修改：
class NovelSpider(RedisSpider):
name = 'novel2'
redis_key = 'novel2:start_urls'
start_urls = ['http://www.daomubiji.com/']
