15.scrapy-redis多页抓取,使用redis一页一页的
2019-04-01 本文已影响0人
starrymusic
和前一篇类似,这里是修改后的csdn2.py文件,代码如下:
# -*- coding: utf-8 -*-
import scrapy
import example.items
from scrapy_redis.spiders import RedisSpider
class Csdn2Spider(RedisSpider):
name = 'csdn2'
redis_key = 'csdn2:start_urls'
def __init__(self, *args, **kwargs):
# Dynamically define the allowed domains list.
domain = kwargs.pop('https://edu.csdn.net', '')
self.allowed_domains = filter(None, domain.split(','))
super(Csdn2Spider, self).__init__(*args, **kwargs)
def parse(self, response):
for pagedata in response.xpath("//dl[@class='lector_list']"):
item = example.items.Csdn2Item()
item['teacher'] = pagedata.xpath("./dd[1]/ul/li/a/text()").extract()
item['lessons'] = pagedata.xpath("./dd[1]/ul/li[2]/span/text()").extract()
item['student'] = pagedata.xpath("./dd[1]/ul/li[3]/span/text()").extract()
item['describe'] = pagedata.xpath("./dd[1]/p/text()").extract()
yield item
这里是修改后的items.py文件,代码如下:
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/topics/items.html
from scrapy.item import Item, Field
from scrapy.loader import ItemLoader
from scrapy.loader.processors import MapCompose, TakeFirst, Join
class Csdn2Item(Item):
teacher = Field()
lessons = Field()
student = Field()
describe = Field()
crawled = Field()
spider = Field()
class ExampleItem(Item):
name = Field()
description = Field()
link = Field()
crawled = Field()
spider = Field()
url = Field()
class ExampleLoader(ItemLoader):
default_item_class = ExampleItem
default_input_processor = MapCompose(lambda s: s.strip())
default_output_processor = TakeFirst()
description_out = Join()
其他文件无需更改,如果想看到输出结果,可以在pipelines.py文件里加一句打印的代码。打开redis-cli.exe,连接上redis服务器。
在cmd.exe里切换到项目所在位置,然后执行“scrapy crawl csdn2”,执行后会显示等待的状态,这时在redis-cli.exe里输入
lpush csdn2:start_urls https://edu.csdn.net/lecturer?&page=1
在cmd.exe里就会显示抓取的第一页的教师信息。
如果想让他继续执行,就在redis客户端里不断地输入要爬取的url地址即可。